This post is the other order. The service keeps running on the machine it was built on, and a Cloudflare Tunnel gives it a public HTTPS hostname. Nothing on the laptop listens to the internet: no port forwarding, no router settings, no exposed IP address. The machine dials out to Cloudflare, and Cloudflare sends requests back down that connection. The tunnel is free, and the TLS certificate, DDoS protection and caching come with the Cloudflare zone you probably already have.

The example is real. One of my products runs on Fly.io, and its mobile app gets over-the-air JavaScript updates from the backend (self-hosted Expo Updates). I wanted a second copy of that update server running on my Mac for development builds, with production left exactly as it was. The outputs below are from that afternoon, including the part where it didn’t work.

What a Cloudflare Tunnel actually is

#

A normal server waits for connections. To reach a laptop from the internet, you’d have to open a port on your router, forward it to the laptop, and hope your ISP gives you a public IPv4 address (in a lot of places, mine included, it doesn’t). Then everything listening on that port is reachable by every scanner on the internet, all day.

A tunnel turns that around. cloudflared runs next to your app and opens outbound connections to Cloudflare’s edge, four of them to two nearby data centres. A DNS record in your zone points a hostname at the tunnel. When a request arrives for that hostname, Cloudflare terminates TLS and forwards the request down one of those connections. cloudflared then hands it to whichever local service your config names.

For security, the important part is that your router has no inbound rule and your IP address never appears in DNS. Someone scanning your home connection finds nothing, because nothing on it is listening.

Setting up cloudflared, start to finish

#

You need a domain on Cloudflare (the free plan is enough) and cloudflared installed (brew install cloudflared on a Mac).

1. Log in once. This opens a browser, you pick the zone, and it saves a certificate that lets this machine create tunnels and DNS records in that zone:

cloudflared tunnel login

2. Create the tunnel. I keep everything for this project in its own folder, because the credentials file is the tunnel’s identity: anyone who has it can serve traffic for your hostnames.

mkdir -p ~/.cloudflared/myapp && cd ~/.cloudflared/myapp
cloudflared tunnel create --credentials-file ./creds.json myapp-local
chmod 600 creds.json

Tunnel credentials written to ./creds.json. Keep this file secret. To revoke these credentials, delete the tunnel.
Created tunnel myapp-local with id 1ade030c-…

3. Point hostnames at it. One command per hostname creates a proxied CNAME to <tunnel-id>.cfargotunnel.com:

cloudflared tunnel route dns myapp-local ota.example.com
cloudflared tunnel route dns myapp-local checkin-local.example.com

4. Write the ingress rules. This is the part that matters for security, so it gets its own section below.

5. Run it. My app already runs under Docker Compose, so cloudflared is one more container on the same network. It reaches the app by service name, and the app doesn’t need a port published on the host at all:

# compose.tunnel.yml, used alongside the project's docker-compose.yml
services:
  cloudflared:
    image: cloudflare/cloudflared:2026.10.7
    command: tunnel --config /etc/cloudflared/config.yml run
    volumes:
      - ${HOME}/.cloudflared/myapp:/etc/cloudflared:ro
    restart: unless-stopped
    depends_on: [api, edge]

docker compose -f docker-compose.yml -f ~/.cloudflared/myapp/compose.tunnel.yml up -d cloudflared

INF Registered tunnel connection connIndex=0 … location=icn06 protocol=quic
INF Registered tunnel connection connIndex=1 … location=icn07 protocol=quic
INF Registered tunnel connection connIndex=2 … location=icn07 protocol=quic
INF Registered tunnel connection connIndex=3 … location=icn06 protocol=quic

Four connections to Seoul, and the laptop is on the internet, from the outside in. restart: unless-stopped brings it back after a reboot as soon as Docker starts.

Expose paths, not servers

#

This is the mistake I nearly made, and I’d guess it’s the most common one. The easy config is a single line: hostname → http://localhost:8000. That publishes the whole development server.

Development servers aren’t built for that. Mine runs with DEBUG=True, which shows full tracebacks and settings on any error. It also runs with authentication switched off (AUTH_MODE=disabled), so that any bearer token logs you in as a test user. That’s exactly right on localhost and a disaster on a public hostname. Yours probably has something similar: an admin with a default password, a seeding endpoint, a debug toolbar, or an API that trusts its caller because only you ever call it.

So the tunnel config is an allowlist. cloudflared checks each request against the ingress rules in order, by hostname and by a regular expression on the path. The first match wins, and the last rule catches everything else:

tunnel: myapp-local
credentials-file: /etc/cloudflared/creds.json

ingress:
  # OTA: the two read-only endpoints the phone calls, and the files they link.
  - hostname: ota.example.com
    path: ^/api/expo-updates/(manifest|status)/$
    service: http://api:8000
  - hostname: ota.example.com
    path: ^/media/
    service: http://api:8000
  # Device check-ins, authenticated by each device's own key.
  - hostname: checkin-local.example.com
    path: ^/c/$
    service: http://edge:8081
  # Everything else, including /admin/ and the publish endpoint: 404 at the edge.
  - service: http_status:404

The publishing endpoint lives on the same server under the same prefix (/api/expo-updates/import-update/), and it isn’t in the list. Neither is the admin, or the rest of the API. Those requests never leave Cloudflare.

You don’t have to trust your regexes; cloudflared will tell you which rule a URL matches:

cloudflared tunnel --config config.yml ingress rule https://ota.example.com/admin/

https://ota.example.com/api/expo-updates/manifest/        service: http://api:8000
https://ota.example.com/api/expo-updates/import-update/   service: http_status:404
https://ota.example.com/admin/                            service: http_status:404
https://ota.example.com/media/expo-updates/x.js           service: http://api:8000
https://checkin-local.example.com/c/                      service: http://edge:8081
https://checkin-local.example.com/health                  service: http_status:404

I’d put that check in a script and run it every time the config changes. It takes seconds, and a regex that matches more than you meant is the kind of mistake you only find out about from someone else.

Two more rules of thumb:

• What you expose should authenticate itself. The OTA endpoints serve public, read-only data. Device check-ins are signed with per-device keys. Neither one depends on the network to keep it safe. If you want to show someone a page that has no login, put Cloudflare Access in front of that hostname (free for up to 50 users) instead of building a login.
• Don’t run the tunnel as a background part of your dev workflow. Run it on purpose, for the hostname you mean, and stop it when you’re done. docker compose stop cloudflared takes everything off the internet in one command.

The redirect that ate my hostnames

#

With the tunnel connected and the rules verified, I tried it from outside:

https://ota.example.com/api/expo-updates/status/?…   301
https://ota.example.com/api/expo-updates/manifest/   301
https://ota.example.com/admin/                       301
https://checkin-local.example.com/c/                 301

Every path returned the same 301, including ones the tunnel should have answered with 404. That means the request never reached cloudflared. The Location header said https://www.example.com/, and the server: cloudflare header settled it: Cloudflare was answering on its own.

I’d done this to myself. Months ago I added a zone-level redirect rule, “any other subdomain → www”, as a sinkhole for scanners that walk invented subdomains. Redirect rules run in Cloudflare’s request pipeline before the request is sent to the origin, and a tunnel is just an origin. The same rule caught my preview environments before Workers could see them. The fix is the same as last time: one more exception in the rule’s expression.

… and not http.host in {"ota.example.com" "checkin-local.example.com"}

If your tunnel hostname answers with a redirect or an error page from Cloudflare instead of your app, check Rules → Redirect Rules, Page Rules and any WAF custom rules for the zone before you debug the tunnel. cloudflared logs every request it forwards, so an empty log means the request never got that far.

What it can’t do

#

• UDP from the public internet. Public tunnel hostnames carry HTTP(S) (plus WebSockets and gRPC). My product’s devices normally check in over CoAP on UDP, and that can’t go through a tunnel; the devices on the local copy fall back to HTTPS check-ins, which cost them a few seconds on each one. If your idea is a game server or VoIP, a tunnel isn’t the tool.
• Uptime. It’s your laptop. Close the lid and the site is down; Cloudflare shows a 530 error page. That’s fine for a demo, a beta with ten people, or a staging copy. It isn’t fine for anything someone depends on.
• Things baked into a build. A mobile app’s update URL is native config, so only builds made with the tunnel’s URL use it. My store builds keep talking to production, and that’s why this setup was safe to try.

The cost: is Cloudflare Tunnel free?

#

A Cloudflare zone on the free plan, the tunnel, its certificate, and the DNS records: $0. The machine was already running. The only real cost is that you’re now the uptime team.

What it saves you is the phase between “it works on my machine” and “it’s worth paying to host”. You can send a link to a tester, a designer or a first customer the same afternoon. If nobody cares, you’ve lost nothing. If they do, moving to a real host is one change: point the CNAME somewhere else.

I first met them in September, when a Cloudflare dashboard said 22% of my requests were failing while every health check stayed green; I switched Early Hints off and moved on. In November the same zone showed another 2,600 504s in a day, with Early Hints on again, and this time I wanted to know exactly what they are and whether the right fix was to turn the feature off a second time or to make it do its job. The zone fronts one Fly.io origin serving a Django API, an Expo web app and a Hugo marketing site.

The short answer

#

• Those 504s are written by Cloudflare’s Early Hints cache, not by your visitors. In Cloudflare’s logs they carry ClientRequestSource = earlyHintsCache.
• A 504 there means a cache miss: Cloudflare had no hints stored for that URL. A hit is logged as a 2xx.
• They never reach your origin. OriginResponseStatus is 0.
• Early Hints runs on the Link HTTP response header. <link rel="preload"> tags in your HTML don’t feed it.
• Send the header, and those misses turn into hits and real 103 Early Hints responses on the pages that have something to preload.

Three more things worth knowing up front. Hint only the render-critical CSS, scripts and fonts, and keep crossorigin on the fonts. Verify with curl -sv --http2: the 103 should arrive before the 200. And keep earlyHintsCache out of your alerting whatever else you decide; if you’ll never send the header, turn the feature off rather than paying for it in log noise.

How to tell a fake Cloudflare 504 from a real one

#

If you use Logpush or Instant Logs, look at two fields before anything else. ClientRequestSource is eyeball for a real visitor’s request and earlyHintsCache for the Early Hints lookups. OriginResponseStatus is 0 when the origin was never contacted.

In the GraphQL Analytics API the same split is the requestSource dimension. This is the query I now run before believing any 5xx spike:

query ($zone: String!, $since: Time!) {
  viewer {
    zones(filter: { zoneTag: $zone }) {
      httpRequestsAdaptiveGroups(
        limit: 20
        filter: { datetime_geq: $since, edgeResponseStatus: 504 }
        orderBy: [count_DESC]
      ) {
        count
        dimensions { requestSource userAgent clientRequestPath }
      }
    }
  }
}

Grouping by user agent tells the same story. In September the Early Hints lookups showed up as bastion early hints and nginx-ssl early hints. In November they were fl2 early hints and nginx-ssl early hints (FL2 is the newer generation of Cloudflare’s proxy). Across 24 hours, all 2,631 of the zone’s 504s came from those user agents. None came from a browser, the app, curl or a search crawler.

A real 504 that a visitor actually saw almost always came from your side of Cloudflare. A proxy or load balancer in front of your application gave up waiting: nginx, a cloud load balancer, or a platform proxy waiting for a cold machine to boot. Cloudflare’s own timeouts use different codes. 522 means it couldn’t connect to your origin, and 524 means it connected but got no response within 125 seconds (the default proxy read timeout on plans below Enterprise).

So if you landed here searching for how to raise Cloudflare’s timeout because of 504s: the timeout behind a 504 is almost never Cloudflare’s. Check proxy_read_timeout in nginx, your load balancer’s idle timeout, or how long your platform takes to wake a machine. A useful clue: if requests fail at a round number well under 125 seconds, such as 30 or 60, it’s your own proxy’s limit, not Cloudflare’s. (Only Enterprise plans can raise Cloudflare’s 125-second limit, up to 6,000 seconds, and Cloudflare itself treats that as a stopgap.)

What those 504s actually are

#

On a zone with Early Hints enabled, every request for an HTML page or an extensionless URL (which, on an API-heavy zone, is most API calls) gets an extra internal lookup: Cloudflare checking its own cache for a set of hints to send. That lookup is logged as its own line, with ClientRequestSource = earlyHintsCache. It isn’t an end-user request, and it is never sent to your origin.

The status on that line is the result of the cache lookup, not an error:

• 2xx is a hit. Cloudflare had Link headers stored for that URL and sent them to the browser as a 103. On my zone, hits show up as both 200 and 204.
• 504 is a miss. Nothing was stored, so there was nothing to send early.

If your origin never sends the header Early Hints runs on, every lookup is a miss and every request writes a 504 line. In the three hours before I changed anything, the zone logged 173 Early Hints lookups and all 173 were misses. Since the feature had been switched on, it had not served a single hint.

One correction to my September post: I described these as probes that “get no response from the Fly origin”. That isn’t quite right. They never go to the origin at all. The 504 records a cache miss, not a timeout.

Cloudflare documents this, briefly, on its Early Hints page: an Early Hints lookup logged with ClientRequestSource: earlyHintsCache is “expected and benign”, a 200 is a cache hit and a 504 a miss, and Log Explorer hides them with ClientRequestSource != "earlyHintsCache". Cache Analytics already leaves them out. The HTTP traffic chart that scared me doesn’t.

These fake 504s talked me into a server upgrade

#

This is why the split matters. On 3 November, the day before I started on this post, Cloudflare logged 3,292 504s against about 13,300 visitor requests: an apparent error rate of 25%. The API runs on Fly.io’s shared-cpu-1x machines, which are burst-credited. They get a small guaranteed share of a CPU and spend credits to run faster, and a load test in August had shown them throttling to roughly 20 to 30 requests a second under sustained load. “A quarter of requests timing out at peak” looked exactly like that throttling, so I moved the API to shared-cpu-2x.

Every one of those 504s was an Early Hints cache miss. Here is what the requestSource split and the origin’s own logs showed:

• Every Cloudflare 5xx from 28 October to 4 November was an earlyHintsCache lookup, apart from about one 525 a day (the one I traced came from a security scanner).
• The origin’s access logs for 24 October to 4 November had no 5xx responses at all, and one request slower than 10 seconds.
• Fly’s metrics never showed more than about 7 requests a second, and the machine never used more than a fifth of one CPU core. It never got near the throttling ceiling.

The bigger machines fixed nothing, because nothing was broken. The API is back on shared-cpu-1x.

Don’t make capacity decisions from a 5xx chart you haven’t split by request source. Check it against your origin’s own logs as well: if Cloudflare reports a quarter of requests failing and your server logged no errors, find out what Cloudflare is counting before you change the server.

Updated 5 November 2026: the first version of this section said the API had really fallen over on shared-cpu-1x that week. It hadn’t. The numbers above are what I found when I checked.

Is HTTP 103 an error?

#

No. Status codes in the 1xx range are informational: interim responses a server can send before the real one. 103 Early Hints, defined in RFC 8297, carries Link headers so the browser can start fetching critical CSS, JavaScript and fonts, or open connections to other origins, while the server is still building the page. The final 200 follows on the same request.

Support is uneven but safe. Chromium-based browsers (Chrome since version 103, and Edge) and current Firefox act on both preload and preconnect hints. Safari (17 and later) uses only preconnect. Browsers process 103 only over HTTP/2 and HTTP/3. A browser that ignores it simply waits for the 200 as it always did, so a 103 is a head start for most visitors and a no-op for the rest.

Why mine was empty: the Link header vs <link rel=preload>

#

This is the part that cost me the most time, because it looks like it should already work.

Early Hints caches exactly one thing: the HTTP Link response header with a rel=preload or rel=preconnect relation. Cloudflare reads it from your normal responses (status 200, 301 or 302, on HTML or extensionless URLs), stores it, and replays it to the next visitor as a 103 before your full response is ready.

My pages already had preloads, but as tags in the HTML <head>:

<link rel="preload" href="/_expo/static/css/global-7d77bd.css" as="style">
<link rel="preload" href="/_expo/static/js/web/entry-fd7fa4.js" as="script">
<link rel="preload" href="/fonts/manrope-latin.woff2" as="font" type="font/woff2" crossorigin>

Those tags still do something: the browser acts on them. But it only sees them after it has received the HTML and started parsing it, which is exactly the moment Early Hints exists to beat. The Expo export and my Hugo theme both write these tags for you, and neither sets the header. So the feature was enabled, the preloads were declared, and the two never met.

The fix: send the header you already have the data for

#

The cleanest fix doesn’t involve maintaining a hand-written list of critical assets that goes stale every time a bundle hash changes. Each page already declares what it wants preloaded, in those tags. So mirror them into a Link header when you serve the page. The standard library’s html.parser copes with quoted and unquoted attributes (Hugo’s minifier strips the quotes), so there’s no regex to get wrong:

from html.parser import HTMLParser


class PreloadLinks(HTMLParser):
    def __init__(self):
        super().__init__()
        self.hints = []

    def handle_starttag(self, tag, attrs):
        a = dict(attrs)
        if tag != "link" or "preload" not in (a.get("rel") or "").split() or not a.get("href"):
            return
        hint = f"<{a['href']}>; rel=preload"
        if a.get("as"):
            hint += f"; as={a['as']}"
        if a.get("as") == "font" or "crossorigin" in a:
            hint += "; crossorigin"
        self.hints.append(hint)


def link_header(html: str, limit: int = 12) -> str | None:
    parser = PreloadLinks()
    parser.feed(html)
    return ", ".join(list(dict.fromkeys(parser.hints))[:limit]) or None

And on the HTML response, in Django:

response = HttpResponse(html, content_type="text/html")
if (link := link_header(html)) and "Link" not in response:
    response["Link"] = link

One detail bit me, and it’s worth keeping: keep crossorigin on fonts. A font preload without it is treated as a different request, so the font gets downloaded twice and the page gets slower, not faster.

You don’t write the 103 yourself. There’s no ASGI, Granian or uvicorn interim-response code to add. Cloudflare builds the 103 from your header; your only job is to send the header.

On other stacks

#

The rule is the same everywhere: put a Link header on the HTML response and let Cloudflare do the rest.

• nginx, on the location that serves your HTML:

  location = / {
      add_header Link "</css/app.css>; rel=preload; as=style" always;
      add_header Link "</fonts/inter.woff2>; rel=preload; as=font; crossorigin" always;
  }

• Laravel with Vite already has middleware for this, AddLinkHeadersForPreloadedAssets, which turns Vite’s preloads into Link headers. Two issues reported against it are the same traps as above: its modulepreload hints have gone out without crossorigin, so browsers can discard them and download the script twice, and on large pages the header has grown past Apache’s roughly 8 KB header limit. Check both on your version.

• Anything else: set one Link header, comma-separated, or several Link headers. Either works.

What to hint is where people overdo it. Hint the render-critical CSS, the entry script and the one or two fonts above the fold, not every chunk. A long list makes the browser fetch things it doesn’t need yet, ahead of things it does. I cap mine at twelve, and only because the Expo export already curates its preload list.

If you’re not behind Cloudflare, or your CDN doesn’t do Early Hints, the Link header still helps a little: the browser can start those fetches as soon as the response headers arrive, before it parses the body. But no 103 is sent unless your server or CDN emits one.

How to check it’s working

#

curl shows the 103 directly. Ask for HTTP/2 and keep only the status lines and Link headers:

$ curl -sv --http2 -o /dev/null https://example.com/register 2>&1 | grep -E '^< (HTTP|link)'
< HTTP/2 103
< link: </_expo/static/css/reset-f5d396.css>; as=style; rel=preload, …
< HTTP/2 200
< link: </_expo/static/css/reset-f5d396.css>; rel=preload; as=style, …

The 103 arrives first, then the 200. Cloudflare reorders the attributes in the hint, which is harmless. Don’t expect every request to get a 103. Two of my first three did. The first visit after a change only stores the header, and Cloudflare skips the 103 when the final response headers are already available.

Then go back to the analytics query above and filter on requestSource: "earlyHintsCache", grouped by status and path. On the pages you fixed, misses should give way to hits.

What changed, and the caveat

#

Minutes after the header went live, the zone logged its first Early Hints hits ever, and curl showed real 103 responses on the web pages. One header fixed both problems: the feature started working, and a slice of the “504s” went away.

The caveat is that a JSON API has nothing to preload. The API endpoints keep missing and keep writing 504 lines, and on an API-heavy zone that’s still most of the volume. You have three sensible options. Send the header on the pages that benefit and accept that API paths keep logging misses; the feature works, and the charts are quieter but not silent. Exclude earlyHintsCache from your 5xx alerting and dashboards whatever else you decide, because an internal cache lookup should never page anyone. Or turn Early Hints off (Speed → Settings → Content Optimization in the dashboard, or the zone’s early_hints setting in the API) if you’ll never send Link headers; a single-page app whose HTML is tiny may get little from it, and then every lookup stops and the noise goes with it.

I kept it on. The web pages now get a head start, the API’s misses are filtered out of everything that alerts, and that’s what I should have done in September instead of switching the feature off.

If you came here to run Docker Compose in GitHub Actions: I did that for two years and stopped. CI now declares the same Postgres and Valkey images as services: and runs the same uv run … and npm run … commands straight on the runner; Compose stays what a developer runs. The rest is why, with the files.

In 2024 I wrote Use Docker Compose for CI/CD Instead of Platform-Specific YAML, which said to keep the build logic in Compose and let the CI tool do nothing but call it. Two years on, I went back through every repository I ship from. Four have workflows, each with one committer, me, and none of those workflows calls docker compose any more. The principle held; the mechanism didn’t. Here’s what replaced it, with cut-down example versions of the files, the step timings GitHub recorded, and the parts that are less automated than a CI/CD diagram would like.

Every pipeline I run, in one table

#

In textbook terms the Hugo sites have continuous deployment, and the app (a marketplace and community app I build) has continuous integration plus one-command releases, which is continuous delivery: the checks tell me something is wrong, but they don’t stop me deploying. That’s deliberate. An OTA update reaches every phone on the binary version it targets, so publishing one should be a decision rather than a side effect of a merge, and the API is where the database migrations run. My other two Django apps have no workflows at all.

The CI half: one workflow, two jobs

#

The app’s CI is one file, .github/workflows/checks.yml. Until 11 October there wasn’t any. This is a trimmed example of its shape:

# Lint, types and tests on every push and pull request to main.
# Nothing here deploys.
name: Checks

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

concurrency:
  group: checks-${{ github.ref }}
  cancel-in-progress: true

jobs:
  python:
    runs-on: ubuntu-latest
    timeout-minutes: 30
    services:
      postgres:
        image: postgres:17
        env:
          POSTGRES_USER: myapp
          POSTGRES_PASSWORD: myapp
          POSTGRES_DB: myapp
        ports: ["5432:5432"]
        options: >-
          --health-cmd "pg_isready -U myapp"
          --health-interval 5s --health-timeout 5s --health-retries 20
      valkey:
        image: valkey/valkey:8-alpine
        ports: ["6379:6379"]
    env:
      DEBUG: "True"
      DATABASE_URL: postgresql://myapp:myapp@localhost:5432/myapp
      REDIS_URL: redis://localhost:6379
      PYTHON_JIT: "0"   # match production
    steps:
      - uses: actions/checkout@v7
      - uses: astral-sh/setup-uv@v10
        with:
          enable-cache: true
          prune-cache: true
      - run: uv sync --locked
      - run: uv run ruff check .
      - run: uv run mypy myapp/
      - run: uv run python manage.py test --parallel 4 --noinput

  app:
    runs-on: ubuntu-latest
    timeout-minutes: 20
    defaults:
      run:
        working-directory: frontend
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-node@v7
        with:
          node-version: 24
          cache: npm
          cache-dependency-path: frontend/package-lock.json
      - run: npm ci
      # Expo writes this on `expo start` and it is gitignored,
      # so a fresh checkout doesn't have it.
      - run: printf '/// <reference types="expo/types" />\n' > expo-env.d.ts
      - run: npm run typecheck
      - run: npx jest --ci

The two jobs run side by side. Across the 35 green runs so far, the app job’s median is about seven minutes and the Python job’s about twelve and a half.

The services: block is where the Compose file went, which is the next section. The environment is a few variables and a flag; everything else falls back to the development default docker compose up gives a developer, so the tests need no secrets. I checked by running part of the suite with exactly that environment and nothing else.

CI runs the same interpreter as my laptop and the production image, uv’s own build of Python 3.15.0, because uv sync reads .python-version and downloads exactly that build. (My real file also passes setup-uv a python-version-file input, which does nothing: setup-uv v6 has no such input, and every run carries a warning saying so. I only noticed while writing this.)

Matching interpreters earned their keep eight days after the workflow went in. On 19 October I switched on 3.15’s JIT compiler, and the first run with it failed a test: a plain yield from, on a code path the app takes on every save, raised AttributeError: 'list' object has no attribute 'send'. The same commit with the JIT off passed. My Mac is arm64 and never hit it; the runner is x86_64, like the production machines. The JIT is off again, in CI and in the Dockerfile, and the bug never reached production. It’s python/cpython#155823, fixed upstream on 13 October, after 3.15.0 shipped.

The very first run failed too, for a duller reason. Expo writes expo-env.d.ts when you run expo start, and the file is gitignored, so every working copy I’d used had one and a fresh checkout didn’t. TypeScript reported five errors that had never appeared locally. The step that writes the file is the fix.

concurrency cancels a run when a newer push to the same branch arrives, which is how all twelve cancelled runs so far ended. A pre-commit hook runs the linter, plus the type checker when Python or TypeScript is staged, so most mistakes never reach a push.

Where the Docker Compose file went

#

Here is the part of the app’s docker-compose.yml that CI cares about, as an example. The real one has more services and a lot more environment.

services:
  valkey:
    image: valkey/valkey:8-alpine
    ports:
      - "6379:6379"

  # Host port 55432, not 5432, so it can't collide with a local Postgres.
  postgres:
    image: postgres:17-alpine
    environment:
      - POSTGRES_USER=myapp
      - POSTGRES_PASSWORD=myapp
      - POSTGRES_DB=myapp
    ports:
      - "55432:5432"
    volumes:
      - postgres_data:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U myapp -d myapp"]
      interval: 5s
      retries: 10

  api:
    build: .
    command: python manage.py runserver 0.0.0.0:8000
    volumes:
      - .:/app
    ports:
      - "8000:8000"
    environment:
      - DATABASE_URL=postgresql://myapp:myapp@postgres:5432/myapp
      - REDIS_URL=redis://valkey:6379
    depends_on:
      postgres:
        condition: service_healthy
      valkey:
        condition: service_started

volumes:
  postgres_data:

Put it next to the services: block in checks.yml and it’s the same two images: Postgres 17 (the Alpine build here, the Debian one in CI) and valkey/valkey:8-alpine. That’s the relationship now. The Compose file is what a developer runs; CI declares the same images as service containers and runs the same commands a developer runs, uv run … and npm run …, straight on the runner.

Against the 2024 post, the part I cared about most held: the CI tool is thin, and the checks and deploys are commands I can run myself. What didn’t survive is CI calling Compose.

The app’s CI doesn’t build the Docker image at all; Fly’s builder does that once, at deploy time, and CI only has to say whether the code is good, which with a cached toolchain takes a six-second install. The Compose file also bind-mounts the working copy and seeds data, which a test run shouldn’t, so CI would have needed a Compose file of its own: a second description of the same services, kept in step by hand.

And the 2024 example had a bug I didn’t notice at the time. docker compose build builds the image and never runs the service’s command, so the npm test in that file never ran in CI. The workflow went green whether the tests passed or not.

The risk that post worried about most, switching CI vendors, never came up. The move that did happen was at the other end: this blog and the client’s site moved hosts to Cloudflare Pages, and for the blog that took one new 45-line workflow, because the build was already one command. And the three-hundred-line workflow I warned about? I have one, pyhtml2text’s. It’s long because the wheels have to be built on Linux, macOS and Windows runners, for several architectures each, which makes it the one job where platform-specific YAML is the right tool.

Caching: read the step timings first

#

GitHub keeps the duration of every step, and gh api repos/<owner>/<repo>/actions/runs/<run-id>/jobs hands it over. These are from the app’s latest green run, on 20 October:

Both dependency caches are on, and both are the cheap part. The uv cache GitHub keeps for the repository is about 200 KB, because setup-uv prunes whatever uv can simply download again; the npm one is nearly 400 MB. The time is in the tests, which already run four ways in parallel. If I wanted the run faster, I’d change the suite, not the cache.

The blog is the opposite case. Hugo’s processed images live in the gitignored resources/_gen, so every fresh runner re-encoded every cover, body image and placeholder. The workflow further down restores them with actions/cache, keyed on a hash of the pictures and falling back to the newest older cache, since Hugo only processes what’s missing. In the last eight runs before the cache, the build step took 33 to 76 seconds; in the 26 since, 6 to 20, with one exception at 94, even though the same commit added placeholders and a second card size.

The app’s image builds on Fly’s builder, with the dependency files copied before the code and uv’s download cache in a BuildKit cache mount (RUN --mount=type=cache,target=/root/.cache/uv). The bigger fix wasn’t a cache. The old .dockerignore was a blocklist, and it missed a folder of extra git worktrees, each with its own dependencies and build output, so 7.7 GB across 277,000 files went to the builder on every deploy. Rewritten as an allowlist, the context is about 40 MB. An example of the pattern:

# Exclude everything, then let back in only what the Dockerfile COPYs.
*

!pyproject.toml
!uv.lock
!manage.py
!myapp
!static

# Rules match the whole relative path and the last match wins,
# so these re-excludes apply inside the allowed directories too.
**/__pycache__/
**/*.pyc
**/.DS_Store
**/*.log

Secrets, in as few places as possible

#

Secrets live in three places. GitHub Actions secrets are for what workflows do, and the two Hugo repositories have exactly one each, CLOUDFLARE_API_TOKEN. Fly secrets are the app’s runtime configuration, set with flyctl secrets set, and they never pass through Actions. And a gitignored .env on my laptop holds the token the deploy script uses to purge Cloudflare’s cache.

PyPI needs no secret at all. pyhtml2text uses trusted publishing: PyPI accepts the short-lived OIDC token GitHub issues to the workflow, so the release job asks for id-token: write and the repository’s secrets page is empty. There’s nothing to leak and nothing to rotate.

Tokens get the least they need. The app’s Fly token in GitHub serves a scheduled check that only lists machines, so it’s created with fly tokens create readonly. The Pages deploys ask for contents: read and deployments: write, and nothing more.

Inputs go through env:, never straight into a script. The OTA workflow takes free-text release notes. Written as ${{ inputs.notes }} inside run:, the text would be pasted into the script before bash reads it, so a note with a quote and a semicolon in it would become code. Passed as an environment variable, it stays data. An example of the pattern:

      - name: Publish
        env:
          OTA_API_TOKEN: ${{ secrets.OTA_API_TOKEN }}
          INPUT_PLATFORM: ${{ inputs.platform }}
          INPUT_ROLLOUT: ${{ inputs.rollout }}
          INPUT_NOTES: ${{ inputs.notes }}
        run: |
          args=("$INPUT_PLATFORM")
          [ -n "$INPUT_ROLLOUT" ] && args+=(--rollout "$INPUT_ROLLOUT")
          [ -n "$INPUT_NOTES" ] && args+=(--notes "$INPUT_NOTES")
          ./scripts/publish-ota.sh "${args[@]}"

Some keys stay off GitHub altogether. One scheduled report needs a key with admin rights over the app’s sign-in backend, so it moved from a scheduled workflow to a scheduled job on my Mac. The workflow is still there as a manual fallback, and leaves out the sections that need that key.

Deploys: on every push, by script, and on a tag

#

The Hugo sites deploy on every push

#

An example version of the blog’s workflow, with the first-run step that creates the Pages project left out:

name: Deploy to Cloudflare Pages

on:
  push:
  workflow_dispatch:

env:
  HUGO_VERSION: "0.165.0"
  PAGES_PROJECT: my-site

concurrency:
  group: deploy-${{ github.ref }}
  cancel-in-progress: true

jobs:
  deploy:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      deployments: write
    steps:
      - uses: actions/checkout@v7
        with:
          submodules: true

      - uses: peaceiris/actions-hugo@v3
        with:
          hugo-version: ${{ env.HUGO_VERSION }}
          extended: true

      # Processed images are gitignored; restore them so Hugo only
      # re-encodes what is new. An older cache still helps.
      - uses: actions/cache@v6
        with:
          path: resources/_gen
          key: hugo-resources-${{ hashFiles('content/**/*.jpg', 'content/**/*.png', 'assets/img/**') }}
          restore-keys: |
            hugo-resources-

      # A multilingual site has no root 404.html; without one, Pages
      # answers unknown paths with index.html and a 200.
      - name: Build
        run: |
          hugo --gc
          cp public/en/404.html public/404.html

      # ...

      - name: Deploy
        uses: cloudflare/wrangler-action@v4
        with:
          apiToken: ${{ secrets.CLOUDFLARE_API_TOKEN }}
          accountId: ${{ vars.CLOUDFLARE_ACCOUNT_ID }}
          command: pages deploy public --project-name=${{ env.PAGES_PROJECT }} --branch=${{ github.ref_name }}

--branch=${{ github.ref_name }} makes main production and gives every other branch a preview URL. The client site’s workflow mirrors this one, plus an rm -rf public before the build, because its public/ is committed and removed pages would otherwise linger in the upload.

Hugo is pinned here and nowhere else. In October a package-manager upgrade moved my laptop to 0.168, one past the newest version the theme supports. The deploys stayed on 0.165 and never noticed.

And the build is the test. On 14 September the preview deploy for one branch failed three times running: a post on it linked, through Hugo’s ref, to one that was still a draft. CI builds what production builds, drafts excluded, so the link pointed at nothing and Hugo stopped with REF_NOT_FOUND. main carried on deploying in the meantime.

The app’s API deploys from a script

#

The core of the deploy script, as an example:

IMAGE_LABEL="deploy-$(git rev-parse --short HEAD)-$(date +%Y%m%d%H%M%S)"

# Build first, then wake a spare machine, then roll out exactly that image.
flyctl deploy --config fly.toml --build-only --push --image-label "$IMAGE_LABEL"
curl -fsS --retry 5 https://myapp.fly.dev/health > /dev/null
flyctl deploy --config fly.toml --image "registry.fly.io/myapp:$IMAGE_LABEL"

And the matching lines in fly.toml:

[deploy]
  # max_unavailable only counts machines that are running,
  # which is why the script wakes one first.
  strategy = "rolling"
  max_unavailable = 1

The image is built once, pushed to Fly’s registry under a label made of the commit and the time, and deployed by that label, so what rolls out is exactly what was built.

Waking a spare first is what makes it zero-downtime. max_unavailable = 1 only counts running machines, and the app keeps none running when it’s idle (a suspended machine resumes in about a second), so a lone awake machine used to be replaced in the same batch as the sleeping ones, with nothing warm behind it. On 28 September a probe against the origin saw 7 timeouts and a 7.3-second gap during a plain flyctl deploy, and no failures in 2,514 requests during the scripted deploy, which builds first and has a machine awake before the rollout starts.

The rest of the script marks the deploy in the team chat channel the error alerts go to, so a burst of errors can be read against what just changed, purges Cloudflare’s cache and warms the machines. The web front end skips the image entirely; that’s in Deploy Frontend and Backend Separately, Without Redeploying the API.

Migrations aren’t a pipeline step. Fly’s release_command hung for me, so the first machine to boot on a new image takes a lock and migrates while the others wait. Every migration therefore has to work with the code that’s still running: add a column in one deploy, stop using it in the next, drop it in a third.

Nothing in the script needs my laptop as such, but a couple of local-only files keep it there: the Cloudflare credentials, and a record of what was last deployed, which a fresh runner wouldn’t have. It will also deploy uncommitted changes, and says so in the chat channel, where the web release script refuses a dirty tree.

In Preview Environments per Branch on Fly.io I wrote that doing this by hand is fine while you’re the only one doing it, and that from the second developer on it has to happen because something happened in git. The app is still at one developer; the branch-environment workflows from that post live on the branch I built them on.

The OTA workflow is manual-only because publishing should be a decision, and because the update service refuses a release while a partial rollout is open, which an on-push trigger would hit during every canary. It has never run. Every OTA update so far has gone out from my laptop through the same script the workflow calls, which is the case for thin workflows in one line: however a release starts, it’s the same script.

pyhtml2text releases on a tag

#

The release job, with the package name made generic:

  release:
    runs-on: ubuntu-latest
    if: ${{ startsWith(github.ref, 'refs/tags/') || github.event_name == 'workflow_dispatch' }}
    needs: [linux, macos, windows, sdist]
    environment:
      name: pypi
      url: https://pypi.org/p/mypackage
    permissions:
      id-token: write       # trusted publishing and signing
      contents: write
      attestations: write
    steps:
      - uses: actions/download-artifact@v8
      - uses: actions/attest@v4
        with:
          subject-path: 'wheels-*/*'
      - uses: astral-sh/setup-uv@v10
        if: ${{ startsWith(github.ref, 'refs/tags/') }}
        with:
          prune-cache: true
      - name: Publish to PyPI
        if: ${{ startsWith(github.ref, 'refs/tags/') }}
        run: uv publish --trusted-publishing always 'wheels-*/*'

The release job waits for every build job, and only a tag publishes; a manual run builds and attests but uploads nothing. actions/attest records which workflow built each wheel, and uv publish --trusted-publishing always uploads them with the OIDC token rather than a password. The package itself is in A Closed Pull Request, a Fork and a PyPI Package.

Scheduled workflows are best effort

#

The app has one scheduled workflow, a check that Fly hasn’t started more machines than the baseline, which would otherwise first show up on the bill. It alerts by failing, since GitHub emails the owner about a failed scheduled run.

It’s set to cron: '*/15 * * * *', 96 runs a day. Over the past two weeks it ran between three and eight times a day, with a median gap of about three hours forty. The only explanation GitHub’s documentation offers is load: scheduled runs can be delayed when Actions is busy, and dropped when it’s busy enough. For a sustained scale-out, a few hours is fine. For anything that has to fire within minutes, a GitHub cron isn’t a monitor.

What I’d ask of a pipeline

#

If you’re building one, or looking at one you already have:

1. If GitHub Actions were down for a day, could you still release? If not, your release logic lives in YAML, and the 2024 post still applies.
2. Does CI run the interpreter, the CPU architecture and the database major version that production runs? Mine caught a compiler bug only because the runner and production are both x86_64 and my laptop isn’t.
3. Does a fresh checkout pass? Generated, gitignored files are the usual reason it doesn’t.
4. Where does a run’s time actually go? Read the step timings before you add a cache.
5. For each secret, how many places does it live, and does it need to be in GitHub at all?

There’s no hosted APM, no tracing in production and no dashboard I have to remember to open, and the only part that costs extra is the log archive, at about US$2 a month. Here’s the whole list, and the things I dropped or stopped trusting, one of which had me chasing an outage no user ever saw.

What I mean by monitoring, and how small this is

#

By DevOps monitoring I mean finding out that something’s broken before somebody tells me, and being able to work out why without guessing. Every tool below is judged on those two things.

Last year I wrote about setting up monitoring that developers actually use, mostly with teams in mind: a cloud provider’s built-in monitoring for a small team, a hosted APM once there are hundreds of services. This is the small end of the same idea. Both products are Django on Fly’s shared-cpu-1x machines, as described in My Current Stack, and I’m the only developer on either. If monitoring is a product and developers are its users, mine has a user base of one. A baseline I pulled from Fly’s metrics in September had the marketplace’s API averaging 1% CPU. This isn’t an enterprise stack, and I won’t dress it up as one.

Of the three pillars, metrics, logs and traces, I have two. Neither product has distributed tracing: each is one Django codebase and one database, with no service boundary for a trace to cross. The only traces I read are the reverse proxy’s, on my workstation.

The list

#

Everything in the table is the marketplace unless it says both apps. The estimating app has less: Fly’s health check, logs and metrics, app analytics on iOS, and a daily Parquet export of its database that I query with DuckDB. It has no crash reporting and no alerts channel, which is the biggest gap here. (A disclosure for the update-service row: the company behind it pays me to write for their blog.)

Errors: a logging handler posting to chat

#

Until September, flyctl logs was the only place an unhandled 500, a failed push notification or a broken WebSocket frame showed up, and nobody reads that stream unprompted. Now a logging handler sends every record at ERROR or above to a chat channel, from the API, the WebSocket server and the background worker. Warnings deliberately don’t alert.

The handler posts on a background thread, so it never slows a request. Nothing in it logs, so it can’t recurse. And the same error goes out at most once every ten minutes, with a cap of 30 an hour, so an outage reads as one message rather than a thousand.

The channel also carries what I want to read errors against: deploys post when they start, when they’re up and where they failed, with the commit and who ran it, and so do over-the-air updates and web releases. A burst of wrong sign-in codes on one account logs at ERROR too, so someone guessing shows up next to the outages.

In October an alert arrived that matched no line in any log. Now each alert carries a short random id, the hostname and the process’s start command, and writes the same line to that process’s log, so an id I can’t find means a process that doesn’t log there, like an SSH shell. Everything in the channel also goes into a feed that a coding agent I work with can read, so the agent working on the code sees the errors when I do.

Logs: Fly’s shipper to S3, queried with DuckDB

#

Fly’s own buffer holds a few minutes of output, roughly the last hundred lines, which is fine for watching a deploy and no use for yesterday. So a third Fly app runs Fly’s log shipper, which is Vector subscribed to the organisation’s log stream, with one change to its S3 sink:

[sinks.aws_s3]
  type = "aws_s3"
  bucket = "myapp-logs"
  compression = "gzip"
  key_prefix = "logs/{{fly.app.name}}/%F/"   # one folder per app per day
  batch.timeout_secs = 300                  # a quiet app still lands its lines

Every line from the marketplace’s apps lands in an S3-compatible bucket as gzipped JSON. The shipper is one shared-cpu-1x machine with 256 MB, about US$2 a month plus pennies of storage. DuckDB reads the files where they sit through one macro, and a day comes back in about a second:

CREATE MACRO logs(app, day := '*') AS TABLE
  SELECT * FROM read_json_auto('s3://myapp-logs/logs/' || app || '/' || day || '/*.log.gz',
                               union_by_name := true);

SELECT timestamp, message FROM logs('myapp-api', '2026-09-06') WHERE message LIKE '% 500 %';

That macro is how I measured the cold starts in the scale-to-zero post: the startup script announces each phase, every line carries the machine’s id, and the rest is a GROUP BY.

So it’s three levels of one stream, by who reads them: the archive keeps INFO and up, which is why the apps log at INFO in production, a live-tail script shows warnings and errors, and the chat channel gets ERROR. None of them knows the caller, though. The access log has no client IP or user agent, and for that I go to Cloudflare.

Metrics: Fly’s built-in Prometheus

#

Fly runs a managed Prometheus for each organisation, so I don’t run a metrics stack of my own. I query its API when I have a question: memory, CPU and burst credit per machine (fly_instance_cpu_balance matters on a shared CPU), responses and concurrency per app. Two things I wish I’d known first. A token from fly tokens create goes in as Authorization: FlyV1 <token>; Bearer only works with the personal token from flyctl auth token, and a read-only token gets “something went wrong resolving organization”. And it keeps about two weeks (Fly says roughly 15 days).

It earned its place in September, when the logs had “Could not open a database connection” several times a day, which reads exactly like an app bug. The managed Postgres cluster is in the same Prometheus, and at every one of those minutes the primary’s scrape time jumped from about 80 milliseconds to between one and ten seconds, while the replica was fine and Postgres’s start time never moved. Postgres hadn’t restarted; the host under it was stalling. I sent Fly’s support the timestamps, and after a maintenance restart in early October moved the primary to another host, the daily stalls stopped.

Crash reports: Sentry’s SDK, pointed at my own API

#

A crash on a phone used to leave no trace anywhere I could see. The obvious answer was Sentry, and I took half of it: the app carries Sentry’s open-source SDK, but its DSN points at an endpoint in my own Django API that accepts Sentry’s envelope format, groups the crashes, and alerts through the same chat handler. Tracing, sessions and replays are off. It’s errors only.

I didn’t sign up for sentry.io because, at a few reports a day, it would have been another vendor holding user data, another line in the privacy policy and another bill. The price is symbolication, which I haven’t replaced. Stacks are minified JavaScript or raw native addresses, so an issue tells me what crashed, on which release and device, and what the person did just before, but not where in the source.

The only dashboard is a message

#

I haven’t built a dashboard for either app. Last year’s post had a test for them: if nobody opens one unprompted, it isn’t earning its place. A message that arrives on its own doesn’t need opening, so the closest thing I have is a post in the team chat every morning.

It starts with the last 24 hours of production errors, read from the log archive rather than from the chat channel, because throttled alerts can’t say how often something happened. Numbers, ids and addresses are masked before grouping, so a hundred copies of one failure are one row with a count and no email reaches the channel.

Then failed health checks, yesterday’s usage against the same weekday a week earlier (a Sunday is never like a Monday), the last week of search traffic, chats waiting on a reply, the task board and what I’ve spent. A section that can’t load says so, and the post still goes out.

Reading app analytics through an API teaches you to distrust them a little. In August, 121 Android first-opens from Azerbaijan were followed by 119 uninstalls, and I read that month’s Android totals with them taken out.

Releases have their own screen. When over-the-air updates came from my own server, my view of a release was an admin list with an “is active” checkbox, and I had no idea how many phones had taken one. The hosted service’s dashboard shows which release devices run, how installs went, and a crash rollback when a bad bundle crashes and the SDK restores the previous one. I wrote up the move to a hosted update service separately.

What I dropped, or stopped believing

#

None of this was a paid tool. Mostly it was things I’d been treating as signals that weren’t, and one job running in the wrong place.

Cloudflare’s 504 rate, until I grouped it by user agent

#

In September the Cloudflare dashboard showed what looked like 22% of a day’s requests failing with 504s, concentrated on the endpoints the app polls every 30 seconds, while every health check was green and the app’s own logs said those requests were fast. I spent days on it, found that machines took 25 to 33 seconds to cold-start and did it 92 times a day, fixed that, and wrote it up in the scale-to-zero post as the cause.

The next morning I grouped the same 504s by user agent. All 7,344 of them between 5 and 9 September came from Cloudflare’s own Early Hints cache lookups (user agents bastion early hints and nginx-ssl early hints), which never reach the origin: a 504 on one of those lines is a cache miss, not a timeout. They copy real request paths, so they landed on the busiest endpoints and looked exactly like users failing. The app, browsers, curl and search crawlers had none. Early Hints wasn’t doing anything for me because the origin sent no Link header, so I turned it off, and later made it work.

The cold-start work still paid off, since deploys and wakes are faster for it, but the outage I thought I was fixing never reached a user.

So this is the rule I’d most like to pass on: before you believe a CDN’s error rate, group it by user agent and check it isn’t the CDN’s own machinery. On Cloudflare’s free plan that’s a GraphQL query on httpRequestsAdaptiveGroups, a day at a time. The same query settled the next scare, when one polling endpoint went from 70–260 requests a day to 2,880. It wasn’t a scraper. It was my own Mac, with the app left running in the iOS simulator after testing.

A green health check

#

Fly checks /api/health/ on both apps every 30 seconds and uses the answer to decide which machines get traffic, which it’s good at. But a health check runs against a machine that’s already up, so it can’t see what happens before one is: a cold start, a queue at the proxy, a request that never arrives. Routing is all I ask of it now.

Deploys get the probe script instead, which hits the origin throughout and exits 1 if anything failed. It caught a plain flyctl deploy leaving a 7.3-second gap, then counted no failures in 2,514 requests through the deploy script, which wakes a spare machine first.

The health check has also caused two outages of its own. In July, Django’s SECURE_SSL_REDIRECT answered Fly’s plain-HTTP probe with a 301, every machine failed its check, and everyone got a 503. In August, narrowing an except Exception in the health view let a “connection already closed” error escape as a 500, and Fly pulled the machine in exactly the case the view’s retry was there for. Now the check only asks whether the process can serve, and a database stall no longer takes both apps out of routing.

The morning summary had to learn the same lesson about counting. The marketplace suspends idle machines, so a machine woken by a request fails its first check a few hundred milliseconds after starting and passes the next. Counting those made every morning read like an outage, so a failure that passes again within a minute is now listed separately, as machines waking up and deploys.

A scheduled job in CI

#

The morning summary started as a CI workflow on a cron schedule and moved within a day. Once it read the analytics and search data, the key it needed was the analytics project’s service-account key, which can do anything in that project, and I didn’t want that sitting in CI secrets. Now a scheduler on my Mac runs it every 20 minutes, and it posts once it’s past eight in Auckland and today’s post hasn’t gone out, so a Mac asleep at eight posts when it wakes. The workflow stays as a manual fallback, minus the analytics and search sections.

If I were starting again

#

I’d set up three things before the first user rather than after the first incident: errors to a chat channel the team already reads, with a rate limit; a log archive in object storage I can query with SQL; and whatever metrics the platform already collects. That’s a couple of dollars a month. Crash reporting comes the week an app goes onto phones, because otherwise a crash on a phone tells nobody anything. A morning summary comes once there are enough places to check that I’d stop checking them.

I’d leave tracing until there’s a second service for a request to cross, and a hosted APM until there’s a team that needs shared dashboards, an on-call rota and a budget for someone else’s bill. That’s when the tools in last year’s post start paying for themselves. Before then the most useful habit is free: when a number looks alarming, find out what produced it before deciding what broke. Several of the scares in this post came from the monitoring, not the product.

My working setup is three machines that are rarely in the same place. There’s a MacBook Air, which goes wherever I go. There’s a workstation in the office called devbox, with a real GPU, which runs Ollama, my fine-tuning tools, a small Kubernetes cluster with its own container registry, and whatever agent I’ve left grinding through a long job. And there’s my phone, which is where the apps I build actually get used, and which spends most of its day on mobile data rather than wifi.

For a long time those three didn’t really talk to each other. The laptop could reach devbox when it was on the office LAN and not otherwise. The phone could reach my laptop’s dev server when both were on the same wifi, and then only if I remembered to bind to 0.0.0.0 and typed in whatever DHCP had given the Mac that morning. Everything interesting on the workstation, the Ollama API, the cluster dashboards, the traces, the registry, was bound to localhost or to the cluster network, deliberately, because none of it belongs on the internet.

This post is how I joined them up with Cloudflare WARP and a pair of Cloudflare Tunnels, what it changed about how I build AI features and mobile apps, and the handful of things that broke without telling me.

What I actually wanted

#

Stated as requirements, it’s short:

1. The laptop should be a thin client. Anything that needs a GPU, a lot of RAM or hours of runtime should run on devbox, and the laptop should reach it the same way from the office, from home and from a café.
2. The phone should reach my dev servers and my shells from anywhere, on mobile data, without a VPN I have to remember to switch on and off.
3. The things I use to see how a system is behaving, dashboards, traces, docker info, model lists, should be reachable by me and nobody else, and should never get a public URL.
4. No inbound ports. Not on the office router, not on my home router, not anywhere.

The fourth one rules out the old answer, which is a port forward and a dynamic DNS name. It also rules out a lot of cleverness: both devbox and my laptop sit behind NAT I don’t control, and the laptop is behind a different NAT every few hours.

Before: a repo of SSH tunnels

#

The first version of this was a small repo of my own, and it’s still the base layer. It solves the “clean names for things I reach over SSH” problem on the laptop and nothing more. dnsmasq answers *.test with 127.0.0.1, Caddy routes by hostname to a local port with tls internal so the browser is happy, and a set of autossh launch agents hold the SSH forwards open and reconnect when they drop. The hostnames say where a thing lives:

So https://devbox-ollama.ssh.test/api/tags lists the models on the workstation, and https://api.local.test is the API of the app I’m working on, running on my own machine.

Subdomains of .test, not new top-level names like .ssh or .cloud, and not .local. RFC 6761 reserves .test for exactly this, and .local belongs to mDNS; take it over and Bonjour discovery stops working on your Mac. dnsmasq already wildcards the whole of *.test, so the extra label costs nothing: one /etc/resolver/test file pointing at 127.0.0.1 and no further DNS changes, ever.

HTTPS on names that don’t exist

#

Every one of those names is served over HTTPS with a certificate the browser trusts, which sounds impossible for a hostname that exists only on my laptop. Caddy does it with its own internal certificate authority. tls internal on a site tells Caddy to issue that site’s certificate from a local CA instead of asking Let’s Encrypt, and caddy trust, run once by the installer, puts the CA’s root into the macOS System keychain (Chrome and Safari read it) and into Firefox’s own store. From then on https://anything.test opens with no certificate warning, plain http:// redirects to it, and nothing leaves the machine.

It isn’t just cosmetic. A lot of the web platform now refuses to work on plain HTTP: secure cookies, service workers, the clipboard API, anything with SameSite=None. Testing a web build on plain http://localhost hides the bugs that show up on the real HTTPS domain. Testing on https://app.local.test surfaces them on my laptop instead.

A site block, which is the same for every service apart from the port:

devbox-ollama.ssh.test, devbox-ollama.test {
	tls internal
	import cors
	reverse_proxy 127.0.0.1:11435 {
		import proxyhdr
		transport http {
			keepalive off
			dial_timeout 5s
		}
	}
}

The two snippets carry the fiddly parts. cors reflects the caller’s Origin back rather than sending *, because * isn’t allowed once credentials are involved, and it answers OPTIONS preflights itself with a 204 so the upstream never has to. proxyhdr rewrites the Host header to the dial address. Fussy backends, router admin pages and Chrome’s remote-debugging endpoint among them, validate Host, and would otherwise reject devbox-ollama.ssh.test as a name they’ve never heard of. Ollama’s own origin check accepts loopback on any port, so rewriting Host to 127.0.0.1:11435 gets it through.

keepalive off is the one I’d have got wrong. The upstreams are SSH forwards that autossh tears down and rebuilds whenever the network changes. With keep-alive on, Caddy holds a pooled connection to a tunnel that has since died, and it returns baffling 404s until it’s reloaded. Off, every request dials fresh, which on loopback costs nothing measurable.

Each service also answers on its old flat name (devbox-ollama.test), so bookmarks from before the suffix scheme still work.

The menu bar

#

With a dozen forwards held open by launchd, the question I kept asking was “is it up?”, and the answer lived in three places: launchctl list, lsof and a curl. So a small menu-bar plugin puts it in one place, with a dot next to every hostname:

• 🟢 the service answered a request through the tunnel
• 🟡 the tunnel is open but the service behind it is silent
• 🔴 nothing is listening on the local port, so the tunnel itself is down

The yellow state is the useful one. A green/red check can’t tell “the SSH forward died” from “Ollama crashed on the far end”, and those have entirely different fixes. The plugin keeps no list of its own; it reads the same proxy and SSH config as everything else, so adding a service there adds it to the menu too.

The first version was greedy. It refreshed every ten seconds and spawned a few processes per service, about seventy a run, and macOS promptly listed it under “apps using significant energy”. Taking one snapshot per run and refreshing every few minutes brought it down to about 0.1% of a core. A status light that drains the battery is not helping, so it’s worth budgeting for from the start.

The whole setup had one hole, and it was the one that mattered. devbox has no public name. Its SSH forward dialled a 192.168.x address, so it only worked while the laptop was on the office LAN. Away from the office the menu bar went red and stayed red, and everything I’d moved onto the workstation to save the laptop was out of reach at exactly the times I wasn’t sitting next to it.

Two tunnels, pointing the right way

#

The fix has three parts, and the first two are Cloudflare Tunnels.

A Cloudflare Tunnel is cloudflared holding an outbound connection to Cloudflare’s edge. Cloudflare hands requests back down that connection, so a machine behind any amount of NAT gets reachable without anything listening on a router. It follows a laptop onto hotel wifi and it survives CGNAT.

I run two of them, one per machine, and the reason is worth a sentence. The obvious shortcut was a single tunnel on the laptop with one more ingress rule pointing at devbox’s SSH port on the office LAN. That needs nothing installed on devbox at all. It also routes through the laptop, and the laptop is the machine that leaves the office. It would have broken in exactly the case it existed to fix. devbox stays put, so devbox holds its own connection to the edge.

# config.yml on devbox (runs as a systemd service)
protocol: auto        # prefer QUIC, fall back to HTTP/2 over TCP 443
ha-connections: 4     # mains-powered, its job is to stay reachable

warp-routing:
  connectTimeout: 5s

ingress:
  - hostname: ssh-devbox.example.com
    service: tcp://127.0.0.1:22
  - service: http_status:404

tcp:// rather than ssh:// is an easy one to get wrong. ssh:// selects Cloudflare’s browser-rendered terminal, which wraps the stream for a web page; a normal ssh client fed through it gets the banner and then dies at key exchange. tcp:// is the plain byte pipe cloudflared access ssh expects, and as a bonus it carries scp, rsync, git over SSH and port forwards.

That last part is what closed the hole in the SSH layer. The SSH config now picks a route per connection:

Match originalhost devbox !exec "nc -z -G 1 <office-lan-ip> 22 >/dev/null 2>&1"
    ProxyCommand cloudflared access ssh --hostname ssh-devbox.example.com ...

On the office LAN the probe succeeds and SSH goes direct, one hop. Anywhere else it fails and SSH goes through Cloudflare, authenticated with an Access service token so there’s no browser login in the way. It’s Match originalhost, not Match host: host matches after HostName has been substituted, so by then the name is already an IP and the block never fires. The upshot is that devbox-ollama.ssh.test works in the office, at home and on a train, and the menu bar dot stays green.

The laptop’s tunnel goes the other direction. It gives dev servers on the Mac real public HTTPS hostnames: for each app I’m working on, its API, its WebSocket server and its web build. Every one of them sits behind Cloudflare Access, which I’ll come back to, because it’s the part that makes this safe rather than reckless.

The menu bar learned about both tunnels too, and it couldn’t do it by probing. An inbound tunnel has no local port to check, one of the two connectors doesn’t even run on the laptop, and the obvious test, curling ssh-devbox.example.com, gets an answer from Cloudflare’s edge whether or not a connector is attached, so it would read green on a dead tunnel. launchd is no better: cloudflared sits there “loaded, exit 0” with every edge connection dropped. So for tunnels the menu asks the Cloudflare API whether each one has a healthy connector, since that’s what decides whether an ssh lands. If the API can’t be reached, the section is left out rather than drawn red. The menu is read at a glance, and a false alarm there costs more than a missing line.

The third part: WARP makes it a network

#

Tunnels with hostnames are great for anything that speaks HTTPS, and for SSH from a machine that can run cloudflared. A phone can’t. And a tunnel hostname isn’t a TCP endpoint: Cloudflare’s edge only terminates HTTPS and WebSocket on it, so port 22 at ssh.example.com is closed and no amount of fiddling with Access will open it.

That’s the job for WARP. The Cloudflare One Agent (the enterprise WARP client, not the consumer “1.1.1.1” app, which can’t join an organisation) is enrolled on the laptop and the phone in my Zero Trust organisation. Each tunnel advertises private routes, and a device running WARP can then open a plain socket to those addresses and land on the right machine. It stops being “reach this service by name” and becomes “these machines are on a network with me”:

Those addresses are loopback aliases on each machine, created at boot, so they owe nothing to DHCP or whatever router I’m behind. 198.18.42.2 is devbox in the office, from home, and from a phone on 5G. The Mac’s real LAN address changes every time I move; this one never does.

The split tunnel is in include mode with only those routes in it (plus the Cloudflare ranges WARP needs for itself). That’s what makes the client liveable. The default is exclude mode, where WARP carries everything except a list of private ranges, so every byte the phone sends detours through Cloudflare. That has a real battery and latency cost, enough that I’d have ended up turning it on only when I needed it, which defeats the point. In include mode, traffic to my three machines goes through WARP and everything else goes out the normal path untouched. So the client just stays on. warp-cli status on the laptop says Connected all day and I don’t think about it.

For transport I have it on MASQUE, which is HTTP/3 on UDP 443 with an HTTP/2 fallback, rather than WireGuard on UDP 2408. Some carriers and guest networks block 2408; almost nobody blocks 443. My organisation’s policy also has post-quantum key agreement switched on for MASQUE, which I didn’t go looking for but won’t turn down.

Why 198.18, and why there’s an IPv6 address

#

The addresses look odd, and the first ones I picked were wrong in an instructive way.

My first attempt was 10.10.0.1, because that’s what you reach for. It fails in both split-tunnel modes. In exclude mode, WARP’s default list excludes every RFC 1918 range plus 100.64.0.0/10, so the client would have dropped traffic to it before it ever left the device. In include mode it would collide with any real 10.x network I happen to join; cafés, co-working spaces and plenty of offices use exactly that range. 198.18.0.0/15 is the RFC 2544 benchmarking range. It’s reserved, it’s never routed on the public internet, it isn’t in WARP’s exclude list, and no ordinary network uses it. The IPv6 addresses are ULA (fd00::/8) for the same reason.

Why there’s IPv6 at all is the more interesting story. Once the v4 address worked I tested it from the phone on wifi, it was fine, and I called it done. On 5G it didn’t work. The Agent said connected, the tunnel was healthy, Access was fine; the wifi test proved all of that. The problem was that mobile networks are mostly IPv6-only now. The phone had no native IPv4 at all and was reaching the v4 literal through the carrier’s 464XLAT translator, which sits outside the tunnel when the include list holds only a v4 prefix. The traffic went out to the public internet, where 198.18.0.0/15 doesn’t route, and was dropped without a sound. No error on the phone, nothing in the connector’s log.

The fix is to give every machine a second address the phone can reach natively over IPv6, and add it to the routes and to the include list. Now the fd12:3456:789a:: addresses are what my SSH client on the phone actually uses on mobile data.

Offloading: the laptop as a terminal

#

This is the part the whole setup exists for.

Models and fine-tuning. Ollama runs on devbox and binds to 127.0.0.1:11434. It’s invisible to the rest of the office LAN, let alone the internet. On the laptop it appears at https://devbox-ollama.ssh.test, or as a bare port for tools that want one:

OLLAMA_HOST=127.0.0.1:11435 ollama list

11435, not 11434, on purpose: 11434 is Ollama’s own default, and if the laptop ever runs a local Ollama too, the two would race for the port and I’d silently get devbox’s answers from the “local” one depending on which started first.

The practical effect is that when I’m building an AI feature, the model it calls is a workstation GPU rather than the laptop’s, from wherever I am. Evaluation runs, embedding a pile of documents, a fine-tune: they run on devbox, and the laptop’s fan stays off. When I’m prototyping an agent, it can point at a local model on the workstation and burn through tool-calling iterations without an API bill, and switch to a hosted model once the loop is right.

Long-running agents. An agent session that runs for an hour shouldn’t depend on my laptop lid staying open. Run it on devbox instead and the laptop is just a window onto it: close the lid, and later ssh devbox from wherever I am, or from the phone, and it’s still there. Before the tunnel this only worked in the office, which is exactly where I didn’t need it.

Builds and the cluster. The cluster on devbox has its own registry, so images can be built and pushed on the workstation, next to where they run, rather than uploaded over hotel wifi from a laptop. The cluster dashboard, the ingress dashboard and the registry UI are all *.ssh.test hostnames on the laptop.

The mobile loop. The phone is on the same private network as the laptop, and the laptop’s dev servers also have public hostnames behind Access. So I can take a native build of the app out of the building, on 5G, and it talks to the API running on my desk. Or it talks to the one on devbox, if that’s where I left it. The feedback loop on a real device over a real mobile network used to mean deploying to a shared staging server. Now it means saving a file.

There’s one trap here worth knowing before you debug it as an app bug. Access protects the WebSocket hostname too, and a native WebSocket client isn’t a browser. It carries no CF_Authorization cookie, so the native build’s connection to the WebSocket server is rejected at the edge with a 403 that looks nothing like an error from your own server. Browser tabs are fine because they already hold the cookie from logging in. For anything that isn’t a browser (the native app, a script, an agent calling a dev API) the answer is an Access service token, sent as two headers:

CF-Access-Client-Id:     <client_id>
CF-Access-Client-Secret: <client_secret>

The token grants nothing on its own. Access only honours it on apps whose policy names it, so its reach is exactly the list of dev hostnames and nothing wider. I file its policy under the non_identity decision rather than allow, which keeps it visible in the audit log as a bearer secret and not a person who logged in.

The stats the internet never sees

#

The other half of what I wanted was visibility. Most of what tells you how a system is behaving is, correctly, not public. On devbox that’s:

• the ingress proxy’s dashboard, and the tracing UI that holds its traces
• a dashboard for the Kubernetes cluster
• the container registry’s UI (read-only)
• Ollama’s model list and what’s loaded right now
• docker info and container stats

None of those have a public hostname, and none ever will. They’re bound to loopback or to cluster service IPs, reached over an SSH forward that itself rides the tunnel. A monitoring app on my phone reads docker info --format '{{json .}}' over SSH across WARP, so I can see from anywhere whether my containers are up or something has eaten all the memory.

That app is how I found a small bug worth mentioning. The tunnel sshd on the Mac was at LogLevel DEBUG2 from when I was debugging it. With -e, sshd logs to stderr, and for a session that stderr is the SSH channel’s stderr. Every command’s output came back with debug2: do_setup_env: set TMPDIR appended, and the phone app, which merges the two streams, got valid JSON followed by a log line, failed to parse it, and showed the whole thing as an error. LogLevel INFO fixed it. Put log levels back after debugging.

Just as deliberate is what I don’t put on this network. The old autossh tunnels also reach a few routers on other people’s networks that I help look after. Those get no public hostname and no WARP route. A router admin page on the internet is a liability even behind a login, and it isn’t my network to widen.

Access is not optional

#

A Cloudflare Tunnel hostname is on the public internet the moment its DNS record exists. There’s no firewall in front of it; that’s the whole point of not opening a port. The protection has to come from Cloudflare’s edge, and that’s what Access does. It refuses to forward a request at all until the visitor has proved who they are, so an unauthenticated scanner never reaches the laptop. That matters, because these are dev servers: a debug-mode traceback with settings in it, an open JavaScript bundler. Nothing behind these hostnames was built to face the internet.

The policy allows my own email addresses and nothing else, with a one-time PIN by email and single sign-on as the two ways to prove it. Sessions last eight hours, because the realistic threat is a laptop left open in a café with a day-long cookie, not someone brute-forcing a PIN.

SSH is authenticated twice, by two systems that don’t know about each other. Access decides whether a TCP stream may reach sshd at all; sshd still demands my key. Losing either doesn’t open the machine. The Mac’s tunnel sshd also trusts Cloudflare’s SSH CA, so Access can log me in with a short-lived certificate minted from my SSO session, while authorized_keys still works alongside it, so a broken Access config can’t lock me out of my own laptop.

The piece I’m happiest with is small. Exposure happens when three things line up: a DNS record, a running connector and an active zone. They become true at different times, in an order I don’t control; the zone flips to active on its own, some time after the nameservers change, quite possibly while the Mac is asleep. So the launch agent doesn’t run cloudflared directly. It runs a preflight that reads the hostnames out of the tunnel’s own config, asks the Access API which ones are covered, and refuses to start if any aren’t:

HOSTNAMES=$(grep -E '^\s*-\s*hostname:' "$CONFIG" | sed -E 's/.*hostname:\s*//' | sort -u)
PROTECTED=$(curl -fsS -H "Authorization: Bearer $CLOUDFLARE_API_TOKEN" \
    "https://api.cloudflare.com/client/v4/accounts/$CLOUDFLARE_ACCOUNT_ID/access/apps?per_page=100" \
    | python3 -c 'import sys,json; print("\n".join(a.get("domain","") for a in json.load(sys.stdin)["result"]))') \
    || { echo "could not reach the Access API. refusing (fail closed)."; exit 1; }

for h in $HOSTNAMES; do
    grep -qxF "$h" <<<"$PROTECTED" || { echo "NOT starting: $h has no Access app"; exit 1; }
done
exec cloudflared tunnel --config "$CONFIG" run

The launch agent has KeepAlive, so a refusal retries by itself every 30 seconds and the tunnel comes up on its own once the Access apps exist. Failing closed costs an outage. Failing open costs an exposure. I know which of those I’d rather explain.

Four things that bit me

#

Everything above works now. Getting there, three of the four had the same pattern: the thing that was broken produced no error anywhere, and the parts I could see all reported healthy.

The route and the include list are two different lists. A tunnel route makes Cloudflare willing to carry an address to the right connector. The WARP include list is what makes the client send it in the first place. devbox was unreachable from the phone because its routes existed but the include list still only had the Mac’s addresses. The phone dropped the packets locally, the connector never saw a flow, and every dashboard was green. Every new machine means adding its addresses to both.

IPv4 literals on IPv6-only mobile networks. Covered above: works on wifi, fails on 5G, no error. If a WARP route works on wifi and not on mobile data, and the Agent shows connected, it’s routing, not the tunnel.

macOS sshd behind cloudflared. macOS starts its sshd with launchd socket activation, one short-lived sshd -i per connection. Behind cloudflared, the banner reaches the client and then the stream is torn down before the client’s identification string gets back, so every login dies at kex_exchange_identification. I confirmed it from both ends and reproduced it with a plain TCP relay, which ruled out Cloudflare. The fix is a second, standalone sshd -D on 127.0.0.1:2222 and the WARP loopback addresses, running as my user, that exists only to be the tunnel’s origin. Remote Login in System Settings is untouched. devbox, being Linux with a normal long-lived sshd, never had the problem.

Config fields that stop existing. Older examples have warp-routing: enabled: true in the tunnel config. cloudflared 2026.8 rejects that field outright. WARP routing is on whenever the block exists and a route points at the tunnel. That one at least produced an error, which after the other three felt almost generous.

What it changed

#

The honest summary is that I stopped choosing where to work based on where the hardware is. The laptop is a good keyboard and a good screen with a long battery life, and it doesn’t need to be more than that. The workstation does the heavy lifting from wherever I am. The phone is a real test device on a real network rather than something tethered to office wifi. And the parts of the system I use to see what it’s doing are one click away for me and don’t exist for anyone else.

At this size it fits in Cloudflare’s free Zero Trust plan. What it cost me was time and a README I’m glad I wrote, because every one of the failures above would have been a fresh mystery the second time.

If you’re thinking about something similar, these are the questions I’d ask of your setup:

1. What runs on your laptop only because the laptop is the one machine you can always reach?
2. Can your phone reach your dev server on mobile data, not just on the office wifi? Test it on 5G, not wifi.
3. Which of your dashboards are public because that was the easy way to see them from outside? Each one is a candidate for a private route instead.
4. If a tunnel hostname came up without its Access app tomorrow, what would stop it? If the answer is “I’d remember”, write the preflight.
5. Are your private addresses in a range the networks you visit will never use?

Picture a small team working on one app. Two developers, or five; not specialists, but people who each take a feature from the database migration through the API to the screen in the mobile app, end to end, on their own branch. It’s a good way to run a team, and from the second developer onward it has one recurring failure: the message “Is anyone on staging?”

Behind that message is a developer who has finished a feature and wants a tester or a designer to see it, and can’t, because staging is running the second developer’s branch, and dev is running the third’s, which broke login two days ago and hasn’t been redeployed since. A handful of people, two shared environments, and a chat thread doing the scheduling.

That isn’t a people problem. It’s a topology problem. Every developer is treating the environment as the place where their branch becomes real, so the environment is doing the job the merge is supposed to do, one branch at a time, and everyone else is queueing for it.

I described the production shape of my stack in My Current Stack: Django on Fly.io’s cheapest machines, Postgres behind a connection pooler, Redis for the cache and the Channels layer, a separate WebSocket app, a background worker, an S3-compatible bucket for files, a hosted auth provider, Cloudflare in front, and a mobile app talking to all of it. This post is how I turned that stack into something where any branch gets its own public, HTTPS-fronted copy of the whole thing a couple of minutes after its developer puts one label on the PR, with nobody else asked and nobody provisioning it, and without a second Postgres cluster, a second Redis or a second bucket. The infrastructure stays shared. What changes is that every shared resource learns to be sliced by a name, and the name comes from the branch. Nothing in it assumes a team size: it’s the same design for two developers as for twenty, and what grows is the app list, which the sweeper handles. I built it, ran three branches at once, broke it in two places I hadn’t predicted, and tore it down, all in an afternoon; the outputs below are from that run.

Why one shared environment can’t scale past one developer

#

A shared dev environment has an assumption baked in: that there is one “current version of the code” that everyone wants to look at. That’s true for one developer, or for a team working a single stream of changes in lockstep. It stops being true the moment the second developer starts a feature on a different timeline, and with end-to-end ownership that’s every feature. From then on, everything the environment holds is contended:

• The deployed code. Whoever deployed last wins. Everyone else’s work is invisible until they redeploy, which undoes the first person’s.
• The database schema. This is the one that bites hardest when every developer owns their feature’s data model. The first branch adds a column. The second, deployed an hour later, doesn’t know about it and its ORM starts failing on inserts. Or both branches have a 0042_* migration, the shared database ends up in a state that matches nobody’s migration history, and someone resets it, deleting the test data the third developer spent an afternoon setting up.
• The cache. One branch’s serialiser change is cached under the same Redis keys another branch’s code reads. The bug reports that produces are baffling and cost a morning each.
• Third-party integrations. One OAuth callback, one webhook URL, one push certificate. Whoever’s branch is deployed gets the events.
• The tester. They can only test what’s deployed, so they test one feature at a time in the order the environment happened to be claimed, not the order the features are ready.

The team responds with a booking system: a chat thread, a pinned message, a bot with /claim staging. That’s a lock, and a lock on the only place your work can be seen is a lock on delivery rate. With two developers it’s an irritation. With three it’s tolerable on a good week. Add a fourth, or have one feature hold staging for three days of back-and-forth with a designer, and it stops being tolerable.

The first fix people reach for is a third shared environment, “dev2” or “uat”. That buys a few weeks. The queue just has two lanes.

The real fix is to notice that dev and staging were never environments. They were the current state of someone’s branch, with a URL. If that’s what they are, the right number of them is the number of branches the team cares about right now, and the right lifetime is the lifetime of the branch.

The model: the branch is the environment

#

Here’s the target state.

Read the Created by and Destroyed by columns again: every entry is a git event or a clock. Git is the control plane. The set of environments that should exist is a function of the set of branches that exist, and the automation’s only job is to keep reality matching it: a label on the PR creates, every push after that updates, closing or merging the PR destroys, and a nightly job reconciles whatever the first three missed. Nor does every branch get one: the label is the gate, one click by the developer, so a backup push or a bot’s dependency bump costs nothing. Nobody provisions an environment. Not a DevOps engineer with a ticket, not a bot you ask in chat, not an AI agent you prompt each time. The tenth developer’s first branch gets an environment the same way the first developer’s did, by labelling its PR, and if a person has to be asked, it hasn’t scaled; the queue has just moved.

And the rule that makes it work: every resource an environment touches is named after the branch, deterministically. Not after the developer, not after a ticket, not after a PR number that doesn’t exist until someone opens one. After the branch, because the branch is what developers actually work with, and because a deterministic name means the second run of ./scripts/env up on the same branch finds the environment it already made rather than making another.

The name is a slug for humans and a short hash for uniqueness:

branch=$(git rev-parse --abbrev-ref HEAD)              # feature/payments-retry
slug=$(printf '%s' "$branch" | tr '[:upper:]/_ ' '[:lower:]---' | tr -cd 'a-z0-9-' | cut -c1-20)
hash=$(printf '%s' "$branch" | sha256sum | cut -c1-6)
ENV="${slug%-}-${hash}"                                 # feature-payments-retr-3f9a1c

feature-payments-retr-3f9a1c is the environment. That string becomes the Fly app name, the Postgres database name (with hyphens turned to underscores), the Redis key prefix, the bucket’s object prefix, the error tracker’s environment, and the left-hand label of the hostname. Anyone who knows the branch name can compute it. Nothing has to be looked up.

That last sentence is the design. There is no registry of environments and no coordinator. env up, env down, the Cloudflare Worker that routes hostnames and the nightly sweeper each recompute the name from the branch independently and agree by construction. Cloudflare in particular is not coordinating anything: the Worker is a pure function from hostname to Fly app, and it never learns that an environment was created or destroyed. If the app exists the request lands; if it doesn’t, Fly answers with its own error.

Three things fall out of this.

First, “who’s on staging” stops being a question. There is feature-payments-retr-3f9a1c-dev.example.com and fix-login-redirect-8b21e0-dev.example.com, and each belongs to whoever is on that branch.

Second, staging becomes an honest rehearsal of prod rather than a dumping ground. It’s built from the exact image that will go to prod, against a database cloned from a prod-shaped snapshot minutes earlier. If it works there, the only variable left in prod is prod’s data.

Third, each developer gets power without getting root on anything shared. The blast radius of a branch environment is that branch environment. A fourth hire on their first day can deploy whatever they like to their branch, and the worst outcome is that their branch is broken.

fly.io/dashboard/my-org

Shared physically, separated logically

#

The tempting version of “an environment per branch” is a full copy: a Postgres cluster each, a Redis each, a bucket each. It’s clean and it’s the wrong trade. A Fly Postgres cluster is a machine plus a volume plus a restore; a Redis is another machine; you’d be waiting minutes for each environment and paying for a dozen idle databases. The sweeper would have four kinds of thing to clean up instead of one.

Instead, each shared service is provisioned once for the whole preview org, sized like a small prod, and every environment gets a slice of it keyed by $ENV. This is the table I’d want on the wall:

Two of those need a word.

Postgres: database, not schema. The other way to slice one cluster is a schema per environment, with search_path set per connection. Django will do it if you push a -c search_path=... through the connection options, but a connection pooler in transaction mode discards startup options unless you tell it not to, and then every connection in the pool has to belong to one environment anyway. It’s a source of “why is my query hitting the wrong tables” bugs that a database per environment simply doesn’t have. A database is a hard boundary, CREATE DATABASE ... TEMPLATE makes one in seconds, and DROP DATABASE makes it go away with nothing left behind. Branch environments connect to Postgres directly on port 5433 rather than through the pooler, the same way the background worker does in prod; the pooler earns its keep at production concurrency, which a branch environment never sees.

Secrets: generated, not shared. It’d be easy to have one set of dev secrets and copy it into every environment. Don’t. The env up script generates a fresh SECRET_KEY (session signing), ENCRYPTION_KEY (anything encrypted at rest) and ENV_TOKEN (the bearer token the mobile app presents, more on that below) for each environment and stores them only in that Fly app’s secret store. It means a session cookie from one branch is worthless on another, a leaked preview token opens one preview, and the developer never sees the values unless they go looking. If you keep secrets in a separate secrets manager rather than only in Fly, the same rule applies: the path is myapp/preview/$ENV/, and the sweeper deletes the path with the environment. The only secrets that are shared across environments are the credentials for the shared infrastructure itself (the Postgres admin URL, the Redis URL, the bucket key), and those live in CI, not in any environment.

The Redis row is the easiest to doubt, so here is the test. Set a cache key from inside one environment, read it from the other, then list every key in the shared Redis:

$ fly ssh console -a myapp-feature-profile-tagl-fa8832 -C "python manage.py shell -c \"from django.core.cache import cache; cache.set('who-am-i', 'tagline-branch', 600)\""
$ fly ssh console -a myapp-feature-profile-webs-dcb2e7 -C "python manage.py shell -c \"from django.core.cache import cache; print(cache.get('who-am-i'))\""
None

$ # every key in the shared Redis
  feature-profile-tagl-fa8832:1:who-am-i
  ...

Same Redis, same key name, the other environment sees nothing.

Everything else in this post is the script that stamps $ENV onto those eight rows, and the rules that keep it honest.

./scripts/env up

#

This is the whole idea, as a script a developer runs from their branch. CI runs the same script; there’s no separate CI path to drift. It needs fly and git on the laptop and nothing else: SQL goes to the Postgres app over fly ssh, and the environment’s Redis keys and bucket objects are removed by the environment’s own machine before it’s destroyed, so the laptop never holds the shared Redis or bucket credentials beyond a gitignored env file. Trimmed to the lines that matter:

#!/usr/bin/env bash
# scripts/env up|down|name|url: one environment per branch on shared preview infrastructure
set -euo pipefail
ORG=${PREVIEW_ORG:-myapp-preview}; PG_APP=${PREVIEW_PG_APP:-myapp-preview-pg}; DOMAIN=${BASE_DOMAIN:-example.com}

branch=${BRANCH:-$(git rev-parse --abbrev-ref HEAD)}
[[ "$branch" == "main" ]] && { echo "main has staging, not a branch environment" >&2; exit 1; }
slug=$(printf '%s' "$branch" | tr '[:upper:]/_ ' '[:lower:]---' | tr -cd 'a-z0-9-' | cut -c1-20); slug=${slug%-}
ENV=${ENV_OVERRIDE:-${slug}-$(printf '%s' "$branch" | shasum -a 256 | cut -c1-6)}   # the sweeper passes the name
APP="myapp-${ENV}"; DB="env_${ENV//-/_}"; HOST="${ENV}-dev.${DOMAIN}"

pg() { fly ssh console -a "$PG_APP" -q -C "psql postgres://postgres:${PG_PASSWORD}@localhost:5433/postgres -At -c \"$1\""; }
exists() { fly apps list --org "$ORG" --json | grep -q "\"Name\": *\"$APP\""; }

up() {
  echo "== $branch -> $ENV"
  exists || fly apps create "$APP" --org "$ORG"                         # idempotent
  [[ "$(pg "SELECT 1 FROM pg_database WHERE datname='$DB'")" == "1" ]] || pg "CREATE DATABASE $DB TEMPLATE seed_template"

  have=$(fly secrets list -a "$APP" --json | grep -o '"Name": *"[^"]*"' | cut -d'"' -f4 || true)
  gen() { grep -qx "$1" <<<"$have" || echo "$1=$(openssl rand -hex 32)"; }   # generated once, never rotated by a later `up`
  { gen SECRET_KEY; gen ENCRYPTION_KEY; gen ENV_TOKEN
    echo "ENV_NAME=$ENV"; echo "ALLOWED_HOSTS=$HOST,$APP.fly.dev"
    echo "DATABASE_URL=postgres://postgres:${PG_PASSWORD}@${PG_APP}.flycast:5432/${DB}"
    # ... plus the shared Redis URL and bucket credentials
  } | fly secrets import -a "$APP" --stage

  fly deploy -a "$APP" --config fly.preview.toml --remote-only --ha=false \
    --build-arg GIT_SHA="$(git rev-parse --short HEAD)" --build-arg GIT_BRANCH="$branch"
  echo "https://$HOST"
}

down() {
  [[ "$APP" =~ ^myapp-[a-z0-9-]+-[0-9a-f]{6}$ ]] || { echo "refusing to destroy $APP" >&2; exit 1; }
  if exists; then
    fly machine start -a "$APP" >/dev/null 2>&1 || true                 # it may be suspended
    fly ssh console -a "$APP" -q -C "python manage.py teardown_env" || echo "warning: cleanup failed; the sweeper will catch it" >&2
    fly apps destroy "$APP" --yes
  fi
  pg "DROP DATABASE IF EXISTS $DB WITH (FORCE)" || true
}

case "${1:-}" in up) up ;; down) down ;; name) echo "$ENV" ;; url) echo "https://$HOST" ;; esac

For the demo I ran it from my laptop rather than pushing, because the three branches were throwaways I didn’t want in the repository’s history and I wanted to watch the output as it went. In the team setup nobody types this: the label workflow further down runs exactly this script, and what follows is what would land in the Actions log. Here is what it printed the first time, on the branch that carries the tooling itself:

$ ./scripts/env up
== feature/branch-environments -> feature-branch-envir-c65cb4
New app created: myapp-feature-branch-envir-c65cb4
CREATE DATABASE
Secrets have been staged, but not set on VMs. Deploy or update machines in this app for the secrets to take effect.
==> Building image
image size: 139 MB
✔ Machine 1a2b3c4d5e6f78 [app] update finished: success
https://feature-branch-envir-c65cb4-dev.example.com

$ curl -s https://feature-branch-envir-c65cb4-dev.example.com/api/version/
{"environment": "feature-branch-envir-c65cb4", "branch": "feature/branch-environments", "commit": "a1b2c3d4", "region": "syd"}

About two and a half minutes from command to URL, most of it the image build. The second up on the same branch, after another commit, printed no “New app created” and no “CREATE DATABASE”, staged the same secret names without regenerating the generated ones, and rebuilt: same URL, new commit.

Some of the choices in there deserve a sentence each.

fly.preview.toml is not fly.toml. Prod runs the API, the WebSocket server and the background worker as separate Fly apps and process groups so each scales on its own metric, which is the whole argument of the stack post. A branch environment doesn’t need to scale; it needs to be one machine. So the preview config has a single process: the container’s entrypoint gets a preview mode that runs migrations and then starts one ASGI server with WebSockets enabled. Channels already routes both HTTP and WebSocket traffic within one ASGI application, so one process serves the whole app. Sync views run thread-sensitive in that mode, roughly one request at a time, which is exactly the trade I moved away from in prod and exactly right here, because nobody testing a branch will notice.

# fly.preview.toml (the app name comes from `fly deploy -a`)
app = 'placeholder'
primary_region = 'syd'

[processes]
  app = "/app/entrypoint.sh preview"   # migrate, then one ASGI server for HTTP and WebSockets

[env]
  DJANGO_SETTINGS_MODULE = 'myapp.settings'
  DEBUG = 'False'

[http_service]
  internal_port = 8000
  force_https = true
  auto_stop_machines = 'suspend'
  auto_start_machines = true
  min_machines_running = 0
  [[http_service.checks]]
    path = '/healthz'
    [http_service.checks.headers]
      Host = '127.0.0.1'               # the app's own hostname isn't known to a static file

[[vm]]
  size = 'shared-cpu-1x'
  memory = '1gb'

auto_stop_machines = "suspend", min_machines_running = 0. A branch environment nobody has opened since lunch should cost nothing. Suspend rather than stop gives the ~3 second resume I ended up with after chasing 504s instead of a 30-second cold start, and here even 30 seconds would be tolerable. Two of the three demo environments were already suspended in fly apps list a minute after their deploy finished.

fly.io/apps/myapp-feature-profile-tagl-fa8832

DEBUG = False, same settings module as prod. A branch environment is a public URL. It should behave like prod in every way that matters for testing (real HTTPS, real cookies, the same health check) and unlike prod in every way that could hurt someone. Rather than a separate preview settings file, the difference is carried by the secrets env up sets: sandbox payment keys, no admin credentials for the auth provider, and the ENV_NAME that switches on every prefix below.

One setting slices everything. ENV_NAME is the only new knob in the settings, and it’s applied in four places:

ENV_NAME = os.getenv("ENV_NAME", "")
_prefix = f"{ENV_NAME}/" if ENV_NAME else ""

CACHES["default"]["KEY_PREFIX"] = ENV_NAME                           # every cache key
CHANNEL_LAYERS["default"]["CONFIG"]["prefix"] = f"{ENV_NAME}:asgi"  # Channels group and channel names
STORAGES["default"]["OPTIONS"]["location"] = f"{_prefix}media"      # uploads
STORAGES["staticfiles"]["OPTIONS"]["location"] = f"{_prefix}static"

Miss the channels one and two branches’ WebSocket servers share group names, and a message sent in one environment turns up in another. That is the kind of bug that costs a day.

There was a fifth place I only found by reading the shared Redis after the fact. The startup lock from the 504s post, the thing that lets one machine per image run migrations while the others wait, was keyed on the image reference. Two branch environments built from the same commit have the same image. The second one would have found the first’s “done” marker and skipped its own migrations against its own, un-migrated database. Now the environment name is part of the key and the key lives under the prefix. It’s the sort of thing a shared-infrastructure design keeps producing: anything that was implicitly “per deployment” has to be made explicitly per environment, and you don’t know what was implicit until you look.

The database is cloned, not migrated from empty. That’s the next section.

The database: a template, cloned in seconds

#

A branch environment’s database can be one of three things. Shared with everyone, which is what got us here. Empty, migrated from zero and seeded with fixtures: deterministic, but fixtures are never as good as real-shaped data and are always eighteen months stale. Or a clone of a prod-shaped snapshot: realistic row counts, realistic nulls in columns the fixtures assumed were always filled, the migration that’s instant on an empty table and takes forty minutes on a real one shows itself.

The third, with the second kept as a CI-only check. And what makes the third fast enough to do per branch is a Postgres feature most people forget exists:

-- Nightly, on the preview cluster. Restore last night's prod backup into seed_raw,
-- run the anonymiser, then freeze it as a template nobody can connect to.
ALTER DATABASE seed_raw RENAME TO seed_template;
UPDATE pg_database SET datistemplate = true, datallowconn = false WHERE datname = 'seed_template';

-- Per branch, in the time it takes to copy the files:
CREATE DATABASE env_feature_payments_retr_3f9a1c TEMPLATE seed_template;

CREATE DATABASE ... TEMPLATE is a file-level copy within the cluster. A few gigabytes is seconds, not the minutes a pg_restore takes. The template can’t have open connections while it’s being cloned, which is what datallowconn = false enforces, and it’s also why the nightly job builds into seed_raw and renames at the end rather than restoring over the live template.

For the demo the template was built by hand rather than from a prod dump: a tunnel to the preview cluster with fly proxy, manage.py migrate, then the repo’s own seed commands for a small but realistic data set. Nineteen megabytes. Three clones of it appeared in the cluster as three branches came up, each 19 MB, each taking under a second:

$ fly ssh console -a myapp-preview-pg -C "psql ... -c '\l'"
             datname             | template | size
---------------------------------+----------+-------
 env_feature_branch_envir_c65cb4 | f        | 19 MB
 env_feature_profile_tagl_fa8832 | f        | 19 MB
 env_feature_profile_webs_dcb2e7 | f        | 19 MB
 seed_template                   | t        | 19 MB

The preview cluster is one unmanaged Fly Postgres node, shared-cpu-1x with a 1 GB volume, about two dollars a month. The branch environments connect to it over .flycast on the private network; nothing about it is public.

The anonymiser is not optional. The moment prod data is reachable on a -dev hostname, it needs to not be prod data: names, emails, phone numbers, addresses, free text, payment references, tokens, all overwritten before the rename. If the anonymiser fails, the previous night’s template stays and someone gets a message. It’s a small amount of work and it’s the thing that makes the whole approach acceptable to whoever answers your privacy questionnaire.

The entrypoint runs migrate on the first boot of every image, so the branch’s own migrations apply on top of the template. That’s the first honest test of “does my migration work on data shaped like prod”, and it happens before anyone looks at the URL.

Hostnames: a wildcard, a Worker, and two things the docs didn’t tell me

#

Every environment needs a public HTTPS hostname, and “public” is the point: a tester on a phone, a designer on another network, a product manager who won’t install a VPN. Fly gives each app myapp-$ENV.fly.dev for free, and that alone works. But *.fly.dev hostnames have three problems: they’re ugly to send around, Cloudflare isn’t in front of them so none of the protections you have on prod apply, and if you want them on your own domain you’re doing a DNS record and a certificate per environment, which is one more thing for the sweeper.

The version that removes all of that is one wildcard and one Cloudflare Worker. I’d planned *.dev.example.com. Two things stopped that, and both are worth knowing before you plan the same.

Universal SSL covers one wildcard level. Cloudflare’s free certificate is issued for example.com and *.example.com. It does not cover *.dev.example.com; a second-level wildcard needs Advanced Certificate Manager at $10 a month. The first request to feature-branch-envir-c65cb4.dev.example.com died with an SSL handshake failure before anything of mine ran. The fix is to keep the environment in the first label: feature-branch-envir-c65cb4-dev.example.com. That’s covered by the certificate that already exists, and by the proxied *.example.com record that already exists, the one I set up as a sinkhole for the scanners walking invented subdomains. So an environment needs no DNS at all.

Redirect Rules run before Workers. With the route in place, every -dev hostname answered 301 to the www homepage without the Worker running. The sinkhole is a zone-level redirect rule, “any other subdomain → www”, and Cloudflare’s redirect phase executes before the Workers phase. The rule needed one clause: and not ends_with(http.host, "-dev.example.com"). The deploy script adds it on up and removes it on down, because a rule someone edited by hand is the thing that gets forgotten.

dash.cloudflare.com/…/example.com/rules/redirect-rules/…/edit

The DNS side is the record that was already there:

dash.cloudflare.com/…/example.com/dns/records

With those two out of the way the routing is a single Worker on *-dev.example.com/*:

// dev-router.js: <env>-dev.example.com -> myapp-<env>.fly.dev
const DEV_SUFFIX = "-dev.example.com";
const ENV_NAME = /^[a-z0-9-]+-[0-9a-f]{6}$/;   // slug-hash, as scripts/env names them

export default {
  async fetch(request) {
    const url = new URL(request.url);
    if (!url.hostname.endsWith(DEV_SUFFIX)) return new Response("not a dev hostname", { status: 404 });
    const env = url.hostname.slice(0, -DEV_SUFFIX.length);
    if (!ENV_NAME.test(env)) return new Response(`no such environment: ${env}`, { status: 404 });

    url.hostname = `myapp-${env}.fly.dev`;
    const upstream = new Request(url, request);
    upstream.headers.set("X-Forwarded-Host", request.headers.get("host") ?? "");

    const res = await fetch(upstream);
    const out = new Response(res.body, res);
    out.headers.set("X-Robots-Tag", "noindex, nofollow");   // a half-finished branch is not for search engines
    out.headers.set("X-Dev-Environment", env);
    return out;
  },
};

Cloudflare terminates TLS on the wildcard, the Worker computes the Fly hostname from the label, and Fly’s own *.fly.dev certificate covers the hop to the machine. WebSocket upgrades pass through fetch unchanged. And because the route is on the zone, everything Cloudflare does for prod is available: WAF rules, rate limits, bot fight mode, and Access.

dash.cloudflare.com/…/example.com/workers

Three branches up at once, one Worker, and what each hostname answered:

Every 200 came back with X-Robots-Tag: noindex, nofollow and X-Dev-Environment: <env> added by the Worker. The 404 is the Worker’s own, before Fly is ever asked.

Cloudflare Access is what turns “public” into “reachable by the people who should reach it”. One application on *-dev.example.com with a policy of “anyone with an @example.com email” and browser visitors get a login page once and never think about it again. That covers designers, PMs and testers on laptops. I didn’t add it for the demo; the environments were up for an hour with noindex and nothing on them but seed data.

It doesn’t cover the mobile app, because a native app can’t complete an Access login redirect. Two ways through. Access supports service tokens (a CF-Access-Client-Id / CF-Access-Client-Secret header pair) that bypass the login, and the mobile app’s dev menu can carry a pair for the dev hostnames. Or, simpler, add an Access bypass for /api/ and have Django reject any request without either a signed-in user or the environment’s ENV_TOKEN, the one env up generated for exactly this environment. Either way, the mobile side is a dev-menu screen with two fields, base URL and token, and the app a tester already has installed is now talking to that branch. No new build. The over-the-air update channel stays pointed at whatever the app was built with; the API base URL is a runtime setting, not a build-time one, and it should have been anyway.

Auth is the one shared service that’s shared on purpose. Identity isn’t environment-specific; a tester wants to log in with the same account on every branch. One dev project at the auth provider with example.com in its authorised domains covers every -dev hostname, and every environment verifies ID tokens against it. The users table in each environment’s database is a copy of the template’s users, so the tester’s account exists everywhere the template did.

Automating preview environments with GitHub Actions: events in, environments out

#

Everything above by hand is fine when you’re the only one doing it. The moment there’s a second developer it has to happen because something happened in git, not because someone remembered. The whole automation is four workflows, one repository setting, one ruleset and one GitHub environment, and every one of them maps a git event to a call of the same scripts/env the developer runs by hand. There is no CI-only path to drift.

The label, push and delete workflow

#

# .github/workflows/branch-env.yml
name: branch environment
on:
  pull_request:
    types: [labeled, unlabeled, synchronize, closed]
  delete:
  workflow_dispatch:
    inputs:
      branch: { description: branch to bring up without a PR, required: true }

concurrency:
  group: env-${{ github.head_ref || github.event.ref || inputs.branch }}
  cancel-in-progress: true

env:
  FLY_API_TOKEN: ${{ secrets.FLY_PREVIEW_TOKEN }}   # org-scoped deploy token; a separate preview org keeps it away from prod
  PG_PASSWORD: ${{ secrets.PREVIEW_PG_PASSWORD }}
  # ... the shared Redis URL and bucket credentials, the same way

jobs:
  up:
    # the gate: a PR carrying the `env` label, or someone asking for a branch by hand
    if: >-
      github.event_name == 'workflow_dispatch' ||
      (github.event_name == 'pull_request' && github.event.action != 'closed'
        && contains(github.event.pull_request.labels.*.name, 'env'))
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with: { ref: "${{ github.head_ref || inputs.branch }}" }
      - uses: superfly/flyctl-actions/setup-flyctl@master
      - id: env
        env: { BRANCH: "${{ github.head_ref || inputs.branch }}" }
        run: scripts/env up && echo "url=$(scripts/env url)" >> "$GITHUB_OUTPUT"
      - if: github.event_name == 'pull_request'
        uses: marocchino/sticky-pull-request-comment@v2
        with:
          header: env
          message: "Branch environment: ${{ steps.env.outputs.url }}"

  down:
    # label removed, labelled PR closed or merged, or the branch itself deleted
    if: >-
      (github.event_name == 'pull_request' &&
        ((github.event.action == 'closed' && contains(github.event.pull_request.labels.*.name, 'env')) ||
         (github.event.action == 'unlabeled' && github.event.label.name == 'env'))) ||
      (github.event_name == 'delete' && github.event.ref_type == 'branch')
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: superfly/flyctl-actions/setup-flyctl@master
      - run: BRANCH="${{ github.head_ref || github.event.ref }}" scripts/env down

The gate is the env label, and it’s the part I’d argue for hardest. Without one, every branch costs an app and a database clone: the branch someone pushes to back up a laptop, the spike that gets squashed tomorrow, the dependency bump a bot opened at 3 a.m. So a push alone does nothing and opening a PR does nothing. Adding env to the PR brings the environment up, every push while the label is on updates it, and removing the label, closing the PR or merging it takes it down. Opting in is still a GitHub event, not a request to a person, and it’s one click made by the one person who knows whether anyone is going to look. The sticky comment is the user interface: label the PR, a comment appears with a URL they can send to anyone, and it updates in place on every push rather than stacking. workflow_dispatch covers the branch you want up before there’s a PR, and ./scripts/env up from a laptop covers the rest.

Two other gates are worth knowing. A branch-name filter (branches: ['feature/**', 'fix/**']) needs no PR but makes every feature branch pay. “Any PR that isn’t a draft” is the most automatic, and the right choice if your team opens PRs late and only when they’re ready. I went with the label because it puts the cost decision with the person who has the context, and because it can be taken back without closing anything. One side effect I like: pull_request runs from forks don’t get secrets, so a fork can’t bring up an environment.

concurrency matters more than it looks. A run of quick pushes to one branch cancels the earlier deploys rather than racing them; two fly deploys to the same app at once end with one of them losing.

closed covers teardown for anything that had a PR, merged or not. The delete event is the backstop for a branch brought up by workflow_dispatch that never got one, and it only fires if branches actually get deleted. That’s a repository setting, Automatically delete head branches, and it’s off by default. With it on, clicking Merge closes the PR and deletes the branch, and either event alone would have run env down. The merge button is the teardown.

gh api -X PATCH repos/org/myapp -f delete_branch_on_merge=true

The FLY_PREVIEW_TOKEN is the dangerous line in the file. It can create and destroy apps, and if $APP were ever anything but a branch environment it could destroy prod. Two guards, and I’d keep both: the token is scoped to a separate Fly org so it literally cannot see the production org (for the demo I used the production org, to skip setting up billing on a new one, and relied on the second guard alone); and down() refuses any app name that doesn’t match the slug-hash pattern.

The check a branch environment can’t do for itself

#

A branch only ever sees its own migrations, so the check that matters runs on the pull request against a merge with main, and it’s the one status check main requires:

# .github/workflows/migrations-check.yml
name: migrations
on:
  pull_request:
    branches: [main]

jobs:
  check:
    runs-on: ubuntu-latest
    services:
      postgres:
        image: postgres:18
        env: { POSTGRES_PASSWORD: ci, POSTGRES_DB: ci }
        ports: ['5432:5432']
        options: --health-cmd pg_isready --health-interval 5s --health-retries 10
    env:
      DATABASE_URL: postgres://postgres:ci@localhost:5432/ci
      SECRET_KEY: ci-only
    steps:
      - uses: actions/checkout@v4
        with: { fetch-depth: 0 }
      - uses: actions/setup-python@v5
        with: { python-version: '3.14' }
      - run: pip install -r requirements.txt
      - name: Merge main into the branch (a conflict here fails the check too)
        run: |
          git config user.email ci@example.com && git config user.name ci
          git merge --no-edit origin/main
      - name: One leaf in the migration graph
        run: python manage.py makemigrations --check --dry-run
      - name: The full chain applies from nothing
        run: python manage.py migrate --noinput

Thirty seconds. It’s what turns the 0042 / 0042 collision from a release-day surprise into a red check on the second developer’s PR; the migrations section below shows it firing.

The rules on main

#

None of this holds up if branches live for a month or if someone pushes to main directly. The rules are less about protection and more about keeping environments small and their migrations current:

• main is protected and always deployable. Merge via PR with the migrations check green against the current main (strict status checks, so a PR that was green last week re-runs after main moves). Linear history, no force-push, no deletion.
• Feature branches are short. A branch open for two weeks has an environment two weeks behind main and a migration two weeks more likely to collide with someone else’s. A month-long feature merges to main behind a flag in pieces. The flag is the long-lived thing, not the branch.
• One branch, one environment, one database. Already enforced by the naming.
• Release tags are the only thing that goes to prod. The image is built once from the tag; staging and prod run that image, not a rebuild.
• A PR that touches migrations/ is reviewed by one of the other two. Below.

As a ruleset, applied once with gh api -X POST repos/org/myapp/rulesets --input ruleset-main.json:

{
  "name": "main", "target": "branch", "enforcement": "active",
  "conditions": { "ref_name": { "include": ["~DEFAULT_BRANCH"], "exclude": [] } },
  "rules": [
    { "type": "deletion" },
    { "type": "non_fast_forward" },
    { "type": "required_linear_history" },
    { "type": "pull_request",
      "parameters": { "required_approving_review_count": 1, "dismiss_stale_reviews_on_push": true,
                      "required_review_thread_resolution": true } },
    { "type": "required_status_checks",
      "parameters": { "strict_required_status_checks_policy": true,
                      "required_status_checks": [ { "context": "check" } ] } }
  ]
}

One honest wrinkle. This repository is private on GitHub’s Free plan, and on Free, branch protection and rulesets are only available for public repositories; the API answers 403 Upgrade to GitHub Pro or make this repository public. The delete_branch_on_merge setting, the environments and the workflows all work on Free. So for a small team on a private Free repo, the ruleset above is a convention: the migrations workflow still runs and still goes red, but nothing greys out the merge button. That’s a $4-a-month decision, and I’d make it the moment a second person can merge.

There’s no develop branch and no release/* branches. Git-flow was designed around integration being expensive and needing to be batched. Branch environments make integration cheap, so the batching goes.

Staging is a tag’s environment, and prod is a promotion

#

Staging reuses the whole machinery. A v* tag on main runs scripts/env up with the branch name staging/v1.42.0, which the naming turns into staging-v1-42-0-<hash>: same clone-from-template database, same suspend-when-idle machine, same hostname scheme. The difference is what comes after:

# .github/workflows/release.yml (abridged)
on:
  push:
    tags: ['v*']
jobs:
  staging:
    steps:
      - run: BRANCH="staging/${GITHUB_REF_NAME}" scripts/env up
      - run: echo "image=$(fly image show -a "myapp-$(BRANCH=staging/$GITHUB_REF_NAME scripts/env name)" --json | jq -r '…')" >> "$GITHUB_OUTPUT"
  production:
    needs: staging
    environment: production        # required reviewer on this environment = the promotion gate
    steps:
      - run: fly deploy --config fly.api.toml --image "${{ needs.staging.outputs.image }}"
      - run: fly deploy --config fly.ws.toml  --image "${{ needs.staging.outputs.image }}"
      - run: BRANCH="staging/${GITHUB_REF_NAME}" scripts/env down

The production job waits on a GitHub environment with a required reviewer. Someone looks at staging, clicks Approve, and prod receives the same image staging ran, by reference, not a rebuild. The last step destroys staging because it’s done its job. If nobody approves, the release candidate was rejected, and the sweeper removes staging when its “branch” (staging/v1.42.0 never exists on origin) fails the liveness check. Nothing gets to exist without a reason the sweeper can check.

The nightly sweeper

#

# .github/workflows/sweep-envs.yml
on:
  schedule:
    - cron: '0 14 * * *'     # 02:00 NZST
  workflow_dispatch:
jobs:
  sweep:
    steps:
      - uses: actions/checkout@v4
        with: { fetch-depth: 0 }
      - uses: superfly/flyctl-actions/setup-flyctl@master
      - run: scripts/sweep-envs

fetch-depth: 0 because the sweeper needs every remote branch and its last commit date, not the one commit Actions checks out by default. workflow_dispatch so it can be run by hand after an incident.

What a developer actually does

#

Push a branch, open a PR, and put the env label on it when they want someone to look. That’s the list. A URL appears on the PR within three minutes, updates on every push, and disappears when the PR is merged or closed. The migration check tells them before anyone else does if they’ve collided with a teammate. Nobody claims staging because there is no staging to claim, and nobody tears anything down because tearing down is what merging does.

And what the person who set it up does per branch, afterwards: nothing. The shared Postgres, Redis and bucket are provisioned once, the secrets go into Actions once, and from then on the recurring work is reading the sweeper’s log on a Monday. No ticket queue, no bot to ask, nobody to prompt.

Teardown that doesn’t trust the teardown

#

env down is the ordinary case, and the interesting part of it is who does the cleaning. The Redis keys and the bucket objects belong to the shared services, and the laptop running down doesn’t need credentials for either, because the environment’s own machine already has them and already knows its prefix. So down starts the machine if it’s suspended, runs a small cleanup command on it, and only then destroys the app and drops the database. The environment cleans up after itself:

$ BRANCH=feature/profile-website ./scripts/env down
redis: deleted 0 keys under feature-profile-webs-dcb2e7:*
bucket: deleted 278 objects under feature-profile-webs-dcb2e7/
Destroyed app myapp-feature-profile-webs-dcb2e7
DROP DATABASE

$ curl -s -o /dev/null -w '%{http_code}\n' https://feature-profile-webs-dcb2e7-dev.example.com/api/version/
530

The 530 is the point about coordination again: the Worker still routes the hostname, Fly has nothing there, and nobody had to tell Cloudflare.

The cleanup command refuses to run without ENV_NAME, because with an empty prefix it would be a wipe of the shared bucket. That guard is the whole reason it exists as a command inside the app rather than three lines in the shell script.

Environments leak anyway. A branch is deleted while GitHub Actions is having an outage. A developer runs env up from a laptop for a spike and never pushes. A fly apps destroy fails transiently and the || true swallows it. Three months later there are forty apps and a bill that makes someone ask what happened.

So there’s a second mechanism that doesn’t trust the first one: the nightly sweeper from the table above. It lists every myapp-* app, works out which branch each one belongs to, and destroys any whose branch is gone or hasn’t been pushed to in seven days. A week-idle branch doesn’t need a live URL; the next push recreates it in two minutes. A hash isn’t reversible, so the sweeper computes the environment name for every live branch and destroys any app that isn’t in that set, then drops any env_* database with no app behind it:

#!/usr/bin/env bash
# scripts/sweep-envs: nightly
set -euo pipefail
ORG=${PREVIEW_ORG:-myapp-preview}; TTL_DAYS=${TTL_DAYS:-7}
cutoff=$(( $(date +%s) - TTL_DAYS*86400 ))

git fetch --prune --quiet origin
live=""
while IFS=$'\t' read -r ts branch; do
  [[ "$branch" == "main" ]] && continue
  (( ts >= cutoff )) || continue
  live+="$(BRANCH="$branch" scripts/env name)"$'\n'
done < <(git for-each-ref --format='%(committerdate:unix)%09%(refname:short)' refs/remotes/origin | sed 's#^\([0-9]*\)\torigin/#\1\t#')

apps=$(fly apps list --org "$ORG" --json | grep -o '"Name": *"myapp-[^"]*"' | cut -d'"' -f4)
for app in $apps; do
  env="${app#myapp-}"
  [[ "$env" =~ ^[a-z0-9-]+-[0-9a-f]{6}$ ]] || continue       # not a branch environment (e.g. myapp-preview-pg)
  grep -qx "$env" <<<"$live" && continue
  echo "sweeping $app: branch gone or idle > ${TTL_DAYS}d"
  ENV_OVERRIDE="$env" scripts/env down || true
done
# ...then any env_* database with no app: DROP DATABASE.

None of the demo branches had ever been pushed (the laptop run above), so from the sweeper’s point of view all of them were gone, which is exactly the state a spike someone brought up from a laptop and never pushed would be in a week later. It took the remaining two apart, including the Redis keys the tagline environment had written:

$ ./scripts/sweep-envs
sweeping myapp-feature-branch-envir-c65cb4: branch gone or idle > 7d
redis: deleted 0 keys under feature-branch-envir-c65cb4:*
bucket: deleted 322 objects under feature-branch-envir-c65cb4/
Destroyed app myapp-feature-branch-envir-c65cb4
DROP DATABASE
sweeping myapp-feature-profile-tagl-fa8832: branch gone or idle > 7d
redis: deleted 3 keys under feature-profile-tagl-fa8832:*
bucket: deleted 278 objects under feature-profile-tagl-fa8832/
Destroyed app myapp-feature-profile-tagl-fa8832
DROP DATABASE

$ fly apps list --org myapp-preview | grep myapp-
 myapp-preview-pg │ myapp-preview │ deployed

(One small thing that bit: macOS ships bash 3, which has no associative arrays. The first version of the sweeper used one and died on declare -A on my laptop. The version above uses a newline-separated list and runs on anything.)

The cost is the pleasant part. A suspended machine costs its rootfs, which is cents a month. A database is 19 MB on a volume that’s already paid for. The Redis is on a pay-as-you-go plan, twenty cents per hundred thousand commands, which at branch-testing volumes rounds to nothing. A bucket prefix costs what it stores. The shared Postgres node is the only fixed line, about two dollars a month. Twenty active branches would cost roughly what one always-on shared-cpu-1x costs, and the sweeper keeps it at twenty rather than two hundred. The whole demo, three environments for an hour plus the shared pieces, cost less than the coffee I drank during it.

Migrations across branches, without the shared database to warn you

#

When each developer owns a feature end to end, every branch has a migration in it. That makes this section the important one.

Here’s the thing people miss when they leave the shared-environment world. When everyone deployed to one dev database, migration conflicts surfaced immediately and painfully, but they surfaced. One migration ran, the next deploy failed, someone fixed it that afternoon. With a database per branch, each migration works perfectly in its own environment, and two of them first meet when the second one merges to main. Done carelessly, this moves the conflict from dev, where it was cheap, to a release, where it isn’t.

Four rules, all enforceable by CI so nobody has to remember them.

1. CI fails on two leaves in the migration graph, against a merge with main. makemigrations --check catches models disagreeing with migrations, but the conflict that matters is two branches both adding 0042_*. Check it against the branch merged into main, not the branch alone:

git fetch origin main
git merge --no-commit --no-ff origin/main || { echo "merge conflict"; exit 1; }
python manage.py makemigrations --check --dry-run
python manage.py migrate --plan 2>&1 | grep -q "Conflicting migrations" && exit 1

This is the one I made happen on purpose. Two of the three demo branches, feature/profile-tagline and feature/profile-website, each added one field to the user profile model with one migration, and each environment applied its own on the way up. The new columns were only where they should be:

env_feature_branch_envir_c65cb4      (no new column)
env_feature_profile_tagl_fa8832      tagline
env_feature_profile_webs_dcb2e7      website

Both migrations are 0042_*. Neither branch can see the other’s, so neither environment can tell you there’s a problem. Merging one into the other, the check does:

$ git merge --no-commit --no-ff feature/profile-tagline
$ python manage.py makemigrations --check --dry-run
CommandError: Conflicting migrations detected; multiple leaf nodes in the migration graph:
  (0042_profile_tagline, 0042_profile_website in accounts).
To fix them run 'python manage.py makemigrations --merge'

$ python manage.py makemigrations --merge --noinput
Created new merge migration accounts/migrations/0043_merge_20260919_0324.py
    dependencies = [
        ('accounts', '0042_profile_tagline'),
        ('accounts', '0042_profile_website'),
    ]

The second author runs that --merge on their branch, in their environment, bothering nobody, and the next env up applies 0043 on top of their clone. (In this case git also flagged a textual conflict first, because both fields had been added at the same line of the model file. Realistic, and worth knowing that the textual conflict hides the migration one until it’s resolved.)

2. Migrations are backwards-compatible with the previous release’s code. During a deploy there’s a window where the new schema and the old code coexist, or the reverse if a machine rolls back. Add columns nullable or with defaults, never rename in one step, never drop what the current release still reads: add, deploy, backfill, switch code, drop in a later release. Expand and contract. The reason it matters more here is that disposable environments make deploying more frequent, so you’re inside that window more often. Staging-per-tag is where it’s rehearsed.

3. CI also migrates from zero into an empty database. The template clone tests “does my migration apply to realistic data”. The empty run tests “does the full chain still work from nothing”, which catches the migration that assumed a previous data migration had populated something. Thirty seconds, and it catches a class of bug that otherwise waits for the next new laptop.

4. Schema changes get a second pair of eyes. In a small team this doesn’t need a CODEOWNERS file, but it does need to be a rule: a PR that touches migrations/ is reviewed by a teammate before merge, and the reviewer’s job isn’t to check the SQL. It’s to know that they, or someone else on the team, are touching the same table this fortnight and say so. Past a dozen developers that knowledge stops fitting in one head, and a CODEOWNERS entry on migrations/ earns its place. No tooling replaces this. With the tooling catching everything mechanical, it’s a two-minute job, and it’s the one part of the old shared-environment world worth keeping: the moment where two people find out they’re about to collide.

Versioning, so a URL tells you what it’s running

#

With one shared environment, “what’s on staging” was a chat question. With many, it has to be a property of the environment or every bug report starts with archaeology.

• Every image carries the commit SHA and the app exposes it. fly deploy --build-arg GIT_SHA=... --build-arg GIT_BRANCH=... stamps them into the image as environment variables, and /api/version/ returns them with the environment name and the region (see the output above). The first thing anyone does with “it’s broken on feature-profile-tagl-fa8832” is check it’s running the commit they think. The first demo environment was deployed from the working tree a commit before the tooling was committed, and the endpoint said so: it reported the previous commit, not the SHA I expected.
• ENVIRONMENT=$ENV is in every log line and the error tracker’s environment tag. Branch errors don’t pollute prod’s error tracking and are filterable in one click.
• Branches get SHAs, releases get tags. feature-payments-retr-3f9a1c never has a version number; it has commits. Staging and prod have v1.42.0.
• The mobile app carries a minimum API version, not a URL. The dev-menu URL field is how a tester points a real phone at a branch, and the version endpoint carrying an API schema version is how the app knows whether it’s talking to something it understands.

What this doesn’t solve

#

It doesn’t solve two features that only make sense together. A branch environment shows one branch. The answer is to merge both behind flags and look at main, which you can do with one standing environment that tracks main (main-dev.example.com, redeployed on every merge). That’s the one long-lived non-prod environment I’d keep, and nobody deploys to it; it’s for seeing what the next release will be.

It doesn’t solve providers that want a registered callback and don’t accept wildcards. My auth provider does accept a parent domain, so example.com covers every -dev branch hostname. Some OAuth and payment providers don’t, and for those you keep one shared hostname that the Worker routes to whichever environment last claimed it. That’s a small lock over a small thing, and it’s a lot better than a lock over everything.

It doesn’t solve the human side of migrations. The tooling catches mechanical conflicts. It doesn’t catch two of you modelling the same concept incompatibly in the same fortnight, and the only fix for that is the migration review where one of you notices.

What I’d check in your setup

#

If you have a shared dev or staging and a chat thread for booking it:

1. How long does a finished branch wait before someone other than its author can see it running? More than an hour and you’re paying for the lock in delivery time, and on a small team that’s a big fraction of it waiting.
2. How often does the shared database get reset, and who loses work when it does?
3. Could a new developer, on day one, get a public URL running their branch without asking anyone for anything? Whatever they’d have to ask for is the thing to automate first.
4. Does a fresh environment start from realistic data? If it starts empty, the first bug it won’t catch is the migration that’s fine on an empty table and takes forty minutes on the real one.
5. Is there anything in your non-prod hosting that would still exist a month after everyone forgot about it? If so, write the sweeper before anything else. The leak starts on day one.

The chat question goes away not because people stop asking it but because the answer is always “you are, on yours.”

Search traffic to this blog has been flat for a couple of years and I’ve stopped expecting it to move. So I went looking at where visitors come from instead, and whether the AI assistants are among them. They are, barely. In the twelve months to 16 September 2026, my analytics counted 5,372 sessions here. Twenty-seven of them came from an AI assistant: twelve from chatgpt.com, nine from gemini.google.com, six from Perplexity, and none from claude.ai. Half a percent. The ChatGPT visitors were the good kind, though. They stayed five minutes on average, against 27 seconds for direct traffic and three minutes for Google, because they’d already read the answer and wanted the detail.

So I’ve spent some time on how a small blog gets picked as the source for those answers. This post is what I’ve done on this site and what the logs say about it. Where I have numbers I give them, and where I’m guessing I say so. Three sentences in the first draft turned out to be wrong once I looked, and I’ve left the corrections in.

Two kinds of bot, and only one of them cites you

#

The first thing to get straight is that “AI bots” is two different jobs with different user agents, and blocking one does nothing to the other.

Training crawlers pull pages in bulk to build the next model. OpenAI’s is GPTBot, Anthropic’s is ClaudeBot, Google’s is controlled by the Google-Extended token, Apple’s by Applebot-Extended, and Common Crawl’s CCBot feeds most of the open datasets. Whatever they take ends up as weights, with no link back to you. There’s a case for blocking these and I don’t think it’s a silly one.

Retrieval bots are the other kind. When someone asks ChatGPT or Claude or Perplexity a question, the assistant runs a search, fetches a handful of pages right then, reads them, and writes an answer with citations. Those fetches arrive as OAI-SearchBot and ChatGPT-User, Claude-SearchBot and Claude-User, PerplexityBot and Perplexity-User. Google’s AI Overviews use plain Googlebot, so there’s no separate switch for them at all. These are the bots that send a visitor back. If the aim is to be cited, these are the ones you’re optimising for, and it’s worth knowing that blocking GPTBot does nothing to ChatGPT’s search results in either direction.

This site’s robots.txt is Hugo’s default, which is allow everything:

User-agent: *
Allow: /
Sitemap: https://jared.lynskey.co.nz/sitemap.xml

I’ve thought about splitting it, letting the retrieval bots in and keeping the training ones out, and decided it wasn’t worth doing for a blog whose whole purpose is to be read. If I did want that, it would look like the block below. The reason to show it is the same lesson as in Cloudflare Served Googlebot an Empty Page: the list has to name bots, not vendors. There is no line you can write that says “OpenAI”.

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
Disallow: /

User-agent: *
Allow: /

One thing to check if the site sits behind Cloudflare. Since the middle of 2025, new zones there block AI crawlers by default, and there’s a one-click toggle under the bot settings that does the same to existing ones. It blocks both kinds. This blog moved to Cloudflare Pages last week, so that was on my list. If you turned it on because “AI bots” sounded like something to keep out, you’ve also turned off the citations. Cloudflare has since started writing a Content-Signal line into the robots.txt files it manages, with separate search, ai-input and ai-train flags. Nobody is obliged to honour it, but it’s the right vocabulary: the same split as above, expressed as a preference rather than a wall.

What actually knocks on the door

#

The move to Cloudflare had a side effect: for the first time I can see every request by user agent. Here is what came in over the four days from 13 to 16 September 2026, after throwing out the fakes, which I’ll get to.

Two things stand out. The training crawlers read everything, twice, in the time it took the retrieval bots to read a few dozen pages. And the retrieval bots are not fetching at random: ChatGPT-User and Claude-User both went for the update server post, which is the most specific thing on the site, and the Japanese cold-start post got fetched in Japanese.

Now the fakes. Of 20 requests claiming to be GPTBot, 18 came from addresses outside OpenAI’s published ranges and asked for /.env, /.git/HEAD and /firebase-adminsdk.json. Fifteen of 36 “ChatGPT-User” requests were the same kind of thing. Most of the “PerplexityBot” traffic wanted /id_ed25519. A user agent is a string anyone can send, and vulnerability scanners have noticed that AI bot names get waved through. OpenAI, Anthropic and Perplexity all publish their IP ranges. If you’re going to treat a bot specially, in either direction, check the address, not the name. The 20, 59 and 8 above are the requests that came from the right addresses.

The page has to be readable without a browser

#

Retrieval bots fetch at answer time, with a timeout, and most of them don’t run JavaScript. A page that is an empty div until a bundle loads is a page they don’t read. Hugo writes finished HTML, so this blog gets that for free, and it’s the main reason I’ve never been tempted to move it to anything cleverer. The marketplace I run is the opposite case, a React Native web app, and getting crawlers a real document there took a dynamic-rendering branch and then a caching bug that quietly undid it. The Cloudflare post above is that story.

Speed counts the same way, and here the logs corrected me. I’d assumed a static site behind Cloudflare was served from the nearest edge. It isn’t. This blog is on Cloudflare Pages, the HTML goes out uncached, and from Seoul the response comes back from Los Angeles in about half a second. That’s still fine for a bot with a timeout of several seconds, but it isn’t the tens of milliseconds I’d have claimed, and I only know because I measured it. An assistant assembling an answer from six pages will drop the one that takes three seconds, and it doesn’t tell you it did.

Write so a paragraph can be lifted out

#

The way these systems use a page is not the way a reader does. They split it into chunks, score each chunk against the question, and quote the ones that score. A post that builds up to its point over eight hundred words scores badly on every chunk. A post where each section states its claim in the first sentence and then supports it scores on the section that matches.

That changed how I write here more than any technical thing did. The description in the front matter is now a summary of what the post says, not a hook to get you to read it, because the description is often the only part an assistant shows. Section headings say what the section concludes. Numbers go in the sentence with their units rather than in a chart. Dates are written out, because “last week” means nothing to a model reading the page in a year. And a post is about one thing.

The other half is having something worth quoting, and here the data is mixed. In the logs, the pages the answer-time bots fetch are the specific ones: cold starts, the update server, iOS debugging. Naver’s crawler is called Yeti and Daum’s is Daumoa, a restart on my host is a thirty-second window for 504s, Cloudflare’s on-the-fly Brotli measured worse than gzip on my bundle. Those facts aren’t written down in many places, so a question that needs one of them lands here. But the people who click through from an assistant mostly land on the 2023 and 2024 posts about management, one or two sessions each. The specific posts get read by the machine. The general ones get the click. I don’t have a tidy explanation for that and I’d rather report it than invent one. The lever, if there is one, is still to write down the specific thing you found out this week that you couldn’t find when you searched for it, because that’s what the machines are demonstrably reading.

Say who wrote it, and keep saying the same thing

#

An assistant deciding whether to cite a page is doing something like what a careful reader does: who wrote this, when, and does it agree with everything else I’ve found? The mechanics of that on this site are mostly the theme’s doing. Every post carries Article structured data with an author, a published date and a modified date, and the author is a Person with links to the same public profiles. That’s a few lines in a template, and it’s the difference between “a page” and “a page by a named person with a history”.

The part that isn’t the theme’s doing is consistency. The about page, the llms.txt, my public profiles and the line at the top of every post all describe the same person with the same job history. When something changes, all of them change, or the one I forgot becomes the one that’s wrong. That sounds fussy, but one page saying I’m an engineering manager and another saying I’m freelance is exactly the kind of contradiction that makes a model hedge, or pick someone else.

The modified date has to be honest too. It changes when the content changes and I don’t touch it otherwise. Bumping it to look fresh is the kind of thing that works once and then gets your whole domain discounted.

What llms.txt is, and the fact that nothing reads it

#

llms.txt is a proposal from September 2024 for a Markdown file at the site root that gives a language model a summary of the site and a curated list of links. I added one in February this year. It’s a paragraph about who I am, then sections for management, infrastructure, mobile and community, with one line per post: the title, the URL, and a sentence saying what the post concludes. It reads like a table of contents written for someone in a hurry.

The caveat is that nothing reads it. In the four days of logs I have, the only requests for /llms.txt were my own. Common Crawl has captured 76 pages of this site in its August 2026 crawl and never that file. No major assistant has said it reads it. So it is not why anything here gets cited, and I’d stop short of recommending it to anyone on those grounds. I keep it for two reasons. It cost an hour, and writing one sentence per post that says what the post actually concludes was worth more than the file, because it showed me which posts didn’t conclude anything. It’s also the one page on the site that tells a machine what the site is for, and I’d rather have that and not need it. It has drifted since, because I write it by hand and forget, which is the argument for generating it from the front matter as a Hugo output format instead. That’s on the list.

Feeds, sitemaps and five languages

#

The boring plumbing still counts. The sitemap is how a crawler finds a new post the day it goes up. RSS is how it learns a post changed. Both are Hugo defaults and I’ve never had to think about them. ClaudeBot read the sitemap 36 times in two days, which is how it found 1,380 pages so quickly.

The languages are the part I didn’t expect to matter. Everything here is in English, Japanese, Korean, Simplified Chinese and Vietnamese, with alternate links so a crawler knows they’re the same post. The crawlers treat them as five sites: ClaudeBot’s crawl split almost evenly across the five, and of the 73 post pages Common Crawl took in August, only four were the English ones. Assistants answer in the language they’re asked in and lean towards sources in that language, so a Korean question about cold starts can be answered from the Korean page rather than from a translation of the English one done on the fly, and the Japanese page is the one ChatGPT actually fetched. The condition is that the translation reads as if it were written in that language. The pass I did on that earlier this year, with native speakers checking, was for human readers, but it’s also what makes the Vietnamese page a source rather than a curiosity.

What I can measure

#

More than I thought, once I stopped guessing. Cloudflare’s free plan gives request logs by user agent and path, one day per query, which is where the bot table came from; referrers need the paid plan. The analytics tool has the referrers, and pulling them through its reporting API rather than squinting at a dashboard is what produced the numbers at the top. Together they show the shape of it. In four days the retrieval bots fetched about ninety pages from OpenAI and Anthropic addresses. In the last month, two people arrived from ChatGPT. Read a lot, cited sometimes, clicked rarely. There’s still no way I know of to see the answers that used a page and didn’t get clicked. The most useful check is still the manual one: ask the assistant the question your post answers and see whose page it quotes.

If I had to keep one thing from all of this, it’s the writing rule. The robots.txt, the structured data and the llms.txt are an afternoon’s work and then they’re done. Writing so each section says its claim first, and writing about the specific thing rather than the general one, is a habit, and it’s the habit that decides whether a page is worth quoting at all. The second thing is to measure before you claim. Being trusted by a machine turns out to look a lot like being trusted by a person: say who you are, say what you know, and say it the same way every time.

Until recently, every change to the web side of my marketplace app went out the same way as a change to the Django API: build the Docker image, push it, and roll it across every machine on Fly.io. The web front end (the mobile app’s web export, plus the marketing pages a static site generator builds) was copied into the image next to the API, so there was no other way to get it out. A CSS fix took about seven minutes and restarted eleven machines. One day in August I did that six times.

The rest of this post walks through that deploy script step by step, including how it waits until every machine is serving the new version, and then the Django side that reads the pointer.

Why the old way was a problem

#

The restart was the biggest cost. Eleven machines rolling means eleven cold boots, and on a shared CPU Django takes about thirteen seconds to come up, seven of which is just importing Django. I had a warm-up step at the end of the deploy script that hit every machine so a real visitor wouldn’t be the one to pay for that, which meant every web release ended with waiting on the slowest machine. I’ve written before about what those cold boots cost on Fly.io, in Scale to Zero, and Every Restart Is a 30-Second Outage. The short version is that every restart is a window where a visitor can get a 504, so a deploy path that restarts nothing closes that window for web releases entirely.

The second problem was that the static site’s output was split. The HTML went into the image, but the RSS feeds and the search index went straight to the bucket. They were built from the same source, just not at the same moment, so they could disagree with each other and nobody would notice until they did.

The third one is more about habit. When a release costs seven minutes and a restart, you stop doing small ones. I’d let three or four fixes pile up rather than ship them one at a time, and that’s exactly how a one-line change ends up going out next to something broken.

The layout in the bucket

#

Everything lives in an S3-compatible bucket under a web/ prefix:

web/current                              ← a text file containing one version string
web/a1b2c3d-20260914021500/manifest.json ← every file in this version
web/a1b2c3d-20260914021500/app/...       ← the app's web export
web/a1b2c3d-20260914021500/site/...      ← the static site's output
web/9f8e7d6-20260912093012/...           ← the previous version, still there

A version is the short git SHA plus a UTC timestamp, so you can read off what commit it is and when it went up. web/current is the only thing that changes when a release goes live.

What the deploy script does

#

The whole thing is one script, run from the repo root. In order:

1. Refuse a dirty tree. The first thing it does is git status --porcelain, and if anything comes back it exits. Whatever is on disk is what gets uploaded, and the old deploy script for the marketing site once shipped uncommitted copy that way. There’s an override flag for when you know what you’re doing, but I haven’t used it. It then computes the version string from git rev-parse --short HEAD and the current time.

2. Build both trees. For the app it runs the web export with the production API and WebSocket URLs baked in, followed by a Brotli precompression pass. The precompression is worth explaining: Cloudflare will Brotli-compress on the fly, but at a low quality level, and on this bundle it measured worse than gzip (2.04MB against 1.66MB). Compressing once at build time with quality 11 gets it to 1.28MB, so the script writes a .br twin next to every JS, CSS and JSON file over 1KB. For the marketing site it runs the static site generator with minification on.

3. Upload. It walks both build directories and uploads every file to web/<version>/app/... and web/<version>/site/... with sixteen threads. Each object gets a Content-Type guessed from its filename (with the .br stripped first), and the Brotli twins get Content-Encoding: br as well. Only when every file is up does it write manifest.json, a JSON object with two lists of relative paths. The order matters: the Django side treats a version without a manifest as not existing, so a half-uploaded version can never be made current. Right now that’s a few hundred files and the upload takes about six seconds.

4. Flip the pointer. One put_object writes the version string to web/current. That’s the release.

5. Wait for the machines. Each API machine caches the pointer for five seconds, so for a moment after the flip some of them are still on the old version. The script polls a version endpoint on the API every half second and counts consecutive answers that carry the new version. Fly load-balances across all the machines, so a run of sixteen matching answers in a row (more than the machine count) is how it knows every one of them has re-read the pointer. One stale answer resets the count. It gives up after 90 seconds and tells you to check by hand; the first real flip took a bit over 30 seconds, which is why the limit isn’t lower.

6. Purge Cloudflare. One API call, covered below. If it fails, the script prints a warning and carries on.

7. Post to the team chat. The same webhook the runtime error alerts use gets a one-liner with the version and who ran it, so a burst of errors can be read against what just changed. Rollbacks post in amber.

Start to finish, a release is a bit over a minute, most of which is the web export.

The Django side

#

Django was already the thing serving the front end. Every non-API URL is answered with the prerendered page from the web export for that route, and the marketing, privacy and support pages come from the static site’s output. Those views used to open files on disk. Now they read through one small layer that knows about the bucket.

Per request it resolves three things: the current version (from web/current, cached for five seconds), the manifest for that version (cached for as long as that version is current), and the file itself (a small in-memory LRU of file bodies, so a page’s HTML and its main bundle stay in memory). The manifest is what makes 404s cheap: if a path isn’t listed, the answer is no without asking the bucket. If the client accepts Brotli, it reads the .br twin instead and sets the encoding header. Assets go out with a one-year immutable cache header, since their filenames are content-hashed. Anything derived from the file list, such as the table that maps routes to prerendered pages, is cached per version too, so a new deploy rebuilds it on the first request and not before.

Then there’s the endpoint the deploy script polls:

def web_version(request):
    # bundle: the layer that reads web/current and the manifest
    response = JsonResponse({"source": bundle.source(), "version": bundle.version()})
    response["Cache-Control"] = "no-store"
    return response

It answers with bucket:<version> in production or filesystem when it’s serving from the image. Each Fly machine answers for itself, so hitting it a dozen times tells you whether the fleet has converged. It’s also the quickest way to check what a machine thinks is live when something looks off.

Purging the Cloudflare cache through the API

#

HTML on the site is cached at Cloudflare’s edge for five minutes. Without a purge, a deploy would be live on the machines but most visitors would keep getting the old page for up to five minutes, which defeats the purpose.

The purge is a single API call:

urllib.request.Request(
    f"https://api.cloudflare.com/client/v4/zones/{zone}/purge_cache",
    data=json.dumps({"hosts": ["example.com", "www.example.com"]}).encode(),
    headers={"Authorization": f"Bearer {token}", "Content-Type": "application/json"},
)

I purge by host rather than by URL so I don’t have to list every page. The assets don’t need purging at all: the filenames are content-hashed, so a new build points at new filenames and the old ones just stop being referenced. Only the HTML that references them needs to be refreshed.

The script waits for the machines before purging, on purpose. If it purged first, Cloudflare could refill its cache from a machine that was still on the old version, and that page would then sit at the edge for another five minutes.

If the purge fails, the script prints a warning and finishes anyway. The cache expires on its own within five minutes, so the worst case is a slow release rather than a broken one. I’d rather that than have the deploy abort at the last step.

Checking a build before it goes live

#

A no-flip mode does everything up to step 3 and stops. The version is in the bucket but nothing serves it. To look at it, I set an environment variable on one machine that pins it to that build regardless of what web/current says. When it looks right, the rollback command makes it current. It’s called rollback but it’s really “point current at this version”, and going forward and going back are the same operation.

Rolling back

#

Old versions stay in the bucket, so rolling back is the same command with an earlier version, and a list command shows what’s there, newest first. It does the same wait and purge as a forward deploy. I haven’t needed it yet. Nothing prunes old versions at the moment; they’re small and I’d rather have them than not.

What stayed the same

#

The API still deploys the old way, through its own deploy script, and it should: it’s stateful and it’s where the migrations run. The Dockerfile still copies both front-end builds into the image too. That’s deliberate. When there’s no bucket configured (local development, the test suite) or the bucket can’t be read, the serving layer falls back to those directories on disk, so a storage outage means visitors get the front end that shipped with the last image rather than an error page. One environment variable forces that behaviour in production if the bucket ever needs to be taken out of the loop.

The front end changes far more often than the API does, and each release now takes about a minute instead of seven. I’ve stopped batching without trying to.

My marketplace app is a single-page app built with React Native Web. Every listing page is one script tag and an empty div until JavaScript runs. That is fine for people and fatal for search: Google can sometimes execute the JS, Bing will not, and Naver’s crawler — the one that matters most when nearly all your listings are in Korean — certainly will not.

This is the story of making those pages visible, and then of the caching rule that undid it in a way I did not see coming.

Dynamic rendering, which is not cloaking

#

The fix is old and Google-sanctioned: detect a crawler, and serve it server-rendered HTML containing the same content the SPA would have painted. Not different content. The same content, earlier.

So a listing URL like /listings/view?id=... branches. A browser gets the SPA shell. A crawler gets a real document: a heading, the listing’s key facts, an image per photo with alt text, a body excerpt, and links to a handful of related listings so the page is not an orphan.

That last part mattered more than I expected. Before those internal links existed, roughly a thousand listing pages sat with nothing pointing at them, and Search Console showed the crawl settling to seven to eleven requests a day. A crawler that cannot find your pages does not care how good they are.

The trap: a vendor’s name is not their bot’s name

#

Crawler detection is a user-agent match, and here is where it bit.

Korean platforms crawl under names that look nothing like the company. Naver’s crawler is Yeti. Daum’s is Daumoa. Kakao’s link scraper is kakaotalk-scrap. None of them is the vendor name.

Meanwhile those same vendors ship apps with embedded browsers, and those send a real browser user-agent with the app’s name appended. KakaoTalk’s in-app browser says KAKAOTALK/10.5.1. Kakao’s scraper says kakaotalk-scrap/1.0.

Match on kakao and you match both. Which is exactly what I did, and so every KakaoTalk, Naver-app and Daum-app user who opened a link got the crawler page: no JavaScript, no navigation, no login. A perfectly good document, and completely useless to a human.

The fix is an ordering rule, and it is worth stating as one:

import re

# In-app browsers append the app's name to an ordinary browser UA.
INAPP_BROWSER_RE = re.compile(r"kakaotalk/|naver\(inapp|daumapps", re.I)

# Name the bot, never the vendor: Naver crawls as Yeti, Daum as Daumoa,
# Kakao's link-preview scraper as kakaotalk-scrap.
CRAWLER_RE = re.compile(r"googlebot|bingbot|yeti/|daumoa|kakaotalk-scrap", re.I)


def is_crawler(user_agent: str) -> bool:
    # An in-app browser is never a crawler, so that check comes first.
    if not user_agent or INAPP_BROWSER_RE.search(user_agent):
        return False
    return bool(CRAWLER_RE.search(user_agent))

In-app browsers are checked first and win. And every crawler token has to name a bot, never a vendor.

Then I put Cloudflare in front of it

#

The origin is a single shared-cpu machine in Sydney that sustains twenty to thirty requests a second. Googlebot has about 995 URLs to walk. Every one of them was reaching Sydney, because cf-cache-status read DYNAMIC on all of them: Cloudflare does not cache HTML unless a rule tells it to. About 16% of the zone was cached, and none of the part that mattered.

So: a cache rule on the crawler pages. Straightforward. Except that a listing URL does not have one body. It has three.

• A crawler asking for HTML gets the server-rendered page.
• A human gets the SPA shell.
• An agent sending Accept: text/markdown gets markdown, built from the database, because converting an empty HTML shell would produce an empty document.

The origin says Vary: Accept and Vary on the user-agent would be hopeless anyway. And here is the thing I did not know:

Cloudflare ignores Vary on HTML. All three bodies share one cache key.

Whichever body arrives first wins that URL for every later visitor in that edge location, for the hour the origin’s s-maxage allows, plus a day of stale-while-revalidate.

I got to observe this in production in both directions:

• A Naver-app user warmed a listing URL with the crawler page, and plain iPhone Safari then got that page on a HIT. A human staring at a JavaScript-less document.
• A human warmed a URL with the SPA shell, and Googlebot got the empty shell. The exact failure the whole SEO effort existed to prevent, now served faster and from the edge.

The rule that fixes it by narrowing

#

You cannot make Cloudflare respect Vary on HTML. What you can do is make sure only one of those three bodies is ever eligible to enter the cache:

(starts_with(http.request.uri.path, "/listings/")
 and not any(http.request.headers["accept"][*] contains "text/markdown")
 and (lower(http.user_agent) contains "googlebot" or ...)
 and not (lower(http.user_agent) contains "kakaotalk/" or ...))

Match crawler user-agents only, and exclude markdown requests. Now a human request matches no rule at all, so it reads DYNAMIC and goes to the origin as before. Only crawler requests read or write the cache, and the only body that can ever be in there is the crawler one.

Edge TTL is respect_origin, so the lifetime stays owned by the code that renders the page rather than being pinned in a dashboard.

One operational detail that is easy to miss: narrowing a rule does not evict what is already in the cache. You have to purge, or you keep serving the poisoned entries you just stopped creating.

The two lists, and why their errors are not symmetric

#

The rule needs a list of crawler user-agents, and Django already has one. Two copies of the same knowledge in two systems is usually a smell, but here the interesting part is what happens when they drift, because it is not symmetric:

• The bot list in Cloudflare must stay a subset of the one in Django. A token Cloudflare thinks is a bot but Django does not would cache an SPA shell under the crawler-only key, and then serve that to a real crawler.
• The in-app browser exclusion list must stay a superset of Django’s. Excluding too much only costs you a cache hit.

Get either one wrong in the safe direction and you lose some caching. Get the first one wrong in the other direction and you are back to serving Googlebot an empty page. That asymmetry is why the lists use contains rather than a regex, incidentally: matches needs a Business plan, and contains is expressive enough for a list that is allowed to be approximate in a known-safe direction.

An invariant that fails the build instead of the page

#

The crawler pages embed presigned image URLs, valid for seven days. The pages themselves are cached at the edge for an hour, plus a day of stale-while-revalidate.

Those two numbers are related, and if the cache lifetime ever exceeds the image lifetime, the edge starts serving pages whose every photo 403s. Nothing would alert on it; the HTML is fine.

So the code that defines those numbers refuses to import if the relationship breaks:

CRAWLER_IMAGE_TTL = 7 * 24 * 3600  # presigned image URLs
EDGE_S_MAXAGE = 3600
EDGE_STALE_WHILE_REVALIDATE = 24 * 3600

# Fail at import time if the edge cache could outlive the image URLs.
if EDGE_S_MAXAGE + EDGE_STALE_WHILE_REVALIDATE >= CRAWLER_IMAGE_TTL:
    raise RuntimeError("edge cache lifetime must stay below the image URL lifetime")

A constraint between two constants in different files is invisible in code review and obvious at process start. Put it where the process starts.

Two things Cloudflare did that I had to undo

#

Vary: Origin kept the bundle out of the cache. django-cors-headers stamps Vary: Origin on every response it touches, and Cloudflare declines to cache anything that varies on more than Accept-Encoding. That quietly excluded the 1.6 MB JavaScript bundle — the single biggest thing on the critical path — from the edge entirely. The fix is a middleware placed first in the list, so it runs last on the way out, rewriting Vary on the static asset paths.

Compression at the edge was worse than compression at the origin. Letting Cloudflare do Brotli on the fly produced 2,041,742 bytes where the origin’s own gzip produced 1,658,115. Twenty-three percent larger, for the privilege of not thinking about it. On-the-fly compression optimises for CPU, not for ratio; precompressed assets from the origin win.

Telling search engines without waiting to be crawled

#

Two protocols, and they cover disappointingly little between them.

IndexNow takes a ping and covers Bing, Yandex and Seznam. Google and Naver do not participate. I ping it in the background whenever a listing is saved, with a short timeout, and skip listings that are sold or closed. A network failure logs a warning and never fails the save that triggered it, which is the only acceptable behaviour for a best-effort side channel.

Google’s Indexing API accepts exactly two schema types, and one of them is JobPosting, which I have. Live jobs send URL_UPDATED, closed ones send URL_DELETED. That deletion is only honest because closed listings also emit noindex, and the code says so, so that whoever removes the noindex finds out what else they broke.

Two things cost me time there. The quota is 200 URLs a day, and a bulk import of listings would fire it hundreds of times — so imported listings are skipped, or one import spends the entire day’s budget. And the endpoint is urlNotifications:publish, with a colon. Use a slash and Google returns a plain HTML 404, which looks exactly like a disabled API.

What the structured data taught me

#

Three rules, each learned by having Search Console complain:

1. Omit what you cannot parse. Salaries arrive as free text. I used to emit the raw string as baseSalary.value, which is invalid, and flagged every job for a missing unit. Now an unparseable salary is simply absent.
2. Never fabricate a field to satisfy a schema. Job ads rarely carry a street address. The temptation is to invent something plausible for PostalAddress. A fabricated address on a job ad is worse than an incomplete one, so only locality and country go in.
3. Expiry is not optional. Google drops a JobPosting with no validThrough, so one is derived, with a floor so a stale listing does not claim to have expired yesterday.

And a fourth, about sitemaps: do not claim changefreq: daily for everything. Asking Googlebot to re-crawl a thousand listings every day reads as an overloaded host, and it responds by cutting the crawl rate for the whole site. Derive it from how recently the thing actually changed.

The sitemap also has a quality floor: forty characters of title and body combined. Search Console had reported around a thousand listings as “Crawled – currently not indexed,” and one used-goods page’s entire unique content was a one-word title and a phone number. Forty characters is deliberately low, because 471 of 484 listings are in Korean and Korean says more per character; an English-tuned threshold would have dropped half the marketplace. Thin listings stay crawlable through the index pages, and re-enter the sitemap automatically when someone edits them.

What I would take from this

#

• Serving different bodies at one URL is a cache-key problem before it is anything else. Work out what your CDN keys on, and assume it ignores Vary until you have proven otherwise on that exact content type.
• When two systems must agree on a list, work out which direction of error is safe and write that down next to both copies. “These can drift, and here is the only way they are allowed to” beats pretending they cannot.
• Match bots by bot name, never by vendor name. Especially outside the Anglosphere, where the vendor also ships the browser your users are holding.
• Put invariants between distant constants somewhere that fails loudly. Import time is free and nobody can skip it.

Update, 4 November 2026: one more correction. Those 504s aren’t probes the origin fails to answer. They’re Cloudflare’s own Early Hints cache lookups, which never reach the origin at all, and a 504 there only means nothing was cached for that URL. Turning Early Hints off wasn’t the best fix either: sending a Link response header made the feature work and turned those misses into hits. The full story is in Cloudflare 504 errors that aren’t real: Early Hints and HTTP 103.

With auto_stop_machines on, any request that lands on a stopped Fly.io machine waits for a full boot, and mine took 25 to 33 seconds because the entrypoint ran Django migrations and collectstatic on every machine start, and nothing had ever cached the bytecode. Running that release work once per image (a Redis lock keyed on FLY_IMAGE_REF) and adding compileall to the build got a cold boot down to about nine seconds. Switching auto_stop_machines from 'stop' to 'suspend', which meant dropping a swap file Fly won’t suspend with, got wake-ups to about three.

I went looking for a traffic spike on my Cloudflare dashboard and found something much worse underneath it: 22% of every request my app served in a day had failed. No alert had fired. Every health check was green. The logs looked perfectly healthy, because the failing requests never reached the application at all.

The cause turned out to be my own container startup script, doing sensible work at exactly the wrong time.

The spike was a red herring

#

The thing I set out to explain was one hour that took about 4,900 requests against a median of 121. Forty-two unique visitors. That is 110 requests per visitor, which is not how humans browse.

It was a scanner. A single bot walking through invented subdomains, asking each one for secrets:

api-stage-asia.example.com/.env.local
api-us-east-1-demo.example.com/.env
api-eu-uat.example.com/wp-config.php~
app-test-us-east-1.example.com/actuator/configprops
prod-api-use2.example.com/.env.local
relay.example.com/.env.production

Thirty or so hostnames, every .env variant you can think of, plus a Java framework’s actuator endpoints on a Django app. All of them got a 301 to my canonical host and nothing was served. Boring, and genuinely harmless.

But while I had the analytics open, I grouped the same day by status code. That is where the real problem was.

Twenty-two percent. And not spread evenly across the site, but concentrated exactly where it hurts:

Those are the endpoints my mobile app polls every 30 seconds. So this was not an abstract error-rate number. It was every user’s phone, quietly failing to load their unread badge, all day, for weeks probably.

Why the logs looked fine

#

My first instinct was that the app had become slow. It had not. I pulled the application’s own request lines for one of the worst hours out of the log archive:

[2026-09-08 01:00:29] "GET /api/notifications/unread/" 304 61.615
[2026-09-08 01:00:29] "GET /api/notifications/"        304 63.568
[2026-09-08 01:00:59] "GET /api/notifications/unread/" 304 58.742
[2026-09-08 01:00:59] "GET /api/notifications/"        304 105.438

Forty-four to a hundred milliseconds, 304s all the way down. The origin was fast.

That is the important clue, and it is worth stating as a rule: if your edge reports errors your origin has never heard of, the requests are dying before they arrive. Stop reading application logs and start looking at what sits in front of them.

In my case that is Fly.io’s proxy, and the answer was in how my machines are scheduled.

Scale to zero is a promise about money, not latency

#

My config looks like this, and on the face of it it is reasonable:

[http_service]
  auto_stop_machines = 'stop'
  auto_start_machines = true
  min_machines_running = 1

Idle machines stop. Traffic starts them again. You pay for what you use. For a small app with bursty traffic this is exactly what you want, and it is why I had ten machines defined but almost always one running.

The bill it hides is latency. When a request arrives for a stopped machine, somebody has to wait for that machine to boot. If booting takes longer than the proxy’s patience, that somebody gets a 504.

So: how long did my machines take to boot? I had never measured. The log archive knows, because the startup script announces each phase, and every line carries the machine id:

WITH s AS (
  SELECT instance, timestamp,
         CASE WHEN message ILIKE '%Starting app server%' THEN 'begin'
              WHEN message ILIKE '%Migrations complete%' THEN 'migrated'
              WHEN message LIKE  '%"GET /healthz%'       THEN 'serving' END AS phase
  FROM logs
  WHERE app = 'myapp' AND timestamp::date = '2026-09-08'
)
SELECT instance,
       min(timestamp) FILTER (WHERE phase = 'begin')    AS started,
       min(timestamp) FILTER (WHERE phase = 'migrated') AS migrated,
       min(timestamp) FILTER (WHERE phase = 'serving')  AS first_serve
FROM s GROUP BY instance

The answer, consistently, across every machine:

Then I counted how often that happened:

SELECT count(*) AS starts, count(DISTINCT instance) AS machines
FROM logs
WHERE app = 'myapp' AND timestamp::date = '2026-09-08'
  AND message ILIKE '%Starting app server%'
-- starts: 92, machines: 10

Ninety-two cold starts in one day. Each one a half-minute window in which anything routed to that machine timed out. Twenty-two percent stopped looking mysterious.

The cause was in my own entrypoint

#

Here is what every single machine ran before it would answer a request:

# entrypoint.sh, run on every machine start
echo "Running migrations..."
python manage.py migrate --noinput

echo "Running collectstatic in background..."
python manage.py collectstatic --noinput &

exec "$@"    # then start the app server

Read that again with “92 times a day” in mind.

migrate is a full Django boot: import every model, connect to Postgres, query the migrations table, decide there is nothing to do. On a shared CPU that is ten seconds to accomplish nothing. collectstatic is a second full Django boot that then uploads static files to object storage, competing for the same single core that the web server is trying to start on. The app server is the third boot.

Three Python interpreters, two of them doing work that had already been done, every time a machine woke up.

The reason it was written that way is good, and it is probably why yours looks similar: migrations have to run somewhere, Fly’s release_command had hung on me before, and the entrypoint is the one place guaranteed to execute on deploy. It is a correct place to put deploy work. It is just not a correct place to put it per machine start, and once your machines stop and start on their own, those are no longer the same event.

That is the whole bug, and I think it is a common one:

Work that belongs to a release was being run once per machine start. Scale-to-zero turned one event into ninety-two.

The fix: claim the release once

#

Migrations do not need to run on the eighth machine to boot a given image. They need to run once for that image, and everyone else needs to know it has happened.

That is a lock with a name that changes per deploy. Fly hands you the name for free in FLY_IMAGE_REF, and I already had Redis. The whole thing is a small script that runs before Django exists, so the common answer costs milliseconds rather than a framework boot:

# release_gate.py: runs before Django exists. Exit code 0 means "prepare this release".
import os, time

PREPARE, SKIP = 0, 1
LOCK_TTL, WAIT, POLL = 300, 240, 2
_sleep, _monotonic = time.sleep, time.monotonic    # aliases, so tests can replace them


def release_id() -> str:
    """Something that changes exactly when the deployed image does."""
    return os.getenv("FLY_IMAGE_REF") or os.getenv("FLY_MACHINE_VERSION") or ""


def claim(client, release: str) -> int:
    if client is None or not release:   # no Redis, or no idea which release: behave as before
        return PREPARE
    done_key, lock_key = f"release:{release}:done", f"release:{release}:lock"
    if client.get(done_key):            # somebody already did it for this image
        return SKIP

    owner = os.getenv("FLY_MACHINE_ID", "unknown")
    if client.set(lock_key, owner, nx=True, ex=LOCK_TTL):
        return PREPARE                  # we won the race; we do the work

    # Someone else is preparing. Wait for them, because serving requests
    # against a half-migrated schema is worse than starting slowly.
    deadline = _monotonic() + WAIT
    while _monotonic() < deadline:
        _sleep(POLL)
        if client.get(done_key):
            return SKIP
        if client.set(lock_key, owner, nx=True, ex=LOCK_TTL):
            return PREPARE              # the owner died and its lock expired; take over
    return PREPARE                      # waited long enough; do it ourselves

# ... plus `done` (sets done_key) and a main() that treats any Redis error as PREPARE