The fast way is a presigned URL. The API signs a URL, the client PUTs the file straight to the bucket, and the bytes never touch the API at all. It’s the standard answer for good reason. But the moment you do it you’ve cut your backend out of the loop. It handed out a URL and then heard nothing. Did the file arrive? Is it the size you expected? Is it even a JPEG? Something has to tell the backend, and the usual options are all a bit sad: trust the client to call back and say “done” (clients lie, crash and lose signal), poll the bucket on a schedule (slow and wasteful), or process lazily on the first read (the first viewer pays, and you still don’t know when the file is safe to reference). This is what makes uploading directly to S3 look harder than it is. The upload is easy. The signalling is what’s missing.

Object notifications are that signal. Tigris, the S3-compatible storage I use on Fly.io, will POST a small JSON body to your app every time an object is created or deleted in a bucket. On AWS the equivalent is S3 event notifications, which usually means wiring up a Lambda, an SQS queue or EventBridge before anything reaches your code (SNS can POST to an HTTPS endpoint, but then you’re confirming subscriptions and verifying signatures first). Tigris keeps it to one HTTP request to a URL you own, which is about as simple as storage-to-backend signalling gets. The feature exists so the bucket can talk to your backend without you building a bridge for it.

This post is the whole flow: the setup, a Django endpoint that receives the events, a presigned upload that keeps every bit of server-side processing, and the gotchas (CORS, size limits, expiry, multiple files). At the end, the same signal doing back-office work, where log files landing in the bucket became Discord alerts and let me delete a cron job I’d been nursing for months.

The short answer

#

In the Tigris dashboard, open the bucket, then Settings, then Object notifications. Give it an HTTPS URL on your app and pick basic auth (a username and password Tigris sends in the Authorization header; a bearer token is the other option). Add a filter so you only hear about the keys you care about. The filter is a SQL-like expression, so “only the three prefixes this post handles” is:

WHERE `key` REGEXP "^(uploads/raw|logs/api|media)/"

Tigris then sends a batch like this on every change:

{
  "events": [
    {
      "eventName": "OBJECT_CREATED_PUT",
      "eventTime": "2026-11-07T09:12:44Z",
      "bucket": "example-bucket",
      "object": { "key": "uploads/raw/1f3c.jpg", "size": 48213, "eTag": "9b2c…" }
    }
  ]
}

The event names you’ll meet are OBJECT_CREATED_PUT, OBJECT_CREATED_MULTIPART, OBJECT_CREATED_COPY and OBJECT_DELETED, plus rename and storage-tier transitions. Your endpoint checks the auth, routes each event by key prefix, and returns a 2xx within ten seconds. Three things to plan for from the start: delivery is at least once, so you will see repeats; events are batched, so one request can carry several; and they can arrive out of order, so don’t read “latest” off the order they came in.

A Django webhook for the bucket’s events

#

The view is boring on purpose. It checks the auth, parses the body and hands each event to a router:

import base64, hmac, json, logging
from django.conf import settings
from django.http import JsonResponse
from django.views.decorators.csrf import csrf_exempt
from django.views.decorators.http import require_POST

from .bucket_events import handle, parse

logger = logging.getLogger(__name__)

def _authorised(request, secret) -> bool:
    header = request.headers.get("Authorization", "")
    if not header.startswith("Basic "):
        return False
    try:
        user, _, password = base64.b64decode(header[6:]).decode().partition(":")
    except (ValueError, UnicodeDecodeError):
        return False
    return hmac.compare_digest(user.encode(), b"tigris") and hmac.compare_digest(
        password.encode(), secret.encode()
    )

@csrf_exempt
@require_POST
def bucket_webhook(request):
    secret = settings.BUCKET_WEBHOOK_SECRET
    if not secret:
        return JsonResponse({"error": "not configured"}, status=503)
    if not _authorised(request, secret):
        return JsonResponse({"error": "unauthorised"}, status=401)
    try:
        body = json.loads(request.body or b"{}")
    except ValueError:
        return JsonResponse({"error": "invalid json"}, status=400)
    for event in parse(body):
        try:
            handle(event)
        except Exception:
            logger.exception("bucket event failed: %s", event.key)
    return JsonResponse({"ok": True})

compare_digest is there so the check takes the same time whatever you send it (on bytes, because on strings it raises for non-ASCII input, and a crafted header would get a 500 instead of a 401). If the secret isn’t set, say on a fresh staging box, the endpoint refuses everything rather than accepting everything. And a handler that throws is logged and still answered 200: Tigris retries anything that isn’t a 2xx, a retry would hit the same bug, and by then the event has been claimed (two sections down) so the retry would do nothing anyway. The log line is what gets it fixed. That assumes the failure is deterministic. A transient storage timeout consumes the event for good: the photos don’t mind, because the save can process them itself, but a log file’s alert would be lost. If that matters to you, release the claim in the except and answer 500 so Tigris retries, and accept that a real bug then retries until it’s fixed.

Routing by prefix

#

Everything in the bucket is already organised by prefix, so that’s the natural way to route. Each use is one function and one line in a table:

HANDLERS = [
    ("uploads/raw/",   on_raw_photo),
    ("logs/api/",      on_api_log_file),
    ("media/",         on_media_change),
]

def handle(event):
    for prefix, fn in HANDLERS:
        if event.key.startswith(prefix):
            if claim(event):  # see below
                fn(event)
            return

Adding a fourth thing later is a new function and a new row. I like that more than I expected to.

At least once means “maybe twice”

#

Tigris retries if your endpoint is slow or returns an error, and like most webhook senders it makes no promise that you’ll see an event only once. So before a handler runs, the event is claimed by (key, eTag, eventName). The eTag matters: if a file is overwritten, it’s a new version with a new eTag and should be handled again. A repeat of the same version shouldn’t.

from django.core.cache import cache

def claim(event, ttl=7 * 24 * 3600) -> bool:
    k = f"bucket-event:{event.name}:{event.key}:{event.etag}"
    return cache.add(k, 1, ttl)  # False if someone already claimed it

cache.add only sets the key if it isn’t there already, which in Redis is atomic, so two machines receiving the same retry can’t both win. I use a database-backed lease for this in the real thing because I wanted it to survive a Redis flush, but the idea is identical.

Direct-to-S3 image uploads, with the pipeline intact

#

This is the part that pays for the plumbing.

Previously, picking photos for a listing meant the app held them, and when you hit Post it sent them all to the API as base64. The API resized them, made thumbnails, worked out colours and a blurhash, and only then replied. Ten photos on a slow connection was a long spinner. And a request carrying ten base64 photos is a big, slow body for an API server to hold in memory, which was quietly why I kept photos small.

Now it’s four steps, and the backend is told about every one of them.

1. The client shrinks the photo and asks for a slot

#

The app still resizes on the device, to the longest edge the server keeps (1080px) as a JPEG at 0.7 quality. That resize is for the user’s bandwidth, nothing else. Then it asks the API for an upload slot:

import uuid
from datetime import timedelta
import boto3
from django.utils import timezone

s3 = boto3.client("s3", endpoint_url="https://fly.storage.tigris.dev")
BUCKET = "example-bucket"

def create_upload_slot(user):
    public_id = uuid.uuid4()
    key = f"uploads/raw/{public_id}.jpg"
    upload = PhotoUpload.objects.create(
        user=user,
        public_id=public_id,
        raw_key=key,
        expires_at=timezone.now() + timedelta(days=2),
    )
    url = s3.generate_presigned_url(
        "put_object",
        Params={"Bucket": BUCKET, "Key": key, "ContentType": "image/jpeg"},
        ExpiresIn=15 * 60,
    )
    return {
        "upload_id": str(upload.public_id),
        "url": url,
        "method": "PUT",
        "headers": {"Content-Type": "image/jpeg"},
        "expires_in": 15 * 60,
    }

The slot is a row before it’s a file. That’s what lets the server know who asked for it, when it expires, and later whether it was ever used.

2. The client PUTs straight to the bucket

#

const slot = await api.post("/photo-uploads/");
const res = await fetch(slot.url, {
  method: slot.method,
  headers: slot.headers,
  body: blob,
});
if (!res.ok) throw new Error("upload refused");
// keep slot.upload_id for the save

The API never sees the bytes. This is the same from a browser, from React Native or from a native app; the only difference is that a browser needs a CORS rule on the bucket, which is in the gotchas below.

3. Tigris tells the backend, and the pipeline runs

#

Within a few seconds Tigris posts an OBJECT_CREATED_PUT for uploads/raw/…, and the handler runs the same image pipeline the base64 path always used, while the person is still typing their title:

from io import BytesIO
from PIL import Image, ImageOps
from django.core.files.base import ContentFile
from django.utils import timezone
import blurhash

MAX_BYTES = 15 * 1024 * 1024

def on_raw_photo(event):
    if not event.name.startswith("OBJECT_CREATED"):
        return
    if event.size > MAX_BYTES:
        s3.delete_object(Bucket=event.bucket, Key=event.key)
        return
    upload = PhotoUpload.objects.filter(raw_key=event.key, processed_at=None).first()
    if upload is None:
        return
    data = s3.get_object(Bucket=event.bucket, Key=event.key)["Body"].read()
    process(upload, data)

def to_webp(im, longest, quality):
    im = im.copy()
    im.thumbnail((longest, longest))
    buf = BytesIO()
    im.save(buf, "WEBP", quality=quality, icc_profile=im.info.get("icc_profile"))
    return ContentFile(buf.getvalue())

def process(upload, data):
    with Image.open(BytesIO(data)) as im:
        im = ImageOps.exif_transpose(im).convert("RGB")
        im.thumbnail((1080, 1080))
        upload.width, upload.height = im.size
        upload.dominant_color = "#%02x%02x%02x" % im.resize((1, 1)).getpixel((0, 0))
        upload.image.save(f"{upload.public_id}.webp", to_webp(im, 1080, 80), save=False)
        upload.thumbnail.save(f"{upload.public_id}_t.webp", to_webp(im, 320, 70), save=False)
        upload.blurhash = blurhash.encode(im.resize((32, 32)), 4, 3)
    upload.processed_at = timezone.now()
    upload.save()

The first line of on_raw_photo matters more than it looks. The router sends every event under uploads/raw/ here, including the deletes from the orphan sweep and from the size check itself, and get_object on a key that’s gone would only raise.

So the client’s shrink was only ever for the pipe. The server still decides the format (WebP), the sizes, the thumbnail, the placeholder, and the dimensions it stores. Re-encoding through Pillow also drops the EXIF block, which is where phone cameras put the GPS position, so the privacy strip comes free. The ICC profile would go the same way, which shifts wide-gamut phone photos a touch, so to_webp passes it through. If you’d rather keep pixel-perfect originals, don’t shrink on the client; the flow is identical, the PUTs are just bigger, and bigger PUTs are precisely the thing this design makes cheap. One catch: originals from an iPhone are HEIC, which Pillow only opens with pillow-heif installed.

4. The save attaches the photos and never waits on the webhook

#

When they hit Post, the request carries upload_ids instead of image data. The server looks each one up, with three outcomes:

def take(user, upload_ids):
    uploads = list(PhotoUpload.objects.filter(
        user=user, public_id__in=upload_ids, expires_at__gt=timezone.now()
    ))
    if len(uploads) != len(set(upload_ids)):     # expired, or not this user's
        raise UploadNotReadyError(upload_ids)
    for upload in uploads:
        if upload.processed_at is None:          # the webhook hasn't got here yet
            data = read_raw(upload)              # None if the PUT never landed
            if data is None:
                raise UploadNotReadyError(upload.public_id)
            process(upload, data)
    return uploads

If the notification already ran, the processed files are attached as they are. If the file is in the bucket but the notification hasn’t arrived, the save processes it inline, right there, so a post never waits on the webhook. The notification is a head start, not a dependency. And if the file isn’t there at all, or an id has expired or belongs to someone else, the save returns a validation error the app retries after a moment; the app waits for its own PUTs before sending, so in practice that’s rare.

A few details that made it hold up. The slot belongs to a user, so the server only accepts upload ids the same person created; otherwise anyone who saw an id could attach your photo to their listing. Everything falls back: if the slot request fails, the PUT is refused, or the API is an older version, the app sends base64 exactly as before. It only takes the upload path when every new photo has a slot, so a post’s pictures are all uploads or all base64, and their order stays one list. And orphans expire: people pick photos and then give up, so an upload that’s never attached expires after two days and the existing orphaned-media sweep cleans its files up.

Presigned URL upload gotchas: CORS, size limits, expiry, multiple files

#

Most of what goes wrong with presigned uploads goes wrong before the notification ever fires. These are the ones I’d want on one page.

CORS, for browsers only

#

A native app doesn’t care. A browser sends a preflight OPTIONS to the bucket before the PUT, and the upload fails with a CORS error unless the bucket’s CORS rules allow PUT from your origin with the Content-Type header. Tigris takes the standard S3 CORS configuration, so the rule looks the same as on AWS. It’s the first thing to check when a presigned PUT works from curl and not from the page.

The content type is part of the signature

#

Signing with ContentType means the client must send exactly that header, or the bucket answers 403 SignatureDoesNotMatch. That’s why the slot returns the headers along with the URL. Sign it on purpose, but know what it buys you: the header is enforced, the content isn’t, so a PDF labelled image/jpeg gets through. The pipeline is the real check; Pillow refuses to open what isn’t an image, and the row is never marked processed.

Size limits

#

A presigned PUT URL carries no size limit. If you need a hard stop before the bytes land, a presigned POST with a content-length-range policy condition does that. I didn’t bother, because the notification carries the object’s size: the handler deletes anything over the limit before reading it, and the save-time path reads at most the limit plus one byte so it can tell. The app sends a few hundred kilobytes; anything near fifteen megabytes isn’t ours.

Expiry

#

Two clocks. The presigned URL is good for fifteen minutes, which is plenty for one PUT and short enough that a leaked URL is useless soon. The slot row is good for two days, which covers someone picking photos, putting the phone down and finishing the listing tomorrow. The client treats a pending upload as stale after a day and starts a fresh one rather than hoping.

Multiple files, and bigger ones

#

One slot per file. Ten photos is ten slots and ten parallel PUTs, each with its own progress (an XMLHttpRequest gives you upload.onprogress; fetch doesn’t), which you can’t get from one giant base64 POST. Past a few megabytes, use multipart (parts are 5 MiB minimum, so it’s for files well beyond that): create the multipart upload on the server, presign each upload_part, and complete it when the parts are in. Tigris sends OBJECT_CREATED_MULTIPART when the upload is completed, so the same handler fires and nothing else changes.

Out of order

#

Events can arrive out of order, because the object may have been written in more than one region. For uploads it doesn’t matter: one key, one event. For a key that gets overwritten, trust the eTag and the object’s Last-Modified, not the order the webhooks reached you.

The same signal for logs: alerts to Discord

#

Once the bucket can talk to the backend, anything that lands in it can be reacted to the same way, and the second thing I pointed it at was logs.

The app’s logs are shipped to the bucket as small gzipped files, one every five minutes or so per app. For most of the year a GitHub Action on a 15-minute cron read the last 20 minutes of that archive with DuckDB, found the slow-request lines, and posted a digest to Discord. On paper that’s fine. In practice GitHub’s scheduled workflows don’t run on time when the platform is busy, so “every 15 minutes” was more like “every few hours”, and when it finally ran it had a bigger window to chew through and occasionally ran out of memory. The overlap I’d added so nothing slipped between runs meant the same slow request sometimes showed up twice. It was a lot of machinery for “tell me when a request is slow”.

Now each log file is read once, as it lands. They’re a few kilobytes gzipped. The handler pulls the slow-request lines out, and if there are enough of them, or one really slow one, it posts a digest to a Discord channel through a webhook. Same for a burst of 5xx responses in one file. A Discord webhook from Python is one call:

import requests
from django.conf import settings

def discord_alert(text: str) -> None:
    requests.post(
        settings.DISCORD_WEBHOOK_URL,
        json={"content": text[:1900]},  # Discord caps a message at 2000 chars
        timeout=5,
    )

No cron, no overlap, no duplicates, and the alert arrives within a few minutes of the request instead of whenever GitHub got round to it. The Action is gone.

While each file is open anyway, I also count what’s in it (errors, warnings, health checks) into an hourly tally table. The daily summary used to scan a whole day of the archive. Now it reads a few hundred rows, and only falls back to the archive if the tally has a gap.

A guard on deletes

#

This one I didn’t plan. It fell out of having OBJECT_DELETED events. When a media file is deleted, the handler checks whether any row still points at it. If one does, that’s a bug somewhere (or a person in the bucket console with a heavy finger), so it clears the column (the app shows a placeholder instead of a broken image) and reports it.

It also counts deletions per hour. Normal housekeeping removes a handful. If it ever sees a couple of hundred in an hour, I want to hear about that while it’s happening, not when users start reporting blank photos.

Things worth knowing

#

Respond fast. Tigris gives each delivery ten seconds. Reading a 5 KB log file or processing one phone photo fits; anything that might take longer goes to a queue and the view returns 200, or Tigris times out and retries, and you’re back to deduplicating.

Filter at the bucket. My log shipper writes a file every few minutes per app, forever. If you only care about one prefix, say so in the notification’s filter so your app isn’t woken for everything else.

Test with a fake body. The payload is small and stable, so the tests post JSON at the view with the right Authorization header and assert what the handler did. No real storage involved.

Don’t make it load-bearing on day one. Every use here has a fallback: the save can process a photo itself, the daily summary can still scan the archive, the app can still send base64. Notifications made each of them faster and cheaper, but none of them falls over if a webhook goes missing.

The best bit of the week was deleting that workflow file. Forty lines of YAML, a Python script, two repository secrets and a recurring “why didn’t the alert fire” are gone, and the replacement is one handler that reads one small file when it arrives. The uploads were the point, though. The photos go up as fast as the network allows, the server still gets the last word on every one of them, and nobody had to build a bridge between the bucket and the backend, because the bucket now just says what happened.

If you came here to run Docker Compose in GitHub Actions: I did that for two years and stopped. CI now declares the same Postgres and Valkey images as services: and runs the same uv run … and npm run … commands straight on the runner; Compose stays what a developer runs. The rest is why, with the files.

In 2024 I wrote Use Docker Compose for CI/CD Instead of Platform-Specific YAML, which said to keep the build logic in Compose and let the CI tool do nothing but call it. Two years on, I went back through every repository I ship from. Four have workflows, each with one committer, me, and none of those workflows calls docker compose any more. The principle held; the mechanism didn’t. Here’s what replaced it, with cut-down example versions of the files, the step timings GitHub recorded, and the parts that are less automated than a CI/CD diagram would like.

Every pipeline I run, in one table

#

In textbook terms the Hugo sites have continuous deployment, and the app (a marketplace and community app I build) has continuous integration plus one-command releases, which is continuous delivery: the checks tell me something is wrong, but they don’t stop me deploying. That’s deliberate. An OTA update reaches every phone on the binary version it targets, so publishing one should be a decision rather than a side effect of a merge, and the API is where the database migrations run. My other two Django apps have no workflows at all.

The CI half: one workflow, two jobs

#

The app’s CI is one file, .github/workflows/checks.yml. Until 11 October there wasn’t any. This is a trimmed example of its shape:

# Lint, types and tests on every push and pull request to main.
# Nothing here deploys.
name: Checks

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

concurrency:
  group: checks-${{ github.ref }}
  cancel-in-progress: true

jobs:
  python:
    runs-on: ubuntu-latest
    timeout-minutes: 30
    services:
      postgres:
        image: postgres:17
        env:
          POSTGRES_USER: myapp
          POSTGRES_PASSWORD: myapp
          POSTGRES_DB: myapp
        ports: ["5432:5432"]
        options: >-
          --health-cmd "pg_isready -U myapp"
          --health-interval 5s --health-timeout 5s --health-retries 20
      valkey:
        image: valkey/valkey:8-alpine
        ports: ["6379:6379"]
    env:
      DEBUG: "True"
      DATABASE_URL: postgresql://myapp:myapp@localhost:5432/myapp
      REDIS_URL: redis://localhost:6379
      PYTHON_JIT: "0"   # match production
    steps:
      - uses: actions/checkout@v7
      - uses: astral-sh/setup-uv@v10
        with:
          enable-cache: true
          prune-cache: true
      - run: uv sync --locked
      - run: uv run ruff check .
      - run: uv run mypy myapp/
      - run: uv run python manage.py test --parallel 4 --noinput

  app:
    runs-on: ubuntu-latest
    timeout-minutes: 20
    defaults:
      run:
        working-directory: frontend
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-node@v7
        with:
          node-version: 24
          cache: npm
          cache-dependency-path: frontend/package-lock.json
      - run: npm ci
      # Expo writes this on `expo start` and it is gitignored,
      # so a fresh checkout doesn't have it.
      - run: printf '/// <reference types="expo/types" />\n' > expo-env.d.ts
      - run: npm run typecheck
      - run: npx jest --ci

The two jobs run side by side. Across the 35 green runs so far, the app job’s median is about seven minutes and the Python job’s about twelve and a half.

The services: block is where the Compose file went, which is the next section. The environment is a few variables and a flag; everything else falls back to the development default docker compose up gives a developer, so the tests need no secrets. I checked by running part of the suite with exactly that environment and nothing else.

CI runs the same interpreter as my laptop and the production image, uv’s own build of Python 3.15.0, because uv sync reads .python-version and downloads exactly that build. (My real file also passes setup-uv a python-version-file input, which does nothing: setup-uv v6 has no such input, and every run carries a warning saying so. I only noticed while writing this.)

Matching interpreters earned their keep eight days after the workflow went in. On 19 October I switched on 3.15’s JIT compiler, and the first run with it failed a test: a plain yield from, on a code path the app takes on every save, raised AttributeError: 'list' object has no attribute 'send'. The same commit with the JIT off passed. My Mac is arm64 and never hit it; the runner is x86_64, like the production machines. The JIT is off again, in CI and in the Dockerfile, and the bug never reached production. It’s python/cpython#155823, fixed upstream on 13 October, after 3.15.0 shipped.

The very first run failed too, for a duller reason. Expo writes expo-env.d.ts when you run expo start, and the file is gitignored, so every working copy I’d used had one and a fresh checkout didn’t. TypeScript reported five errors that had never appeared locally. The step that writes the file is the fix.

concurrency cancels a run when a newer push to the same branch arrives, which is how all twelve cancelled runs so far ended. A pre-commit hook runs the linter, plus the type checker when Python or TypeScript is staged, so most mistakes never reach a push.

Where the Docker Compose file went

#

Here is the part of the app’s docker-compose.yml that CI cares about, as an example. The real one has more services and a lot more environment.

services:
  valkey:
    image: valkey/valkey:8-alpine
    ports:
      - "6379:6379"

  # Host port 55432, not 5432, so it can't collide with a local Postgres.
  postgres:
    image: postgres:17-alpine
    environment:
      - POSTGRES_USER=myapp
      - POSTGRES_PASSWORD=myapp
      - POSTGRES_DB=myapp
    ports:
      - "55432:5432"
    volumes:
      - postgres_data:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U myapp -d myapp"]
      interval: 5s
      retries: 10

  api:
    build: .
    command: python manage.py runserver 0.0.0.0:8000
    volumes:
      - .:/app
    ports:
      - "8000:8000"
    environment:
      - DATABASE_URL=postgresql://myapp:myapp@postgres:5432/myapp
      - REDIS_URL=redis://valkey:6379
    depends_on:
      postgres:
        condition: service_healthy
      valkey:
        condition: service_started

volumes:
  postgres_data:

Put it next to the services: block in checks.yml and it’s the same two images: Postgres 17 (the Alpine build here, the Debian one in CI) and valkey/valkey:8-alpine. That’s the relationship now. The Compose file is what a developer runs; CI declares the same images as service containers and runs the same commands a developer runs, uv run … and npm run …, straight on the runner.

Against the 2024 post, the part I cared about most held: the CI tool is thin, and the checks and deploys are commands I can run myself. What didn’t survive is CI calling Compose.

The app’s CI doesn’t build the Docker image at all; Fly’s builder does that once, at deploy time, and CI only has to say whether the code is good, which with a cached toolchain takes a six-second install. The Compose file also bind-mounts the working copy and seeds data, which a test run shouldn’t, so CI would have needed a Compose file of its own: a second description of the same services, kept in step by hand.

And the 2024 example had a bug I didn’t notice at the time. docker compose build builds the image and never runs the service’s command, so the npm test in that file never ran in CI. The workflow went green whether the tests passed or not.

The risk that post worried about most, switching CI vendors, never came up. The move that did happen was at the other end: this blog and the client’s site moved hosts to Cloudflare Pages, and for the blog that took one new 45-line workflow, because the build was already one command. And the three-hundred-line workflow I warned about? I have one, pyhtml2text’s. It’s long because the wheels have to be built on Linux, macOS and Windows runners, for several architectures each, which makes it the one job where platform-specific YAML is the right tool.

Caching: read the step timings first

#

GitHub keeps the duration of every step, and gh api repos/<owner>/<repo>/actions/runs/<run-id>/jobs hands it over. These are from the app’s latest green run, on 20 October:

Both dependency caches are on, and both are the cheap part. The uv cache GitHub keeps for the repository is about 200 KB, because setup-uv prunes whatever uv can simply download again; the npm one is nearly 400 MB. The time is in the tests, which already run four ways in parallel. If I wanted the run faster, I’d change the suite, not the cache.

The blog is the opposite case. Hugo’s processed images live in the gitignored resources/_gen, so every fresh runner re-encoded every cover, body image and placeholder. The workflow further down restores them with actions/cache, keyed on a hash of the pictures and falling back to the newest older cache, since Hugo only processes what’s missing. In the last eight runs before the cache, the build step took 33 to 76 seconds; in the 26 since, 6 to 20, with one exception at 94, even though the same commit added placeholders and a second card size.

The app’s image builds on Fly’s builder, with the dependency files copied before the code and uv’s download cache in a BuildKit cache mount (RUN --mount=type=cache,target=/root/.cache/uv). The bigger fix wasn’t a cache. The old .dockerignore was a blocklist, and it missed a folder of extra git worktrees, each with its own dependencies and build output, so 7.7 GB across 277,000 files went to the builder on every deploy. Rewritten as an allowlist, the context is about 40 MB. An example of the pattern:

# Exclude everything, then let back in only what the Dockerfile COPYs.
*

!pyproject.toml
!uv.lock
!manage.py
!myapp
!static

# Rules match the whole relative path and the last match wins,
# so these re-excludes apply inside the allowed directories too.
**/__pycache__/
**/*.pyc
**/.DS_Store
**/*.log

Secrets, in as few places as possible

#

Secrets live in three places. GitHub Actions secrets are for what workflows do, and the two Hugo repositories have exactly one each, CLOUDFLARE_API_TOKEN. Fly secrets are the app’s runtime configuration, set with flyctl secrets set, and they never pass through Actions. And a gitignored .env on my laptop holds the token the deploy script uses to purge Cloudflare’s cache.

PyPI needs no secret at all. pyhtml2text uses trusted publishing: PyPI accepts the short-lived OIDC token GitHub issues to the workflow, so the release job asks for id-token: write and the repository’s secrets page is empty. There’s nothing to leak and nothing to rotate.

Tokens get the least they need. The app’s Fly token in GitHub serves a scheduled check that only lists machines, so it’s created with fly tokens create readonly. The Pages deploys ask for contents: read and deployments: write, and nothing more.

Inputs go through env:, never straight into a script. The OTA workflow takes free-text release notes. Written as ${{ inputs.notes }} inside run:, the text would be pasted into the script before bash reads it, so a note with a quote and a semicolon in it would become code. Passed as an environment variable, it stays data. An example of the pattern:

      - name: Publish
        env:
          OTA_API_TOKEN: ${{ secrets.OTA_API_TOKEN }}
          INPUT_PLATFORM: ${{ inputs.platform }}
          INPUT_ROLLOUT: ${{ inputs.rollout }}
          INPUT_NOTES: ${{ inputs.notes }}
        run: |
          args=("$INPUT_PLATFORM")
          [ -n "$INPUT_ROLLOUT" ] && args+=(--rollout "$INPUT_ROLLOUT")
          [ -n "$INPUT_NOTES" ] && args+=(--notes "$INPUT_NOTES")
          ./scripts/publish-ota.sh "${args[@]}"

Some keys stay off GitHub altogether. One scheduled report needs a key with admin rights over the app’s sign-in backend, so it moved from a scheduled workflow to a scheduled job on my Mac. The workflow is still there as a manual fallback, and leaves out the sections that need that key.

Deploys: on every push, by script, and on a tag

#

The Hugo sites deploy on every push

#

An example version of the blog’s workflow, with the first-run step that creates the Pages project left out:

name: Deploy to Cloudflare Pages

on:
  push:
  workflow_dispatch:

env:
  HUGO_VERSION: "0.165.0"
  PAGES_PROJECT: my-site

concurrency:
  group: deploy-${{ github.ref }}
  cancel-in-progress: true

jobs:
  deploy:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      deployments: write
    steps:
      - uses: actions/checkout@v7
        with:
          submodules: true

      - uses: peaceiris/actions-hugo@v3
        with:
          hugo-version: ${{ env.HUGO_VERSION }}
          extended: true

      # Processed images are gitignored; restore them so Hugo only
      # re-encodes what is new. An older cache still helps.
      - uses: actions/cache@v6
        with:
          path: resources/_gen
          key: hugo-resources-${{ hashFiles('content/**/*.jpg', 'content/**/*.png', 'assets/img/**') }}
          restore-keys: |
            hugo-resources-

      # A multilingual site has no root 404.html; without one, Pages
      # answers unknown paths with index.html and a 200.
      - name: Build
        run: |
          hugo --gc
          cp public/en/404.html public/404.html

      # ...

      - name: Deploy
        uses: cloudflare/wrangler-action@v4
        with:
          apiToken: ${{ secrets.CLOUDFLARE_API_TOKEN }}
          accountId: ${{ vars.CLOUDFLARE_ACCOUNT_ID }}
          command: pages deploy public --project-name=${{ env.PAGES_PROJECT }} --branch=${{ github.ref_name }}

--branch=${{ github.ref_name }} makes main production and gives every other branch a preview URL. The client site’s workflow mirrors this one, plus an rm -rf public before the build, because its public/ is committed and removed pages would otherwise linger in the upload.

Hugo is pinned here and nowhere else. In October a package-manager upgrade moved my laptop to 0.168, one past the newest version the theme supports. The deploys stayed on 0.165 and never noticed.

And the build is the test. On 14 September the preview deploy for one branch failed three times running: a post on it linked, through Hugo’s ref, to one that was still a draft. CI builds what production builds, drafts excluded, so the link pointed at nothing and Hugo stopped with REF_NOT_FOUND. main carried on deploying in the meantime.

The app’s API deploys from a script

#

The core of the deploy script, as an example:

IMAGE_LABEL="deploy-$(git rev-parse --short HEAD)-$(date +%Y%m%d%H%M%S)"

# Build first, then wake a spare machine, then roll out exactly that image.
flyctl deploy --config fly.toml --build-only --push --image-label "$IMAGE_LABEL"
curl -fsS --retry 5 https://myapp.fly.dev/health > /dev/null
flyctl deploy --config fly.toml --image "registry.fly.io/myapp:$IMAGE_LABEL"

And the matching lines in fly.toml:

[deploy]
  # max_unavailable only counts machines that are running,
  # which is why the script wakes one first.
  strategy = "rolling"
  max_unavailable = 1

The image is built once, pushed to Fly’s registry under a label made of the commit and the time, and deployed by that label, so what rolls out is exactly what was built.

Waking a spare first is what makes it zero-downtime. max_unavailable = 1 only counts running machines, and the app keeps none running when it’s idle (a suspended machine resumes in about a second), so a lone awake machine used to be replaced in the same batch as the sleeping ones, with nothing warm behind it. On 28 September a probe against the origin saw 7 timeouts and a 7.3-second gap during a plain flyctl deploy, and no failures in 2,514 requests during the scripted deploy, which builds first and has a machine awake before the rollout starts.

The rest of the script marks the deploy in the team chat channel the error alerts go to, so a burst of errors can be read against what just changed, purges Cloudflare’s cache and warms the machines. The web front end skips the image entirely; that’s in Deploy Frontend and Backend Separately, Without Redeploying the API.

Migrations aren’t a pipeline step. Fly’s release_command hung for me, so the first machine to boot on a new image takes a lock and migrates while the others wait. Every migration therefore has to work with the code that’s still running: add a column in one deploy, stop using it in the next, drop it in a third.

Nothing in the script needs my laptop as such, but a couple of local-only files keep it there: the Cloudflare credentials, and a record of what was last deployed, which a fresh runner wouldn’t have. It will also deploy uncommitted changes, and says so in the chat channel, where the web release script refuses a dirty tree.

In Preview Environments per Branch on Fly.io I wrote that doing this by hand is fine while you’re the only one doing it, and that from the second developer on it has to happen because something happened in git. The app is still at one developer; the branch-environment workflows from that post live on the branch I built them on.

The OTA workflow is manual-only because publishing should be a decision, and because the update service refuses a release while a partial rollout is open, which an on-push trigger would hit during every canary. It has never run. Every OTA update so far has gone out from my laptop through the same script the workflow calls, which is the case for thin workflows in one line: however a release starts, it’s the same script.

pyhtml2text releases on a tag

#

The release job, with the package name made generic:

  release:
    runs-on: ubuntu-latest
    if: ${{ startsWith(github.ref, 'refs/tags/') || github.event_name == 'workflow_dispatch' }}
    needs: [linux, macos, windows, sdist]
    environment:
      name: pypi
      url: https://pypi.org/p/mypackage
    permissions:
      id-token: write       # trusted publishing and signing
      contents: write
      attestations: write
    steps:
      - uses: actions/download-artifact@v8
      - uses: actions/attest@v4
        with:
          subject-path: 'wheels-*/*'
      - uses: astral-sh/setup-uv@v10
        if: ${{ startsWith(github.ref, 'refs/tags/') }}
        with:
          prune-cache: true
      - name: Publish to PyPI
        if: ${{ startsWith(github.ref, 'refs/tags/') }}
        run: uv publish --trusted-publishing always 'wheels-*/*'

The release job waits for every build job, and only a tag publishes; a manual run builds and attests but uploads nothing. actions/attest records which workflow built each wheel, and uv publish --trusted-publishing always uploads them with the OIDC token rather than a password. The package itself is in A Closed Pull Request, a Fork and a PyPI Package.

Scheduled workflows are best effort

#

The app has one scheduled workflow, a check that Fly hasn’t started more machines than the baseline, which would otherwise first show up on the bill. It alerts by failing, since GitHub emails the owner about a failed scheduled run.

It’s set to cron: '*/15 * * * *', 96 runs a day. Over the past two weeks it ran between three and eight times a day, with a median gap of about three hours forty. The only explanation GitHub’s documentation offers is load: scheduled runs can be delayed when Actions is busy, and dropped when it’s busy enough. For a sustained scale-out, a few hours is fine. For anything that has to fire within minutes, a GitHub cron isn’t a monitor.

What I’d ask of a pipeline

#

If you’re building one, or looking at one you already have:

1. If GitHub Actions were down for a day, could you still release? If not, your release logic lives in YAML, and the 2024 post still applies.
2. Does CI run the interpreter, the CPU architecture and the database major version that production runs? Mine caught a compiler bug only because the runner and production are both x86_64 and my laptop isn’t.
3. Does a fresh checkout pass? Generated, gitignored files are the usual reason it doesn’t.
4. Where does a run’s time actually go? Read the step timings before you add a cache.
5. For each secret, how many places does it live, and does it need to be in GitHub at all?

There’s no hosted APM, no tracing in production and no dashboard I have to remember to open, and the only part that costs extra is the log archive, at about US$2 a month. Here’s the whole list, and the things I dropped or stopped trusting, one of which had me chasing an outage no user ever saw.

What I mean by monitoring, and how small this is

#

By DevOps monitoring I mean finding out that something’s broken before somebody tells me, and being able to work out why without guessing. Every tool below is judged on those two things.

Last year I wrote about setting up monitoring that developers actually use, mostly with teams in mind: a cloud provider’s built-in monitoring for a small team, a hosted APM once there are hundreds of services. This is the small end of the same idea. Both products are Django on Fly’s shared-cpu-1x machines, as described in My Current Stack, and I’m the only developer on either. If monitoring is a product and developers are its users, mine has a user base of one. A baseline I pulled from Fly’s metrics in September had the marketplace’s API averaging 1% CPU. This isn’t an enterprise stack, and I won’t dress it up as one.

Of the three pillars, metrics, logs and traces, I have two. Neither product has distributed tracing: each is one Django codebase and one database, with no service boundary for a trace to cross. The only traces I read are the reverse proxy’s, on my workstation.

The list

#

Everything in the table is the marketplace unless it says both apps. The estimating app has less: Fly’s health check, logs and metrics, app analytics on iOS, and a daily Parquet export of its database that I query with DuckDB. It has no crash reporting and no alerts channel, which is the biggest gap here. (A disclosure for the update-service row: the company behind it pays me to write for their blog.)

Errors: a logging handler posting to chat

#

Until September, flyctl logs was the only place an unhandled 500, a failed push notification or a broken WebSocket frame showed up, and nobody reads that stream unprompted. Now a logging handler sends every record at ERROR or above to a chat channel, from the API, the WebSocket server and the background worker. Warnings deliberately don’t alert.

The handler posts on a background thread, so it never slows a request. Nothing in it logs, so it can’t recurse. And the same error goes out at most once every ten minutes, with a cap of 30 an hour, so an outage reads as one message rather than a thousand.

The channel also carries what I want to read errors against: deploys post when they start, when they’re up and where they failed, with the commit and who ran it, and so do over-the-air updates and web releases. A burst of wrong sign-in codes on one account logs at ERROR too, so someone guessing shows up next to the outages.

In October an alert arrived that matched no line in any log. Now each alert carries a short random id, the hostname and the process’s start command, and writes the same line to that process’s log, so an id I can’t find means a process that doesn’t log there, like an SSH shell. Everything in the channel also goes into a feed that a coding agent I work with can read, so the agent working on the code sees the errors when I do.

Logs: Fly’s shipper to S3, queried with DuckDB

#

Fly’s own buffer holds a few minutes of output, roughly the last hundred lines, which is fine for watching a deploy and no use for yesterday. So a third Fly app runs Fly’s log shipper, which is Vector subscribed to the organisation’s log stream, with one change to its S3 sink:

[sinks.aws_s3]
  type = "aws_s3"
  bucket = "myapp-logs"
  compression = "gzip"
  key_prefix = "logs/{{fly.app.name}}/%F/"   # one folder per app per day
  batch.timeout_secs = 300                  # a quiet app still lands its lines

Every line from the marketplace’s apps lands in an S3-compatible bucket as gzipped JSON. The shipper is one shared-cpu-1x machine with 256 MB, about US$2 a month plus pennies of storage. DuckDB reads the files where they sit through one macro, and a day comes back in about a second:

CREATE MACRO logs(app, day := '*') AS TABLE
  SELECT * FROM read_json_auto('s3://myapp-logs/logs/' || app || '/' || day || '/*.log.gz',
                               union_by_name := true);

SELECT timestamp, message FROM logs('myapp-api', '2026-09-06') WHERE message LIKE '% 500 %';

That macro is how I measured the cold starts in the scale-to-zero post: the startup script announces each phase, every line carries the machine’s id, and the rest is a GROUP BY.

So it’s three levels of one stream, by who reads them: the archive keeps INFO and up, which is why the apps log at INFO in production, a live-tail script shows warnings and errors, and the chat channel gets ERROR. None of them knows the caller, though. The access log has no client IP or user agent, and for that I go to Cloudflare.

Metrics: Fly’s built-in Prometheus

#

Fly runs a managed Prometheus for each organisation, so I don’t run a metrics stack of my own. I query its API when I have a question: memory, CPU and burst credit per machine (fly_instance_cpu_balance matters on a shared CPU), responses and concurrency per app. Two things I wish I’d known first. A token from fly tokens create goes in as Authorization: FlyV1 <token>; Bearer only works with the personal token from flyctl auth token, and a read-only token gets “something went wrong resolving organization”. And it keeps about two weeks (Fly says roughly 15 days).

It earned its place in September, when the logs had “Could not open a database connection” several times a day, which reads exactly like an app bug. The managed Postgres cluster is in the same Prometheus, and at every one of those minutes the primary’s scrape time jumped from about 80 milliseconds to between one and ten seconds, while the replica was fine and Postgres’s start time never moved. Postgres hadn’t restarted; the host under it was stalling. I sent Fly’s support the timestamps, and after a maintenance restart in early October moved the primary to another host, the daily stalls stopped.

Crash reports: Sentry’s SDK, pointed at my own API

#

A crash on a phone used to leave no trace anywhere I could see. The obvious answer was Sentry, and I took half of it: the app carries Sentry’s open-source SDK, but its DSN points at an endpoint in my own Django API that accepts Sentry’s envelope format, groups the crashes, and alerts through the same chat handler. Tracing, sessions and replays are off. It’s errors only.

I didn’t sign up for sentry.io because, at a few reports a day, it would have been another vendor holding user data, another line in the privacy policy and another bill. The price is symbolication, which I haven’t replaced. Stacks are minified JavaScript or raw native addresses, so an issue tells me what crashed, on which release and device, and what the person did just before, but not where in the source.

The only dashboard is a message

#

I haven’t built a dashboard for either app. Last year’s post had a test for them: if nobody opens one unprompted, it isn’t earning its place. A message that arrives on its own doesn’t need opening, so the closest thing I have is a post in the team chat every morning.

It starts with the last 24 hours of production errors, read from the log archive rather than from the chat channel, because throttled alerts can’t say how often something happened. Numbers, ids and addresses are masked before grouping, so a hundred copies of one failure are one row with a count and no email reaches the channel.

Then failed health checks, yesterday’s usage against the same weekday a week earlier (a Sunday is never like a Monday), the last week of search traffic, chats waiting on a reply, the task board and what I’ve spent. A section that can’t load says so, and the post still goes out.

Reading app analytics through an API teaches you to distrust them a little. In August, 121 Android first-opens from Azerbaijan were followed by 119 uninstalls, and I read that month’s Android totals with them taken out.

Releases have their own screen. When over-the-air updates came from my own server, my view of a release was an admin list with an “is active” checkbox, and I had no idea how many phones had taken one. The hosted service’s dashboard shows which release devices run, how installs went, and a crash rollback when a bad bundle crashes and the SDK restores the previous one. I wrote up the move to a hosted update service separately.

What I dropped, or stopped believing

#

None of this was a paid tool. Mostly it was things I’d been treating as signals that weren’t, and one job running in the wrong place.

Cloudflare’s 504 rate, until I grouped it by user agent

#

In September the Cloudflare dashboard showed what looked like 22% of a day’s requests failing with 504s, concentrated on the endpoints the app polls every 30 seconds, while every health check was green and the app’s own logs said those requests were fast. I spent days on it, found that machines took 25 to 33 seconds to cold-start and did it 92 times a day, fixed that, and wrote it up in the scale-to-zero post as the cause.

The next morning I grouped the same 504s by user agent. All 7,344 of them between 5 and 9 September came from Cloudflare’s own Early Hints cache lookups (user agents bastion early hints and nginx-ssl early hints), which never reach the origin: a 504 on one of those lines is a cache miss, not a timeout. They copy real request paths, so they landed on the busiest endpoints and looked exactly like users failing. The app, browsers, curl and search crawlers had none. Early Hints wasn’t doing anything for me because the origin sent no Link header, so I turned it off, and later made it work.

The cold-start work still paid off, since deploys and wakes are faster for it, but the outage I thought I was fixing never reached a user.

So this is the rule I’d most like to pass on: before you believe a CDN’s error rate, group it by user agent and check it isn’t the CDN’s own machinery. On Cloudflare’s free plan that’s a GraphQL query on httpRequestsAdaptiveGroups, a day at a time. The same query settled the next scare, when one polling endpoint went from 70–260 requests a day to 2,880. It wasn’t a scraper. It was my own Mac, with the app left running in the iOS simulator after testing.

A green health check

#

Fly checks /api/health/ on both apps every 30 seconds and uses the answer to decide which machines get traffic, which it’s good at. But a health check runs against a machine that’s already up, so it can’t see what happens before one is: a cold start, a queue at the proxy, a request that never arrives. Routing is all I ask of it now.

Deploys get the probe script instead, which hits the origin throughout and exits 1 if anything failed. It caught a plain flyctl deploy leaving a 7.3-second gap, then counted no failures in 2,514 requests through the deploy script, which wakes a spare machine first.

The health check has also caused two outages of its own. In July, Django’s SECURE_SSL_REDIRECT answered Fly’s plain-HTTP probe with a 301, every machine failed its check, and everyone got a 503. In August, narrowing an except Exception in the health view let a “connection already closed” error escape as a 500, and Fly pulled the machine in exactly the case the view’s retry was there for. Now the check only asks whether the process can serve, and a database stall no longer takes both apps out of routing.

The morning summary had to learn the same lesson about counting. The marketplace suspends idle machines, so a machine woken by a request fails its first check a few hundred milliseconds after starting and passes the next. Counting those made every morning read like an outage, so a failure that passes again within a minute is now listed separately, as machines waking up and deploys.

A scheduled job in CI

#

The morning summary started as a CI workflow on a cron schedule and moved within a day. Once it read the analytics and search data, the key it needed was the analytics project’s service-account key, which can do anything in that project, and I didn’t want that sitting in CI secrets. Now a scheduler on my Mac runs it every 20 minutes, and it posts once it’s past eight in Auckland and today’s post hasn’t gone out, so a Mac asleep at eight posts when it wakes. The workflow stays as a manual fallback, minus the analytics and search sections.

If I were starting again

#

I’d set up three things before the first user rather than after the first incident: errors to a chat channel the team already reads, with a rate limit; a log archive in object storage I can query with SQL; and whatever metrics the platform already collects. That’s a couple of dollars a month. Crash reporting comes the week an app goes onto phones, because otherwise a crash on a phone tells nobody anything. A morning summary comes once there are enough places to check that I’d stop checking them.

I’d leave tracing until there’s a second service for a request to cross, and a hosted APM until there’s a team that needs shared dashboards, an on-call rota and a budget for someone else’s bill. That’s when the tools in last year’s post start paying for themselves. Before then the most useful habit is free: when a number looks alarming, find out what produced it before deciding what broke. Several of the scares in this post came from the monitoring, not the product.

Picture a small team working on one app. Two developers, or five; not specialists, but people who each take a feature from the database migration through the API to the screen in the mobile app, end to end, on their own branch. It’s a good way to run a team, and from the second developer onward it has one recurring failure: the message “Is anyone on staging?”

Behind that message is a developer who has finished a feature and wants a tester or a designer to see it, and can’t, because staging is running the second developer’s branch, and dev is running the third’s, which broke login two days ago and hasn’t been redeployed since. A handful of people, two shared environments, and a chat thread doing the scheduling.

That isn’t a people problem. It’s a topology problem. Every developer is treating the environment as the place where their branch becomes real, so the environment is doing the job the merge is supposed to do, one branch at a time, and everyone else is queueing for it.

I described the production shape of my stack in My Current Stack: Django on Fly.io’s cheapest machines, Postgres behind a connection pooler, Redis for the cache and the Channels layer, a separate WebSocket app, a background worker, an S3-compatible bucket for files, a hosted auth provider, Cloudflare in front, and a mobile app talking to all of it. This post is how I turned that stack into something where any branch gets its own public, HTTPS-fronted copy of the whole thing a couple of minutes after its developer puts one label on the PR, with nobody else asked and nobody provisioning it, and without a second Postgres cluster, a second Redis or a second bucket. The infrastructure stays shared. What changes is that every shared resource learns to be sliced by a name, and the name comes from the branch. Nothing in it assumes a team size: it’s the same design for two developers as for twenty, and what grows is the app list, which the sweeper handles. I built it, ran three branches at once, broke it in two places I hadn’t predicted, and tore it down, all in an afternoon; the outputs below are from that run.

Why one shared environment can’t scale past one developer

#

A shared dev environment has an assumption baked in: that there is one “current version of the code” that everyone wants to look at. That’s true for one developer, or for a team working a single stream of changes in lockstep. It stops being true the moment the second developer starts a feature on a different timeline, and with end-to-end ownership that’s every feature. From then on, everything the environment holds is contended:

• The deployed code. Whoever deployed last wins. Everyone else’s work is invisible until they redeploy, which undoes the first person’s.
• The database schema. This is the one that bites hardest when every developer owns their feature’s data model. The first branch adds a column. The second, deployed an hour later, doesn’t know about it and its ORM starts failing on inserts. Or both branches have a 0042_* migration, the shared database ends up in a state that matches nobody’s migration history, and someone resets it, deleting the test data the third developer spent an afternoon setting up.
• The cache. One branch’s serialiser change is cached under the same Redis keys another branch’s code reads. The bug reports that produces are baffling and cost a morning each.
• Third-party integrations. One OAuth callback, one webhook URL, one push certificate. Whoever’s branch is deployed gets the events.
• The tester. They can only test what’s deployed, so they test one feature at a time in the order the environment happened to be claimed, not the order the features are ready.

The team responds with a booking system: a chat thread, a pinned message, a bot with /claim staging. That’s a lock, and a lock on the only place your work can be seen is a lock on delivery rate. With two developers it’s an irritation. With three it’s tolerable on a good week. Add a fourth, or have one feature hold staging for three days of back-and-forth with a designer, and it stops being tolerable.

The first fix people reach for is a third shared environment, “dev2” or “uat”. That buys a few weeks. The queue just has two lanes.

The real fix is to notice that dev and staging were never environments. They were the current state of someone’s branch, with a URL. If that’s what they are, the right number of them is the number of branches the team cares about right now, and the right lifetime is the lifetime of the branch.

The model: the branch is the environment

#

Here’s the target state.

Read the Created by and Destroyed by columns again: every entry is a git event or a clock. Git is the control plane. The set of environments that should exist is a function of the set of branches that exist, and the automation’s only job is to keep reality matching it: a label on the PR creates, every push after that updates, closing or merging the PR destroys, and a nightly job reconciles whatever the first three missed. Nor does every branch get one: the label is the gate, one click by the developer, so a backup push or a bot’s dependency bump costs nothing. Nobody provisions an environment. Not a DevOps engineer with a ticket, not a bot you ask in chat, not an AI agent you prompt each time. The tenth developer’s first branch gets an environment the same way the first developer’s did, by labelling its PR, and if a person has to be asked, it hasn’t scaled; the queue has just moved.

And the rule that makes it work: every resource an environment touches is named after the branch, deterministically. Not after the developer, not after a ticket, not after a PR number that doesn’t exist until someone opens one. After the branch, because the branch is what developers actually work with, and because a deterministic name means the second run of ./scripts/env up on the same branch finds the environment it already made rather than making another.

The name is a slug for humans and a short hash for uniqueness:

branch=$(git rev-parse --abbrev-ref HEAD)              # feature/payments-retry
slug=$(printf '%s' "$branch" | tr '[:upper:]/_ ' '[:lower:]---' | tr -cd 'a-z0-9-' | cut -c1-20)
hash=$(printf '%s' "$branch" | sha256sum | cut -c1-6)
ENV="${slug%-}-${hash}"                                 # feature-payments-retr-3f9a1c

feature-payments-retr-3f9a1c is the environment. That string becomes the Fly app name, the Postgres database name (with hyphens turned to underscores), the Redis key prefix, the bucket’s object prefix, the error tracker’s environment, and the left-hand label of the hostname. Anyone who knows the branch name can compute it. Nothing has to be looked up.

That last sentence is the design. There is no registry of environments and no coordinator. env up, env down, the Cloudflare Worker that routes hostnames and the nightly sweeper each recompute the name from the branch independently and agree by construction. Cloudflare in particular is not coordinating anything: the Worker is a pure function from hostname to Fly app, and it never learns that an environment was created or destroyed. If the app exists the request lands; if it doesn’t, Fly answers with its own error.

Three things fall out of this.

First, “who’s on staging” stops being a question. There is feature-payments-retr-3f9a1c-dev.example.com and fix-login-redirect-8b21e0-dev.example.com, and each belongs to whoever is on that branch.

Second, staging becomes an honest rehearsal of prod rather than a dumping ground. It’s built from the exact image that will go to prod, against a database cloned from a prod-shaped snapshot minutes earlier. If it works there, the only variable left in prod is prod’s data.

Third, each developer gets power without getting root on anything shared. The blast radius of a branch environment is that branch environment. A fourth hire on their first day can deploy whatever they like to their branch, and the worst outcome is that their branch is broken.

fly.io/dashboard/my-org

Shared physically, separated logically

#

The tempting version of “an environment per branch” is a full copy: a Postgres cluster each, a Redis each, a bucket each. It’s clean and it’s the wrong trade. A Fly Postgres cluster is a machine plus a volume plus a restore; a Redis is another machine; you’d be waiting minutes for each environment and paying for a dozen idle databases. The sweeper would have four kinds of thing to clean up instead of one.

Instead, each shared service is provisioned once for the whole preview org, sized like a small prod, and every environment gets a slice of it keyed by $ENV. This is the table I’d want on the wall:

Two of those need a word.

Postgres: database, not schema. The other way to slice one cluster is a schema per environment, with search_path set per connection. Django will do it if you push a -c search_path=... through the connection options, but a connection pooler in transaction mode discards startup options unless you tell it not to, and then every connection in the pool has to belong to one environment anyway. It’s a source of “why is my query hitting the wrong tables” bugs that a database per environment simply doesn’t have. A database is a hard boundary, CREATE DATABASE ... TEMPLATE makes one in seconds, and DROP DATABASE makes it go away with nothing left behind. Branch environments connect to Postgres directly on port 5433 rather than through the pooler, the same way the background worker does in prod; the pooler earns its keep at production concurrency, which a branch environment never sees.

Secrets: generated, not shared. It’d be easy to have one set of dev secrets and copy it into every environment. Don’t. The env up script generates a fresh SECRET_KEY (session signing), ENCRYPTION_KEY (anything encrypted at rest) and ENV_TOKEN (the bearer token the mobile app presents, more on that below) for each environment and stores them only in that Fly app’s secret store. It means a session cookie from one branch is worthless on another, a leaked preview token opens one preview, and the developer never sees the values unless they go looking. If you keep secrets in a separate secrets manager rather than only in Fly, the same rule applies: the path is myapp/preview/$ENV/, and the sweeper deletes the path with the environment. The only secrets that are shared across environments are the credentials for the shared infrastructure itself (the Postgres admin URL, the Redis URL, the bucket key), and those live in CI, not in any environment.

The Redis row is the easiest to doubt, so here is the test. Set a cache key from inside one environment, read it from the other, then list every key in the shared Redis:

$ fly ssh console -a myapp-feature-profile-tagl-fa8832 -C "python manage.py shell -c \"from django.core.cache import cache; cache.set('who-am-i', 'tagline-branch', 600)\""
$ fly ssh console -a myapp-feature-profile-webs-dcb2e7 -C "python manage.py shell -c \"from django.core.cache import cache; print(cache.get('who-am-i'))\""
None

$ # every key in the shared Redis
  feature-profile-tagl-fa8832:1:who-am-i
  ...

Same Redis, same key name, the other environment sees nothing.

Everything else in this post is the script that stamps $ENV onto those eight rows, and the rules that keep it honest.

./scripts/env up

#

This is the whole idea, as a script a developer runs from their branch. CI runs the same script; there’s no separate CI path to drift. It needs fly and git on the laptop and nothing else: SQL goes to the Postgres app over fly ssh, and the environment’s Redis keys and bucket objects are removed by the environment’s own machine before it’s destroyed, so the laptop never holds the shared Redis or bucket credentials beyond a gitignored env file. Trimmed to the lines that matter:

#!/usr/bin/env bash
# scripts/env up|down|name|url: one environment per branch on shared preview infrastructure
set -euo pipefail
ORG=${PREVIEW_ORG:-myapp-preview}; PG_APP=${PREVIEW_PG_APP:-myapp-preview-pg}; DOMAIN=${BASE_DOMAIN:-example.com}

branch=${BRANCH:-$(git rev-parse --abbrev-ref HEAD)}
[[ "$branch" == "main" ]] && { echo "main has staging, not a branch environment" >&2; exit 1; }
slug=$(printf '%s' "$branch" | tr '[:upper:]/_ ' '[:lower:]---' | tr -cd 'a-z0-9-' | cut -c1-20); slug=${slug%-}
ENV=${ENV_OVERRIDE:-${slug}-$(printf '%s' "$branch" | shasum -a 256 | cut -c1-6)}   # the sweeper passes the name
APP="myapp-${ENV}"; DB="env_${ENV//-/_}"; HOST="${ENV}-dev.${DOMAIN}"

pg() { fly ssh console -a "$PG_APP" -q -C "psql postgres://postgres:${PG_PASSWORD}@localhost:5433/postgres -At -c \"$1\""; }
exists() { fly apps list --org "$ORG" --json | grep -q "\"Name\": *\"$APP\""; }

up() {
  echo "== $branch -> $ENV"
  exists || fly apps create "$APP" --org "$ORG"                         # idempotent
  [[ "$(pg "SELECT 1 FROM pg_database WHERE datname='$DB'")" == "1" ]] || pg "CREATE DATABASE $DB TEMPLATE seed_template"

  have=$(fly secrets list -a "$APP" --json | grep -o '"Name": *"[^"]*"' | cut -d'"' -f4 || true)
  gen() { grep -qx "$1" <<<"$have" || echo "$1=$(openssl rand -hex 32)"; }   # generated once, never rotated by a later `up`
  { gen SECRET_KEY; gen ENCRYPTION_KEY; gen ENV_TOKEN
    echo "ENV_NAME=$ENV"; echo "ALLOWED_HOSTS=$HOST,$APP.fly.dev"
    echo "DATABASE_URL=postgres://postgres:${PG_PASSWORD}@${PG_APP}.flycast:5432/${DB}"
    # ... plus the shared Redis URL and bucket credentials
  } | fly secrets import -a "$APP" --stage

  fly deploy -a "$APP" --config fly.preview.toml --remote-only --ha=false \
    --build-arg GIT_SHA="$(git rev-parse --short HEAD)" --build-arg GIT_BRANCH="$branch"
  echo "https://$HOST"
}

down() {
  [[ "$APP" =~ ^myapp-[a-z0-9-]+-[0-9a-f]{6}$ ]] || { echo "refusing to destroy $APP" >&2; exit 1; }
  if exists; then
    fly machine start -a "$APP" >/dev/null 2>&1 || true                 # it may be suspended
    fly ssh console -a "$APP" -q -C "python manage.py teardown_env" || echo "warning: cleanup failed; the sweeper will catch it" >&2
    fly apps destroy "$APP" --yes
  fi
  pg "DROP DATABASE IF EXISTS $DB WITH (FORCE)" || true
}

case "${1:-}" in up) up ;; down) down ;; name) echo "$ENV" ;; url) echo "https://$HOST" ;; esac

For the demo I ran it from my laptop rather than pushing, because the three branches were throwaways I didn’t want in the repository’s history and I wanted to watch the output as it went. In the team setup nobody types this: the label workflow further down runs exactly this script, and what follows is what would land in the Actions log. Here is what it printed the first time, on the branch that carries the tooling itself:

$ ./scripts/env up
== feature/branch-environments -> feature-branch-envir-c65cb4
New app created: myapp-feature-branch-envir-c65cb4
CREATE DATABASE
Secrets have been staged, but not set on VMs. Deploy or update machines in this app for the secrets to take effect.
==> Building image
image size: 139 MB
✔ Machine 1a2b3c4d5e6f78 [app] update finished: success
https://feature-branch-envir-c65cb4-dev.example.com

$ curl -s https://feature-branch-envir-c65cb4-dev.example.com/api/version/
{"environment": "feature-branch-envir-c65cb4", "branch": "feature/branch-environments", "commit": "a1b2c3d4", "region": "syd"}

About two and a half minutes from command to URL, most of it the image build. The second up on the same branch, after another commit, printed no “New app created” and no “CREATE DATABASE”, staged the same secret names without regenerating the generated ones, and rebuilt: same URL, new commit.

Some of the choices in there deserve a sentence each.

fly.preview.toml is not fly.toml. Prod runs the API, the WebSocket server and the background worker as separate Fly apps and process groups so each scales on its own metric, which is the whole argument of the stack post. A branch environment doesn’t need to scale; it needs to be one machine. So the preview config has a single process: the container’s entrypoint gets a preview mode that runs migrations and then starts one ASGI server with WebSockets enabled. Channels already routes both HTTP and WebSocket traffic within one ASGI application, so one process serves the whole app. Sync views run thread-sensitive in that mode, roughly one request at a time, which is exactly the trade I moved away from in prod and exactly right here, because nobody testing a branch will notice.

# fly.preview.toml (the app name comes from `fly deploy -a`)
app = 'placeholder'
primary_region = 'syd'

[processes]
  app = "/app/entrypoint.sh preview"   # migrate, then one ASGI server for HTTP and WebSockets

[env]
  DJANGO_SETTINGS_MODULE = 'myapp.settings'
  DEBUG = 'False'

[http_service]
  internal_port = 8000
  force_https = true
  auto_stop_machines = 'suspend'
  auto_start_machines = true
  min_machines_running = 0
  [[http_service.checks]]
    path = '/healthz'
    [http_service.checks.headers]
      Host = '127.0.0.1'               # the app's own hostname isn't known to a static file

[[vm]]
  size = 'shared-cpu-1x'
  memory = '1gb'

auto_stop_machines = "suspend", min_machines_running = 0. A branch environment nobody has opened since lunch should cost nothing. Suspend rather than stop gives the ~3 second resume I ended up with after chasing 504s instead of a 30-second cold start, and here even 30 seconds would be tolerable. Two of the three demo environments were already suspended in fly apps list a minute after their deploy finished.

fly.io/apps/myapp-feature-profile-tagl-fa8832

DEBUG = False, same settings module as prod. A branch environment is a public URL. It should behave like prod in every way that matters for testing (real HTTPS, real cookies, the same health check) and unlike prod in every way that could hurt someone. Rather than a separate preview settings file, the difference is carried by the secrets env up sets: sandbox payment keys, no admin credentials for the auth provider, and the ENV_NAME that switches on every prefix below.

One setting slices everything. ENV_NAME is the only new knob in the settings, and it’s applied in four places:

ENV_NAME = os.getenv("ENV_NAME", "")
_prefix = f"{ENV_NAME}/" if ENV_NAME else ""

CACHES["default"]["KEY_PREFIX"] = ENV_NAME                           # every cache key
CHANNEL_LAYERS["default"]["CONFIG"]["prefix"] = f"{ENV_NAME}:asgi"  # Channels group and channel names
STORAGES["default"]["OPTIONS"]["location"] = f"{_prefix}media"      # uploads
STORAGES["staticfiles"]["OPTIONS"]["location"] = f"{_prefix}static"

Miss the channels one and two branches’ WebSocket servers share group names, and a message sent in one environment turns up in another. That is the kind of bug that costs a day.

There was a fifth place I only found by reading the shared Redis after the fact. The startup lock from the 504s post, the thing that lets one machine per image run migrations while the others wait, was keyed on the image reference. Two branch environments built from the same commit have the same image. The second one would have found the first’s “done” marker and skipped its own migrations against its own, un-migrated database. Now the environment name is part of the key and the key lives under the prefix. It’s the sort of thing a shared-infrastructure design keeps producing: anything that was implicitly “per deployment” has to be made explicitly per environment, and you don’t know what was implicit until you look.

The database is cloned, not migrated from empty. That’s the next section.

The database: a template, cloned in seconds

#

A branch environment’s database can be one of three things. Shared with everyone, which is what got us here. Empty, migrated from zero and seeded with fixtures: deterministic, but fixtures are never as good as real-shaped data and are always eighteen months stale. Or a clone of a prod-shaped snapshot: realistic row counts, realistic nulls in columns the fixtures assumed were always filled, the migration that’s instant on an empty table and takes forty minutes on a real one shows itself.

The third, with the second kept as a CI-only check. And what makes the third fast enough to do per branch is a Postgres feature most people forget exists:

-- Nightly, on the preview cluster. Restore last night's prod backup into seed_raw,
-- run the anonymiser, then freeze it as a template nobody can connect to.
ALTER DATABASE seed_raw RENAME TO seed_template;
UPDATE pg_database SET datistemplate = true, datallowconn = false WHERE datname = 'seed_template';

-- Per branch, in the time it takes to copy the files:
CREATE DATABASE env_feature_payments_retr_3f9a1c TEMPLATE seed_template;

CREATE DATABASE ... TEMPLATE is a file-level copy within the cluster. A few gigabytes is seconds, not the minutes a pg_restore takes. The template can’t have open connections while it’s being cloned, which is what datallowconn = false enforces, and it’s also why the nightly job builds into seed_raw and renames at the end rather than restoring over the live template.

For the demo the template was built by hand rather than from a prod dump: a tunnel to the preview cluster with fly proxy, manage.py migrate, then the repo’s own seed commands for a small but realistic data set. Nineteen megabytes. Three clones of it appeared in the cluster as three branches came up, each 19 MB, each taking under a second:

$ fly ssh console -a myapp-preview-pg -C "psql ... -c '\l'"
             datname             | template | size
---------------------------------+----------+-------
 env_feature_branch_envir_c65cb4 | f        | 19 MB
 env_feature_profile_tagl_fa8832 | f        | 19 MB
 env_feature_profile_webs_dcb2e7 | f        | 19 MB
 seed_template                   | t        | 19 MB

The preview cluster is one unmanaged Fly Postgres node, shared-cpu-1x with a 1 GB volume, about two dollars a month. The branch environments connect to it over .flycast on the private network; nothing about it is public.

The anonymiser is not optional. The moment prod data is reachable on a -dev hostname, it needs to not be prod data: names, emails, phone numbers, addresses, free text, payment references, tokens, all overwritten before the rename. If the anonymiser fails, the previous night’s template stays and someone gets a message. It’s a small amount of work and it’s the thing that makes the whole approach acceptable to whoever answers your privacy questionnaire.

The entrypoint runs migrate on the first boot of every image, so the branch’s own migrations apply on top of the template. That’s the first honest test of “does my migration work on data shaped like prod”, and it happens before anyone looks at the URL.

Hostnames: a wildcard, a Worker, and two things the docs didn’t tell me

#

Every environment needs a public HTTPS hostname, and “public” is the point: a tester on a phone, a designer on another network, a product manager who won’t install a VPN. Fly gives each app myapp-$ENV.fly.dev for free, and that alone works. But *.fly.dev hostnames have three problems: they’re ugly to send around, Cloudflare isn’t in front of them so none of the protections you have on prod apply, and if you want them on your own domain you’re doing a DNS record and a certificate per environment, which is one more thing for the sweeper.

The version that removes all of that is one wildcard and one Cloudflare Worker. I’d planned *.dev.example.com. Two things stopped that, and both are worth knowing before you plan the same.

Universal SSL covers one wildcard level. Cloudflare’s free certificate is issued for example.com and *.example.com. It does not cover *.dev.example.com; a second-level wildcard needs Advanced Certificate Manager at $10 a month. The first request to feature-branch-envir-c65cb4.dev.example.com died with an SSL handshake failure before anything of mine ran. The fix is to keep the environment in the first label: feature-branch-envir-c65cb4-dev.example.com. That’s covered by the certificate that already exists, and by the proxied *.example.com record that already exists, the one I set up as a sinkhole for the scanners walking invented subdomains. So an environment needs no DNS at all.

Redirect Rules run before Workers. With the route in place, every -dev hostname answered 301 to the www homepage without the Worker running. The sinkhole is a zone-level redirect rule, “any other subdomain → www”, and Cloudflare’s redirect phase executes before the Workers phase. The rule needed one clause: and not ends_with(http.host, "-dev.example.com"). The deploy script adds it on up and removes it on down, because a rule someone edited by hand is the thing that gets forgotten.

dash.cloudflare.com/…/example.com/rules/redirect-rules/…/edit

The DNS side is the record that was already there:

dash.cloudflare.com/…/example.com/dns/records

With those two out of the way the routing is a single Worker on *-dev.example.com/*:

// dev-router.js: <env>-dev.example.com -> myapp-<env>.fly.dev
const DEV_SUFFIX = "-dev.example.com";
const ENV_NAME = /^[a-z0-9-]+-[0-9a-f]{6}$/;   // slug-hash, as scripts/env names them

export default {
  async fetch(request) {
    const url = new URL(request.url);
    if (!url.hostname.endsWith(DEV_SUFFIX)) return new Response("not a dev hostname", { status: 404 });
    const env = url.hostname.slice(0, -DEV_SUFFIX.length);
    if (!ENV_NAME.test(env)) return new Response(`no such environment: ${env}`, { status: 404 });

    url.hostname = `myapp-${env}.fly.dev`;
    const upstream = new Request(url, request);
    upstream.headers.set("X-Forwarded-Host", request.headers.get("host") ?? "");

    const res = await fetch(upstream);
    const out = new Response(res.body, res);
    out.headers.set("X-Robots-Tag", "noindex, nofollow");   // a half-finished branch is not for search engines
    out.headers.set("X-Dev-Environment", env);
    return out;
  },
};

Cloudflare terminates TLS on the wildcard, the Worker computes the Fly hostname from the label, and Fly’s own *.fly.dev certificate covers the hop to the machine. WebSocket upgrades pass through fetch unchanged. And because the route is on the zone, everything Cloudflare does for prod is available: WAF rules, rate limits, bot fight mode, and Access.

dash.cloudflare.com/…/example.com/workers

Three branches up at once, one Worker, and what each hostname answered:

Every 200 came back with X-Robots-Tag: noindex, nofollow and X-Dev-Environment: <env> added by the Worker. The 404 is the Worker’s own, before Fly is ever asked.

Cloudflare Access is what turns “public” into “reachable by the people who should reach it”. One application on *-dev.example.com with a policy of “anyone with an @example.com email” and browser visitors get a login page once and never think about it again. That covers designers, PMs and testers on laptops. I didn’t add it for the demo; the environments were up for an hour with noindex and nothing on them but seed data.

It doesn’t cover the mobile app, because a native app can’t complete an Access login redirect. Two ways through. Access supports service tokens (a CF-Access-Client-Id / CF-Access-Client-Secret header pair) that bypass the login, and the mobile app’s dev menu can carry a pair for the dev hostnames. Or, simpler, add an Access bypass for /api/ and have Django reject any request without either a signed-in user or the environment’s ENV_TOKEN, the one env up generated for exactly this environment. Either way, the mobile side is a dev-menu screen with two fields, base URL and token, and the app a tester already has installed is now talking to that branch. No new build. The over-the-air update channel stays pointed at whatever the app was built with; the API base URL is a runtime setting, not a build-time one, and it should have been anyway.

Auth is the one shared service that’s shared on purpose. Identity isn’t environment-specific; a tester wants to log in with the same account on every branch. One dev project at the auth provider with example.com in its authorised domains covers every -dev hostname, and every environment verifies ID tokens against it. The users table in each environment’s database is a copy of the template’s users, so the tester’s account exists everywhere the template did.

Automating preview environments with GitHub Actions: events in, environments out

#

Everything above by hand is fine when you’re the only one doing it. The moment there’s a second developer it has to happen because something happened in git, not because someone remembered. The whole automation is four workflows, one repository setting, one ruleset and one GitHub environment, and every one of them maps a git event to a call of the same scripts/env the developer runs by hand. There is no CI-only path to drift.

The label, push and delete workflow

#

# .github/workflows/branch-env.yml
name: branch environment
on:
  pull_request:
    types: [labeled, unlabeled, synchronize, closed]
  delete:
  workflow_dispatch:
    inputs:
      branch: { description: branch to bring up without a PR, required: true }

concurrency:
  group: env-${{ github.head_ref || github.event.ref || inputs.branch }}
  cancel-in-progress: true

env:
  FLY_API_TOKEN: ${{ secrets.FLY_PREVIEW_TOKEN }}   # org-scoped deploy token; a separate preview org keeps it away from prod
  PG_PASSWORD: ${{ secrets.PREVIEW_PG_PASSWORD }}
  # ... the shared Redis URL and bucket credentials, the same way

jobs:
  up:
    # the gate: a PR carrying the `env` label, or someone asking for a branch by hand
    if: >-
      github.event_name == 'workflow_dispatch' ||
      (github.event_name == 'pull_request' && github.event.action != 'closed'
        && contains(github.event.pull_request.labels.*.name, 'env'))
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with: { ref: "${{ github.head_ref || inputs.branch }}" }
      - uses: superfly/flyctl-actions/setup-flyctl@master
      - id: env
        env: { BRANCH: "${{ github.head_ref || inputs.branch }}" }
        run: scripts/env up && echo "url=$(scripts/env url)" >> "$GITHUB_OUTPUT"
      - if: github.event_name == 'pull_request'
        uses: marocchino/sticky-pull-request-comment@v2
        with:
          header: env
          message: "Branch environment: ${{ steps.env.outputs.url }}"

  down:
    # label removed, labelled PR closed or merged, or the branch itself deleted
    if: >-
      (github.event_name == 'pull_request' &&
        ((github.event.action == 'closed' && contains(github.event.pull_request.labels.*.name, 'env')) ||
         (github.event.action == 'unlabeled' && github.event.label.name == 'env'))) ||
      (github.event_name == 'delete' && github.event.ref_type == 'branch')
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: superfly/flyctl-actions/setup-flyctl@master
      - run: BRANCH="${{ github.head_ref || github.event.ref }}" scripts/env down

The gate is the env label, and it’s the part I’d argue for hardest. Without one, every branch costs an app and a database clone: the branch someone pushes to back up a laptop, the spike that gets squashed tomorrow, the dependency bump a bot opened at 3 a.m. So a push alone does nothing and opening a PR does nothing. Adding env to the PR brings the environment up, every push while the label is on updates it, and removing the label, closing the PR or merging it takes it down. Opting in is still a GitHub event, not a request to a person, and it’s one click made by the one person who knows whether anyone is going to look. The sticky comment is the user interface: label the PR, a comment appears with a URL they can send to anyone, and it updates in place on every push rather than stacking. workflow_dispatch covers the branch you want up before there’s a PR, and ./scripts/env up from a laptop covers the rest.

Two other gates are worth knowing. A branch-name filter (branches: ['feature/**', 'fix/**']) needs no PR but makes every feature branch pay. “Any PR that isn’t a draft” is the most automatic, and the right choice if your team opens PRs late and only when they’re ready. I went with the label because it puts the cost decision with the person who has the context, and because it can be taken back without closing anything. One side effect I like: pull_request runs from forks don’t get secrets, so a fork can’t bring up an environment.

concurrency matters more than it looks. A run of quick pushes to one branch cancels the earlier deploys rather than racing them; two fly deploys to the same app at once end with one of them losing.

closed covers teardown for anything that had a PR, merged or not. The delete event is the backstop for a branch brought up by workflow_dispatch that never got one, and it only fires if branches actually get deleted. That’s a repository setting, Automatically delete head branches, and it’s off by default. With it on, clicking Merge closes the PR and deletes the branch, and either event alone would have run env down. The merge button is the teardown.

gh api -X PATCH repos/org/myapp -f delete_branch_on_merge=true

The FLY_PREVIEW_TOKEN is the dangerous line in the file. It can create and destroy apps, and if $APP were ever anything but a branch environment it could destroy prod. Two guards, and I’d keep both: the token is scoped to a separate Fly org so it literally cannot see the production org (for the demo I used the production org, to skip setting up billing on a new one, and relied on the second guard alone); and down() refuses any app name that doesn’t match the slug-hash pattern.

The check a branch environment can’t do for itself

#

A branch only ever sees its own migrations, so the check that matters runs on the pull request against a merge with main, and it’s the one status check main requires:

# .github/workflows/migrations-check.yml
name: migrations
on:
  pull_request:
    branches: [main]

jobs:
  check:
    runs-on: ubuntu-latest
    services:
      postgres:
        image: postgres:18
        env: { POSTGRES_PASSWORD: ci, POSTGRES_DB: ci }
        ports: ['5432:5432']
        options: --health-cmd pg_isready --health-interval 5s --health-retries 10
    env:
      DATABASE_URL: postgres://postgres:ci@localhost:5432/ci
      SECRET_KEY: ci-only
    steps:
      - uses: actions/checkout@v4
        with: { fetch-depth: 0 }
      - uses: actions/setup-python@v5
        with: { python-version: '3.14' }
      - run: pip install -r requirements.txt
      - name: Merge main into the branch (a conflict here fails the check too)
        run: |
          git config user.email ci@example.com && git config user.name ci
          git merge --no-edit origin/main
      - name: One leaf in the migration graph
        run: python manage.py makemigrations --check --dry-run
      - name: The full chain applies from nothing
        run: python manage.py migrate --noinput

Thirty seconds. It’s what turns the 0042 / 0042 collision from a release-day surprise into a red check on the second developer’s PR; the migrations section below shows it firing.

The rules on main

#

None of this holds up if branches live for a month or if someone pushes to main directly. The rules are less about protection and more about keeping environments small and their migrations current:

• main is protected and always deployable. Merge via PR with the migrations check green against the current main (strict status checks, so a PR that was green last week re-runs after main moves). Linear history, no force-push, no deletion.
• Feature branches are short. A branch open for two weeks has an environment two weeks behind main and a migration two weeks more likely to collide with someone else’s. A month-long feature merges to main behind a flag in pieces. The flag is the long-lived thing, not the branch.
• One branch, one environment, one database. Already enforced by the naming.
• Release tags are the only thing that goes to prod. The image is built once from the tag; staging and prod run that image, not a rebuild.
• A PR that touches migrations/ is reviewed by one of the other two. Below.

As a ruleset, applied once with gh api -X POST repos/org/myapp/rulesets --input ruleset-main.json:

{
  "name": "main", "target": "branch", "enforcement": "active",
  "conditions": { "ref_name": { "include": ["~DEFAULT_BRANCH"], "exclude": [] } },
  "rules": [
    { "type": "deletion" },
    { "type": "non_fast_forward" },
    { "type": "required_linear_history" },
    { "type": "pull_request",
      "parameters": { "required_approving_review_count": 1, "dismiss_stale_reviews_on_push": true,
                      "required_review_thread_resolution": true } },
    { "type": "required_status_checks",
      "parameters": { "strict_required_status_checks_policy": true,
                      "required_status_checks": [ { "context": "check" } ] } }
  ]
}

One honest wrinkle. This repository is private on GitHub’s Free plan, and on Free, branch protection and rulesets are only available for public repositories; the API answers 403 Upgrade to GitHub Pro or make this repository public. The delete_branch_on_merge setting, the environments and the workflows all work on Free. So for a small team on a private Free repo, the ruleset above is a convention: the migrations workflow still runs and still goes red, but nothing greys out the merge button. That’s a $4-a-month decision, and I’d make it the moment a second person can merge.

There’s no develop branch and no release/* branches. Git-flow was designed around integration being expensive and needing to be batched. Branch environments make integration cheap, so the batching goes.

Staging is a tag’s environment, and prod is a promotion

#

Staging reuses the whole machinery. A v* tag on main runs scripts/env up with the branch name staging/v1.42.0, which the naming turns into staging-v1-42-0-<hash>: same clone-from-template database, same suspend-when-idle machine, same hostname scheme. The difference is what comes after:

# .github/workflows/release.yml (abridged)
on:
  push:
    tags: ['v*']
jobs:
  staging:
    steps:
      - run: BRANCH="staging/${GITHUB_REF_NAME}" scripts/env up
      - run: echo "image=$(fly image show -a "myapp-$(BRANCH=staging/$GITHUB_REF_NAME scripts/env name)" --json | jq -r '…')" >> "$GITHUB_OUTPUT"
  production:
    needs: staging
    environment: production        # required reviewer on this environment = the promotion gate
    steps:
      - run: fly deploy --config fly.api.toml --image "${{ needs.staging.outputs.image }}"
      - run: fly deploy --config fly.ws.toml  --image "${{ needs.staging.outputs.image }}"
      - run: BRANCH="staging/${GITHUB_REF_NAME}" scripts/env down

The production job waits on a GitHub environment with a required reviewer. Someone looks at staging, clicks Approve, and prod receives the same image staging ran, by reference, not a rebuild. The last step destroys staging because it’s done its job. If nobody approves, the release candidate was rejected, and the sweeper removes staging when its “branch” (staging/v1.42.0 never exists on origin) fails the liveness check. Nothing gets to exist without a reason the sweeper can check.

The nightly sweeper

#

# .github/workflows/sweep-envs.yml
on:
  schedule:
    - cron: '0 14 * * *'     # 02:00 NZST
  workflow_dispatch:
jobs:
  sweep:
    steps:
      - uses: actions/checkout@v4
        with: { fetch-depth: 0 }
      - uses: superfly/flyctl-actions/setup-flyctl@master
      - run: scripts/sweep-envs

fetch-depth: 0 because the sweeper needs every remote branch and its last commit date, not the one commit Actions checks out by default. workflow_dispatch so it can be run by hand after an incident.

What a developer actually does

#

Push a branch, open a PR, and put the env label on it when they want someone to look. That’s the list. A URL appears on the PR within three minutes, updates on every push, and disappears when the PR is merged or closed. The migration check tells them before anyone else does if they’ve collided with a teammate. Nobody claims staging because there is no staging to claim, and nobody tears anything down because tearing down is what merging does.

And what the person who set it up does per branch, afterwards: nothing. The shared Postgres, Redis and bucket are provisioned once, the secrets go into Actions once, and from then on the recurring work is reading the sweeper’s log on a Monday. No ticket queue, no bot to ask, nobody to prompt.

Teardown that doesn’t trust the teardown

#

env down is the ordinary case, and the interesting part of it is who does the cleaning. The Redis keys and the bucket objects belong to the shared services, and the laptop running down doesn’t need credentials for either, because the environment’s own machine already has them and already knows its prefix. So down starts the machine if it’s suspended, runs a small cleanup command on it, and only then destroys the app and drops the database. The environment cleans up after itself:

$ BRANCH=feature/profile-website ./scripts/env down
redis: deleted 0 keys under feature-profile-webs-dcb2e7:*
bucket: deleted 278 objects under feature-profile-webs-dcb2e7/
Destroyed app myapp-feature-profile-webs-dcb2e7
DROP DATABASE

$ curl -s -o /dev/null -w '%{http_code}\n' https://feature-profile-webs-dcb2e7-dev.example.com/api/version/
530

The 530 is the point about coordination again: the Worker still routes the hostname, Fly has nothing there, and nobody had to tell Cloudflare.

The cleanup command refuses to run without ENV_NAME, because with an empty prefix it would be a wipe of the shared bucket. That guard is the whole reason it exists as a command inside the app rather than three lines in the shell script.

Environments leak anyway. A branch is deleted while GitHub Actions is having an outage. A developer runs env up from a laptop for a spike and never pushes. A fly apps destroy fails transiently and the || true swallows it. Three months later there are forty apps and a bill that makes someone ask what happened.

So there’s a second mechanism that doesn’t trust the first one: the nightly sweeper from the table above. It lists every myapp-* app, works out which branch each one belongs to, and destroys any whose branch is gone or hasn’t been pushed to in seven days. A week-idle branch doesn’t need a live URL; the next push recreates it in two minutes. A hash isn’t reversible, so the sweeper computes the environment name for every live branch and destroys any app that isn’t in that set, then drops any env_* database with no app behind it:

#!/usr/bin/env bash
# scripts/sweep-envs: nightly
set -euo pipefail
ORG=${PREVIEW_ORG:-myapp-preview}; TTL_DAYS=${TTL_DAYS:-7}
cutoff=$(( $(date +%s) - TTL_DAYS*86400 ))

git fetch --prune --quiet origin
live=""
while IFS=$'\t' read -r ts branch; do
  [[ "$branch" == "main" ]] && continue
  (( ts >= cutoff )) || continue
  live+="$(BRANCH="$branch" scripts/env name)"$'\n'
done < <(git for-each-ref --format='%(committerdate:unix)%09%(refname:short)' refs/remotes/origin | sed 's#^\([0-9]*\)\torigin/#\1\t#')

apps=$(fly apps list --org "$ORG" --json | grep -o '"Name": *"myapp-[^"]*"' | cut -d'"' -f4)
for app in $apps; do
  env="${app#myapp-}"
  [[ "$env" =~ ^[a-z0-9-]+-[0-9a-f]{6}$ ]] || continue       # not a branch environment (e.g. myapp-preview-pg)
  grep -qx "$env" <<<"$live" && continue
  echo "sweeping $app: branch gone or idle > ${TTL_DAYS}d"
  ENV_OVERRIDE="$env" scripts/env down || true
done
# ...then any env_* database with no app: DROP DATABASE.

None of the demo branches had ever been pushed (the laptop run above), so from the sweeper’s point of view all of them were gone, which is exactly the state a spike someone brought up from a laptop and never pushed would be in a week later. It took the remaining two apart, including the Redis keys the tagline environment had written:

$ ./scripts/sweep-envs
sweeping myapp-feature-branch-envir-c65cb4: branch gone or idle > 7d
redis: deleted 0 keys under feature-branch-envir-c65cb4:*
bucket: deleted 322 objects under feature-branch-envir-c65cb4/
Destroyed app myapp-feature-branch-envir-c65cb4
DROP DATABASE
sweeping myapp-feature-profile-tagl-fa8832: branch gone or idle > 7d
redis: deleted 3 keys under feature-profile-tagl-fa8832:*
bucket: deleted 278 objects under feature-profile-tagl-fa8832/
Destroyed app myapp-feature-profile-tagl-fa8832
DROP DATABASE

$ fly apps list --org myapp-preview | grep myapp-
 myapp-preview-pg │ myapp-preview │ deployed

(One small thing that bit: macOS ships bash 3, which has no associative arrays. The first version of the sweeper used one and died on declare -A on my laptop. The version above uses a newline-separated list and runs on anything.)

The cost is the pleasant part. A suspended machine costs its rootfs, which is cents a month. A database is 19 MB on a volume that’s already paid for. The Redis is on a pay-as-you-go plan, twenty cents per hundred thousand commands, which at branch-testing volumes rounds to nothing. A bucket prefix costs what it stores. The shared Postgres node is the only fixed line, about two dollars a month. Twenty active branches would cost roughly what one always-on shared-cpu-1x costs, and the sweeper keeps it at twenty rather than two hundred. The whole demo, three environments for an hour plus the shared pieces, cost less than the coffee I drank during it.

Migrations across branches, without the shared database to warn you

#

When each developer owns a feature end to end, every branch has a migration in it. That makes this section the important one.

Here’s the thing people miss when they leave the shared-environment world. When everyone deployed to one dev database, migration conflicts surfaced immediately and painfully, but they surfaced. One migration ran, the next deploy failed, someone fixed it that afternoon. With a database per branch, each migration works perfectly in its own environment, and two of them first meet when the second one merges to main. Done carelessly, this moves the conflict from dev, where it was cheap, to a release, where it isn’t.

Four rules, all enforceable by CI so nobody has to remember them.

1. CI fails on two leaves in the migration graph, against a merge with main. makemigrations --check catches models disagreeing with migrations, but the conflict that matters is two branches both adding 0042_*. Check it against the branch merged into main, not the branch alone:

git fetch origin main
git merge --no-commit --no-ff origin/main || { echo "merge conflict"; exit 1; }
python manage.py makemigrations --check --dry-run
python manage.py migrate --plan 2>&1 | grep -q "Conflicting migrations" && exit 1

This is the one I made happen on purpose. Two of the three demo branches, feature/profile-tagline and feature/profile-website, each added one field to the user profile model with one migration, and each environment applied its own on the way up. The new columns were only where they should be:

env_feature_branch_envir_c65cb4      (no new column)
env_feature_profile_tagl_fa8832      tagline
env_feature_profile_webs_dcb2e7      website

Both migrations are 0042_*. Neither branch can see the other’s, so neither environment can tell you there’s a problem. Merging one into the other, the check does:

$ git merge --no-commit --no-ff feature/profile-tagline
$ python manage.py makemigrations --check --dry-run
CommandError: Conflicting migrations detected; multiple leaf nodes in the migration graph:
  (0042_profile_tagline, 0042_profile_website in accounts).
To fix them run 'python manage.py makemigrations --merge'

$ python manage.py makemigrations --merge --noinput
Created new merge migration accounts/migrations/0043_merge_20260919_0324.py
    dependencies = [
        ('accounts', '0042_profile_tagline'),
        ('accounts', '0042_profile_website'),
    ]

The second author runs that --merge on their branch, in their environment, bothering nobody, and the next env up applies 0043 on top of their clone. (In this case git also flagged a textual conflict first, because both fields had been added at the same line of the model file. Realistic, and worth knowing that the textual conflict hides the migration one until it’s resolved.)

2. Migrations are backwards-compatible with the previous release’s code. During a deploy there’s a window where the new schema and the old code coexist, or the reverse if a machine rolls back. Add columns nullable or with defaults, never rename in one step, never drop what the current release still reads: add, deploy, backfill, switch code, drop in a later release. Expand and contract. The reason it matters more here is that disposable environments make deploying more frequent, so you’re inside that window more often. Staging-per-tag is where it’s rehearsed.

3. CI also migrates from zero into an empty database. The template clone tests “does my migration apply to realistic data”. The empty run tests “does the full chain still work from nothing”, which catches the migration that assumed a previous data migration had populated something. Thirty seconds, and it catches a class of bug that otherwise waits for the next new laptop.

4. Schema changes get a second pair of eyes. In a small team this doesn’t need a CODEOWNERS file, but it does need to be a rule: a PR that touches migrations/ is reviewed by a teammate before merge, and the reviewer’s job isn’t to check the SQL. It’s to know that they, or someone else on the team, are touching the same table this fortnight and say so. Past a dozen developers that knowledge stops fitting in one head, and a CODEOWNERS entry on migrations/ earns its place. No tooling replaces this. With the tooling catching everything mechanical, it’s a two-minute job, and it’s the one part of the old shared-environment world worth keeping: the moment where two people find out they’re about to collide.

Versioning, so a URL tells you what it’s running

#

With one shared environment, “what’s on staging” was a chat question. With many, it has to be a property of the environment or every bug report starts with archaeology.

• Every image carries the commit SHA and the app exposes it. fly deploy --build-arg GIT_SHA=... --build-arg GIT_BRANCH=... stamps them into the image as environment variables, and /api/version/ returns them with the environment name and the region (see the output above). The first thing anyone does with “it’s broken on feature-profile-tagl-fa8832” is check it’s running the commit they think. The first demo environment was deployed from the working tree a commit before the tooling was committed, and the endpoint said so: it reported the previous commit, not the SHA I expected.
• ENVIRONMENT=$ENV is in every log line and the error tracker’s environment tag. Branch errors don’t pollute prod’s error tracking and are filterable in one click.
• Branches get SHAs, releases get tags. feature-payments-retr-3f9a1c never has a version number; it has commits. Staging and prod have v1.42.0.
• The mobile app carries a minimum API version, not a URL. The dev-menu URL field is how a tester points a real phone at a branch, and the version endpoint carrying an API schema version is how the app knows whether it’s talking to something it understands.

What this doesn’t solve

#

It doesn’t solve two features that only make sense together. A branch environment shows one branch. The answer is to merge both behind flags and look at main, which you can do with one standing environment that tracks main (main-dev.example.com, redeployed on every merge). That’s the one long-lived non-prod environment I’d keep, and nobody deploys to it; it’s for seeing what the next release will be.

It doesn’t solve providers that want a registered callback and don’t accept wildcards. My auth provider does accept a parent domain, so example.com covers every -dev branch hostname. Some OAuth and payment providers don’t, and for those you keep one shared hostname that the Worker routes to whichever environment last claimed it. That’s a small lock over a small thing, and it’s a lot better than a lock over everything.

It doesn’t solve the human side of migrations. The tooling catches mechanical conflicts. It doesn’t catch two of you modelling the same concept incompatibly in the same fortnight, and the only fix for that is the migration review where one of you notices.

What I’d check in your setup

#

If you have a shared dev or staging and a chat thread for booking it:

1. How long does a finished branch wait before someone other than its author can see it running? More than an hour and you’re paying for the lock in delivery time, and on a small team that’s a big fraction of it waiting.
2. How often does the shared database get reset, and who loses work when it does?
3. Could a new developer, on day one, get a public URL running their branch without asking anyone for anything? Whatever they’d have to ask for is the thing to automate first.
4. Does a fresh environment start from realistic data? If it starts empty, the first bug it won’t catch is the migration that’s fine on an empty table and takes forty minutes on the real one.
5. Is there anything in your non-prod hosting that would still exist a month after everyone forgot about it? If so, write the sweeper before anything else. The leak starts on day one.

The chat question goes away not because people stop asking it but because the answer is always “you are, on yours.”

Until recently, every change to the web side of my marketplace app went out the same way as a change to the Django API: build the Docker image, push it, and roll it across every machine on Fly.io. The web front end (the mobile app’s web export, plus the marketing pages a static site generator builds) was copied into the image next to the API, so there was no other way to get it out. A CSS fix took about seven minutes and restarted eleven machines. One day in August I did that six times.

The rest of this post walks through that deploy script step by step, including how it waits until every machine is serving the new version, and then the Django side that reads the pointer.

Why the old way was a problem

#

The restart was the biggest cost. Eleven machines rolling means eleven cold boots, and on a shared CPU Django takes about thirteen seconds to come up, seven of which is just importing Django. I had a warm-up step at the end of the deploy script that hit every machine so a real visitor wouldn’t be the one to pay for that, which meant every web release ended with waiting on the slowest machine. I’ve written before about what those cold boots cost on Fly.io, in Scale to Zero, and Every Restart Is a 30-Second Outage. The short version is that every restart is a window where a visitor can get a 504, so a deploy path that restarts nothing closes that window for web releases entirely.

The second problem was that the static site’s output was split. The HTML went into the image, but the RSS feeds and the search index went straight to the bucket. They were built from the same source, just not at the same moment, so they could disagree with each other and nobody would notice until they did.

The third one is more about habit. When a release costs seven minutes and a restart, you stop doing small ones. I’d let three or four fixes pile up rather than ship them one at a time, and that’s exactly how a one-line change ends up going out next to something broken.

The layout in the bucket

#

Everything lives in an S3-compatible bucket under a web/ prefix:

web/current                              ← a text file containing one version string
web/a1b2c3d-20260914021500/manifest.json ← every file in this version
web/a1b2c3d-20260914021500/app/...       ← the app's web export
web/a1b2c3d-20260914021500/site/...      ← the static site's output
web/9f8e7d6-20260912093012/...           ← the previous version, still there

A version is the short git SHA plus a UTC timestamp, so you can read off what commit it is and when it went up. web/current is the only thing that changes when a release goes live.

What the deploy script does

#

The whole thing is one script, run from the repo root. In order:

1. Refuse a dirty tree. The first thing it does is git status --porcelain, and if anything comes back it exits. Whatever is on disk is what gets uploaded, and the old deploy script for the marketing site once shipped uncommitted copy that way. There’s an override flag for when you know what you’re doing, but I haven’t used it. It then computes the version string from git rev-parse --short HEAD and the current time.

2. Build both trees. For the app it runs the web export with the production API and WebSocket URLs baked in, followed by a Brotli precompression pass. The precompression is worth explaining: Cloudflare will Brotli-compress on the fly, but at a low quality level, and on this bundle it measured worse than gzip (2.04MB against 1.66MB). Compressing once at build time with quality 11 gets it to 1.28MB, so the script writes a .br twin next to every JS, CSS and JSON file over 1KB. For the marketing site it runs the static site generator with minification on.

3. Upload. It walks both build directories and uploads every file to web/<version>/app/... and web/<version>/site/... with sixteen threads. Each object gets a Content-Type guessed from its filename (with the .br stripped first), and the Brotli twins get Content-Encoding: br as well. Only when every file is up does it write manifest.json, a JSON object with two lists of relative paths. The order matters: the Django side treats a version without a manifest as not existing, so a half-uploaded version can never be made current. Right now that’s a few hundred files and the upload takes about six seconds.

4. Flip the pointer. One put_object writes the version string to web/current. That’s the release.

5. Wait for the machines. Each API machine caches the pointer for five seconds, so for a moment after the flip some of them are still on the old version. The script polls a version endpoint on the API every half second and counts consecutive answers that carry the new version. Fly load-balances across all the machines, so a run of sixteen matching answers in a row (more than the machine count) is how it knows every one of them has re-read the pointer. One stale answer resets the count. It gives up after 90 seconds and tells you to check by hand; the first real flip took a bit over 30 seconds, which is why the limit isn’t lower.

6. Purge Cloudflare. One API call, covered below. If it fails, the script prints a warning and carries on.

7. Post to the team chat. The same webhook the runtime error alerts use gets a one-liner with the version and who ran it, so a burst of errors can be read against what just changed. Rollbacks post in amber.

Start to finish, a release is a bit over a minute, most of which is the web export.

The Django side

#

Django was already the thing serving the front end. Every non-API URL is answered with the prerendered page from the web export for that route, and the marketing, privacy and support pages come from the static site’s output. Those views used to open files on disk. Now they read through one small layer that knows about the bucket.

Per request it resolves three things: the current version (from web/current, cached for five seconds), the manifest for that version (cached for as long as that version is current), and the file itself (a small in-memory LRU of file bodies, so a page’s HTML and its main bundle stay in memory). The manifest is what makes 404s cheap: if a path isn’t listed, the answer is no without asking the bucket. If the client accepts Brotli, it reads the .br twin instead and sets the encoding header. Assets go out with a one-year immutable cache header, since their filenames are content-hashed. Anything derived from the file list, such as the table that maps routes to prerendered pages, is cached per version too, so a new deploy rebuilds it on the first request and not before.

Then there’s the endpoint the deploy script polls:

def web_version(request):
    # bundle: the layer that reads web/current and the manifest
    response = JsonResponse({"source": bundle.source(), "version": bundle.version()})
    response["Cache-Control"] = "no-store"
    return response

It answers with bucket:<version> in production or filesystem when it’s serving from the image. Each Fly machine answers for itself, so hitting it a dozen times tells you whether the fleet has converged. It’s also the quickest way to check what a machine thinks is live when something looks off.

Purging the Cloudflare cache through the API

#

HTML on the site is cached at Cloudflare’s edge for five minutes. Without a purge, a deploy would be live on the machines but most visitors would keep getting the old page for up to five minutes, which defeats the purpose.

The purge is a single API call:

urllib.request.Request(
    f"https://api.cloudflare.com/client/v4/zones/{zone}/purge_cache",
    data=json.dumps({"hosts": ["example.com", "www.example.com"]}).encode(),
    headers={"Authorization": f"Bearer {token}", "Content-Type": "application/json"},
)

I purge by host rather than by URL so I don’t have to list every page. The assets don’t need purging at all: the filenames are content-hashed, so a new build points at new filenames and the old ones just stop being referenced. Only the HTML that references them needs to be refreshed.

The script waits for the machines before purging, on purpose. If it purged first, Cloudflare could refill its cache from a machine that was still on the old version, and that page would then sit at the edge for another five minutes.

If the purge fails, the script prints a warning and finishes anyway. The cache expires on its own within five minutes, so the worst case is a slow release rather than a broken one. I’d rather that than have the deploy abort at the last step.

Checking a build before it goes live

#

A no-flip mode does everything up to step 3 and stops. The version is in the bucket but nothing serves it. To look at it, I set an environment variable on one machine that pins it to that build regardless of what web/current says. When it looks right, the rollback command makes it current. It’s called rollback but it’s really “point current at this version”, and going forward and going back are the same operation.

Rolling back

#

Old versions stay in the bucket, so rolling back is the same command with an earlier version, and a list command shows what’s there, newest first. It does the same wait and purge as a forward deploy. I haven’t needed it yet. Nothing prunes old versions at the moment; they’re small and I’d rather have them than not.

What stayed the same

#

The API still deploys the old way, through its own deploy script, and it should: it’s stateful and it’s where the migrations run. The Dockerfile still copies both front-end builds into the image too. That’s deliberate. When there’s no bucket configured (local development, the test suite) or the bucket can’t be read, the serving layer falls back to those directories on disk, so a storage outage means visitors get the front end that shipped with the last image rather than an error page. One environment variable forces that behaviour in production if the bucket ever needs to be taken out of the loop.

The front end changes far more often than the API does, and each release now takes about a minute instead of seven. I’ve stopped batching without trying to.

Update, 4 November 2026: one more correction. Those 504s aren’t probes the origin fails to answer. They’re Cloudflare’s own Early Hints cache lookups, which never reach the origin at all, and a 504 there only means nothing was cached for that URL. Turning Early Hints off wasn’t the best fix either: sending a Link response header made the feature work and turned those misses into hits. The full story is in Cloudflare 504 errors that aren’t real: Early Hints and HTTP 103.

With auto_stop_machines on, any request that lands on a stopped Fly.io machine waits for a full boot, and mine took 25 to 33 seconds because the entrypoint ran Django migrations and collectstatic on every machine start, and nothing had ever cached the bytecode. Running that release work once per image (a Redis lock keyed on FLY_IMAGE_REF) and adding compileall to the build got a cold boot down to about nine seconds. Switching auto_stop_machines from 'stop' to 'suspend', which meant dropping a swap file Fly won’t suspend with, got wake-ups to about three.

I went looking for a traffic spike on my Cloudflare dashboard and found something much worse underneath it: 22% of every request my app served in a day had failed. No alert had fired. Every health check was green. The logs looked perfectly healthy, because the failing requests never reached the application at all.

The cause turned out to be my own container startup script, doing sensible work at exactly the wrong time.

The spike was a red herring

#

The thing I set out to explain was one hour that took about 4,900 requests against a median of 121. Forty-two unique visitors. That is 110 requests per visitor, which is not how humans browse.

It was a scanner. A single bot walking through invented subdomains, asking each one for secrets:

api-stage-asia.example.com/.env.local
api-us-east-1-demo.example.com/.env
api-eu-uat.example.com/wp-config.php~
app-test-us-east-1.example.com/actuator/configprops
prod-api-use2.example.com/.env.local
relay.example.com/.env.production

Thirty or so hostnames, every .env variant you can think of, plus a Java framework’s actuator endpoints on a Django app. All of them got a 301 to my canonical host and nothing was served. Boring, and genuinely harmless.

But while I had the analytics open, I grouped the same day by status code. That is where the real problem was.

Twenty-two percent. And not spread evenly across the site, but concentrated exactly where it hurts:

Those are the endpoints my mobile app polls every 30 seconds. So this was not an abstract error-rate number. It was every user’s phone, quietly failing to load their unread badge, all day, for weeks probably.

Why the logs looked fine

#

My first instinct was that the app had become slow. It had not. I pulled the application’s own request lines for one of the worst hours out of the log archive:

[2026-09-08 01:00:29] "GET /api/notifications/unread/" 304 61.615
[2026-09-08 01:00:29] "GET /api/notifications/"        304 63.568
[2026-09-08 01:00:59] "GET /api/notifications/unread/" 304 58.742
[2026-09-08 01:00:59] "GET /api/notifications/"        304 105.438

Forty-four to a hundred milliseconds, 304s all the way down. The origin was fast.

That is the important clue, and it is worth stating as a rule: if your edge reports errors your origin has never heard of, the requests are dying before they arrive. Stop reading application logs and start looking at what sits in front of them.

In my case that is Fly.io’s proxy, and the answer was in how my machines are scheduled.

Scale to zero is a promise about money, not latency

#

My config looks like this, and on the face of it it is reasonable:

[http_service]
  auto_stop_machines = 'stop'
  auto_start_machines = true
  min_machines_running = 1

Idle machines stop. Traffic starts them again. You pay for what you use. For a small app with bursty traffic this is exactly what you want, and it is why I had ten machines defined but almost always one running.

The bill it hides is latency. When a request arrives for a stopped machine, somebody has to wait for that machine to boot. If booting takes longer than the proxy’s patience, that somebody gets a 504.

So: how long did my machines take to boot? I had never measured. The log archive knows, because the startup script announces each phase, and every line carries the machine id:

WITH s AS (
  SELECT instance, timestamp,
         CASE WHEN message ILIKE '%Starting app server%' THEN 'begin'
              WHEN message ILIKE '%Migrations complete%' THEN 'migrated'
              WHEN message LIKE  '%"GET /healthz%'       THEN 'serving' END AS phase
  FROM logs
  WHERE app = 'myapp' AND timestamp::date = '2026-09-08'
)
SELECT instance,
       min(timestamp) FILTER (WHERE phase = 'begin')    AS started,
       min(timestamp) FILTER (WHERE phase = 'migrated') AS migrated,
       min(timestamp) FILTER (WHERE phase = 'serving')  AS first_serve
FROM s GROUP BY instance

The answer, consistently, across every machine:

Then I counted how often that happened:

SELECT count(*) AS starts, count(DISTINCT instance) AS machines
FROM logs
WHERE app = 'myapp' AND timestamp::date = '2026-09-08'
  AND message ILIKE '%Starting app server%'
-- starts: 92, machines: 10

Ninety-two cold starts in one day. Each one a half-minute window in which anything routed to that machine timed out. Twenty-two percent stopped looking mysterious.

The cause was in my own entrypoint

#

Here is what every single machine ran before it would answer a request:

# entrypoint.sh, run on every machine start
echo "Running migrations..."
python manage.py migrate --noinput

echo "Running collectstatic in background..."
python manage.py collectstatic --noinput &

exec "$@"    # then start the app server

Read that again with “92 times a day” in mind.

migrate is a full Django boot: import every model, connect to Postgres, query the migrations table, decide there is nothing to do. On a shared CPU that is ten seconds to accomplish nothing. collectstatic is a second full Django boot that then uploads static files to object storage, competing for the same single core that the web server is trying to start on. The app server is the third boot.

Three Python interpreters, two of them doing work that had already been done, every time a machine woke up.

The reason it was written that way is good, and it is probably why yours looks similar: migrations have to run somewhere, Fly’s release_command had hung on me before, and the entrypoint is the one place guaranteed to execute on deploy. It is a correct place to put deploy work. It is just not a correct place to put it per machine start, and once your machines stop and start on their own, those are no longer the same event.

That is the whole bug, and I think it is a common one:

Work that belongs to a release was being run once per machine start. Scale-to-zero turned one event into ninety-two.

The fix: claim the release once

#

Migrations do not need to run on the eighth machine to boot a given image. They need to run once for that image, and everyone else needs to know it has happened.

That is a lock with a name that changes per deploy. Fly hands you the name for free in FLY_IMAGE_REF, and I already had Redis. The whole thing is a small script that runs before Django exists, so the common answer costs milliseconds rather than a framework boot:

# release_gate.py: runs before Django exists. Exit code 0 means "prepare this release".
import os, time

PREPARE, SKIP = 0, 1
LOCK_TTL, WAIT, POLL = 300, 240, 2
_sleep, _monotonic = time.sleep, time.monotonic    # aliases, so tests can replace them


def release_id() -> str:
    """Something that changes exactly when the deployed image does."""
    return os.getenv("FLY_IMAGE_REF") or os.getenv("FLY_MACHINE_VERSION") or ""


def claim(client, release: str) -> int:
    if client is None or not release:   # no Redis, or no idea which release: behave as before
        return PREPARE
    done_key, lock_key = f"release:{release}:done", f"release:{release}:lock"
    if client.get(done_key):            # somebody already did it for this image
        return SKIP

    owner = os.getenv("FLY_MACHINE_ID", "unknown")
    if client.set(lock_key, owner, nx=True, ex=LOCK_TTL):
        return PREPARE                  # we won the race; we do the work

    # Someone else is preparing. Wait for them, because serving requests
    # against a half-migrated schema is worse than starting slowly.
    deadline = _monotonic() + WAIT
    while _monotonic() < deadline:
        _sleep(POLL)
        if client.get(done_key):
            return SKIP
        if client.set(lock_key, owner, nx=True, ex=LOCK_TTL):
            return PREPARE              # the owner died and its lock expired; take over
    return PREPARE                      # waited long enough; do it ourselves

# ... plus `done` (sets done_key) and a main() that treats any Redis error as PREPARE

The entrypoint becomes an if:

if python3 release_gate.py claim; then
    python manage.py migrate --noinput
    python manage.py collectstatic --noinput &
    python3 release_gate.py done
fi

exec "$@"

Every failure has to end in “prepare it anyway”

#

This is the part worth copying, more than the code.

A gate like this sits between your users and a schema migration. If it is wrong in the direction of skipping, you serve requests against a database that has not been migrated, which is a genuinely bad outage. If it is wrong in the direction of preparing, you run an idempotent no-op migration twice and waste ten seconds.

Those outcomes are not symmetrical, so the code must not treat them as such. Every uncertain path returns PREPARE:

• No Redis, or Redis unreachable. Prepare. This is exactly the old behaviour, so a cache outage degrades to “slow”, not “broken”.
• No image reference in the environment. Prepare. I do not know what release this is, so I cannot know whether it is ready.
• The lock’s owner died mid-migration. Its lock expires, the next waiter takes it over and prepares.
• Waited longer than the timeout. Prepare anyway. A machine that never serves is worse than a duplicated migration.

There is one deliberate asymmetry in the other direction: a machine that finds another machine actively holding the lock waits rather than serving. Starting slowly during a deploy is fine. Answering queries against a half-applied schema is not.

Watch out for one trap if you build this on a cache wrapper that swallows connection errors, as I first did. A wrapper like that returns False from add() when Redis is unreachable, which reads identically to “someone else holds the lock” and would have made every machine skip its migrations. That is precisely the dangerous direction. Distinguish the two by reading the lock back: a lock nobody holds means the cache is broken, not that you lost a race.

Testing it without a Redis

#

The logic is worth unit testing, because the interesting paths are the ones you cannot reproduce by hand. A fake with two methods covers it:

class FakeRedis:
    """Enough of a Redis client for the gate: get, and set with nx/ex."""

    def __init__(self) -> None:
        self.store: dict[str, str] = {}

    def get(self, key): return self.store.get(key)

    def set(self, key, value, nx=False, ex=None):
        if nx and key in self.store:
            return None
        self.store[key] = value
        return True

Then drive the awkward cases by making time pass on your terms. This is why the gate calls module-level _sleep and _monotonic aliases rather than time.sleep directly, so a test can replace them without patching the time module for every other thread in the process:

def test_a_waiter_takes_over_when_the_owner_disappears(self):
    redis = FakeRedis()
    self.assertEqual(release_gate.claim(redis, "img-1"), release_gate.PREPARE)   # the owner takes the lock

    def owner_dies(_seconds):
        redis.store.pop("release:img-1:lock", None)                              # its lock expired

    with patch.object(release_gate, "_sleep", owner_dies):
        self.assertEqual(release_gate.claim(redis, "img-1"), release_gate.PREPARE)

Did it work? Partly

#

Deploy, stop a machine, start it again, watch the clock. The startup now announces its decision and gets out of the way:

15:23:34  Starting app server...
15:23:36  release gate: this release is already prepared, starting straight away
15:23:44  "GET /healthz HTTP/1.1" 200

Ten seconds, and a second machine measured eleven. Against 25 to 33 before, two thirds of the window is gone, and the gate itself costs two seconds because it never imports Django.

But ten seconds is still ten seconds. A request landing in the first of them still fails. So I went looking for the rest.

The bytecode nobody was caching

#

The image sets a line that appears in more or less every Python Dockerfile ever written:

ENV PYTHONDONTWRITEBYTECODE=1

It is good advice. A container should not be scribbling .pyc files into a layer at runtime. What I had never thought through is the other half of the bargain: if the runtime never writes bytecode, and the build never writes it either, then nothing is ever cached, and every process start compiles the entire dependency tree from source.

I counted:

site-packages .py  files : 5,707
site-packages .pyc files : 0

Django, the REST framework, every library, recompiled on each of those 92 daily starts. Measured on a production machine:

The fix is one line, and compileall writes explicitly so it works despite the env var:

RUN python -m compileall -q /app/.venv/lib /app/myapp || true

Sixteen seconds of build time, once, in a cached layer.

While I was there I found something quieter. .dockerignore said:

__pycache__/
*.pyc

Docker matches those against the whole relative path, not each path component, so an unanchored pattern only ever matches at the root of the build context. Every nested __pycache__ was shipping. The image contained 218 .pyc files compiled by Python 3.15 on a laptop, riding along inside an image whose interpreter is 3.14 and quietly ignoring all of them. The patterns want to be **/__pycache__/ and **/*.pyc.

Honest result: end to end this bought about a second, not two. Ten seconds became nine. The isolated benchmark oversold it, as isolated benchmarks do.

Suspend, and the one-line setting that was blocking it

#

At nine seconds I had run out of work to remove. What was left was one unavoidable boot: Python starting, Django importing, the WSGI app coming up on a shared core.

So stop booting. Fly can snapshot a machine’s memory instead of shutting it down:

[http_service]
  auto_stop_machines = 'suspend'   # was 'stop'

I went to test it and Fly refused, with the most useful error message of the whole exercise:

failed to suspend VM: failed_precondition:
Machines with swap cannot be suspended

My config had this near the top, with a comment I had written myself and believed:

# Swap cushion so a memory spike swaps to disk instead of getting OOM-killed.
swap_size_mb = 2048

A reasonable precaution. Was it doing anything? Every machine said:

MemTotal:   985220 kB
SwapTotal: 2097148 kB
SwapFree:  2097148 kB

Not one page had ever been swapped. And the day’s logs held no OOM kills — the eleven lines that matched “oom” turned out to be requests for a JavaScript bundle whose content hash happens to contain those three letters. Meanwhile the actual memory guard was somewhere else entirely, in the app server’s own settings, which respawn any worker past 800MB long before it can reach the 1GB ceiling.

So the cushion was insurance against an event that had never happened, already covered by a different mechanism, and its premium was the ability to suspend. Out it went.

Three trials, all within half a second. Every endpoint answered 200 in 250 to 435 milliseconds immediately after waking.

What happens to your sockets while the machine sleeps

#

Nothing good, and there is no hook to fix it. Fly freezes the VM; the process is not signalled and cannot close anything on the way down. Whatever connections it held are still in its memory when it wakes, pointing at sockets the other end abandoned minutes ago.

You do not solve that going down. You solve it coming up, and mostly by having already made the right decisions:

• Database connections are opened per request (conn_max_age = 0), so there is nothing long-lived to go stale.
• The Redis cache degrades instead of raising. A wrapper catches connection errors and returns a miss.

I got to watch the second one earn its place. One resume logged exactly the failure you would expect:

redis.exceptions.ConnectionError: Error while reading from fly-...

and the request it happened inside returned 200 in 3.6 seconds, because a dead cache is a slow page, not an error page. Later resumes logged nothing at all.

If your cache client raises on a dead connection, suspend will convert every wake into a burst of 500s. Check that before you flip the setting, not after.

The test that lied to me

#

My first suspend measurement said the feature was a disaster: 502, after 30 seconds, twice.

I had pinned the request to a specific machine with fly-force-instance-id, which is a lovely header for load testing one instance and a terrible one for this. Forcing an instance bypasses the proxy logic that would have woken it. Fly told me so, if I had read it as an answer rather than an error:

machine was recently stopped and is unavailable to service request

The number that matters comes from the path real traffic takes. Forty concurrent requests through the edge, well past the soft limit of 12, so the proxy has to scale out onto suspended machines:

status codes:  40 × 200
slowest:       2.1s

Two lessons, and the second one is the expensive one. Measure the path your users take, not the path that is convenient to instrument. And when your tooling refuses to do something, read the refusal: “Machines with swap cannot be suspended” was the sentence that unlocked this entire afternoon.

What I would check in your app

#

If you run anything on a platform that stops idle instances, these questions took me an afternoon and would have saved me weeks of quiet failure:

1. How long does a cold start take? Not how long the container takes to run, how long until it answers a real request. If you cannot answer in seconds, measure it before you need to.
2. What does your entrypoint do per start that belongs to a deploy? Migrations, static file collection, cache warming, index building. All fine once. All expensive ninety-two times.
3. Is anything caching your bytecode? PYTHONDONTWRITEBYTECODE with no compileall in the build means the answer is no.
4. Can you suspend instead of stop? And if not, what is stopping you? Mine was a swap file that had never been touched.
5. Does your edge see errors your origin does not? Compare the two. A gap between them is not a reporting quirk, it is requests dying in the space between, and your application logs will never show it.

The health checks were green the whole time, incidentally. They always are: a health check runs against a machine that is already up. It cannot report on the thirty seconds before it existed.