This post is the other order. The service keeps running on the machine it was built on, and a Cloudflare Tunnel gives it a public HTTPS hostname. Nothing on the laptop listens to the internet: no port forwarding, no router settings, no exposed IP address. The machine dials out to Cloudflare, and Cloudflare sends requests back down that connection. The tunnel is free, and the TLS certificate, DDoS protection and caching come with the Cloudflare zone you probably already have.

The example is real. One of my products runs on Fly.io, and its mobile app gets over-the-air JavaScript updates from the backend (self-hosted Expo Updates). I wanted a second copy of that update server running on my Mac for development builds, with production left exactly as it was. The outputs below are from that afternoon, including the part where it didn’t work.

What a Cloudflare Tunnel actually is

#

A normal server waits for connections. To reach a laptop from the internet, you’d have to open a port on your router, forward it to the laptop, and hope your ISP gives you a public IPv4 address (in a lot of places, mine included, it doesn’t). Then everything listening on that port is reachable by every scanner on the internet, all day.

A tunnel turns that around. cloudflared runs next to your app and opens outbound connections to Cloudflare’s edge, four of them to two nearby data centres. A DNS record in your zone points a hostname at the tunnel. When a request arrives for that hostname, Cloudflare terminates TLS and forwards the request down one of those connections. cloudflared then hands it to whichever local service your config names.

For security, the important part is that your router has no inbound rule and your IP address never appears in DNS. Someone scanning your home connection finds nothing, because nothing on it is listening.

Setting up cloudflared, start to finish

#

You need a domain on Cloudflare (the free plan is enough) and cloudflared installed (brew install cloudflared on a Mac).

1. Log in once. This opens a browser, you pick the zone, and it saves a certificate that lets this machine create tunnels and DNS records in that zone:

cloudflared tunnel login

2. Create the tunnel. I keep everything for this project in its own folder, because the credentials file is the tunnel’s identity: anyone who has it can serve traffic for your hostnames.

mkdir -p ~/.cloudflared/myapp && cd ~/.cloudflared/myapp
cloudflared tunnel create --credentials-file ./creds.json myapp-local
chmod 600 creds.json

Tunnel credentials written to ./creds.json. Keep this file secret. To revoke these credentials, delete the tunnel.
Created tunnel myapp-local with id 1ade030c-…

3. Point hostnames at it. One command per hostname creates a proxied CNAME to <tunnel-id>.cfargotunnel.com:

cloudflared tunnel route dns myapp-local ota.example.com
cloudflared tunnel route dns myapp-local checkin-local.example.com

4. Write the ingress rules. This is the part that matters for security, so it gets its own section below.

5. Run it. My app already runs under Docker Compose, so cloudflared is one more container on the same network. It reaches the app by service name, and the app doesn’t need a port published on the host at all:

# compose.tunnel.yml, used alongside the project's docker-compose.yml
services:
  cloudflared:
    image: cloudflare/cloudflared:2026.10.7
    command: tunnel --config /etc/cloudflared/config.yml run
    volumes:
      - ${HOME}/.cloudflared/myapp:/etc/cloudflared:ro
    restart: unless-stopped
    depends_on: [api, edge]

docker compose -f docker-compose.yml -f ~/.cloudflared/myapp/compose.tunnel.yml up -d cloudflared

INF Registered tunnel connection connIndex=0 … location=icn06 protocol=quic
INF Registered tunnel connection connIndex=1 … location=icn07 protocol=quic
INF Registered tunnel connection connIndex=2 … location=icn07 protocol=quic
INF Registered tunnel connection connIndex=3 … location=icn06 protocol=quic

Four connections to Seoul, and the laptop is on the internet, from the outside in. restart: unless-stopped brings it back after a reboot as soon as Docker starts.

Expose paths, not servers

#

This is the mistake I nearly made, and I’d guess it’s the most common one. The easy config is a single line: hostname → http://localhost:8000. That publishes the whole development server.

Development servers aren’t built for that. Mine runs with DEBUG=True, which shows full tracebacks and settings on any error. It also runs with authentication switched off (AUTH_MODE=disabled), so that any bearer token logs you in as a test user. That’s exactly right on localhost and a disaster on a public hostname. Yours probably has something similar: an admin with a default password, a seeding endpoint, a debug toolbar, or an API that trusts its caller because only you ever call it.

So the tunnel config is an allowlist. cloudflared checks each request against the ingress rules in order, by hostname and by a regular expression on the path. The first match wins, and the last rule catches everything else:

tunnel: myapp-local
credentials-file: /etc/cloudflared/creds.json

ingress:
  # OTA: the two read-only endpoints the phone calls, and the files they link.
  - hostname: ota.example.com
    path: ^/api/expo-updates/(manifest|status)/$
    service: http://api:8000
  - hostname: ota.example.com
    path: ^/media/
    service: http://api:8000
  # Device check-ins, authenticated by each device's own key.
  - hostname: checkin-local.example.com
    path: ^/c/$
    service: http://edge:8081
  # Everything else, including /admin/ and the publish endpoint: 404 at the edge.
  - service: http_status:404

The publishing endpoint lives on the same server under the same prefix (/api/expo-updates/import-update/), and it isn’t in the list. Neither is the admin, or the rest of the API. Those requests never leave Cloudflare.

You don’t have to trust your regexes; cloudflared will tell you which rule a URL matches:

cloudflared tunnel --config config.yml ingress rule https://ota.example.com/admin/

https://ota.example.com/api/expo-updates/manifest/        service: http://api:8000
https://ota.example.com/api/expo-updates/import-update/   service: http_status:404
https://ota.example.com/admin/                            service: http_status:404
https://ota.example.com/media/expo-updates/x.js           service: http://api:8000
https://checkin-local.example.com/c/                      service: http://edge:8081
https://checkin-local.example.com/health                  service: http_status:404

I’d put that check in a script and run it every time the config changes. It takes seconds, and a regex that matches more than you meant is the kind of mistake you only find out about from someone else.

Two more rules of thumb:

• What you expose should authenticate itself. The OTA endpoints serve public, read-only data. Device check-ins are signed with per-device keys. Neither one depends on the network to keep it safe. If you want to show someone a page that has no login, put Cloudflare Access in front of that hostname (free for up to 50 users) instead of building a login.
• Don’t run the tunnel as a background part of your dev workflow. Run it on purpose, for the hostname you mean, and stop it when you’re done. docker compose stop cloudflared takes everything off the internet in one command.

The redirect that ate my hostnames

#

With the tunnel connected and the rules verified, I tried it from outside:

https://ota.example.com/api/expo-updates/status/?…   301
https://ota.example.com/api/expo-updates/manifest/   301
https://ota.example.com/admin/                       301
https://checkin-local.example.com/c/                 301

Every path returned the same 301, including ones the tunnel should have answered with 404. That means the request never reached cloudflared. The Location header said https://www.example.com/, and the server: cloudflare header settled it: Cloudflare was answering on its own.

I’d done this to myself. Months ago I added a zone-level redirect rule, “any other subdomain → www”, as a sinkhole for scanners that walk invented subdomains. Redirect rules run in Cloudflare’s request pipeline before the request is sent to the origin, and a tunnel is just an origin. The same rule caught my preview environments before Workers could see them. The fix is the same as last time: one more exception in the rule’s expression.

… and not http.host in {"ota.example.com" "checkin-local.example.com"}

If your tunnel hostname answers with a redirect or an error page from Cloudflare instead of your app, check Rules → Redirect Rules, Page Rules and any WAF custom rules for the zone before you debug the tunnel. cloudflared logs every request it forwards, so an empty log means the request never got that far.

What it can’t do

#

• UDP from the public internet. Public tunnel hostnames carry HTTP(S) (plus WebSockets and gRPC). My product’s devices normally check in over CoAP on UDP, and that can’t go through a tunnel; the devices on the local copy fall back to HTTPS check-ins, which cost them a few seconds on each one. If your idea is a game server or VoIP, a tunnel isn’t the tool.
• Uptime. It’s your laptop. Close the lid and the site is down; Cloudflare shows a 530 error page. That’s fine for a demo, a beta with ten people, or a staging copy. It isn’t fine for anything someone depends on.
• Things baked into a build. A mobile app’s update URL is native config, so only builds made with the tunnel’s URL use it. My store builds keep talking to production, and that’s why this setup was safe to try.

The cost: is Cloudflare Tunnel free?

#

A Cloudflare zone on the free plan, the tunnel, its certificate, and the DNS records: $0. The machine was already running. The only real cost is that you’re now the uptime team.

What it saves you is the phase between “it works on my machine” and “it’s worth paying to host”. You can send a link to a tester, a designer or a first customer the same afternoon. If nobody cares, you’ve lost nothing. If they do, moving to a real host is one change: point the CNAME somewhere else.

The fast way is a presigned URL. The API signs a URL, the client PUTs the file straight to the bucket, and the bytes never touch the API at all. It’s the standard answer for good reason. But the moment you do it you’ve cut your backend out of the loop. It handed out a URL and then heard nothing. Did the file arrive? Is it the size you expected? Is it even a JPEG? Something has to tell the backend, and the usual options are all a bit sad: trust the client to call back and say “done” (clients lie, crash and lose signal), poll the bucket on a schedule (slow and wasteful), or process lazily on the first read (the first viewer pays, and you still don’t know when the file is safe to reference). This is what makes uploading directly to S3 look harder than it is. The upload is easy. The signalling is what’s missing.

Object notifications are that signal. Tigris, the S3-compatible storage I use on Fly.io, will POST a small JSON body to your app every time an object is created or deleted in a bucket. On AWS the equivalent is S3 event notifications, which usually means wiring up a Lambda, an SQS queue or EventBridge before anything reaches your code (SNS can POST to an HTTPS endpoint, but then you’re confirming subscriptions and verifying signatures first). Tigris keeps it to one HTTP request to a URL you own, which is about as simple as storage-to-backend signalling gets. The feature exists so the bucket can talk to your backend without you building a bridge for it.

This post is the whole flow: the setup, a Django endpoint that receives the events, a presigned upload that keeps every bit of server-side processing, and the gotchas (CORS, size limits, expiry, multiple files). At the end, the same signal doing back-office work, where log files landing in the bucket became Discord alerts and let me delete a cron job I’d been nursing for months.

The short answer

#

In the Tigris dashboard, open the bucket, then Settings, then Object notifications. Give it an HTTPS URL on your app and pick basic auth (a username and password Tigris sends in the Authorization header; a bearer token is the other option). Add a filter so you only hear about the keys you care about. The filter is a SQL-like expression, so “only the three prefixes this post handles” is:

WHERE `key` REGEXP "^(uploads/raw|logs/api|media)/"

Tigris then sends a batch like this on every change:

{
  "events": [
    {
      "eventName": "OBJECT_CREATED_PUT",
      "eventTime": "2026-11-07T09:12:44Z",
      "bucket": "example-bucket",
      "object": { "key": "uploads/raw/1f3c.jpg", "size": 48213, "eTag": "9b2c…" }
    }
  ]
}

The event names you’ll meet are OBJECT_CREATED_PUT, OBJECT_CREATED_MULTIPART, OBJECT_CREATED_COPY and OBJECT_DELETED, plus rename and storage-tier transitions. Your endpoint checks the auth, routes each event by key prefix, and returns a 2xx within ten seconds. Three things to plan for from the start: delivery is at least once, so you will see repeats; events are batched, so one request can carry several; and they can arrive out of order, so don’t read “latest” off the order they came in.

A Django webhook for the bucket’s events

#

The view is boring on purpose. It checks the auth, parses the body and hands each event to a router:

import base64, hmac, json, logging
from django.conf import settings
from django.http import JsonResponse
from django.views.decorators.csrf import csrf_exempt
from django.views.decorators.http import require_POST

from .bucket_events import handle, parse

logger = logging.getLogger(__name__)

def _authorised(request, secret) -> bool:
    header = request.headers.get("Authorization", "")
    if not header.startswith("Basic "):
        return False
    try:
        user, _, password = base64.b64decode(header[6:]).decode().partition(":")
    except (ValueError, UnicodeDecodeError):
        return False
    return hmac.compare_digest(user.encode(), b"tigris") and hmac.compare_digest(
        password.encode(), secret.encode()
    )

@csrf_exempt
@require_POST
def bucket_webhook(request):
    secret = settings.BUCKET_WEBHOOK_SECRET
    if not secret:
        return JsonResponse({"error": "not configured"}, status=503)
    if not _authorised(request, secret):
        return JsonResponse({"error": "unauthorised"}, status=401)
    try:
        body = json.loads(request.body or b"{}")
    except ValueError:
        return JsonResponse({"error": "invalid json"}, status=400)
    for event in parse(body):
        try:
            handle(event)
        except Exception:
            logger.exception("bucket event failed: %s", event.key)
    return JsonResponse({"ok": True})

compare_digest is there so the check takes the same time whatever you send it (on bytes, because on strings it raises for non-ASCII input, and a crafted header would get a 500 instead of a 401). If the secret isn’t set, say on a fresh staging box, the endpoint refuses everything rather than accepting everything. And a handler that throws is logged and still answered 200: Tigris retries anything that isn’t a 2xx, a retry would hit the same bug, and by then the event has been claimed (two sections down) so the retry would do nothing anyway. The log line is what gets it fixed. That assumes the failure is deterministic. A transient storage timeout consumes the event for good: the photos don’t mind, because the save can process them itself, but a log file’s alert would be lost. If that matters to you, release the claim in the except and answer 500 so Tigris retries, and accept that a real bug then retries until it’s fixed.

Routing by prefix

#

Everything in the bucket is already organised by prefix, so that’s the natural way to route. Each use is one function and one line in a table:

HANDLERS = [
    ("uploads/raw/",   on_raw_photo),
    ("logs/api/",      on_api_log_file),
    ("media/",         on_media_change),
]

def handle(event):
    for prefix, fn in HANDLERS:
        if event.key.startswith(prefix):
            if claim(event):  # see below
                fn(event)
            return

Adding a fourth thing later is a new function and a new row. I like that more than I expected to.

At least once means “maybe twice”

#

Tigris retries if your endpoint is slow or returns an error, and like most webhook senders it makes no promise that you’ll see an event only once. So before a handler runs, the event is claimed by (key, eTag, eventName). The eTag matters: if a file is overwritten, it’s a new version with a new eTag and should be handled again. A repeat of the same version shouldn’t.

from django.core.cache import cache

def claim(event, ttl=7 * 24 * 3600) -> bool:
    k = f"bucket-event:{event.name}:{event.key}:{event.etag}"
    return cache.add(k, 1, ttl)  # False if someone already claimed it

cache.add only sets the key if it isn’t there already, which in Redis is atomic, so two machines receiving the same retry can’t both win. I use a database-backed lease for this in the real thing because I wanted it to survive a Redis flush, but the idea is identical.

Direct-to-S3 image uploads, with the pipeline intact

#

This is the part that pays for the plumbing.

Previously, picking photos for a listing meant the app held them, and when you hit Post it sent them all to the API as base64. The API resized them, made thumbnails, worked out colours and a blurhash, and only then replied. Ten photos on a slow connection was a long spinner. And a request carrying ten base64 photos is a big, slow body for an API server to hold in memory, which was quietly why I kept photos small.

Now it’s four steps, and the backend is told about every one of them.

1. The client shrinks the photo and asks for a slot

#

The app still resizes on the device, to the longest edge the server keeps (1080px) as a JPEG at 0.7 quality. That resize is for the user’s bandwidth, nothing else. Then it asks the API for an upload slot:

import uuid
from datetime import timedelta
import boto3
from django.utils import timezone

s3 = boto3.client("s3", endpoint_url="https://fly.storage.tigris.dev")
BUCKET = "example-bucket"

def create_upload_slot(user):
    public_id = uuid.uuid4()
    key = f"uploads/raw/{public_id}.jpg"
    upload = PhotoUpload.objects.create(
        user=user,
        public_id=public_id,
        raw_key=key,
        expires_at=timezone.now() + timedelta(days=2),
    )
    url = s3.generate_presigned_url(
        "put_object",
        Params={"Bucket": BUCKET, "Key": key, "ContentType": "image/jpeg"},
        ExpiresIn=15 * 60,
    )
    return {
        "upload_id": str(upload.public_id),
        "url": url,
        "method": "PUT",
        "headers": {"Content-Type": "image/jpeg"},
        "expires_in": 15 * 60,
    }

The slot is a row before it’s a file. That’s what lets the server know who asked for it, when it expires, and later whether it was ever used.

2. The client PUTs straight to the bucket

#

const slot = await api.post("/photo-uploads/");
const res = await fetch(slot.url, {
  method: slot.method,
  headers: slot.headers,
  body: blob,
});
if (!res.ok) throw new Error("upload refused");
// keep slot.upload_id for the save

The API never sees the bytes. This is the same from a browser, from React Native or from a native app; the only difference is that a browser needs a CORS rule on the bucket, which is in the gotchas below.

3. Tigris tells the backend, and the pipeline runs

#

Within a few seconds Tigris posts an OBJECT_CREATED_PUT for uploads/raw/…, and the handler runs the same image pipeline the base64 path always used, while the person is still typing their title:

from io import BytesIO
from PIL import Image, ImageOps
from django.core.files.base import ContentFile
from django.utils import timezone
import blurhash

MAX_BYTES = 15 * 1024 * 1024

def on_raw_photo(event):
    if not event.name.startswith("OBJECT_CREATED"):
        return
    if event.size > MAX_BYTES:
        s3.delete_object(Bucket=event.bucket, Key=event.key)
        return
    upload = PhotoUpload.objects.filter(raw_key=event.key, processed_at=None).first()
    if upload is None:
        return
    data = s3.get_object(Bucket=event.bucket, Key=event.key)["Body"].read()
    process(upload, data)

def to_webp(im, longest, quality):
    im = im.copy()
    im.thumbnail((longest, longest))
    buf = BytesIO()
    im.save(buf, "WEBP", quality=quality, icc_profile=im.info.get("icc_profile"))
    return ContentFile(buf.getvalue())

def process(upload, data):
    with Image.open(BytesIO(data)) as im:
        im = ImageOps.exif_transpose(im).convert("RGB")
        im.thumbnail((1080, 1080))
        upload.width, upload.height = im.size
        upload.dominant_color = "#%02x%02x%02x" % im.resize((1, 1)).getpixel((0, 0))
        upload.image.save(f"{upload.public_id}.webp", to_webp(im, 1080, 80), save=False)
        upload.thumbnail.save(f"{upload.public_id}_t.webp", to_webp(im, 320, 70), save=False)
        upload.blurhash = blurhash.encode(im.resize((32, 32)), 4, 3)
    upload.processed_at = timezone.now()
    upload.save()

The first line of on_raw_photo matters more than it looks. The router sends every event under uploads/raw/ here, including the deletes from the orphan sweep and from the size check itself, and get_object on a key that’s gone would only raise.

So the client’s shrink was only ever for the pipe. The server still decides the format (WebP), the sizes, the thumbnail, the placeholder, and the dimensions it stores. Re-encoding through Pillow also drops the EXIF block, which is where phone cameras put the GPS position, so the privacy strip comes free. The ICC profile would go the same way, which shifts wide-gamut phone photos a touch, so to_webp passes it through. If you’d rather keep pixel-perfect originals, don’t shrink on the client; the flow is identical, the PUTs are just bigger, and bigger PUTs are precisely the thing this design makes cheap. One catch: originals from an iPhone are HEIC, which Pillow only opens with pillow-heif installed.

4. The save attaches the photos and never waits on the webhook

#

When they hit Post, the request carries upload_ids instead of image data. The server looks each one up, with three outcomes:

def take(user, upload_ids):
    uploads = list(PhotoUpload.objects.filter(
        user=user, public_id__in=upload_ids, expires_at__gt=timezone.now()
    ))
    if len(uploads) != len(set(upload_ids)):     # expired, or not this user's
        raise UploadNotReadyError(upload_ids)
    for upload in uploads:
        if upload.processed_at is None:          # the webhook hasn't got here yet
            data = read_raw(upload)              # None if the PUT never landed
            if data is None:
                raise UploadNotReadyError(upload.public_id)
            process(upload, data)
    return uploads

If the notification already ran, the processed files are attached as they are. If the file is in the bucket but the notification hasn’t arrived, the save processes it inline, right there, so a post never waits on the webhook. The notification is a head start, not a dependency. And if the file isn’t there at all, or an id has expired or belongs to someone else, the save returns a validation error the app retries after a moment; the app waits for its own PUTs before sending, so in practice that’s rare.

A few details that made it hold up. The slot belongs to a user, so the server only accepts upload ids the same person created; otherwise anyone who saw an id could attach your photo to their listing. Everything falls back: if the slot request fails, the PUT is refused, or the API is an older version, the app sends base64 exactly as before. It only takes the upload path when every new photo has a slot, so a post’s pictures are all uploads or all base64, and their order stays one list. And orphans expire: people pick photos and then give up, so an upload that’s never attached expires after two days and the existing orphaned-media sweep cleans its files up.

Presigned URL upload gotchas: CORS, size limits, expiry, multiple files

#

Most of what goes wrong with presigned uploads goes wrong before the notification ever fires. These are the ones I’d want on one page.

CORS, for browsers only

#

A native app doesn’t care. A browser sends a preflight OPTIONS to the bucket before the PUT, and the upload fails with a CORS error unless the bucket’s CORS rules allow PUT from your origin with the Content-Type header. Tigris takes the standard S3 CORS configuration, so the rule looks the same as on AWS. It’s the first thing to check when a presigned PUT works from curl and not from the page.

The content type is part of the signature

#

Signing with ContentType means the client must send exactly that header, or the bucket answers 403 SignatureDoesNotMatch. That’s why the slot returns the headers along with the URL. Sign it on purpose, but know what it buys you: the header is enforced, the content isn’t, so a PDF labelled image/jpeg gets through. The pipeline is the real check; Pillow refuses to open what isn’t an image, and the row is never marked processed.

Size limits

#

A presigned PUT URL carries no size limit. If you need a hard stop before the bytes land, a presigned POST with a content-length-range policy condition does that. I didn’t bother, because the notification carries the object’s size: the handler deletes anything over the limit before reading it, and the save-time path reads at most the limit plus one byte so it can tell. The app sends a few hundred kilobytes; anything near fifteen megabytes isn’t ours.

Expiry

#

Two clocks. The presigned URL is good for fifteen minutes, which is plenty for one PUT and short enough that a leaked URL is useless soon. The slot row is good for two days, which covers someone picking photos, putting the phone down and finishing the listing tomorrow. The client treats a pending upload as stale after a day and starts a fresh one rather than hoping.

Multiple files, and bigger ones

#

One slot per file. Ten photos is ten slots and ten parallel PUTs, each with its own progress (an XMLHttpRequest gives you upload.onprogress; fetch doesn’t), which you can’t get from one giant base64 POST. Past a few megabytes, use multipart (parts are 5 MiB minimum, so it’s for files well beyond that): create the multipart upload on the server, presign each upload_part, and complete it when the parts are in. Tigris sends OBJECT_CREATED_MULTIPART when the upload is completed, so the same handler fires and nothing else changes.

Out of order

#

Events can arrive out of order, because the object may have been written in more than one region. For uploads it doesn’t matter: one key, one event. For a key that gets overwritten, trust the eTag and the object’s Last-Modified, not the order the webhooks reached you.

The same signal for logs: alerts to Discord

#

Once the bucket can talk to the backend, anything that lands in it can be reacted to the same way, and the second thing I pointed it at was logs.

The app’s logs are shipped to the bucket as small gzipped files, one every five minutes or so per app. For most of the year a GitHub Action on a 15-minute cron read the last 20 minutes of that archive with DuckDB, found the slow-request lines, and posted a digest to Discord. On paper that’s fine. In practice GitHub’s scheduled workflows don’t run on time when the platform is busy, so “every 15 minutes” was more like “every few hours”, and when it finally ran it had a bigger window to chew through and occasionally ran out of memory. The overlap I’d added so nothing slipped between runs meant the same slow request sometimes showed up twice. It was a lot of machinery for “tell me when a request is slow”.

Now each log file is read once, as it lands. They’re a few kilobytes gzipped. The handler pulls the slow-request lines out, and if there are enough of them, or one really slow one, it posts a digest to a Discord channel through a webhook. Same for a burst of 5xx responses in one file. A Discord webhook from Python is one call:

import requests
from django.conf import settings

def discord_alert(text: str) -> None:
    requests.post(
        settings.DISCORD_WEBHOOK_URL,
        json={"content": text[:1900]},  # Discord caps a message at 2000 chars
        timeout=5,
    )

No cron, no overlap, no duplicates, and the alert arrives within a few minutes of the request instead of whenever GitHub got round to it. The Action is gone.

While each file is open anyway, I also count what’s in it (errors, warnings, health checks) into an hourly tally table. The daily summary used to scan a whole day of the archive. Now it reads a few hundred rows, and only falls back to the archive if the tally has a gap.

A guard on deletes

#

This one I didn’t plan. It fell out of having OBJECT_DELETED events. When a media file is deleted, the handler checks whether any row still points at it. If one does, that’s a bug somewhere (or a person in the bucket console with a heavy finger), so it clears the column (the app shows a placeholder instead of a broken image) and reports it.

It also counts deletions per hour. Normal housekeeping removes a handful. If it ever sees a couple of hundred in an hour, I want to hear about that while it’s happening, not when users start reporting blank photos.

Things worth knowing

#

Respond fast. Tigris gives each delivery ten seconds. Reading a 5 KB log file or processing one phone photo fits; anything that might take longer goes to a queue and the view returns 200, or Tigris times out and retries, and you’re back to deduplicating.

Filter at the bucket. My log shipper writes a file every few minutes per app, forever. If you only care about one prefix, say so in the notification’s filter so your app isn’t woken for everything else.

Test with a fake body. The payload is small and stable, so the tests post JSON at the view with the right Authorization header and assert what the handler did. No real storage involved.

Don’t make it load-bearing on day one. Every use here has a fallback: the save can process a photo itself, the daily summary can still scan the archive, the app can still send base64. Notifications made each of them faster and cheaper, but none of them falls over if a webhook goes missing.

The best bit of the week was deleting that workflow file. Forty lines of YAML, a Python script, two repository secrets and a recurring “why didn’t the alert fire” are gone, and the replacement is one handler that reads one small file when it arrives. The uploads were the point, though. The photos go up as fast as the network allows, the server still gets the last word on every one of them, and nobody had to build a bridge between the bucket and the backend, because the bucket now just says what happened.

The general technique is called test impact analysis, and testmon is a free, open-source way to get it in Python. Nothing about it depends on git: it compares against what it saw on the last run, committed or not.

I added it to my marketplace app’s Python tests this week, and the reason wasn’t speed for its own sake.

Why I added it

#

A feature request is usually small: one screen, one endpoint, one rule that should work differently. Running the whole test suite made every one of them bigger than it was. The suite is 2,248 Django tests, about three and a half minutes with four processes and 11 in one, and every edit ran all of it, though almost all of it checks things the request never touched. When something unrelated failed, or the run simply took its time, the feature waited. It was being held up by circumstances outside the feature request, and the thing holding it up was my own test suite.

The alternative wasn’t better. Skip the full run, guess which test file covers the change, and now and then guess wrong. That’s quick, but it isn’t precise, and an untested change I ship is a feature request I’ll be doing twice.

The tests I wait for should be exactly the tests the change can reach, no more and no fewer. testmon gives me that set from a record instead of a guess. The full suite still runs, in CI, where it doesn’t hold up whatever I’m working on next.

How it decides what to run

#

testmon splits each Python file into blocks: the module’s top level, each class, each function, with the bodies of nested functions cut out of the block around them. It stores a checksum of each block, never the code itself. While a test runs, coverage.py records the lines it executes, and testmon notes which blocks those lines belong to. There’s no static analysis and no guessing from imports. A test depends on exactly the blocks it ran.

Take a module like this:

# pricing.py
def subtotal(items):
    return sum(item.price * item.quantity for item in items)

def discount(total, code):
    if code == "WELCOME10":
        return total * 0.9
    return total

A test that only calls subtotal() depends on the module’s top level and on subtotal’s body. Rewrite discount() however you like and testmon won’t rerun that test, because it never executed a line of discount. Change one character inside subtotal and it runs again, even if the change can’t alter the result. testmon doesn’t try to be clever about side effects: if a block you executed changed, you run.

It also tracks what’s around the code. The record includes the Python version and the installed packages with their versions. It doesn’t watch environment variables: if the same code runs under two settings modules, name them with --testmon-env (step 8) or they share one record. Upgrade Django and everything that touched Django is affected again, which in a Django app is everything.

What it can’t see

#

testmon tracks Python that a test executes. It doesn’t track the other files your tests read: templates, JSON, text, anything else on disk. It doesn’t track services reached over the network. And it can’t pin anything that runs once before the tests start on a particular test.

In a Django project that means migrations, which run once when the test database is created, so no test’s record includes them (the wrapper below rebuilds the test database when there’s a new one); settings and start-up code, which run while pytest starts up, outside any one test; and templates and data files, since a template isn’t Python, so editing one reruns nothing that renders it.

My suite has a few more of its own. Some tests read a JSON file of fixtures that the mobile app’s tests share. Some check the built web bundle, and some check the output of a static site generated in the same repository. Change any of those and testmon will happily tell you nothing needs to run. That’s why the how-to below wraps testmon in a script that notices these files and runs everything when one has changed.

How to add it to a Django project

#

These steps assume pytest and pytest-django, and uv for the dependencies. With pip, swap uv add --dev for pip install and drop uv run.

1. Install it.

uv add --dev pytest-testmon

2. Make sure pytest can run the suite. testmon is a pytest plugin, so it does nothing under python manage.py test. pytest-django needs the settings module and the file patterns your tests use:

# pytest.ini
[pytest]
DJANGO_SETTINGS_MODULE = myproject.settings
python_files = tests.py test_*.py *_tests.py
python_classes = Test* *Tests *TestCase

If the suite has only ever run under Django’s runner, run it under pytest once and fix whatever differs before adding testmon. Otherwise you’ll be debugging two things at once.

3. Keep the record out of git. testmon writes .testmondata in the pytest root directory. It’s a SQLite database, so you’ll also see -shm and -wal files next to it, and the script below adds a stamp file. One pattern covers all of them:

# .gitignore
.testmondata*

4. Record the whole suite once. testmon can’t select anything until it has seen every test run:

uv run pytest --testmon

In a fresh checkout that first run is the whole suite in one process. Here it takes about 12 minutes for 2,248 tests. From then on, the same command runs only the affected tests. With nothing changed, it runs nothing.

Two catches. On testmon 2.2.0 and earlier, it needs pytest’s cache plugin: my first attempt ran with -p no:cacheprovider and crashed with KeyError: 'lf' (2.3.0 treats a missing --lf as unset). And on Python 3.14 or later, set COVERAGE_CORE=ctrace before you record. “The bug that looked like a green run”, further down, explains why. It’s the most important line in the script below.

Update, 7 November: testmon 2.3.0, released on 2 November, forces coverage’s C tracer itself, so on that version the export is belt and braces. Keep it if you’re on 2.2.0 or earlier, and keep the break-something check whatever the version.

5. Wrap it for the files it can’t see. This is the part that makes it safe to use every day. The script keeps a stamp file from the start of the last run. If any file testmon can’t trace is newer than the stamp, it runs everything and records afresh. A new migration only rebuilds the test database. Otherwise it lets testmon choose. A cut-down example:

#!/usr/bin/env bash
# scripts/test-changed.sh: the tests a change can affect, or all when in doubt.
#   scripts/test-changed.sh            # affected tests
#   scripts/test-changed.sh --all      # everything, re-recorded
#   scripts/test-changed.sh -k orders  # anything else goes to pytest
set -euo pipefail
cd "$(dirname "$0")/.."

# coverage's C tracer: from Python 3.14 coverage defaults to sys.monitoring,
# which reports a line only the first time a process runs it, so testmon
# would credit each line to one test. Changing this means --all.
export COVERAGE_CORE=ctrace

STAMP=.testmondata.started
full=false
rebuild=false
if [ "${1:-}" = "--all" ]; then
    full=true; shift
elif [ ! -f .testmondata ] || [ ! -f "$STAMP" ]; then
    echo "No record yet: running everything once."
    full=true
else
    # Changed since the last run started, and not traceable per test:
    # anything that isn't Python, plus settings and conftest.
    newer="$(find myapp myproject fixtures pytest.ini \
        -type f -newer "$STAMP" ! -path '*/migrations/*' \
        \( ! -name '*.py' -o -path 'myproject/settings*' -o -name 'conftest.py' \) \
        ! -path '*/__pycache__/*' | head -5)"
    if [ -n "$newer" ]; then
        echo "Changed, and not traceable per test:"; printf '  %s\n' $newer
        full=true
    fi
    # A new migration needs a fresh test database, not a full run.
    [ -n "$(find myapp -path '*/migrations/*.py' -newer "$STAMP" | head -1)" ] && rebuild=true
fi

# Stamp the start, so an edit made while the tests run counts next time.
started="$(mktemp .testmondata.run.XXXXXX)"
args=(--testmon --reuse-db)
# --create-db alone deletes the database when the run ends; with
# --reuse-db as well, the rebuilt one is kept for next time.
[ "$rebuild" = true ] && args=(--testmon --reuse-db --create-db)
[ "$full" = true ] && args=(--testmon-noselect --reuse-db --create-db)

status=0
uv run pytest "${args[@]}" -q "$@" || status=$?
# 0: passed, or nothing selected. 1: some failed (still recorded).
# 5: nothing collected at all.
case "$status" in 0|1|5) mv "$started" "$STAMP" ;; *) rm -f "$started" ;; esac
[ "$status" = 5 ] && exit 0
exit "$status"

Two details in there are worth explaining. --reuse-db is pytest-django’s flag for keeping the test database between runs, which matters once a run takes seconds: creating it again each time can take longer than the tests themselves. A full run, or one after a new migration, adds --create-db so the database has the current schema. Keep --reuse-db next to it: --create-db on its own makes pytest-django delete the database when the run ends, so the next run would build it all over again.

And when testmon deselects everything, the run exits 0, so “nothing changed” counts as a pass. Exit code 5 is pytest’s “no tests collected”, which the script treats as fine too.

Notice what’s not in the list: pyproject.toml and the lock file. My first version ran everything whenever the lock file changed, which made the slow fallback fire on most days and the whole thing worse than before. testmon already tracks installed package versions itself, so it doesn’t need the help.

6. Give it its own database and cache. If other things run tests on the same machine, like a second terminal, a coding agent or manage.py test, point the script at its own test database and its own cache number so a kept database never meets someone else’s run:

export DATABASE_URL="postgresql://myapp:myapp@localhost:5432/myapp_changed"
export REDIS_URL="redis://localhost:6379/8"

7. Empty that cache before each run. I learnt this one the hard way. Subset runs started failing with KeyErrors that full runs didn’t. Some category labels are cached with no expiry, and a test database kept from one run met a cache still warm from an earlier one. Clearing the cache at the top of the script fixed it:

uv run python -c "import os, redis; redis.Redis.from_url(os.environ['REDIS_URL']).flushdb()"

That’s testmon doing what its docs say it will: running a different subset each time exposes tests that only passed because of what ran before them. The fix belongs in the test setup, not in testmon.

8. Know the flags for when selection is wrong.

• --testmon-noselect runs every test but puts the likely failures first. testmon switches to it by itself when you also pass -k, -m, --lf or a test’s node ID.
• --testmon-forceselect runs only affected tests that also match your -k or -m.
• --testmon-nocollect uses the existing record but doesn’t add to it. It’s forced under a debugger or when coverage is already running.
• --no-testmon turns it off for one run, for when it’s in addopts.
• --testmon-env=<name> keeps separate records in one file, for a suite you run under two settings modules.

If the project has a JavaScript side, Jest has its own version built in. jest --changedSince=main runs the tests that import anything changed since main, following imports rather than coverage. I added it to the app’s package.json as npm run test:changed so both halves have the same habit. An edit to one utility module selected 7 of the app’s 121 test files.

What it saves

#

I timed the wrapper once it was in place, start-up included, after fixing the bug in the next section:

For an everyday edit that’s about 25 times faster than the parallel suite, and when nothing has changed it’s about two hundred times faster.

The record says the same thing more broadly. Of the 307 non-test Python files the suite runs, a change to the median one reaches 22 tests, about 8 seconds of test time. A few core modules sit at the other end: almost every test runs them, so a change there reaches around 1,500 tests, and testmon reruns most of the suite. That’s the right answer, not a failure. Those figures are upper bounds per file: testmon works per block, so changing one function selects only the tests that ran that function.

The cost shows up when the wrapper falls back. A full run is the same single-process recording as the first one, about 12 minutes, which is slower than the parallel suite it replaces. It’s single-process only because I haven’t tried testmon under pytest-xdist yet, which it has supported since version 1.4; that’s the obvious next thing to measure. That’s why the list of files that trigger it should be as short as it can honestly be.

The bug that looked like a green run

#

My first numbers were better than these: one edit, 6 tests, 3.3 seconds. They were wrong, and nothing on screen said so.

From Python 3.14, coverage.py uses a new core by default, built on the interpreter’s sys.monitoring. That core reports a line only the first time a process runs it. testmon builds its record of which test ran which line from coverage, so each line was credited to the first test that happened to run it, and to no other. Edit a line that eight tests share, and testmon selects one of them.

It was caught on a second, smaller project of mine that copied this setup. Part of adopting it there was breaking a line on purpose and comparing the results, and the numbers didn’t match. Back in this suite I ran the same check. I broke one line in the home-screen banner code, the title of a slide. Plain pytest reported 8 failing tests. testmon selected 4. The other 4 broken tests never ran, and nothing in the output said they’d been left out.

The fix is one line, export COVERAGE_CORE=ctrace, which puts coverage back on its classic C tracer. A different tracer means a different record, so I re-recorded: 12 minutes 3 seconds for 2,248 tests. Afterwards, the same kind of breaking edit ran all 11 tests it broke.

Update, 7 November: testmon 2.3.0, released on 2 November, now sets the C tracer itself, in process and for subprocesses, so a fresh install no longer needs the export. The check in the next paragraph still does.

That’s the lesson I’d pass on before anything else here. A test selector that selects too little looks exactly like success: fewer tests, a faster run, all green. So when you set one up, break something shared on purpose, run the plain suite and the selector, and check they fail the same tests. Then do it again whenever Python or coverage.py changes version.

Where a coding agent comes in

#

I don’t make every change in that repository myself any more. A coding agent, Claude Code in my case, makes a lot of the small ones, and a good agent runs the tests after each edit. Faced with an 11-minute suite, it has the same two bad options I had: wait, or guess which test file matters.

So the repository’s agent instructions now say the wrapper script is the default way to run tests while working, with --all for a full run. The agent gets the same precise set of tests I do, after every edit, in seconds. It’s also why step 6 gives the script its own database and cache: on a busy day there’s more than one session running tests on the same machine, and a kept database must never meet somebody else’s run.

None of this needs anything AI-specific from testmon. It’s the same record and the same selection. It just matters more when whatever is making the changes makes a great many of them.

Why CI got something coarser

#

testmon has no CI mode. It behaves the same on a runner as on a laptop, and its author writes that sharing or caching .testmondata across CI runners is far less effective than their hosted service, testmon.net. A cached record would also be blind in all the ways above, and my CI runs the suite with Django’s runner in four processes, so I left testmon out.

What CI got instead is the same idea at file level. The checks workflow now starts with a small job that diffs the push and decides which half of the repository it touched: the Python API, the mobile app, both or neither. Before this, CI ran both the Python and the app jobs on every push, docs-only ones included. Now a docs-only push runs nothing, and a push to the app alone skips the Django suite. I dry-ran the job against five real commits: the docs-only one ran nothing, the app-only one skipped the Python job, and the backend-only one skipped the app job. A cut-down example:

jobs:
  changes:
    runs-on: ubuntu-latest
    outputs:
      python: ${{ steps.diff.outputs.python }}
    steps:
      - uses: actions/checkout@v7
        with:
          fetch-depth: 0
      - id: diff
        env:
          BEFORE: ${{ github.event.before }}
          HEAD: ${{ github.sha }}
        run: |
          FILES="$(git diff --name-only "$BEFORE" "$HEAD")"
          # Python's inputs include the non-Python files its tests read.
          PY='^(myapp/|myproject/|fixtures/shared\.json$|web/dist/|pyproject\.toml$|uv\.lock$)'
          if printf '%s\n' "$FILES" | grep -qE "$PY"; then
            echo "python=true" >> "$GITHUB_OUTPUT"
          else
            echo "python=false" >> "$GITHUB_OUTPUT"
          fi

  python:
    needs: changes
    if: needs.changes.outputs.python == 'true'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - run: uv run python manage.py test --parallel 4 --noinput

The real one also runs everything when there’s no range to compare: a new branch, a force push or a manual run.

The blind spot is the same one in a different form. testmon can’t see a JSON file, and neither can a path filter unless you list it. So the pattern names the shared fixtures, the built bundle and the generated site explicitly, with a comment asking whoever edits it to keep the list honest. The file-level version is cruder, but it fails in the safe direction: if anything on the Python side changed, the whole suite runs.

Is it worth it?

#

For a suite that takes minutes, yes. The real cost is what it can’t see, and that’s what the wrapper script is for. Even so, the laptop loop is where testmon belongs. CI still runs the whole suite whenever the Python side changes, and that’s the run I trust before a deploy.

If you came here to run Docker Compose in GitHub Actions: I did that for two years and stopped. CI now declares the same Postgres and Valkey images as services: and runs the same uv run … and npm run … commands straight on the runner; Compose stays what a developer runs. The rest is why, with the files.

In 2024 I wrote Use Docker Compose for CI/CD Instead of Platform-Specific YAML, which said to keep the build logic in Compose and let the CI tool do nothing but call it. Two years on, I went back through every repository I ship from. Four have workflows, each with one committer, me, and none of those workflows calls docker compose any more. The principle held; the mechanism didn’t. Here’s what replaced it, with cut-down example versions of the files, the step timings GitHub recorded, and the parts that are less automated than a CI/CD diagram would like.

Every pipeline I run, in one table

#

In textbook terms the Hugo sites have continuous deployment, and the app (a marketplace and community app I build) has continuous integration plus one-command releases, which is continuous delivery: the checks tell me something is wrong, but they don’t stop me deploying. That’s deliberate. An OTA update reaches every phone on the binary version it targets, so publishing one should be a decision rather than a side effect of a merge, and the API is where the database migrations run. My other two Django apps have no workflows at all.

The CI half: one workflow, two jobs

#

The app’s CI is one file, .github/workflows/checks.yml. Until 11 October there wasn’t any. This is a trimmed example of its shape:

# Lint, types and tests on every push and pull request to main.
# Nothing here deploys.
name: Checks

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

concurrency:
  group: checks-${{ github.ref }}
  cancel-in-progress: true

jobs:
  python:
    runs-on: ubuntu-latest
    timeout-minutes: 30
    services:
      postgres:
        image: postgres:17
        env:
          POSTGRES_USER: myapp
          POSTGRES_PASSWORD: myapp
          POSTGRES_DB: myapp
        ports: ["5432:5432"]
        options: >-
          --health-cmd "pg_isready -U myapp"
          --health-interval 5s --health-timeout 5s --health-retries 20
      valkey:
        image: valkey/valkey:8-alpine
        ports: ["6379:6379"]
    env:
      DEBUG: "True"
      DATABASE_URL: postgresql://myapp:myapp@localhost:5432/myapp
      REDIS_URL: redis://localhost:6379
      PYTHON_JIT: "0"   # match production
    steps:
      - uses: actions/checkout@v7
      - uses: astral-sh/setup-uv@v10
        with:
          enable-cache: true
          prune-cache: true
      - run: uv sync --locked
      - run: uv run ruff check .
      - run: uv run mypy myapp/
      - run: uv run python manage.py test --parallel 4 --noinput

  app:
    runs-on: ubuntu-latest
    timeout-minutes: 20
    defaults:
      run:
        working-directory: frontend
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-node@v7
        with:
          node-version: 24
          cache: npm
          cache-dependency-path: frontend/package-lock.json
      - run: npm ci
      # Expo writes this on `expo start` and it is gitignored,
      # so a fresh checkout doesn't have it.
      - run: printf '/// <reference types="expo/types" />\n' > expo-env.d.ts
      - run: npm run typecheck
      - run: npx jest --ci

The two jobs run side by side. Across the 35 green runs so far, the app job’s median is about seven minutes and the Python job’s about twelve and a half.

The services: block is where the Compose file went, which is the next section. The environment is a few variables and a flag; everything else falls back to the development default docker compose up gives a developer, so the tests need no secrets. I checked by running part of the suite with exactly that environment and nothing else.

CI runs the same interpreter as my laptop and the production image, uv’s own build of Python 3.15.0, because uv sync reads .python-version and downloads exactly that build. (My real file also passes setup-uv a python-version-file input, which does nothing: setup-uv v6 has no such input, and every run carries a warning saying so. I only noticed while writing this.)

Matching interpreters earned their keep eight days after the workflow went in. On 19 October I switched on 3.15’s JIT compiler, and the first run with it failed a test: a plain yield from, on a code path the app takes on every save, raised AttributeError: 'list' object has no attribute 'send'. The same commit with the JIT off passed. My Mac is arm64 and never hit it; the runner is x86_64, like the production machines. The JIT is off again, in CI and in the Dockerfile, and the bug never reached production. It’s python/cpython#155823, fixed upstream on 13 October, after 3.15.0 shipped.

The very first run failed too, for a duller reason. Expo writes expo-env.d.ts when you run expo start, and the file is gitignored, so every working copy I’d used had one and a fresh checkout didn’t. TypeScript reported five errors that had never appeared locally. The step that writes the file is the fix.

concurrency cancels a run when a newer push to the same branch arrives, which is how all twelve cancelled runs so far ended. A pre-commit hook runs the linter, plus the type checker when Python or TypeScript is staged, so most mistakes never reach a push.

Where the Docker Compose file went

#

Here is the part of the app’s docker-compose.yml that CI cares about, as an example. The real one has more services and a lot more environment.

services:
  valkey:
    image: valkey/valkey:8-alpine
    ports:
      - "6379:6379"

  # Host port 55432, not 5432, so it can't collide with a local Postgres.
  postgres:
    image: postgres:17-alpine
    environment:
      - POSTGRES_USER=myapp
      - POSTGRES_PASSWORD=myapp
      - POSTGRES_DB=myapp
    ports:
      - "55432:5432"
    volumes:
      - postgres_data:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U myapp -d myapp"]
      interval: 5s
      retries: 10

  api:
    build: .
    command: python manage.py runserver 0.0.0.0:8000
    volumes:
      - .:/app
    ports:
      - "8000:8000"
    environment:
      - DATABASE_URL=postgresql://myapp:myapp@postgres:5432/myapp
      - REDIS_URL=redis://valkey:6379
    depends_on:
      postgres:
        condition: service_healthy
      valkey:
        condition: service_started

volumes:
  postgres_data:

Put it next to the services: block in checks.yml and it’s the same two images: Postgres 17 (the Alpine build here, the Debian one in CI) and valkey/valkey:8-alpine. That’s the relationship now. The Compose file is what a developer runs; CI declares the same images as service containers and runs the same commands a developer runs, uv run … and npm run …, straight on the runner.

Against the 2024 post, the part I cared about most held: the CI tool is thin, and the checks and deploys are commands I can run myself. What didn’t survive is CI calling Compose.

The app’s CI doesn’t build the Docker image at all; Fly’s builder does that once, at deploy time, and CI only has to say whether the code is good, which with a cached toolchain takes a six-second install. The Compose file also bind-mounts the working copy and seeds data, which a test run shouldn’t, so CI would have needed a Compose file of its own: a second description of the same services, kept in step by hand.

And the 2024 example had a bug I didn’t notice at the time. docker compose build builds the image and never runs the service’s command, so the npm test in that file never ran in CI. The workflow went green whether the tests passed or not.

The risk that post worried about most, switching CI vendors, never came up. The move that did happen was at the other end: this blog and the client’s site moved hosts to Cloudflare Pages, and for the blog that took one new 45-line workflow, because the build was already one command. And the three-hundred-line workflow I warned about? I have one, pyhtml2text’s. It’s long because the wheels have to be built on Linux, macOS and Windows runners, for several architectures each, which makes it the one job where platform-specific YAML is the right tool.

Caching: read the step timings first

#

GitHub keeps the duration of every step, and gh api repos/<owner>/<repo>/actions/runs/<run-id>/jobs hands it over. These are from the app’s latest green run, on 20 October:

Both dependency caches are on, and both are the cheap part. The uv cache GitHub keeps for the repository is about 200 KB, because setup-uv prunes whatever uv can simply download again; the npm one is nearly 400 MB. The time is in the tests, which already run four ways in parallel. If I wanted the run faster, I’d change the suite, not the cache.

The blog is the opposite case. Hugo’s processed images live in the gitignored resources/_gen, so every fresh runner re-encoded every cover, body image and placeholder. The workflow further down restores them with actions/cache, keyed on a hash of the pictures and falling back to the newest older cache, since Hugo only processes what’s missing. In the last eight runs before the cache, the build step took 33 to 76 seconds; in the 26 since, 6 to 20, with one exception at 94, even though the same commit added placeholders and a second card size.

The app’s image builds on Fly’s builder, with the dependency files copied before the code and uv’s download cache in a BuildKit cache mount (RUN --mount=type=cache,target=/root/.cache/uv). The bigger fix wasn’t a cache. The old .dockerignore was a blocklist, and it missed a folder of extra git worktrees, each with its own dependencies and build output, so 7.7 GB across 277,000 files went to the builder on every deploy. Rewritten as an allowlist, the context is about 40 MB. An example of the pattern:

# Exclude everything, then let back in only what the Dockerfile COPYs.
*

!pyproject.toml
!uv.lock
!manage.py
!myapp
!static

# Rules match the whole relative path and the last match wins,
# so these re-excludes apply inside the allowed directories too.
**/__pycache__/
**/*.pyc
**/.DS_Store
**/*.log

Secrets, in as few places as possible

#

Secrets live in three places. GitHub Actions secrets are for what workflows do, and the two Hugo repositories have exactly one each, CLOUDFLARE_API_TOKEN. Fly secrets are the app’s runtime configuration, set with flyctl secrets set, and they never pass through Actions. And a gitignored .env on my laptop holds the token the deploy script uses to purge Cloudflare’s cache.

PyPI needs no secret at all. pyhtml2text uses trusted publishing: PyPI accepts the short-lived OIDC token GitHub issues to the workflow, so the release job asks for id-token: write and the repository’s secrets page is empty. There’s nothing to leak and nothing to rotate.

Tokens get the least they need. The app’s Fly token in GitHub serves a scheduled check that only lists machines, so it’s created with fly tokens create readonly. The Pages deploys ask for contents: read and deployments: write, and nothing more.

Inputs go through env:, never straight into a script. The OTA workflow takes free-text release notes. Written as ${{ inputs.notes }} inside run:, the text would be pasted into the script before bash reads it, so a note with a quote and a semicolon in it would become code. Passed as an environment variable, it stays data. An example of the pattern:

      - name: Publish
        env:
          OTA_API_TOKEN: ${{ secrets.OTA_API_TOKEN }}
          INPUT_PLATFORM: ${{ inputs.platform }}
          INPUT_ROLLOUT: ${{ inputs.rollout }}
          INPUT_NOTES: ${{ inputs.notes }}
        run: |
          args=("$INPUT_PLATFORM")
          [ -n "$INPUT_ROLLOUT" ] && args+=(--rollout "$INPUT_ROLLOUT")
          [ -n "$INPUT_NOTES" ] && args+=(--notes "$INPUT_NOTES")
          ./scripts/publish-ota.sh "${args[@]}"

Some keys stay off GitHub altogether. One scheduled report needs a key with admin rights over the app’s sign-in backend, so it moved from a scheduled workflow to a scheduled job on my Mac. The workflow is still there as a manual fallback, and leaves out the sections that need that key.

Deploys: on every push, by script, and on a tag

#

The Hugo sites deploy on every push

#

An example version of the blog’s workflow, with the first-run step that creates the Pages project left out:

name: Deploy to Cloudflare Pages

on:
  push:
  workflow_dispatch:

env:
  HUGO_VERSION: "0.165.0"
  PAGES_PROJECT: my-site

concurrency:
  group: deploy-${{ github.ref }}
  cancel-in-progress: true

jobs:
  deploy:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      deployments: write
    steps:
      - uses: actions/checkout@v7
        with:
          submodules: true

      - uses: peaceiris/actions-hugo@v3
        with:
          hugo-version: ${{ env.HUGO_VERSION }}
          extended: true

      # Processed images are gitignored; restore them so Hugo only
      # re-encodes what is new. An older cache still helps.
      - uses: actions/cache@v6
        with:
          path: resources/_gen
          key: hugo-resources-${{ hashFiles('content/**/*.jpg', 'content/**/*.png', 'assets/img/**') }}
          restore-keys: |
            hugo-resources-

      # A multilingual site has no root 404.html; without one, Pages
      # answers unknown paths with index.html and a 200.
      - name: Build
        run: |
          hugo --gc
          cp public/en/404.html public/404.html

      # ...

      - name: Deploy
        uses: cloudflare/wrangler-action@v4
        with:
          apiToken: ${{ secrets.CLOUDFLARE_API_TOKEN }}
          accountId: ${{ vars.CLOUDFLARE_ACCOUNT_ID }}
          command: pages deploy public --project-name=${{ env.PAGES_PROJECT }} --branch=${{ github.ref_name }}

--branch=${{ github.ref_name }} makes main production and gives every other branch a preview URL. The client site’s workflow mirrors this one, plus an rm -rf public before the build, because its public/ is committed and removed pages would otherwise linger in the upload.

Hugo is pinned here and nowhere else. In October a package-manager upgrade moved my laptop to 0.168, one past the newest version the theme supports. The deploys stayed on 0.165 and never noticed.

And the build is the test. On 14 September the preview deploy for one branch failed three times running: a post on it linked, through Hugo’s ref, to one that was still a draft. CI builds what production builds, drafts excluded, so the link pointed at nothing and Hugo stopped with REF_NOT_FOUND. main carried on deploying in the meantime.

The app’s API deploys from a script

#

The core of the deploy script, as an example:

IMAGE_LABEL="deploy-$(git rev-parse --short HEAD)-$(date +%Y%m%d%H%M%S)"

# Build first, then wake a spare machine, then roll out exactly that image.
flyctl deploy --config fly.toml --build-only --push --image-label "$IMAGE_LABEL"
curl -fsS --retry 5 https://myapp.fly.dev/health > /dev/null
flyctl deploy --config fly.toml --image "registry.fly.io/myapp:$IMAGE_LABEL"

And the matching lines in fly.toml:

[deploy]
  # max_unavailable only counts machines that are running,
  # which is why the script wakes one first.
  strategy = "rolling"
  max_unavailable = 1

The image is built once, pushed to Fly’s registry under a label made of the commit and the time, and deployed by that label, so what rolls out is exactly what was built.

Waking a spare first is what makes it zero-downtime. max_unavailable = 1 only counts running machines, and the app keeps none running when it’s idle (a suspended machine resumes in about a second), so a lone awake machine used to be replaced in the same batch as the sleeping ones, with nothing warm behind it. On 28 September a probe against the origin saw 7 timeouts and a 7.3-second gap during a plain flyctl deploy, and no failures in 2,514 requests during the scripted deploy, which builds first and has a machine awake before the rollout starts.

The rest of the script marks the deploy in the team chat channel the error alerts go to, so a burst of errors can be read against what just changed, purges Cloudflare’s cache and warms the machines. The web front end skips the image entirely; that’s in Deploy Frontend and Backend Separately, Without Redeploying the API.

Migrations aren’t a pipeline step. Fly’s release_command hung for me, so the first machine to boot on a new image takes a lock and migrates while the others wait. Every migration therefore has to work with the code that’s still running: add a column in one deploy, stop using it in the next, drop it in a third.

Nothing in the script needs my laptop as such, but a couple of local-only files keep it there: the Cloudflare credentials, and a record of what was last deployed, which a fresh runner wouldn’t have. It will also deploy uncommitted changes, and says so in the chat channel, where the web release script refuses a dirty tree.

In Preview Environments per Branch on Fly.io I wrote that doing this by hand is fine while you’re the only one doing it, and that from the second developer on it has to happen because something happened in git. The app is still at one developer; the branch-environment workflows from that post live on the branch I built them on.

The OTA workflow is manual-only because publishing should be a decision, and because the update service refuses a release while a partial rollout is open, which an on-push trigger would hit during every canary. It has never run. Every OTA update so far has gone out from my laptop through the same script the workflow calls, which is the case for thin workflows in one line: however a release starts, it’s the same script.

pyhtml2text releases on a tag

#

The release job, with the package name made generic:

  release:
    runs-on: ubuntu-latest
    if: ${{ startsWith(github.ref, 'refs/tags/') || github.event_name == 'workflow_dispatch' }}
    needs: [linux, macos, windows, sdist]
    environment:
      name: pypi
      url: https://pypi.org/p/mypackage
    permissions:
      id-token: write       # trusted publishing and signing
      contents: write
      attestations: write
    steps:
      - uses: actions/download-artifact@v8
      - uses: actions/attest@v4
        with:
          subject-path: 'wheels-*/*'
      - uses: astral-sh/setup-uv@v10
        if: ${{ startsWith(github.ref, 'refs/tags/') }}
        with:
          prune-cache: true
      - name: Publish to PyPI
        if: ${{ startsWith(github.ref, 'refs/tags/') }}
        run: uv publish --trusted-publishing always 'wheels-*/*'

The release job waits for every build job, and only a tag publishes; a manual run builds and attests but uploads nothing. actions/attest records which workflow built each wheel, and uv publish --trusted-publishing always uploads them with the OIDC token rather than a password. The package itself is in A Closed Pull Request, a Fork and a PyPI Package.

Scheduled workflows are best effort

#

The app has one scheduled workflow, a check that Fly hasn’t started more machines than the baseline, which would otherwise first show up on the bill. It alerts by failing, since GitHub emails the owner about a failed scheduled run.

It’s set to cron: '*/15 * * * *', 96 runs a day. Over the past two weeks it ran between three and eight times a day, with a median gap of about three hours forty. The only explanation GitHub’s documentation offers is load: scheduled runs can be delayed when Actions is busy, and dropped when it’s busy enough. For a sustained scale-out, a few hours is fine. For anything that has to fire within minutes, a GitHub cron isn’t a monitor.

What I’d ask of a pipeline

#

If you’re building one, or looking at one you already have:

1. If GitHub Actions were down for a day, could you still release? If not, your release logic lives in YAML, and the 2024 post still applies.
2. Does CI run the interpreter, the CPU architecture and the database major version that production runs? Mine caught a compiler bug only because the runner and production are both x86_64 and my laptop isn’t.
3. Does a fresh checkout pass? Generated, gitignored files are the usual reason it doesn’t.
4. Where does a run’s time actually go? Read the step timings before you add a cache.
5. For each secret, how many places does it live, and does it need to be in GitHub at all?

There’s no hosted APM, no tracing in production and no dashboard I have to remember to open, and the only part that costs extra is the log archive, at about US$2 a month. Here’s the whole list, and the things I dropped or stopped trusting, one of which had me chasing an outage no user ever saw.

What I mean by monitoring, and how small this is

#

By DevOps monitoring I mean finding out that something’s broken before somebody tells me, and being able to work out why without guessing. Every tool below is judged on those two things.

Last year I wrote about setting up monitoring that developers actually use, mostly with teams in mind: a cloud provider’s built-in monitoring for a small team, a hosted APM once there are hundreds of services. This is the small end of the same idea. Both products are Django on Fly’s shared-cpu-1x machines, as described in My Current Stack, and I’m the only developer on either. If monitoring is a product and developers are its users, mine has a user base of one. A baseline I pulled from Fly’s metrics in September had the marketplace’s API averaging 1% CPU. This isn’t an enterprise stack, and I won’t dress it up as one.

Of the three pillars, metrics, logs and traces, I have two. Neither product has distributed tracing: each is one Django codebase and one database, with no service boundary for a trace to cross. The only traces I read are the reverse proxy’s, on my workstation.

The list

#

Everything in the table is the marketplace unless it says both apps. The estimating app has less: Fly’s health check, logs and metrics, app analytics on iOS, and a daily Parquet export of its database that I query with DuckDB. It has no crash reporting and no alerts channel, which is the biggest gap here. (A disclosure for the update-service row: the company behind it pays me to write for their blog.)

Errors: a logging handler posting to chat

#

Until September, flyctl logs was the only place an unhandled 500, a failed push notification or a broken WebSocket frame showed up, and nobody reads that stream unprompted. Now a logging handler sends every record at ERROR or above to a chat channel, from the API, the WebSocket server and the background worker. Warnings deliberately don’t alert.

The handler posts on a background thread, so it never slows a request. Nothing in it logs, so it can’t recurse. And the same error goes out at most once every ten minutes, with a cap of 30 an hour, so an outage reads as one message rather than a thousand.

The channel also carries what I want to read errors against: deploys post when they start, when they’re up and where they failed, with the commit and who ran it, and so do over-the-air updates and web releases. A burst of wrong sign-in codes on one account logs at ERROR too, so someone guessing shows up next to the outages.

In October an alert arrived that matched no line in any log. Now each alert carries a short random id, the hostname and the process’s start command, and writes the same line to that process’s log, so an id I can’t find means a process that doesn’t log there, like an SSH shell. Everything in the channel also goes into a feed that a coding agent I work with can read, so the agent working on the code sees the errors when I do.

Logs: Fly’s shipper to S3, queried with DuckDB

#

Fly’s own buffer holds a few minutes of output, roughly the last hundred lines, which is fine for watching a deploy and no use for yesterday. So a third Fly app runs Fly’s log shipper, which is Vector subscribed to the organisation’s log stream, with one change to its S3 sink:

[sinks.aws_s3]
  type = "aws_s3"
  bucket = "myapp-logs"
  compression = "gzip"
  key_prefix = "logs/{{fly.app.name}}/%F/"   # one folder per app per day
  batch.timeout_secs = 300                  # a quiet app still lands its lines

Every line from the marketplace’s apps lands in an S3-compatible bucket as gzipped JSON. The shipper is one shared-cpu-1x machine with 256 MB, about US$2 a month plus pennies of storage. DuckDB reads the files where they sit through one macro, and a day comes back in about a second:

CREATE MACRO logs(app, day := '*') AS TABLE
  SELECT * FROM read_json_auto('s3://myapp-logs/logs/' || app || '/' || day || '/*.log.gz',
                               union_by_name := true);

SELECT timestamp, message FROM logs('myapp-api', '2026-09-06') WHERE message LIKE '% 500 %';

That macro is how I measured the cold starts in the scale-to-zero post: the startup script announces each phase, every line carries the machine’s id, and the rest is a GROUP BY.

So it’s three levels of one stream, by who reads them: the archive keeps INFO and up, which is why the apps log at INFO in production, a live-tail script shows warnings and errors, and the chat channel gets ERROR. None of them knows the caller, though. The access log has no client IP or user agent, and for that I go to Cloudflare.

Metrics: Fly’s built-in Prometheus

#

Fly runs a managed Prometheus for each organisation, so I don’t run a metrics stack of my own. I query its API when I have a question: memory, CPU and burst credit per machine (fly_instance_cpu_balance matters on a shared CPU), responses and concurrency per app. Two things I wish I’d known first. A token from fly tokens create goes in as Authorization: FlyV1 <token>; Bearer only works with the personal token from flyctl auth token, and a read-only token gets “something went wrong resolving organization”. And it keeps about two weeks (Fly says roughly 15 days).

It earned its place in September, when the logs had “Could not open a database connection” several times a day, which reads exactly like an app bug. The managed Postgres cluster is in the same Prometheus, and at every one of those minutes the primary’s scrape time jumped from about 80 milliseconds to between one and ten seconds, while the replica was fine and Postgres’s start time never moved. Postgres hadn’t restarted; the host under it was stalling. I sent Fly’s support the timestamps, and after a maintenance restart in early October moved the primary to another host, the daily stalls stopped.

Crash reports: Sentry’s SDK, pointed at my own API

#

A crash on a phone used to leave no trace anywhere I could see. The obvious answer was Sentry, and I took half of it: the app carries Sentry’s open-source SDK, but its DSN points at an endpoint in my own Django API that accepts Sentry’s envelope format, groups the crashes, and alerts through the same chat handler. Tracing, sessions and replays are off. It’s errors only.

I didn’t sign up for sentry.io because, at a few reports a day, it would have been another vendor holding user data, another line in the privacy policy and another bill. The price is symbolication, which I haven’t replaced. Stacks are minified JavaScript or raw native addresses, so an issue tells me what crashed, on which release and device, and what the person did just before, but not where in the source.

The only dashboard is a message

#

I haven’t built a dashboard for either app. Last year’s post had a test for them: if nobody opens one unprompted, it isn’t earning its place. A message that arrives on its own doesn’t need opening, so the closest thing I have is a post in the team chat every morning.

It starts with the last 24 hours of production errors, read from the log archive rather than from the chat channel, because throttled alerts can’t say how often something happened. Numbers, ids and addresses are masked before grouping, so a hundred copies of one failure are one row with a count and no email reaches the channel.

Then failed health checks, yesterday’s usage against the same weekday a week earlier (a Sunday is never like a Monday), the last week of search traffic, chats waiting on a reply, the task board and what I’ve spent. A section that can’t load says so, and the post still goes out.

Reading app analytics through an API teaches you to distrust them a little. In August, 121 Android first-opens from Azerbaijan were followed by 119 uninstalls, and I read that month’s Android totals with them taken out.

Releases have their own screen. When over-the-air updates came from my own server, my view of a release was an admin list with an “is active” checkbox, and I had no idea how many phones had taken one. The hosted service’s dashboard shows which release devices run, how installs went, and a crash rollback when a bad bundle crashes and the SDK restores the previous one. I wrote up the move to a hosted update service separately.

What I dropped, or stopped believing

#

None of this was a paid tool. Mostly it was things I’d been treating as signals that weren’t, and one job running in the wrong place.

Cloudflare’s 504 rate, until I grouped it by user agent

#

In September the Cloudflare dashboard showed what looked like 22% of a day’s requests failing with 504s, concentrated on the endpoints the app polls every 30 seconds, while every health check was green and the app’s own logs said those requests were fast. I spent days on it, found that machines took 25 to 33 seconds to cold-start and did it 92 times a day, fixed that, and wrote it up in the scale-to-zero post as the cause.

The next morning I grouped the same 504s by user agent. All 7,344 of them between 5 and 9 September came from Cloudflare’s own Early Hints cache lookups (user agents bastion early hints and nginx-ssl early hints), which never reach the origin: a 504 on one of those lines is a cache miss, not a timeout. They copy real request paths, so they landed on the busiest endpoints and looked exactly like users failing. The app, browsers, curl and search crawlers had none. Early Hints wasn’t doing anything for me because the origin sent no Link header, so I turned it off, and later made it work.

The cold-start work still paid off, since deploys and wakes are faster for it, but the outage I thought I was fixing never reached a user.

So this is the rule I’d most like to pass on: before you believe a CDN’s error rate, group it by user agent and check it isn’t the CDN’s own machinery. On Cloudflare’s free plan that’s a GraphQL query on httpRequestsAdaptiveGroups, a day at a time. The same query settled the next scare, when one polling endpoint went from 70–260 requests a day to 2,880. It wasn’t a scraper. It was my own Mac, with the app left running in the iOS simulator after testing.

A green health check

#

Fly checks /api/health/ on both apps every 30 seconds and uses the answer to decide which machines get traffic, which it’s good at. But a health check runs against a machine that’s already up, so it can’t see what happens before one is: a cold start, a queue at the proxy, a request that never arrives. Routing is all I ask of it now.

Deploys get the probe script instead, which hits the origin throughout and exits 1 if anything failed. It caught a plain flyctl deploy leaving a 7.3-second gap, then counted no failures in 2,514 requests through the deploy script, which wakes a spare machine first.

The health check has also caused two outages of its own. In July, Django’s SECURE_SSL_REDIRECT answered Fly’s plain-HTTP probe with a 301, every machine failed its check, and everyone got a 503. In August, narrowing an except Exception in the health view let a “connection already closed” error escape as a 500, and Fly pulled the machine in exactly the case the view’s retry was there for. Now the check only asks whether the process can serve, and a database stall no longer takes both apps out of routing.

The morning summary had to learn the same lesson about counting. The marketplace suspends idle machines, so a machine woken by a request fails its first check a few hundred milliseconds after starting and passes the next. Counting those made every morning read like an outage, so a failure that passes again within a minute is now listed separately, as machines waking up and deploys.

A scheduled job in CI

#

The morning summary started as a CI workflow on a cron schedule and moved within a day. Once it read the analytics and search data, the key it needed was the analytics project’s service-account key, which can do anything in that project, and I didn’t want that sitting in CI secrets. Now a scheduler on my Mac runs it every 20 minutes, and it posts once it’s past eight in Auckland and today’s post hasn’t gone out, so a Mac asleep at eight posts when it wakes. The workflow stays as a manual fallback, minus the analytics and search sections.

If I were starting again

#

I’d set up three things before the first user rather than after the first incident: errors to a chat channel the team already reads, with a rate limit; a log archive in object storage I can query with SQL; and whatever metrics the platform already collects. That’s a couple of dollars a month. Crash reporting comes the week an app goes onto phones, because otherwise a crash on a phone tells nobody anything. A morning summary comes once there are enough places to check that I’d stop checking them.

I’d leave tracing until there’s a second service for a request to cross, and a hosted APM until there’s a team that needs shared dashboards, an on-call rota and a budget for someone else’s bill. That’s when the tools in last year’s post start paying for themselves. Before then the most useful habit is free: when a number looks alarming, find out what produced it before deciding what broke. Several of the scares in this post came from the monitoring, not the product.

Until this week, the only way I found out our app had crashed was when someone told me. Most people don’t. They reopen the app, and if it happens again they stop opening it. The API side was covered (every ERROR log already lands in a team chat channel), but a crash on a phone left no trace anywhere I could see.

The app did have one reporter already. When it puts an error message in front of someone, it posts that message to a small endpoint on the API, which logs it at ERROR, so it reaches the chat channel too. I added it after a delete that had worked left the row in the list, and tapping it again showed “No Item matches the given query”. The server had logged two ordinary 404s and said nothing. That covers errors the app catches and shows. A crash is different: the JavaScript that would send the report is the thing that just died.

The obvious fix is Sentry. I like Sentry’s SDK a lot; it’s the reason crash reporting on React Native is a solved problem. What I didn’t want was another vendor holding user data, another privacy-policy line, and another bill for something that, at our size, is a few reports a day at most. So I kept the SDK and replaced the service: the app’s DSN points at our own API, and a small piece of the Django app there receives what it sends, groups it, alerts on it, and keeps it.

This post is how that works, and how I checked that a real native crash actually makes it through.

The DSN trick

#

A Sentry DSN looks like https://<public-key>@<host>/<project-id>. The SDK turns that into a POST to https://<host>/api/<project-id>/envelope/, with the key in an X-Sentry-Auth header. Nothing about it is specific to sentry.io. So the DSN baked into the app is:

https://<CRASH_REPORT_KEY>@api.example.com/1

and Django has a route that matches what the SDK will call:

path("api/<int:project_id>/envelope/", crash_envelope, name="crash_envelope"),

The key is in every binary, so it isn’t a secret. It’s there so that if the endpoint ever starts attracting junk I can change it, and so that an unset key on the server refuses everything rather than accepting the world:

def crash_envelope(request, project_id):
    expected = settings.CRASH_REPORT_KEY
    if not expected:
        return Response({"detail": "Crash reporting is not configured"}, status=503)
    if not hmac.compare_digest(_key(request).encode(), expected.encode()):
        return Response({"detail": "Unknown key"}, status=401)
    ...

_key reads sentry_key= out of the X-Sentry-Auth header, falling back to the query string, which some SDKs use instead.

What the SDK sends: envelopes

#

Sentry’s wire format is the envelope: one JSON header line, then any number of items, each one a JSON header line followed by a payload. If the item header has a length, the payload is exactly that many bytes; otherwise it runs to the next newline.

{"event_id":"9ec79c33...","dsn":"https://...@api.example.com/1"}
{"type":"event","length":4123}
{ ...the event JSON, 4123 bytes... }
{"type":"client_report"}
{"timestamp":1791400000.5,"discarded_events":[]}

Two details caught me out:

• The native SDKs gzip the body; the JavaScript one doesn’t. A JS error captured by @sentry/react-native arrives as plain text, while a native crash from sentry-cocoa or sentry-android arrives with Content-Encoding: gzip. The decoder handles gzip, zlib-wrapped deflate and raw deflate, and stops decompressing at 1MB, because a crash event is a few kilobytes and anything bigger isn’t one.
• Length-prefixed payloads can contain newlines. If you split on \n you’ll mangle every event the native SDKs send. You have to honour length when it’s there.

The parser is short. It only keeps event items; sessions, transactions, attachments and client reports are accepted and dropped.

Answering in a way the SDK understands

#

This matters more than I expected. The SDK keeps a queue on disk and retries anything it isn’t sure was delivered, so what the server answers decides whether it keeps trying:

• 200 once stored, or deliberately dropped. If I answered a session item with a 4xx, the SDK would keep it and retry forever.
• 429 with Retry-After when throttled. The endpoint has a per-IP throttle (120 an hour), because a phone in a crash loop sends one report per launch, and the SDK backs off on a 429 the way you’d hope.
• 400 for a malformed envelope, logged as a warning, not an error: a broken report says a client is broken, and the next good one will say what crashed.

The SDK also resends the same event if it isn’t sure the first send arrived, so event_id has a unique constraint and a duplicate is a quiet no-op.

Grouping without source maps

#

This is where a self-built receiver has to make a choice Sentry makes for you. Sentry groups crashes by stack trace. But a release Hermes bundle is minified, so frame names are single letters and line/column positions change with every build. Group by frames, and the same bug opens a new issue every release.

Without source maps (more on that at the end), the most stable thing in a JavaScript error is its type and message. So the fingerprint is the exception type plus the message, with the parts that differ between sightings masked out:

_MASKS = [
    (re.compile(r"0x[0-9a-fA-F]+"), "<addr>"),
    (re.compile(r"\b[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}\b"), "<uuid>"),
    (re.compile(r"\b[0-9a-fA-F]{16,}\b"), "<hex>"),
    (re.compile(r"\d+"), "<n>"),
]

Cannot read property 'title' of undefined (reading index 3) and (reading index 17) become the same issue. If the SDK supplies its own fingerprint without {{ default }} in it, that wins, so I can force grouping from the client when the default is wrong. An event with no exception falls back to its masked log message, and one with neither falls back to the top in-app frame. For a native signal crash the message is something like Signal 6, Code 0, so those group by signal, which is coarse but honest until there’s symbolication.

Alerting without flooding the channel

#

Each fingerprint becomes one issue, and each report is stored as an event under it. The issue only logs at ERROR (which is what the chat alerts listen for) in exactly these cases:

• a new issue
• a resolved issue that comes back, which reopens it
• the 10th, 100th, 1,000th and 10,000th sighting

An ordinary repeat only increments a counter. An issue marked ignored keeps counting and never alerts. With the chat handler’s own rate limit on top, a crash loop on a bad release reads as a handful of messages rather than a wall.

Storage is bounded the same way: each issue keeps its 50 most recent events and deletes older ones as new ones arrive. That broke my first chart, because 50 events can’t tell you how often something happened last week, so each issue also keeps a running count per day for 90 days, updated in the same transaction as the counter. The chart draws from the counts; the events are only for reading.

Two small races are worth handling if you build one of these. When a brand-new crash hits several phones at once, two reports can both try to create the issue, so make the fingerprint unique and have the loser re-read the existing row. And measure the 90-day window from the server’s today, not from the newest report, because a phone with its clock set wrong can date a report anywhere, including next year.

Payloads are capped at 128KB. An event over that loses its breadcrumbs and debug metadata first, and if it’s still too big only the exception, message, release and platform are kept, with a flag so I know it was cut.

Scrubbing

#

The SDK runs with sendDefaultPii: false, so in principle no IP address, cookies or request bodies are sent. The server doesn’t rely on that: before storing anything it keeps only id and username from user and drops request entirely, in case a later SDK version or setting starts sending them.

Testing against a real device turned up two more things to strip:

• Breadcrumbs record full URLs, query strings included. The app’s search requests carry the search term in the query string, so a crash right after a search would have stored what the person searched for. The server now cuts every breadcrumb URL at the ?.
• Console breadcrumbs carry the raw arguments that were passed to console.log, which can be any object the app happened to log. The message text stays (it’s useful), and the arguments are dropped.

I only found those by looking at an actual report, which is the main argument for the next section.

The client side

#

On the app, everything goes through one small module, for the same reason I wrap the over-the-air update SDK: an OTA bundle also runs on binaries built before the native module existed. A top-level import * as Sentry from "@sentry/react-native" on a binary without RNSentry throws on import, and that’s a blank screen. So the SDK is required lazily, and only after checking the native module is actually there:

import { NativeModules, Platform, TurboModuleRegistry } from "react-native";
import Constants from "expo-constants";

const DSN = process.env.EXPO_PUBLIC_CRASH_DSN;

function nativeModulePresent(): boolean {
  try {
    return Boolean(TurboModuleRegistry.get("RNSentry") ?? NativeModules.RNSentry);
  } catch {
    return false;
  }
}

export function initCrashReporting(): boolean {
  if (!DSN || Platform.OS === "web" || !nativeModulePresent()) return false;
  const Sentry = require("@sentry/react-native");
  Sentry.init({
    dsn: DSN,
    release: Constants.expoConfig?.version ?? "unknown",
    sendDefaultPii: false,
    tracesSampleRate: 0,
    enableAutoSessionTracking: false,
    // ...turn off the other tracing options the same way
    maxBreadcrumbs: 50,
    beforeSend(event) { /* tag the OTA bundle */ return event; },
  });
  return true;
}

Errors only. The server doesn’t store traces, sessions, profiles or replays, so none of them are switched on. No DSN (development, E2E builds) means the SDK never starts. The build profiles set the DSN and an environment (staging or production), so both kinds of build report to the same endpoint and can be told apart. The signed-in user is set by account ID, never a name or an email.

The error boundaries needed an explicit call. Expo Router’s root ErrorBoundary catches a render error and shows its own screen, so that error never reaches the global handler the SDK hooks into. The fix is a small wrapper that calls captureException in a useEffect and then renders the router’s boundary as before, used for the root and per-route boundaries alike. It’s in an effect rather than in the render so it reports once rather than on every re-render. The root component is also wrapped with Sentry.wrap, and if that throws it falls back to the bare component.

For the web there’s a .web.ts version of the module with every function switched off. Metro picks the platform file, so the SDK never ends up in the web bundle at all. The web app has no native module to run on.

Proving a native crash arrives

#

The unit tests post hand-written envelopes, which proves the parser handles what I think the SDK sends. Two days after release, production had zero crash reports. Only three people had updated yet, so zero was plausible, but “nobody has crashed” and “crashes don’t arrive” look identical from the server. I wanted to see a real one.

So I built a Release simulator binary with the DSN pointed at a local Django on port 8000 and a throwaway key:

EXPO_PUBLIC_CRASH_DSN=http://localcrashtestkey0000@localhost:8000/1 \
EXPO_PUBLIC_CRASH_ENVIRONMENT=local-verify \
xcodebuild -workspace ios/MyApp.xcworkspace -scheme MyApp -configuration Release \
  -destination 'platform=iOS Simulator,id=<udid>' -derivedDataPath /tmp/iosbuild

(Release, not Debug: Debug builds link the dev launcher and behave differently, and Release is what people run.) Then installed it, launched it, waited a few seconds, and killed it the way a real fault would:

xcrun simctl install booted /tmp/iosbuild/Build/Products/Release-iphonesimulator/MyApp.app
xcrun simctl launch booted com.example.myapp      # prints the pid
kill -ABRT <pid>

Simulator apps are ordinary macOS processes, so a signal reaches SentryCrash’s handler exactly as it would on a device. The handler writes the report to disk; nothing is sent while the process is dying. On the next launch the SDK finds it and sends it. I relaunched, and the simulator’s network log showed a connection to :8000 a second later.

Then nothing in the Django log, which had me confused for a minute: the dev server only logs non-2xx responses at the level I had it on. The database had it:

SIGABRT: Signal 6, Code 0 | cocoa 1.4.0 iOS 27.1 iPhone19,4 handled=False
received 09:19:53, crashed 09:19:38
contexts: app, device, os, ota_updates, trace · 24 breadcrumbs

Fifteen seconds from crash to stored, most of which was me relaunching. And looking at the real payload is what taught me things the tests couldn’t:

• The user ID wasn’t the account ID. The test build isn’t really signed in, so sentry-cocoa fell back to its own per-install ID. Worth knowing, and worth saying in the privacy note: a report carries the account’s ID if you’re signed in and a random install ID if you’re not.
• There’s no time zone. I’d assumed the culture context would be there. The device context has locale, memory, screen size, low-power mode and the kernel version, but no time zone.
• Native crashes skip beforeSend. My JS beforeSend adds an ota tag with the running OTA bundle, but a native crash is assembled by the native SDK on the next launch and never passes through the JS hook. To tag native crashes, the tag has to be set on the scope with setTag once the bundle is known, so it syncs down to the native layer.
• The breadcrumbs showed what the app does. Console warnings, view-controller lifecycle, and HTTP calls, including the analytics SDK’s own requests to its collection endpoint, which is a good reminder of what else the app talks to. And the URLs had query strings, which is how I found the search-term problem above.

Reading them

#

The reports are read in the Django admin, on an issue page loosely modelled on Sentry’s: counts and Resolve / Ignore / Reopen at the top, a bar per day, the split by release, OS, device and OTA bundle, then one event at a time with its exception, its frames (app frames highlighted, library frames folded away), the breadcrumbs and the device context. It didn’t need a new dependency.

One fix came out of building it: for a native crash, the “culprit” frame was the binary’s path on the device, which includes a per-install UUID, so the same crash looked different on every phone. It now shows only the address and the binary name, like 0x1000a1b2c (MyApp).

The privacy policy, all of it

#

Crash reports are personal information, near enough: an account ID, a phone model, and what someone did in the minute before the app died. So the privacy policy needed a paragraph. Writing it properly meant saying exactly what a report carries, so I listed it from the payload I’d just looked at: the error, app version and OTA update, phone model, OS and language, free memory and screen size, the last few dozen steps (screens, web addresses with anything after a ? removed, the app’s own log lines), and the account ID or a random install ID. The policy also says the reports come to our server and not to Sentry or any other crash-reporting company. Retention gets its own line in the table: the 50 most recent reports of each problem, plus a daily count for 90 days.

Once I was checking one paragraph against the code, it was hard not to check the rest. The existing policy didn’t hold up. The delete-account page said push-notification tokens and passkeys were deleted immediately when you close your account, and that server logs were gone within 90 days. None of that was true. Closing an account unlinks the sign-in methods and anonymises the profile, but the push tokens and passkeys stay until someone deletes them by hand, and the log archive has no expiry yet. So the whole policy was rewritten from an inventory of what the code actually does, and those lines now say “not yet removed automatically; email us”. That’s less reassuring, and it’s accurate.

The repo now has a rule for it: the policy describes what the code does, not what we intend, and a change that collects, sends or keeps personal data somewhere new updates it in the same commit. That includes the summary screen in the app, the delete-account page, and the app-store privacy forms, which only live in the store consoles. Crash reporting was the change that made me write that down.

What’s missing

#

Symbolication. This is the real gap, and the one thing sentry.io does that I haven’t replaced. Stacks are minified for JavaScript and unsymbolicated for native code:

• JavaScript: frames point into the Hermes bundle. Fixing it means keeping the composed Hermes source map for every store build and every OTA bundle, keyed by release and bundle, and applying it to JS frames when the page renders. That’s the next piece, because most of our crashes will be JavaScript.
• iOS native: instruction addresses need the build’s dSYMs.
• Android: Java frames are obfuscated by R8 when minification is on (they need mapping.txt), and C++ frames need NDK symbols.

There’s also a smaller gap I found above: native crashes don’t carry the OTA tag yet, because it’s only set in beforeSend. Setting it on the scope once the running bundle is confirmed is a few lines I haven’t written yet.

For now, an issue tells me what crashed, on which release, OTA bundle and device, and what the person was doing just before. It doesn’t tell me where in the source. For a small app that ships a few times a week, the type, the message and the breadcrumbs have been enough to find things so far.

The receiver, the admin page and the tests aren’t a lot of code, and it costs nothing on top of the API that was already running. The SDK does the hard part, which is catching a crash on a phone and getting the report off it on the next launch.

Picture a small team working on one app. Two developers, or five; not specialists, but people who each take a feature from the database migration through the API to the screen in the mobile app, end to end, on their own branch. It’s a good way to run a team, and from the second developer onward it has one recurring failure: the message “Is anyone on staging?”

Behind that message is a developer who has finished a feature and wants a tester or a designer to see it, and can’t, because staging is running the second developer’s branch, and dev is running the third’s, which broke login two days ago and hasn’t been redeployed since. A handful of people, two shared environments, and a chat thread doing the scheduling.

That isn’t a people problem. It’s a topology problem. Every developer is treating the environment as the place where their branch becomes real, so the environment is doing the job the merge is supposed to do, one branch at a time, and everyone else is queueing for it.

I described the production shape of my stack in My Current Stack: Django on Fly.io’s cheapest machines, Postgres behind a connection pooler, Redis for the cache and the Channels layer, a separate WebSocket app, a background worker, an S3-compatible bucket for files, a hosted auth provider, Cloudflare in front, and a mobile app talking to all of it. This post is how I turned that stack into something where any branch gets its own public, HTTPS-fronted copy of the whole thing a couple of minutes after its developer puts one label on the PR, with nobody else asked and nobody provisioning it, and without a second Postgres cluster, a second Redis or a second bucket. The infrastructure stays shared. What changes is that every shared resource learns to be sliced by a name, and the name comes from the branch. Nothing in it assumes a team size: it’s the same design for two developers as for twenty, and what grows is the app list, which the sweeper handles. I built it, ran three branches at once, broke it in two places I hadn’t predicted, and tore it down, all in an afternoon; the outputs below are from that run.

Why one shared environment can’t scale past one developer

#

A shared dev environment has an assumption baked in: that there is one “current version of the code” that everyone wants to look at. That’s true for one developer, or for a team working a single stream of changes in lockstep. It stops being true the moment the second developer starts a feature on a different timeline, and with end-to-end ownership that’s every feature. From then on, everything the environment holds is contended:

• The deployed code. Whoever deployed last wins. Everyone else’s work is invisible until they redeploy, which undoes the first person’s.
• The database schema. This is the one that bites hardest when every developer owns their feature’s data model. The first branch adds a column. The second, deployed an hour later, doesn’t know about it and its ORM starts failing on inserts. Or both branches have a 0042_* migration, the shared database ends up in a state that matches nobody’s migration history, and someone resets it, deleting the test data the third developer spent an afternoon setting up.
• The cache. One branch’s serialiser change is cached under the same Redis keys another branch’s code reads. The bug reports that produces are baffling and cost a morning each.
• Third-party integrations. One OAuth callback, one webhook URL, one push certificate. Whoever’s branch is deployed gets the events.
• The tester. They can only test what’s deployed, so they test one feature at a time in the order the environment happened to be claimed, not the order the features are ready.

The team responds with a booking system: a chat thread, a pinned message, a bot with /claim staging. That’s a lock, and a lock on the only place your work can be seen is a lock on delivery rate. With two developers it’s an irritation. With three it’s tolerable on a good week. Add a fourth, or have one feature hold staging for three days of back-and-forth with a designer, and it stops being tolerable.

The first fix people reach for is a third shared environment, “dev2” or “uat”. That buys a few weeks. The queue just has two lanes.

The real fix is to notice that dev and staging were never environments. They were the current state of someone’s branch, with a URL. If that’s what they are, the right number of them is the number of branches the team cares about right now, and the right lifetime is the lifetime of the branch.

The model: the branch is the environment

#

Here’s the target state.

Read the Created by and Destroyed by columns again: every entry is a git event or a clock. Git is the control plane. The set of environments that should exist is a function of the set of branches that exist, and the automation’s only job is to keep reality matching it: a label on the PR creates, every push after that updates, closing or merging the PR destroys, and a nightly job reconciles whatever the first three missed. Nor does every branch get one: the label is the gate, one click by the developer, so a backup push or a bot’s dependency bump costs nothing. Nobody provisions an environment. Not a DevOps engineer with a ticket, not a bot you ask in chat, not an AI agent you prompt each time. The tenth developer’s first branch gets an environment the same way the first developer’s did, by labelling its PR, and if a person has to be asked, it hasn’t scaled; the queue has just moved.

And the rule that makes it work: every resource an environment touches is named after the branch, deterministically. Not after the developer, not after a ticket, not after a PR number that doesn’t exist until someone opens one. After the branch, because the branch is what developers actually work with, and because a deterministic name means the second run of ./scripts/env up on the same branch finds the environment it already made rather than making another.

The name is a slug for humans and a short hash for uniqueness:

branch=$(git rev-parse --abbrev-ref HEAD)              # feature/payments-retry
slug=$(printf '%s' "$branch" | tr '[:upper:]/_ ' '[:lower:]---' | tr -cd 'a-z0-9-' | cut -c1-20)
hash=$(printf '%s' "$branch" | sha256sum | cut -c1-6)
ENV="${slug%-}-${hash}"                                 # feature-payments-retr-3f9a1c

feature-payments-retr-3f9a1c is the environment. That string becomes the Fly app name, the Postgres database name (with hyphens turned to underscores), the Redis key prefix, the bucket’s object prefix, the error tracker’s environment, and the left-hand label of the hostname. Anyone who knows the branch name can compute it. Nothing has to be looked up.

That last sentence is the design. There is no registry of environments and no coordinator. env up, env down, the Cloudflare Worker that routes hostnames and the nightly sweeper each recompute the name from the branch independently and agree by construction. Cloudflare in particular is not coordinating anything: the Worker is a pure function from hostname to Fly app, and it never learns that an environment was created or destroyed. If the app exists the request lands; if it doesn’t, Fly answers with its own error.

Three things fall out of this.

First, “who’s on staging” stops being a question. There is feature-payments-retr-3f9a1c-dev.example.com and fix-login-redirect-8b21e0-dev.example.com, and each belongs to whoever is on that branch.

Second, staging becomes an honest rehearsal of prod rather than a dumping ground. It’s built from the exact image that will go to prod, against a database cloned from a prod-shaped snapshot minutes earlier. If it works there, the only variable left in prod is prod’s data.

Third, each developer gets power without getting root on anything shared. The blast radius of a branch environment is that branch environment. A fourth hire on their first day can deploy whatever they like to their branch, and the worst outcome is that their branch is broken.

fly.io/dashboard/my-org

Shared physically, separated logically

#

The tempting version of “an environment per branch” is a full copy: a Postgres cluster each, a Redis each, a bucket each. It’s clean and it’s the wrong trade. A Fly Postgres cluster is a machine plus a volume plus a restore; a Redis is another machine; you’d be waiting minutes for each environment and paying for a dozen idle databases. The sweeper would have four kinds of thing to clean up instead of one.

Instead, each shared service is provisioned once for the whole preview org, sized like a small prod, and every environment gets a slice of it keyed by $ENV. This is the table I’d want on the wall:

Two of those need a word.

Postgres: database, not schema. The other way to slice one cluster is a schema per environment, with search_path set per connection. Django will do it if you push a -c search_path=... through the connection options, but a connection pooler in transaction mode discards startup options unless you tell it not to, and then every connection in the pool has to belong to one environment anyway. It’s a source of “why is my query hitting the wrong tables” bugs that a database per environment simply doesn’t have. A database is a hard boundary, CREATE DATABASE ... TEMPLATE makes one in seconds, and DROP DATABASE makes it go away with nothing left behind. Branch environments connect to Postgres directly on port 5433 rather than through the pooler, the same way the background worker does in prod; the pooler earns its keep at production concurrency, which a branch environment never sees.

Secrets: generated, not shared. It’d be easy to have one set of dev secrets and copy it into every environment. Don’t. The env up script generates a fresh SECRET_KEY (session signing), ENCRYPTION_KEY (anything encrypted at rest) and ENV_TOKEN (the bearer token the mobile app presents, more on that below) for each environment and stores them only in that Fly app’s secret store. It means a session cookie from one branch is worthless on another, a leaked preview token opens one preview, and the developer never sees the values unless they go looking. If you keep secrets in a separate secrets manager rather than only in Fly, the same rule applies: the path is myapp/preview/$ENV/, and the sweeper deletes the path with the environment. The only secrets that are shared across environments are the credentials for the shared infrastructure itself (the Postgres admin URL, the Redis URL, the bucket key), and those live in CI, not in any environment.

The Redis row is the easiest to doubt, so here is the test. Set a cache key from inside one environment, read it from the other, then list every key in the shared Redis:

$ fly ssh console -a myapp-feature-profile-tagl-fa8832 -C "python manage.py shell -c \"from django.core.cache import cache; cache.set('who-am-i', 'tagline-branch', 600)\""
$ fly ssh console -a myapp-feature-profile-webs-dcb2e7 -C "python manage.py shell -c \"from django.core.cache import cache; print(cache.get('who-am-i'))\""
None

$ # every key in the shared Redis
  feature-profile-tagl-fa8832:1:who-am-i
  ...

Same Redis, same key name, the other environment sees nothing.

Everything else in this post is the script that stamps $ENV onto those eight rows, and the rules that keep it honest.

./scripts/env up

#

This is the whole idea, as a script a developer runs from their branch. CI runs the same script; there’s no separate CI path to drift. It needs fly and git on the laptop and nothing else: SQL goes to the Postgres app over fly ssh, and the environment’s Redis keys and bucket objects are removed by the environment’s own machine before it’s destroyed, so the laptop never holds the shared Redis or bucket credentials beyond a gitignored env file. Trimmed to the lines that matter:

#!/usr/bin/env bash
# scripts/env up|down|name|url: one environment per branch on shared preview infrastructure
set -euo pipefail
ORG=${PREVIEW_ORG:-myapp-preview}; PG_APP=${PREVIEW_PG_APP:-myapp-preview-pg}; DOMAIN=${BASE_DOMAIN:-example.com}

branch=${BRANCH:-$(git rev-parse --abbrev-ref HEAD)}
[[ "$branch" == "main" ]] && { echo "main has staging, not a branch environment" >&2; exit 1; }
slug=$(printf '%s' "$branch" | tr '[:upper:]/_ ' '[:lower:]---' | tr -cd 'a-z0-9-' | cut -c1-20); slug=${slug%-}
ENV=${ENV_OVERRIDE:-${slug}-$(printf '%s' "$branch" | shasum -a 256 | cut -c1-6)}   # the sweeper passes the name
APP="myapp-${ENV}"; DB="env_${ENV//-/_}"; HOST="${ENV}-dev.${DOMAIN}"

pg() { fly ssh console -a "$PG_APP" -q -C "psql postgres://postgres:${PG_PASSWORD}@localhost:5433/postgres -At -c \"$1\""; }
exists() { fly apps list --org "$ORG" --json | grep -q "\"Name\": *\"$APP\""; }

up() {
  echo "== $branch -> $ENV"
  exists || fly apps create "$APP" --org "$ORG"                         # idempotent
  [[ "$(pg "SELECT 1 FROM pg_database WHERE datname='$DB'")" == "1" ]] || pg "CREATE DATABASE $DB TEMPLATE seed_template"

  have=$(fly secrets list -a "$APP" --json | grep -o '"Name": *"[^"]*"' | cut -d'"' -f4 || true)
  gen() { grep -qx "$1" <<<"$have" || echo "$1=$(openssl rand -hex 32)"; }   # generated once, never rotated by a later `up`
  { gen SECRET_KEY; gen ENCRYPTION_KEY; gen ENV_TOKEN
    echo "ENV_NAME=$ENV"; echo "ALLOWED_HOSTS=$HOST,$APP.fly.dev"
    echo "DATABASE_URL=postgres://postgres:${PG_PASSWORD}@${PG_APP}.flycast:5432/${DB}"
    # ... plus the shared Redis URL and bucket credentials
  } | fly secrets import -a "$APP" --stage

  fly deploy -a "$APP" --config fly.preview.toml --remote-only --ha=false \
    --build-arg GIT_SHA="$(git rev-parse --short HEAD)" --build-arg GIT_BRANCH="$branch"
  echo "https://$HOST"
}

down() {
  [[ "$APP" =~ ^myapp-[a-z0-9-]+-[0-9a-f]{6}$ ]] || { echo "refusing to destroy $APP" >&2; exit 1; }
  if exists; then
    fly machine start -a "$APP" >/dev/null 2>&1 || true                 # it may be suspended
    fly ssh console -a "$APP" -q -C "python manage.py teardown_env" || echo "warning: cleanup failed; the sweeper will catch it" >&2
    fly apps destroy "$APP" --yes
  fi
  pg "DROP DATABASE IF EXISTS $DB WITH (FORCE)" || true
}

case "${1:-}" in up) up ;; down) down ;; name) echo "$ENV" ;; url) echo "https://$HOST" ;; esac

For the demo I ran it from my laptop rather than pushing, because the three branches were throwaways I didn’t want in the repository’s history and I wanted to watch the output as it went. In the team setup nobody types this: the label workflow further down runs exactly this script, and what follows is what would land in the Actions log. Here is what it printed the first time, on the branch that carries the tooling itself:

$ ./scripts/env up
== feature/branch-environments -> feature-branch-envir-c65cb4
New app created: myapp-feature-branch-envir-c65cb4
CREATE DATABASE
Secrets have been staged, but not set on VMs. Deploy or update machines in this app for the secrets to take effect.
==> Building image
image size: 139 MB
✔ Machine 1a2b3c4d5e6f78 [app] update finished: success
https://feature-branch-envir-c65cb4-dev.example.com

$ curl -s https://feature-branch-envir-c65cb4-dev.example.com/api/version/
{"environment": "feature-branch-envir-c65cb4", "branch": "feature/branch-environments", "commit": "a1b2c3d4", "region": "syd"}

About two and a half minutes from command to URL, most of it the image build. The second up on the same branch, after another commit, printed no “New app created” and no “CREATE DATABASE”, staged the same secret names without regenerating the generated ones, and rebuilt: same URL, new commit.

Some of the choices in there deserve a sentence each.

fly.preview.toml is not fly.toml. Prod runs the API, the WebSocket server and the background worker as separate Fly apps and process groups so each scales on its own metric, which is the whole argument of the stack post. A branch environment doesn’t need to scale; it needs to be one machine. So the preview config has a single process: the container’s entrypoint gets a preview mode that runs migrations and then starts one ASGI server with WebSockets enabled. Channels already routes both HTTP and WebSocket traffic within one ASGI application, so one process serves the whole app. Sync views run thread-sensitive in that mode, roughly one request at a time, which is exactly the trade I moved away from in prod and exactly right here, because nobody testing a branch will notice.

# fly.preview.toml (the app name comes from `fly deploy -a`)
app = 'placeholder'
primary_region = 'syd'

[processes]
  app = "/app/entrypoint.sh preview"   # migrate, then one ASGI server for HTTP and WebSockets

[env]
  DJANGO_SETTINGS_MODULE = 'myapp.settings'
  DEBUG = 'False'

[http_service]
  internal_port = 8000
  force_https = true
  auto_stop_machines = 'suspend'
  auto_start_machines = true
  min_machines_running = 0
  [[http_service.checks]]
    path = '/healthz'
    [http_service.checks.headers]
      Host = '127.0.0.1'               # the app's own hostname isn't known to a static file

[[vm]]
  size = 'shared-cpu-1x'
  memory = '1gb'

auto_stop_machines = "suspend", min_machines_running = 0. A branch environment nobody has opened since lunch should cost nothing. Suspend rather than stop gives the ~3 second resume I ended up with after chasing 504s instead of a 30-second cold start, and here even 30 seconds would be tolerable. Two of the three demo environments were already suspended in fly apps list a minute after their deploy finished.

fly.io/apps/myapp-feature-profile-tagl-fa8832

DEBUG = False, same settings module as prod. A branch environment is a public URL. It should behave like prod in every way that matters for testing (real HTTPS, real cookies, the same health check) and unlike prod in every way that could hurt someone. Rather than a separate preview settings file, the difference is carried by the secrets env up sets: sandbox payment keys, no admin credentials for the auth provider, and the ENV_NAME that switches on every prefix below.

One setting slices everything. ENV_NAME is the only new knob in the settings, and it’s applied in four places:

ENV_NAME = os.getenv("ENV_NAME", "")
_prefix = f"{ENV_NAME}/" if ENV_NAME else ""

CACHES["default"]["KEY_PREFIX"] = ENV_NAME                           # every cache key
CHANNEL_LAYERS["default"]["CONFIG"]["prefix"] = f"{ENV_NAME}:asgi"  # Channels group and channel names
STORAGES["default"]["OPTIONS"]["location"] = f"{_prefix}media"      # uploads
STORAGES["staticfiles"]["OPTIONS"]["location"] = f"{_prefix}static"

Miss the channels one and two branches’ WebSocket servers share group names, and a message sent in one environment turns up in another. That is the kind of bug that costs a day.

There was a fifth place I only found by reading the shared Redis after the fact. The startup lock from the 504s post, the thing that lets one machine per image run migrations while the others wait, was keyed on the image reference. Two branch environments built from the same commit have the same image. The second one would have found the first’s “done” marker and skipped its own migrations against its own, un-migrated database. Now the environment name is part of the key and the key lives under the prefix. It’s the sort of thing a shared-infrastructure design keeps producing: anything that was implicitly “per deployment” has to be made explicitly per environment, and you don’t know what was implicit until you look.

The database is cloned, not migrated from empty. That’s the next section.

The database: a template, cloned in seconds

#

A branch environment’s database can be one of three things. Shared with everyone, which is what got us here. Empty, migrated from zero and seeded with fixtures: deterministic, but fixtures are never as good as real-shaped data and are always eighteen months stale. Or a clone of a prod-shaped snapshot: realistic row counts, realistic nulls in columns the fixtures assumed were always filled, the migration that’s instant on an empty table and takes forty minutes on a real one shows itself.

The third, with the second kept as a CI-only check. And what makes the third fast enough to do per branch is a Postgres feature most people forget exists:

-- Nightly, on the preview cluster. Restore last night's prod backup into seed_raw,
-- run the anonymiser, then freeze it as a template nobody can connect to.
ALTER DATABASE seed_raw RENAME TO seed_template;
UPDATE pg_database SET datistemplate = true, datallowconn = false WHERE datname = 'seed_template';

-- Per branch, in the time it takes to copy the files:
CREATE DATABASE env_feature_payments_retr_3f9a1c TEMPLATE seed_template;

CREATE DATABASE ... TEMPLATE is a file-level copy within the cluster. A few gigabytes is seconds, not the minutes a pg_restore takes. The template can’t have open connections while it’s being cloned, which is what datallowconn = false enforces, and it’s also why the nightly job builds into seed_raw and renames at the end rather than restoring over the live template.

For the demo the template was built by hand rather than from a prod dump: a tunnel to the preview cluster with fly proxy, manage.py migrate, then the repo’s own seed commands for a small but realistic data set. Nineteen megabytes. Three clones of it appeared in the cluster as three branches came up, each 19 MB, each taking under a second:

$ fly ssh console -a myapp-preview-pg -C "psql ... -c '\l'"
             datname             | template | size
---------------------------------+----------+-------
 env_feature_branch_envir_c65cb4 | f        | 19 MB
 env_feature_profile_tagl_fa8832 | f        | 19 MB
 env_feature_profile_webs_dcb2e7 | f        | 19 MB
 seed_template                   | t        | 19 MB

The preview cluster is one unmanaged Fly Postgres node, shared-cpu-1x with a 1 GB volume, about two dollars a month. The branch environments connect to it over .flycast on the private network; nothing about it is public.

The anonymiser is not optional. The moment prod data is reachable on a -dev hostname, it needs to not be prod data: names, emails, phone numbers, addresses, free text, payment references, tokens, all overwritten before the rename. If the anonymiser fails, the previous night’s template stays and someone gets a message. It’s a small amount of work and it’s the thing that makes the whole approach acceptable to whoever answers your privacy questionnaire.

The entrypoint runs migrate on the first boot of every image, so the branch’s own migrations apply on top of the template. That’s the first honest test of “does my migration work on data shaped like prod”, and it happens before anyone looks at the URL.

Hostnames: a wildcard, a Worker, and two things the docs didn’t tell me

#

Every environment needs a public HTTPS hostname, and “public” is the point: a tester on a phone, a designer on another network, a product manager who won’t install a VPN. Fly gives each app myapp-$ENV.fly.dev for free, and that alone works. But *.fly.dev hostnames have three problems: they’re ugly to send around, Cloudflare isn’t in front of them so none of the protections you have on prod apply, and if you want them on your own domain you’re doing a DNS record and a certificate per environment, which is one more thing for the sweeper.

The version that removes all of that is one wildcard and one Cloudflare Worker. I’d planned *.dev.example.com. Two things stopped that, and both are worth knowing before you plan the same.

Universal SSL covers one wildcard level. Cloudflare’s free certificate is issued for example.com and *.example.com. It does not cover *.dev.example.com; a second-level wildcard needs Advanced Certificate Manager at $10 a month. The first request to feature-branch-envir-c65cb4.dev.example.com died with an SSL handshake failure before anything of mine ran. The fix is to keep the environment in the first label: feature-branch-envir-c65cb4-dev.example.com. That’s covered by the certificate that already exists, and by the proxied *.example.com record that already exists, the one I set up as a sinkhole for the scanners walking invented subdomains. So an environment needs no DNS at all.

Redirect Rules run before Workers. With the route in place, every -dev hostname answered 301 to the www homepage without the Worker running. The sinkhole is a zone-level redirect rule, “any other subdomain → www”, and Cloudflare’s redirect phase executes before the Workers phase. The rule needed one clause: and not ends_with(http.host, "-dev.example.com"). The deploy script adds it on up and removes it on down, because a rule someone edited by hand is the thing that gets forgotten.

dash.cloudflare.com/…/example.com/rules/redirect-rules/…/edit

The DNS side is the record that was already there:

dash.cloudflare.com/…/example.com/dns/records

With those two out of the way the routing is a single Worker on *-dev.example.com/*:

// dev-router.js: <env>-dev.example.com -> myapp-<env>.fly.dev
const DEV_SUFFIX = "-dev.example.com";
const ENV_NAME = /^[a-z0-9-]+-[0-9a-f]{6}$/;   // slug-hash, as scripts/env names them

export default {
  async fetch(request) {
    const url = new URL(request.url);
    if (!url.hostname.endsWith(DEV_SUFFIX)) return new Response("not a dev hostname", { status: 404 });
    const env = url.hostname.slice(0, -DEV_SUFFIX.length);
    if (!ENV_NAME.test(env)) return new Response(`no such environment: ${env}`, { status: 404 });

    url.hostname = `myapp-${env}.fly.dev`;
    const upstream = new Request(url, request);
    upstream.headers.set("X-Forwarded-Host", request.headers.get("host") ?? "");

    const res = await fetch(upstream);
    const out = new Response(res.body, res);
    out.headers.set("X-Robots-Tag", "noindex, nofollow");   // a half-finished branch is not for search engines
    out.headers.set("X-Dev-Environment", env);
    return out;
  },
};

Cloudflare terminates TLS on the wildcard, the Worker computes the Fly hostname from the label, and Fly’s own *.fly.dev certificate covers the hop to the machine. WebSocket upgrades pass through fetch unchanged. And because the route is on the zone, everything Cloudflare does for prod is available: WAF rules, rate limits, bot fight mode, and Access.

dash.cloudflare.com/…/example.com/workers

Three branches up at once, one Worker, and what each hostname answered:

Every 200 came back with X-Robots-Tag: noindex, nofollow and X-Dev-Environment: <env> added by the Worker. The 404 is the Worker’s own, before Fly is ever asked.

Cloudflare Access is what turns “public” into “reachable by the people who should reach it”. One application on *-dev.example.com with a policy of “anyone with an @example.com email” and browser visitors get a login page once and never think about it again. That covers designers, PMs and testers on laptops. I didn’t add it for the demo; the environments were up for an hour with noindex and nothing on them but seed data.

It doesn’t cover the mobile app, because a native app can’t complete an Access login redirect. Two ways through. Access supports service tokens (a CF-Access-Client-Id / CF-Access-Client-Secret header pair) that bypass the login, and the mobile app’s dev menu can carry a pair for the dev hostnames. Or, simpler, add an Access bypass for /api/ and have Django reject any request without either a signed-in user or the environment’s ENV_TOKEN, the one env up generated for exactly this environment. Either way, the mobile side is a dev-menu screen with two fields, base URL and token, and the app a tester already has installed is now talking to that branch. No new build. The over-the-air update channel stays pointed at whatever the app was built with; the API base URL is a runtime setting, not a build-time one, and it should have been anyway.

Auth is the one shared service that’s shared on purpose. Identity isn’t environment-specific; a tester wants to log in with the same account on every branch. One dev project at the auth provider with example.com in its authorised domains covers every -dev hostname, and every environment verifies ID tokens against it. The users table in each environment’s database is a copy of the template’s users, so the tester’s account exists everywhere the template did.

Automating preview environments with GitHub Actions: events in, environments out

#

Everything above by hand is fine when you’re the only one doing it. The moment there’s a second developer it has to happen because something happened in git, not because someone remembered. The whole automation is four workflows, one repository setting, one ruleset and one GitHub environment, and every one of them maps a git event to a call of the same scripts/env the developer runs by hand. There is no CI-only path to drift.

The label, push and delete workflow

#

# .github/workflows/branch-env.yml
name: branch environment
on:
  pull_request:
    types: [labeled, unlabeled, synchronize, closed]
  delete:
  workflow_dispatch:
    inputs:
      branch: { description: branch to bring up without a PR, required: true }

concurrency:
  group: env-${{ github.head_ref || github.event.ref || inputs.branch }}
  cancel-in-progress: true

env:
  FLY_API_TOKEN: ${{ secrets.FLY_PREVIEW_TOKEN }}   # org-scoped deploy token; a separate preview org keeps it away from prod
  PG_PASSWORD: ${{ secrets.PREVIEW_PG_PASSWORD }}
  # ... the shared Redis URL and bucket credentials, the same way

jobs:
  up:
    # the gate: a PR carrying the `env` label, or someone asking for a branch by hand
    if: >-
      github.event_name == 'workflow_dispatch' ||
      (github.event_name == 'pull_request' && github.event.action != 'closed'
        && contains(github.event.pull_request.labels.*.name, 'env'))
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with: { ref: "${{ github.head_ref || inputs.branch }}" }
      - uses: superfly/flyctl-actions/setup-flyctl@master
      - id: env
        env: { BRANCH: "${{ github.head_ref || inputs.branch }}" }
        run: scripts/env up && echo "url=$(scripts/env url)" >> "$GITHUB_OUTPUT"
      - if: github.event_name == 'pull_request'
        uses: marocchino/sticky-pull-request-comment@v2
        with:
          header: env
          message: "Branch environment: ${{ steps.env.outputs.url }}"

  down:
    # label removed, labelled PR closed or merged, or the branch itself deleted
    if: >-
      (github.event_name == 'pull_request' &&
        ((github.event.action == 'closed' && contains(github.event.pull_request.labels.*.name, 'env')) ||
         (github.event.action == 'unlabeled' && github.event.label.name == 'env'))) ||
      (github.event_name == 'delete' && github.event.ref_type == 'branch')
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: superfly/flyctl-actions/setup-flyctl@master
      - run: BRANCH="${{ github.head_ref || github.event.ref }}" scripts/env down

The gate is the env label, and it’s the part I’d argue for hardest. Without one, every branch costs an app and a database clone: the branch someone pushes to back up a laptop, the spike that gets squashed tomorrow, the dependency bump a bot opened at 3 a.m. So a push alone does nothing and opening a PR does nothing. Adding env to the PR brings the environment up, every push while the label is on updates it, and removing the label, closing the PR or merging it takes it down. Opting in is still a GitHub event, not a request to a person, and it’s one click made by the one person who knows whether anyone is going to look. The sticky comment is the user interface: label the PR, a comment appears with a URL they can send to anyone, and it updates in place on every push rather than stacking. workflow_dispatch covers the branch you want up before there’s a PR, and ./scripts/env up from a laptop covers the rest.

Two other gates are worth knowing. A branch-name filter (branches: ['feature/**', 'fix/**']) needs no PR but makes every feature branch pay. “Any PR that isn’t a draft” is the most automatic, and the right choice if your team opens PRs late and only when they’re ready. I went with the label because it puts the cost decision with the person who has the context, and because it can be taken back without closing anything. One side effect I like: pull_request runs from forks don’t get secrets, so a fork can’t bring up an environment.

concurrency matters more than it looks. A run of quick pushes to one branch cancels the earlier deploys rather than racing them; two fly deploys to the same app at once end with one of them losing.

closed covers teardown for anything that had a PR, merged or not. The delete event is the backstop for a branch brought up by workflow_dispatch that never got one, and it only fires if branches actually get deleted. That’s a repository setting, Automatically delete head branches, and it’s off by default. With it on, clicking Merge closes the PR and deletes the branch, and either event alone would have run env down. The merge button is the teardown.

gh api -X PATCH repos/org/myapp -f delete_branch_on_merge=true

The FLY_PREVIEW_TOKEN is the dangerous line in the file. It can create and destroy apps, and if $APP were ever anything but a branch environment it could destroy prod. Two guards, and I’d keep both: the token is scoped to a separate Fly org so it literally cannot see the production org (for the demo I used the production org, to skip setting up billing on a new one, and relied on the second guard alone); and down() refuses any app name that doesn’t match the slug-hash pattern.

The check a branch environment can’t do for itself

#

A branch only ever sees its own migrations, so the check that matters runs on the pull request against a merge with main, and it’s the one status check main requires:

# .github/workflows/migrations-check.yml
name: migrations
on:
  pull_request:
    branches: [main]

jobs:
  check:
    runs-on: ubuntu-latest
    services:
      postgres:
        image: postgres:18
        env: { POSTGRES_PASSWORD: ci, POSTGRES_DB: ci }
        ports: ['5432:5432']
        options: --health-cmd pg_isready --health-interval 5s --health-retries 10
    env:
      DATABASE_URL: postgres://postgres:ci@localhost:5432/ci
      SECRET_KEY: ci-only
    steps:
      - uses: actions/checkout@v4
        with: { fetch-depth: 0 }
      - uses: actions/setup-python@v5
        with: { python-version: '3.14' }
      - run: pip install -r requirements.txt
      - name: Merge main into the branch (a conflict here fails the check too)
        run: |
          git config user.email ci@example.com && git config user.name ci
          git merge --no-edit origin/main
      - name: One leaf in the migration graph
        run: python manage.py makemigrations --check --dry-run
      - name: The full chain applies from nothing
        run: python manage.py migrate --noinput

Thirty seconds. It’s what turns the 0042 / 0042 collision from a release-day surprise into a red check on the second developer’s PR; the migrations section below shows it firing.

The rules on main

#

None of this holds up if branches live for a month or if someone pushes to main directly. The rules are less about protection and more about keeping environments small and their migrations current:

• main is protected and always deployable. Merge via PR with the migrations check green against the current main (strict status checks, so a PR that was green last week re-runs after main moves). Linear history, no force-push, no deletion.
• Feature branches are short. A branch open for two weeks has an environment two weeks behind main and a migration two weeks more likely to collide with someone else’s. A month-long feature merges to main behind a flag in pieces. The flag is the long-lived thing, not the branch.
• One branch, one environment, one database. Already enforced by the naming.
• Release tags are the only thing that goes to prod. The image is built once from the tag; staging and prod run that image, not a rebuild.
• A PR that touches migrations/ is reviewed by one of the other two. Below.

As a ruleset, applied once with gh api -X POST repos/org/myapp/rulesets --input ruleset-main.json:

{
  "name": "main", "target": "branch", "enforcement": "active",
  "conditions": { "ref_name": { "include": ["~DEFAULT_BRANCH"], "exclude": [] } },
  "rules": [
    { "type": "deletion" },
    { "type": "non_fast_forward" },
    { "type": "required_linear_history" },
    { "type": "pull_request",
      "parameters": { "required_approving_review_count": 1, "dismiss_stale_reviews_on_push": true,
                      "required_review_thread_resolution": true } },
    { "type": "required_status_checks",
      "parameters": { "strict_required_status_checks_policy": true,
                      "required_status_checks": [ { "context": "check" } ] } }
  ]
}

One honest wrinkle. This repository is private on GitHub’s Free plan, and on Free, branch protection and rulesets are only available for public repositories; the API answers 403 Upgrade to GitHub Pro or make this repository public. The delete_branch_on_merge setting, the environments and the workflows all work on Free. So for a small team on a private Free repo, the ruleset above is a convention: the migrations workflow still runs and still goes red, but nothing greys out the merge button. That’s a $4-a-month decision, and I’d make it the moment a second person can merge.

There’s no develop branch and no release/* branches. Git-flow was designed around integration being expensive and needing to be batched. Branch environments make integration cheap, so the batching goes.

Staging is a tag’s environment, and prod is a promotion

#

Staging reuses the whole machinery. A v* tag on main runs scripts/env up with the branch name staging/v1.42.0, which the naming turns into staging-v1-42-0-<hash>: same clone-from-template database, same suspend-when-idle machine, same hostname scheme. The difference is what comes after:

# .github/workflows/release.yml (abridged)
on:
  push:
    tags: ['v*']
jobs:
  staging:
    steps:
      - run: BRANCH="staging/${GITHUB_REF_NAME}" scripts/env up
      - run: echo "image=$(fly image show -a "myapp-$(BRANCH=staging/$GITHUB_REF_NAME scripts/env name)" --json | jq -r '…')" >> "$GITHUB_OUTPUT"
  production:
    needs: staging
    environment: production        # required reviewer on this environment = the promotion gate
    steps:
      - run: fly deploy --config fly.api.toml --image "${{ needs.staging.outputs.image }}"
      - run: fly deploy --config fly.ws.toml  --image "${{ needs.staging.outputs.image }}"
      - run: BRANCH="staging/${GITHUB_REF_NAME}" scripts/env down

The production job waits on a GitHub environment with a required reviewer. Someone looks at staging, clicks Approve, and prod receives the same image staging ran, by reference, not a rebuild. The last step destroys staging because it’s done its job. If nobody approves, the release candidate was rejected, and the sweeper removes staging when its “branch” (staging/v1.42.0 never exists on origin) fails the liveness check. Nothing gets to exist without a reason the sweeper can check.

The nightly sweeper

#

# .github/workflows/sweep-envs.yml
on:
  schedule:
    - cron: '0 14 * * *'     # 02:00 NZST
  workflow_dispatch:
jobs:
  sweep:
    steps:
      - uses: actions/checkout@v4
        with: { fetch-depth: 0 }
      - uses: superfly/flyctl-actions/setup-flyctl@master
      - run: scripts/sweep-envs

fetch-depth: 0 because the sweeper needs every remote branch and its last commit date, not the one commit Actions checks out by default. workflow_dispatch so it can be run by hand after an incident.

What a developer actually does

#

Push a branch, open a PR, and put the env label on it when they want someone to look. That’s the list. A URL appears on the PR within three minutes, updates on every push, and disappears when the PR is merged or closed. The migration check tells them before anyone else does if they’ve collided with a teammate. Nobody claims staging because there is no staging to claim, and nobody tears anything down because tearing down is what merging does.

And what the person who set it up does per branch, afterwards: nothing. The shared Postgres, Redis and bucket are provisioned once, the secrets go into Actions once, and from then on the recurring work is reading the sweeper’s log on a Monday. No ticket queue, no bot to ask, nobody to prompt.

Teardown that doesn’t trust the teardown

#

env down is the ordinary case, and the interesting part of it is who does the cleaning. The Redis keys and the bucket objects belong to the shared services, and the laptop running down doesn’t need credentials for either, because the environment’s own machine already has them and already knows its prefix. So down starts the machine if it’s suspended, runs a small cleanup command on it, and only then destroys the app and drops the database. The environment cleans up after itself:

$ BRANCH=feature/profile-website ./scripts/env down
redis: deleted 0 keys under feature-profile-webs-dcb2e7:*
bucket: deleted 278 objects under feature-profile-webs-dcb2e7/
Destroyed app myapp-feature-profile-webs-dcb2e7
DROP DATABASE

$ curl -s -o /dev/null -w '%{http_code}\n' https://feature-profile-webs-dcb2e7-dev.example.com/api/version/
530