The general technique is called test impact analysis, and testmon is a free, open-source way to get it in Python. Nothing about it depends on git: it compares against what it saw on the last run, committed or not.

I added it to my marketplace app’s Python tests this week, and the reason wasn’t speed for its own sake.

Why I added it

#

A feature request is usually small: one screen, one endpoint, one rule that should work differently. Running the whole test suite made every one of them bigger than it was. The suite is 2,248 Django tests, about three and a half minutes with four processes and 11 in one, and every edit ran all of it, though almost all of it checks things the request never touched. When something unrelated failed, or the run simply took its time, the feature waited. It was being held up by circumstances outside the feature request, and the thing holding it up was my own test suite.

The alternative wasn’t better. Skip the full run, guess which test file covers the change, and now and then guess wrong. That’s quick, but it isn’t precise, and an untested change I ship is a feature request I’ll be doing twice.

The tests I wait for should be exactly the tests the change can reach, no more and no fewer. testmon gives me that set from a record instead of a guess. The full suite still runs, in CI, where it doesn’t hold up whatever I’m working on next.

How it decides what to run

#

testmon splits each Python file into blocks: the module’s top level, each class, each function, with the bodies of nested functions cut out of the block around them. It stores a checksum of each block, never the code itself. While a test runs, coverage.py records the lines it executes, and testmon notes which blocks those lines belong to. There’s no static analysis and no guessing from imports. A test depends on exactly the blocks it ran.

Take a module like this:

# pricing.py
def subtotal(items):
    return sum(item.price * item.quantity for item in items)

def discount(total, code):
    if code == "WELCOME10":
        return total * 0.9
    return total

A test that only calls subtotal() depends on the module’s top level and on subtotal’s body. Rewrite discount() however you like and testmon won’t rerun that test, because it never executed a line of discount. Change one character inside subtotal and it runs again, even if the change can’t alter the result. testmon doesn’t try to be clever about side effects: if a block you executed changed, you run.

It also tracks what’s around the code. The record includes the Python version and the installed packages with their versions. It doesn’t watch environment variables: if the same code runs under two settings modules, name them with --testmon-env (step 8) or they share one record. Upgrade Django and everything that touched Django is affected again, which in a Django app is everything.

What it can’t see

#

testmon tracks Python that a test executes. It doesn’t track the other files your tests read: templates, JSON, text, anything else on disk. It doesn’t track services reached over the network. And it can’t pin anything that runs once before the tests start on a particular test.

In a Django project that means migrations, which run once when the test database is created, so no test’s record includes them (the wrapper below rebuilds the test database when there’s a new one); settings and start-up code, which run while pytest starts up, outside any one test; and templates and data files, since a template isn’t Python, so editing one reruns nothing that renders it.

My suite has a few more of its own. Some tests read a JSON file of fixtures that the mobile app’s tests share. Some check the built web bundle, and some check the output of a static site generated in the same repository. Change any of those and testmon will happily tell you nothing needs to run. That’s why the how-to below wraps testmon in a script that notices these files and runs everything when one has changed.

How to add it to a Django project

#

These steps assume pytest and pytest-django, and uv for the dependencies. With pip, swap uv add --dev for pip install and drop uv run.

1. Install it.

uv add --dev pytest-testmon

2. Make sure pytest can run the suite. testmon is a pytest plugin, so it does nothing under python manage.py test. pytest-django needs the settings module and the file patterns your tests use:

# pytest.ini
[pytest]
DJANGO_SETTINGS_MODULE = myproject.settings
python_files = tests.py test_*.py *_tests.py
python_classes = Test* *Tests *TestCase

If the suite has only ever run under Django’s runner, run it under pytest once and fix whatever differs before adding testmon. Otherwise you’ll be debugging two things at once.

3. Keep the record out of git. testmon writes .testmondata in the pytest root directory. It’s a SQLite database, so you’ll also see -shm and -wal files next to it, and the script below adds a stamp file. One pattern covers all of them:

# .gitignore
.testmondata*

4. Record the whole suite once. testmon can’t select anything until it has seen every test run:

uv run pytest --testmon

In a fresh checkout that first run is the whole suite in one process. Here it takes about 12 minutes for 2,248 tests. From then on, the same command runs only the affected tests. With nothing changed, it runs nothing.

Two catches. On testmon 2.2.0 and earlier, it needs pytest’s cache plugin: my first attempt ran with -p no:cacheprovider and crashed with KeyError: 'lf' (2.3.0 treats a missing --lf as unset). And on Python 3.14 or later, set COVERAGE_CORE=ctrace before you record. “The bug that looked like a green run”, further down, explains why. It’s the most important line in the script below.

Update, 7 November: testmon 2.3.0, released on 2 November, forces coverage’s C tracer itself, so on that version the export is belt and braces. Keep it if you’re on 2.2.0 or earlier, and keep the break-something check whatever the version.

5. Wrap it for the files it can’t see. This is the part that makes it safe to use every day. The script keeps a stamp file from the start of the last run. If any file testmon can’t trace is newer than the stamp, it runs everything and records afresh. A new migration only rebuilds the test database. Otherwise it lets testmon choose. A cut-down example:

#!/usr/bin/env bash
# scripts/test-changed.sh: the tests a change can affect, or all when in doubt.
#   scripts/test-changed.sh            # affected tests
#   scripts/test-changed.sh --all      # everything, re-recorded
#   scripts/test-changed.sh -k orders  # anything else goes to pytest
set -euo pipefail
cd "$(dirname "$0")/.."

# coverage's C tracer: from Python 3.14 coverage defaults to sys.monitoring,
# which reports a line only the first time a process runs it, so testmon
# would credit each line to one test. Changing this means --all.
export COVERAGE_CORE=ctrace

STAMP=.testmondata.started
full=false
rebuild=false
if [ "${1:-}" = "--all" ]; then
    full=true; shift
elif [ ! -f .testmondata ] || [ ! -f "$STAMP" ]; then
    echo "No record yet: running everything once."
    full=true
else
    # Changed since the last run started, and not traceable per test:
    # anything that isn't Python, plus settings and conftest.
    newer="$(find myapp myproject fixtures pytest.ini \
        -type f -newer "$STAMP" ! -path '*/migrations/*' \
        \( ! -name '*.py' -o -path 'myproject/settings*' -o -name 'conftest.py' \) \
        ! -path '*/__pycache__/*' | head -5)"
    if [ -n "$newer" ]; then
        echo "Changed, and not traceable per test:"; printf '  %s\n' $newer
        full=true
    fi
    # A new migration needs a fresh test database, not a full run.
    [ -n "$(find myapp -path '*/migrations/*.py' -newer "$STAMP" | head -1)" ] && rebuild=true
fi

# Stamp the start, so an edit made while the tests run counts next time.
started="$(mktemp .testmondata.run.XXXXXX)"
args=(--testmon --reuse-db)
# --create-db alone deletes the database when the run ends; with
# --reuse-db as well, the rebuilt one is kept for next time.
[ "$rebuild" = true ] && args=(--testmon --reuse-db --create-db)
[ "$full" = true ] && args=(--testmon-noselect --reuse-db --create-db)

status=0
uv run pytest "${args[@]}" -q "$@" || status=$?
# 0: passed, or nothing selected. 1: some failed (still recorded).
# 5: nothing collected at all.
case "$status" in 0|1|5) mv "$started" "$STAMP" ;; *) rm -f "$started" ;; esac
[ "$status" = 5 ] && exit 0
exit "$status"

Two details in there are worth explaining. --reuse-db is pytest-django’s flag for keeping the test database between runs, which matters once a run takes seconds: creating it again each time can take longer than the tests themselves. A full run, or one after a new migration, adds --create-db so the database has the current schema. Keep --reuse-db next to it: --create-db on its own makes pytest-django delete the database when the run ends, so the next run would build it all over again.

And when testmon deselects everything, the run exits 0, so “nothing changed” counts as a pass. Exit code 5 is pytest’s “no tests collected”, which the script treats as fine too.

Notice what’s not in the list: pyproject.toml and the lock file. My first version ran everything whenever the lock file changed, which made the slow fallback fire on most days and the whole thing worse than before. testmon already tracks installed package versions itself, so it doesn’t need the help.

6. Give it its own database and cache. If other things run tests on the same machine, like a second terminal, a coding agent or manage.py test, point the script at its own test database and its own cache number so a kept database never meets someone else’s run:

export DATABASE_URL="postgresql://myapp:myapp@localhost:5432/myapp_changed"
export REDIS_URL="redis://localhost:6379/8"

7. Empty that cache before each run. I learnt this one the hard way. Subset runs started failing with KeyErrors that full runs didn’t. Some category labels are cached with no expiry, and a test database kept from one run met a cache still warm from an earlier one. Clearing the cache at the top of the script fixed it:

uv run python -c "import os, redis; redis.Redis.from_url(os.environ['REDIS_URL']).flushdb()"

That’s testmon doing what its docs say it will: running a different subset each time exposes tests that only passed because of what ran before them. The fix belongs in the test setup, not in testmon.

8. Know the flags for when selection is wrong.

• --testmon-noselect runs every test but puts the likely failures first. testmon switches to it by itself when you also pass -k, -m, --lf or a test’s node ID.
• --testmon-forceselect runs only affected tests that also match your -k or -m.
• --testmon-nocollect uses the existing record but doesn’t add to it. It’s forced under a debugger or when coverage is already running.
• --no-testmon turns it off for one run, for when it’s in addopts.
• --testmon-env=<name> keeps separate records in one file, for a suite you run under two settings modules.

If the project has a JavaScript side, Jest has its own version built in. jest --changedSince=main runs the tests that import anything changed since main, following imports rather than coverage. I added it to the app’s package.json as npm run test:changed so both halves have the same habit. An edit to one utility module selected 7 of the app’s 121 test files.

What it saves

#

I timed the wrapper once it was in place, start-up included, after fixing the bug in the next section:

For an everyday edit that’s about 25 times faster than the parallel suite, and when nothing has changed it’s about two hundred times faster.

The record says the same thing more broadly. Of the 307 non-test Python files the suite runs, a change to the median one reaches 22 tests, about 8 seconds of test time. A few core modules sit at the other end: almost every test runs them, so a change there reaches around 1,500 tests, and testmon reruns most of the suite. That’s the right answer, not a failure. Those figures are upper bounds per file: testmon works per block, so changing one function selects only the tests that ran that function.

The cost shows up when the wrapper falls back. A full run is the same single-process recording as the first one, about 12 minutes, which is slower than the parallel suite it replaces. It’s single-process only because I haven’t tried testmon under pytest-xdist yet, which it has supported since version 1.4; that’s the obvious next thing to measure. That’s why the list of files that trigger it should be as short as it can honestly be.

The bug that looked like a green run

#

My first numbers were better than these: one edit, 6 tests, 3.3 seconds. They were wrong, and nothing on screen said so.

From Python 3.14, coverage.py uses a new core by default, built on the interpreter’s sys.monitoring. That core reports a line only the first time a process runs it. testmon builds its record of which test ran which line from coverage, so each line was credited to the first test that happened to run it, and to no other. Edit a line that eight tests share, and testmon selects one of them.

It was caught on a second, smaller project of mine that copied this setup. Part of adopting it there was breaking a line on purpose and comparing the results, and the numbers didn’t match. Back in this suite I ran the same check. I broke one line in the home-screen banner code, the title of a slide. Plain pytest reported 8 failing tests. testmon selected 4. The other 4 broken tests never ran, and nothing in the output said they’d been left out.

The fix is one line, export COVERAGE_CORE=ctrace, which puts coverage back on its classic C tracer. A different tracer means a different record, so I re-recorded: 12 minutes 3 seconds for 2,248 tests. Afterwards, the same kind of breaking edit ran all 11 tests it broke.

Update, 7 November: testmon 2.3.0, released on 2 November, now sets the C tracer itself, in process and for subprocesses, so a fresh install no longer needs the export. The check in the next paragraph still does.

That’s the lesson I’d pass on before anything else here. A test selector that selects too little looks exactly like success: fewer tests, a faster run, all green. So when you set one up, break something shared on purpose, run the plain suite and the selector, and check they fail the same tests. Then do it again whenever Python or coverage.py changes version.

Where a coding agent comes in

#

I don’t make every change in that repository myself any more. A coding agent, Claude Code in my case, makes a lot of the small ones, and a good agent runs the tests after each edit. Faced with an 11-minute suite, it has the same two bad options I had: wait, or guess which test file matters.

So the repository’s agent instructions now say the wrapper script is the default way to run tests while working, with --all for a full run. The agent gets the same precise set of tests I do, after every edit, in seconds. It’s also why step 6 gives the script its own database and cache: on a busy day there’s more than one session running tests on the same machine, and a kept database must never meet somebody else’s run.

None of this needs anything AI-specific from testmon. It’s the same record and the same selection. It just matters more when whatever is making the changes makes a great many of them.

Why CI got something coarser

#

testmon has no CI mode. It behaves the same on a runner as on a laptop, and its author writes that sharing or caching .testmondata across CI runners is far less effective than their hosted service, testmon.net. A cached record would also be blind in all the ways above, and my CI runs the suite with Django’s runner in four processes, so I left testmon out.

What CI got instead is the same idea at file level. The checks workflow now starts with a small job that diffs the push and decides which half of the repository it touched: the Python API, the mobile app, both or neither. Before this, CI ran both the Python and the app jobs on every push, docs-only ones included. Now a docs-only push runs nothing, and a push to the app alone skips the Django suite. I dry-ran the job against five real commits: the docs-only one ran nothing, the app-only one skipped the Python job, and the backend-only one skipped the app job. A cut-down example:

jobs:
  changes:
    runs-on: ubuntu-latest
    outputs:
      python: ${{ steps.diff.outputs.python }}
    steps:
      - uses: actions/checkout@v7
        with:
          fetch-depth: 0
      - id: diff
        env:
          BEFORE: ${{ github.event.before }}
          HEAD: ${{ github.sha }}
        run: |
          FILES="$(git diff --name-only "$BEFORE" "$HEAD")"
          # Python's inputs include the non-Python files its tests read.
          PY='^(myapp/|myproject/|fixtures/shared\.json$|web/dist/|pyproject\.toml$|uv\.lock$)'
          if printf '%s\n' "$FILES" | grep -qE "$PY"; then
            echo "python=true" >> "$GITHUB_OUTPUT"
          else
            echo "python=false" >> "$GITHUB_OUTPUT"
          fi

  python:
    needs: changes
    if: needs.changes.outputs.python == 'true'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - run: uv run python manage.py test --parallel 4 --noinput

The real one also runs everything when there’s no range to compare: a new branch, a force push or a manual run.

The blind spot is the same one in a different form. testmon can’t see a JSON file, and neither can a path filter unless you list it. So the pattern names the shared fixtures, the built bundle and the generated site explicitly, with a comment asking whoever edits it to keep the list honest. The file-level version is cruder, but it fails in the safe direction: if anything on the Python side changed, the whole suite runs.

Is it worth it?

#

For a suite that takes minutes, yes. The real cost is what it can’t see, and that’s what the wrapper script is for. Even so, the laptop loop is where testmon belongs. CI still runs the whole suite whenever the Python side changes, and that’s the run I trust before a deploy.

If you came here to run Docker Compose in GitHub Actions: I did that for two years and stopped. CI now declares the same Postgres and Valkey images as services: and runs the same uv run … and npm run … commands straight on the runner; Compose stays what a developer runs. The rest is why, with the files.

In 2024 I wrote Use Docker Compose for CI/CD Instead of Platform-Specific YAML, which said to keep the build logic in Compose and let the CI tool do nothing but call it. Two years on, I went back through every repository I ship from. Four have workflows, each with one committer, me, and none of those workflows calls docker compose any more. The principle held; the mechanism didn’t. Here’s what replaced it, with cut-down example versions of the files, the step timings GitHub recorded, and the parts that are less automated than a CI/CD diagram would like.

Every pipeline I run, in one table

#

In textbook terms the Hugo sites have continuous deployment, and the app (a marketplace and community app I build) has continuous integration plus one-command releases, which is continuous delivery: the checks tell me something is wrong, but they don’t stop me deploying. That’s deliberate. An OTA update reaches every phone on the binary version it targets, so publishing one should be a decision rather than a side effect of a merge, and the API is where the database migrations run. My other two Django apps have no workflows at all.

The CI half: one workflow, two jobs

#

The app’s CI is one file, .github/workflows/checks.yml. Until 11 October there wasn’t any. This is a trimmed example of its shape:

# Lint, types and tests on every push and pull request to main.
# Nothing here deploys.
name: Checks

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

concurrency:
  group: checks-${{ github.ref }}
  cancel-in-progress: true

jobs:
  python:
    runs-on: ubuntu-latest
    timeout-minutes: 30
    services:
      postgres:
        image: postgres:17
        env:
          POSTGRES_USER: myapp
          POSTGRES_PASSWORD: myapp
          POSTGRES_DB: myapp
        ports: ["5432:5432"]
        options: >-
          --health-cmd "pg_isready -U myapp"
          --health-interval 5s --health-timeout 5s --health-retries 20
      valkey:
        image: valkey/valkey:8-alpine
        ports: ["6379:6379"]
    env:
      DEBUG: "True"
      DATABASE_URL: postgresql://myapp:myapp@localhost:5432/myapp
      REDIS_URL: redis://localhost:6379
      PYTHON_JIT: "0"   # match production
    steps:
      - uses: actions/checkout@v7
      - uses: astral-sh/setup-uv@v10
        with:
          enable-cache: true
          prune-cache: true
      - run: uv sync --locked
      - run: uv run ruff check .
      - run: uv run mypy myapp/
      - run: uv run python manage.py test --parallel 4 --noinput

  app:
    runs-on: ubuntu-latest
    timeout-minutes: 20
    defaults:
      run:
        working-directory: frontend
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-node@v7
        with:
          node-version: 24
          cache: npm
          cache-dependency-path: frontend/package-lock.json
      - run: npm ci
      # Expo writes this on `expo start` and it is gitignored,
      # so a fresh checkout doesn't have it.
      - run: printf '/// <reference types="expo/types" />\n' > expo-env.d.ts
      - run: npm run typecheck
      - run: npx jest --ci

The two jobs run side by side. Across the 35 green runs so far, the app job’s median is about seven minutes and the Python job’s about twelve and a half.

The services: block is where the Compose file went, which is the next section. The environment is a few variables and a flag; everything else falls back to the development default docker compose up gives a developer, so the tests need no secrets. I checked by running part of the suite with exactly that environment and nothing else.

CI runs the same interpreter as my laptop and the production image, uv’s own build of Python 3.15.0, because uv sync reads .python-version and downloads exactly that build. (My real file also passes setup-uv a python-version-file input, which does nothing: setup-uv v6 has no such input, and every run carries a warning saying so. I only noticed while writing this.)

Matching interpreters earned their keep eight days after the workflow went in. On 19 October I switched on 3.15’s JIT compiler, and the first run with it failed a test: a plain yield from, on a code path the app takes on every save, raised AttributeError: 'list' object has no attribute 'send'. The same commit with the JIT off passed. My Mac is arm64 and never hit it; the runner is x86_64, like the production machines. The JIT is off again, in CI and in the Dockerfile, and the bug never reached production. It’s python/cpython#155823, fixed upstream on 13 October, after 3.15.0 shipped.

The very first run failed too, for a duller reason. Expo writes expo-env.d.ts when you run expo start, and the file is gitignored, so every working copy I’d used had one and a fresh checkout didn’t. TypeScript reported five errors that had never appeared locally. The step that writes the file is the fix.

concurrency cancels a run when a newer push to the same branch arrives, which is how all twelve cancelled runs so far ended. A pre-commit hook runs the linter, plus the type checker when Python or TypeScript is staged, so most mistakes never reach a push.

Where the Docker Compose file went

#

Here is the part of the app’s docker-compose.yml that CI cares about, as an example. The real one has more services and a lot more environment.

services:
  valkey:
    image: valkey/valkey:8-alpine
    ports:
      - "6379:6379"

  # Host port 55432, not 5432, so it can't collide with a local Postgres.
  postgres:
    image: postgres:17-alpine
    environment:
      - POSTGRES_USER=myapp
      - POSTGRES_PASSWORD=myapp
      - POSTGRES_DB=myapp
    ports:
      - "55432:5432"
    volumes:
      - postgres_data:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U myapp -d myapp"]
      interval: 5s
      retries: 10

  api:
    build: .
    command: python manage.py runserver 0.0.0.0:8000
    volumes:
      - .:/app
    ports:
      - "8000:8000"
    environment:
      - DATABASE_URL=postgresql://myapp:myapp@postgres:5432/myapp
      - REDIS_URL=redis://valkey:6379
    depends_on:
      postgres:
        condition: service_healthy
      valkey:
        condition: service_started

volumes:
  postgres_data:

Put it next to the services: block in checks.yml and it’s the same two images: Postgres 17 (the Alpine build here, the Debian one in CI) and valkey/valkey:8-alpine. That’s the relationship now. The Compose file is what a developer runs; CI declares the same images as service containers and runs the same commands a developer runs, uv run … and npm run …, straight on the runner.

Against the 2024 post, the part I cared about most held: the CI tool is thin, and the checks and deploys are commands I can run myself. What didn’t survive is CI calling Compose.

The app’s CI doesn’t build the Docker image at all; Fly’s builder does that once, at deploy time, and CI only has to say whether the code is good, which with a cached toolchain takes a six-second install. The Compose file also bind-mounts the working copy and seeds data, which a test run shouldn’t, so CI would have needed a Compose file of its own: a second description of the same services, kept in step by hand.

And the 2024 example had a bug I didn’t notice at the time. docker compose build builds the image and never runs the service’s command, so the npm test in that file never ran in CI. The workflow went green whether the tests passed or not.

The risk that post worried about most, switching CI vendors, never came up. The move that did happen was at the other end: this blog and the client’s site moved hosts to Cloudflare Pages, and for the blog that took one new 45-line workflow, because the build was already one command. And the three-hundred-line workflow I warned about? I have one, pyhtml2text’s. It’s long because the wheels have to be built on Linux, macOS and Windows runners, for several architectures each, which makes it the one job where platform-specific YAML is the right tool.

Caching: read the step timings first

#

GitHub keeps the duration of every step, and gh api repos/<owner>/<repo>/actions/runs/<run-id>/jobs hands it over. These are from the app’s latest green run, on 20 October:

Both dependency caches are on, and both are the cheap part. The uv cache GitHub keeps for the repository is about 200 KB, because setup-uv prunes whatever uv can simply download again; the npm one is nearly 400 MB. The time is in the tests, which already run four ways in parallel. If I wanted the run faster, I’d change the suite, not the cache.

The blog is the opposite case. Hugo’s processed images live in the gitignored resources/_gen, so every fresh runner re-encoded every cover, body image and placeholder. The workflow further down restores them with actions/cache, keyed on a hash of the pictures and falling back to the newest older cache, since Hugo only processes what’s missing. In the last eight runs before the cache, the build step took 33 to 76 seconds; in the 26 since, 6 to 20, with one exception at 94, even though the same commit added placeholders and a second card size.

The app’s image builds on Fly’s builder, with the dependency files copied before the code and uv’s download cache in a BuildKit cache mount (RUN --mount=type=cache,target=/root/.cache/uv). The bigger fix wasn’t a cache. The old .dockerignore was a blocklist, and it missed a folder of extra git worktrees, each with its own dependencies and build output, so 7.7 GB across 277,000 files went to the builder on every deploy. Rewritten as an allowlist, the context is about 40 MB. An example of the pattern:

# Exclude everything, then let back in only what the Dockerfile COPYs.
*

!pyproject.toml
!uv.lock
!manage.py
!myapp
!static

# Rules match the whole relative path and the last match wins,
# so these re-excludes apply inside the allowed directories too.
**/__pycache__/
**/*.pyc
**/.DS_Store
**/*.log

Secrets, in as few places as possible

#

Secrets live in three places. GitHub Actions secrets are for what workflows do, and the two Hugo repositories have exactly one each, CLOUDFLARE_API_TOKEN. Fly secrets are the app’s runtime configuration, set with flyctl secrets set, and they never pass through Actions. And a gitignored .env on my laptop holds the token the deploy script uses to purge Cloudflare’s cache.

PyPI needs no secret at all. pyhtml2text uses trusted publishing: PyPI accepts the short-lived OIDC token GitHub issues to the workflow, so the release job asks for id-token: write and the repository’s secrets page is empty. There’s nothing to leak and nothing to rotate.

Tokens get the least they need. The app’s Fly token in GitHub serves a scheduled check that only lists machines, so it’s created with fly tokens create readonly. The Pages deploys ask for contents: read and deployments: write, and nothing more.

Inputs go through env:, never straight into a script. The OTA workflow takes free-text release notes. Written as ${{ inputs.notes }} inside run:, the text would be pasted into the script before bash reads it, so a note with a quote and a semicolon in it would become code. Passed as an environment variable, it stays data. An example of the pattern:

      - name: Publish
        env:
          OTA_API_TOKEN: ${{ secrets.OTA_API_TOKEN }}
          INPUT_PLATFORM: ${{ inputs.platform }}
          INPUT_ROLLOUT: ${{ inputs.rollout }}
          INPUT_NOTES: ${{ inputs.notes }}
        run: |
          args=("$INPUT_PLATFORM")
          [ -n "$INPUT_ROLLOUT" ] && args+=(--rollout "$INPUT_ROLLOUT")
          [ -n "$INPUT_NOTES" ] && args+=(--notes "$INPUT_NOTES")
          ./scripts/publish-ota.sh "${args[@]}"

Some keys stay off GitHub altogether. One scheduled report needs a key with admin rights over the app’s sign-in backend, so it moved from a scheduled workflow to a scheduled job on my Mac. The workflow is still there as a manual fallback, and leaves out the sections that need that key.

Deploys: on every push, by script, and on a tag

#

The Hugo sites deploy on every push

#

An example version of the blog’s workflow, with the first-run step that creates the Pages project left out:

name: Deploy to Cloudflare Pages

on:
  push:
  workflow_dispatch:

env:
  HUGO_VERSION: "0.165.0"
  PAGES_PROJECT: my-site

concurrency:
  group: deploy-${{ github.ref }}
  cancel-in-progress: true

jobs:
  deploy:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      deployments: write
    steps:
      - uses: actions/checkout@v7
        with:
          submodules: true

      - uses: peaceiris/actions-hugo@v3
        with:
          hugo-version: ${{ env.HUGO_VERSION }}
          extended: true

      # Processed images are gitignored; restore them so Hugo only
      # re-encodes what is new. An older cache still helps.
      - uses: actions/cache@v6
        with:
          path: resources/_gen
          key: hugo-resources-${{ hashFiles('content/**/*.jpg', 'content/**/*.png', 'assets/img/**') }}
          restore-keys: |
            hugo-resources-

      # A multilingual site has no root 404.html; without one, Pages
      # answers unknown paths with index.html and a 200.
      - name: Build
        run: |
          hugo --gc
          cp public/en/404.html public/404.html

      # ...

      - name: Deploy
        uses: cloudflare/wrangler-action@v4
        with:
          apiToken: ${{ secrets.CLOUDFLARE_API_TOKEN }}
          accountId: ${{ vars.CLOUDFLARE_ACCOUNT_ID }}
          command: pages deploy public --project-name=${{ env.PAGES_PROJECT }} --branch=${{ github.ref_name }}

--branch=${{ github.ref_name }} makes main production and gives every other branch a preview URL. The client site’s workflow mirrors this one, plus an rm -rf public before the build, because its public/ is committed and removed pages would otherwise linger in the upload.

Hugo is pinned here and nowhere else. In October a package-manager upgrade moved my laptop to 0.168, one past the newest version the theme supports. The deploys stayed on 0.165 and never noticed.

And the build is the test. On 14 September the preview deploy for one branch failed three times running: a post on it linked, through Hugo’s ref, to one that was still a draft. CI builds what production builds, drafts excluded, so the link pointed at nothing and Hugo stopped with REF_NOT_FOUND. main carried on deploying in the meantime.

The app’s API deploys from a script

#

The core of the deploy script, as an example:

IMAGE_LABEL="deploy-$(git rev-parse --short HEAD)-$(date +%Y%m%d%H%M%S)"

# Build first, then wake a spare machine, then roll out exactly that image.
flyctl deploy --config fly.toml --build-only --push --image-label "$IMAGE_LABEL"
curl -fsS --retry 5 https://myapp.fly.dev/health > /dev/null
flyctl deploy --config fly.toml --image "registry.fly.io/myapp:$IMAGE_LABEL"

And the matching lines in fly.toml:

[deploy]
  # max_unavailable only counts machines that are running,
  # which is why the script wakes one first.
  strategy = "rolling"
  max_unavailable = 1

The image is built once, pushed to Fly’s registry under a label made of the commit and the time, and deployed by that label, so what rolls out is exactly what was built.

Waking a spare first is what makes it zero-downtime. max_unavailable = 1 only counts running machines, and the app keeps none running when it’s idle (a suspended machine resumes in about a second), so a lone awake machine used to be replaced in the same batch as the sleeping ones, with nothing warm behind it. On 28 September a probe against the origin saw 7 timeouts and a 7.3-second gap during a plain flyctl deploy, and no failures in 2,514 requests during the scripted deploy, which builds first and has a machine awake before the rollout starts.

The rest of the script marks the deploy in the team chat channel the error alerts go to, so a burst of errors can be read against what just changed, purges Cloudflare’s cache and warms the machines. The web front end skips the image entirely; that’s in Deploy Frontend and Backend Separately, Without Redeploying the API.

Migrations aren’t a pipeline step. Fly’s release_command hung for me, so the first machine to boot on a new image takes a lock and migrates while the others wait. Every migration therefore has to work with the code that’s still running: add a column in one deploy, stop using it in the next, drop it in a third.

Nothing in the script needs my laptop as such, but a couple of local-only files keep it there: the Cloudflare credentials, and a record of what was last deployed, which a fresh runner wouldn’t have. It will also deploy uncommitted changes, and says so in the chat channel, where the web release script refuses a dirty tree.

In Preview Environments per Branch on Fly.io I wrote that doing this by hand is fine while you’re the only one doing it, and that from the second developer on it has to happen because something happened in git. The app is still at one developer; the branch-environment workflows from that post live on the branch I built them on.

The OTA workflow is manual-only because publishing should be a decision, and because the update service refuses a release while a partial rollout is open, which an on-push trigger would hit during every canary. It has never run. Every OTA update so far has gone out from my laptop through the same script the workflow calls, which is the case for thin workflows in one line: however a release starts, it’s the same script.

pyhtml2text releases on a tag

#

The release job, with the package name made generic:

  release:
    runs-on: ubuntu-latest
    if: ${{ startsWith(github.ref, 'refs/tags/') || github.event_name == 'workflow_dispatch' }}
    needs: [linux, macos, windows, sdist]
    environment:
      name: pypi
      url: https://pypi.org/p/mypackage
    permissions:
      id-token: write       # trusted publishing and signing
      contents: write
      attestations: write
    steps:
      - uses: actions/download-artifact@v8
      - uses: actions/attest@v4
        with:
          subject-path: 'wheels-*/*'
      - uses: astral-sh/setup-uv@v10
        if: ${{ startsWith(github.ref, 'refs/tags/') }}
        with:
          prune-cache: true
      - name: Publish to PyPI
        if: ${{ startsWith(github.ref, 'refs/tags/') }}
        run: uv publish --trusted-publishing always 'wheels-*/*'

The release job waits for every build job, and only a tag publishes; a manual run builds and attests but uploads nothing. actions/attest records which workflow built each wheel, and uv publish --trusted-publishing always uploads them with the OIDC token rather than a password. The package itself is in A Closed Pull Request, a Fork and a PyPI Package.

Scheduled workflows are best effort

#

The app has one scheduled workflow, a check that Fly hasn’t started more machines than the baseline, which would otherwise first show up on the bill. It alerts by failing, since GitHub emails the owner about a failed scheduled run.

It’s set to cron: '*/15 * * * *', 96 runs a day. Over the past two weeks it ran between three and eight times a day, with a median gap of about three hours forty. The only explanation GitHub’s documentation offers is load: scheduled runs can be delayed when Actions is busy, and dropped when it’s busy enough. For a sustained scale-out, a few hours is fine. For anything that has to fire within minutes, a GitHub cron isn’t a monitor.

What I’d ask of a pipeline

#

If you’re building one, or looking at one you already have:

1. If GitHub Actions were down for a day, could you still release? If not, your release logic lives in YAML, and the 2024 post still applies.
2. Does CI run the interpreter, the CPU architecture and the database major version that production runs? Mine caught a compiler bug only because the runner and production are both x86_64 and my laptop isn’t.
3. Does a fresh checkout pass? Generated, gitignored files are the usual reason it doesn’t.
4. Where does a run’s time actually go? Read the step timings before you add a cache.
5. For each secret, how many places does it live, and does it need to be in GitHub at all?

Picture a small team working on one app. Two developers, or five; not specialists, but people who each take a feature from the database migration through the API to the screen in the mobile app, end to end, on their own branch. It’s a good way to run a team, and from the second developer onward it has one recurring failure: the message “Is anyone on staging?”

Behind that message is a developer who has finished a feature and wants a tester or a designer to see it, and can’t, because staging is running the second developer’s branch, and dev is running the third’s, which broke login two days ago and hasn’t been redeployed since. A handful of people, two shared environments, and a chat thread doing the scheduling.

That isn’t a people problem. It’s a topology problem. Every developer is treating the environment as the place where their branch becomes real, so the environment is doing the job the merge is supposed to do, one branch at a time, and everyone else is queueing for it.

I described the production shape of my stack in My Current Stack: Django on Fly.io’s cheapest machines, Postgres behind a connection pooler, Redis for the cache and the Channels layer, a separate WebSocket app, a background worker, an S3-compatible bucket for files, a hosted auth provider, Cloudflare in front, and a mobile app talking to all of it. This post is how I turned that stack into something where any branch gets its own public, HTTPS-fronted copy of the whole thing a couple of minutes after its developer puts one label on the PR, with nobody else asked and nobody provisioning it, and without a second Postgres cluster, a second Redis or a second bucket. The infrastructure stays shared. What changes is that every shared resource learns to be sliced by a name, and the name comes from the branch. Nothing in it assumes a team size: it’s the same design for two developers as for twenty, and what grows is the app list, which the sweeper handles. I built it, ran three branches at once, broke it in two places I hadn’t predicted, and tore it down, all in an afternoon; the outputs below are from that run.

Why one shared environment can’t scale past one developer

#

A shared dev environment has an assumption baked in: that there is one “current version of the code” that everyone wants to look at. That’s true for one developer, or for a team working a single stream of changes in lockstep. It stops being true the moment the second developer starts a feature on a different timeline, and with end-to-end ownership that’s every feature. From then on, everything the environment holds is contended:

• The deployed code. Whoever deployed last wins. Everyone else’s work is invisible until they redeploy, which undoes the first person’s.
• The database schema. This is the one that bites hardest when every developer owns their feature’s data model. The first branch adds a column. The second, deployed an hour later, doesn’t know about it and its ORM starts failing on inserts. Or both branches have a 0042_* migration, the shared database ends up in a state that matches nobody’s migration history, and someone resets it, deleting the test data the third developer spent an afternoon setting up.
• The cache. One branch’s serialiser change is cached under the same Redis keys another branch’s code reads. The bug reports that produces are baffling and cost a morning each.
• Third-party integrations. One OAuth callback, one webhook URL, one push certificate. Whoever’s branch is deployed gets the events.
• The tester. They can only test what’s deployed, so they test one feature at a time in the order the environment happened to be claimed, not the order the features are ready.

The team responds with a booking system: a chat thread, a pinned message, a bot with /claim staging. That’s a lock, and a lock on the only place your work can be seen is a lock on delivery rate. With two developers it’s an irritation. With three it’s tolerable on a good week. Add a fourth, or have one feature hold staging for three days of back-and-forth with a designer, and it stops being tolerable.

The first fix people reach for is a third shared environment, “dev2” or “uat”. That buys a few weeks. The queue just has two lanes.

The real fix is to notice that dev and staging were never environments. They were the current state of someone’s branch, with a URL. If that’s what they are, the right number of them is the number of branches the team cares about right now, and the right lifetime is the lifetime of the branch.

The model: the branch is the environment

#

Here’s the target state.

Read the Created by and Destroyed by columns again: every entry is a git event or a clock. Git is the control plane. The set of environments that should exist is a function of the set of branches that exist, and the automation’s only job is to keep reality matching it: a label on the PR creates, every push after that updates, closing or merging the PR destroys, and a nightly job reconciles whatever the first three missed. Nor does every branch get one: the label is the gate, one click by the developer, so a backup push or a bot’s dependency bump costs nothing. Nobody provisions an environment. Not a DevOps engineer with a ticket, not a bot you ask in chat, not an AI agent you prompt each time. The tenth developer’s first branch gets an environment the same way the first developer’s did, by labelling its PR, and if a person has to be asked, it hasn’t scaled; the queue has just moved.

And the rule that makes it work: every resource an environment touches is named after the branch, deterministically. Not after the developer, not after a ticket, not after a PR number that doesn’t exist until someone opens one. After the branch, because the branch is what developers actually work with, and because a deterministic name means the second run of ./scripts/env up on the same branch finds the environment it already made rather than making another.

The name is a slug for humans and a short hash for uniqueness:

branch=$(git rev-parse --abbrev-ref HEAD)              # feature/payments-retry
slug=$(printf '%s' "$branch" | tr '[:upper:]/_ ' '[:lower:]---' | tr -cd 'a-z0-9-' | cut -c1-20)
hash=$(printf '%s' "$branch" | sha256sum | cut -c1-6)
ENV="${slug%-}-${hash}"                                 # feature-payments-retr-3f9a1c

feature-payments-retr-3f9a1c is the environment. That string becomes the Fly app name, the Postgres database name (with hyphens turned to underscores), the Redis key prefix, the bucket’s object prefix, the error tracker’s environment, and the left-hand label of the hostname. Anyone who knows the branch name can compute it. Nothing has to be looked up.

That last sentence is the design. There is no registry of environments and no coordinator. env up, env down, the Cloudflare Worker that routes hostnames and the nightly sweeper each recompute the name from the branch independently and agree by construction. Cloudflare in particular is not coordinating anything: the Worker is a pure function from hostname to Fly app, and it never learns that an environment was created or destroyed. If the app exists the request lands; if it doesn’t, Fly answers with its own error.

Three things fall out of this.

First, “who’s on staging” stops being a question. There is feature-payments-retr-3f9a1c-dev.example.com and fix-login-redirect-8b21e0-dev.example.com, and each belongs to whoever is on that branch.

Second, staging becomes an honest rehearsal of prod rather than a dumping ground. It’s built from the exact image that will go to prod, against a database cloned from a prod-shaped snapshot minutes earlier. If it works there, the only variable left in prod is prod’s data.

Third, each developer gets power without getting root on anything shared. The blast radius of a branch environment is that branch environment. A fourth hire on their first day can deploy whatever they like to their branch, and the worst outcome is that their branch is broken.

fly.io/dashboard/my-org

Shared physically, separated logically

#

The tempting version of “an environment per branch” is a full copy: a Postgres cluster each, a Redis each, a bucket each. It’s clean and it’s the wrong trade. A Fly Postgres cluster is a machine plus a volume plus a restore; a Redis is another machine; you’d be waiting minutes for each environment and paying for a dozen idle databases. The sweeper would have four kinds of thing to clean up instead of one.

Instead, each shared service is provisioned once for the whole preview org, sized like a small prod, and every environment gets a slice of it keyed by $ENV. This is the table I’d want on the wall:

Two of those need a word.

Postgres: database, not schema. The other way to slice one cluster is a schema per environment, with search_path set per connection. Django will do it if you push a -c search_path=... through the connection options, but a connection pooler in transaction mode discards startup options unless you tell it not to, and then every connection in the pool has to belong to one environment anyway. It’s a source of “why is my query hitting the wrong tables” bugs that a database per environment simply doesn’t have. A database is a hard boundary, CREATE DATABASE ... TEMPLATE makes one in seconds, and DROP DATABASE makes it go away with nothing left behind. Branch environments connect to Postgres directly on port 5433 rather than through the pooler, the same way the background worker does in prod; the pooler earns its keep at production concurrency, which a branch environment never sees.

Secrets: generated, not shared. It’d be easy to have one set of dev secrets and copy it into every environment. Don’t. The env up script generates a fresh SECRET_KEY (session signing), ENCRYPTION_KEY (anything encrypted at rest) and ENV_TOKEN (the bearer token the mobile app presents, more on that below) for each environment and stores them only in that Fly app’s secret store. It means a session cookie from one branch is worthless on another, a leaked preview token opens one preview, and the developer never sees the values unless they go looking. If you keep secrets in a separate secrets manager rather than only in Fly, the same rule applies: the path is myapp/preview/$ENV/, and the sweeper deletes the path with the environment. The only secrets that are shared across environments are the credentials for the shared infrastructure itself (the Postgres admin URL, the Redis URL, the bucket key), and those live in CI, not in any environment.

The Redis row is the easiest to doubt, so here is the test. Set a cache key from inside one environment, read it from the other, then list every key in the shared Redis:

$ fly ssh console -a myapp-feature-profile-tagl-fa8832 -C "python manage.py shell -c \"from django.core.cache import cache; cache.set('who-am-i', 'tagline-branch', 600)\""
$ fly ssh console -a myapp-feature-profile-webs-dcb2e7 -C "python manage.py shell -c \"from django.core.cache import cache; print(cache.get('who-am-i'))\""
None

$ # every key in the shared Redis
  feature-profile-tagl-fa8832:1:who-am-i
  ...

Same Redis, same key name, the other environment sees nothing.

Everything else in this post is the script that stamps $ENV onto those eight rows, and the rules that keep it honest.

./scripts/env up

#

This is the whole idea, as a script a developer runs from their branch. CI runs the same script; there’s no separate CI path to drift. It needs fly and git on the laptop and nothing else: SQL goes to the Postgres app over fly ssh, and the environment’s Redis keys and bucket objects are removed by the environment’s own machine before it’s destroyed, so the laptop never holds the shared Redis or bucket credentials beyond a gitignored env file. Trimmed to the lines that matter:

#!/usr/bin/env bash
# scripts/env up|down|name|url: one environment per branch on shared preview infrastructure
set -euo pipefail
ORG=${PREVIEW_ORG:-myapp-preview}; PG_APP=${PREVIEW_PG_APP:-myapp-preview-pg}; DOMAIN=${BASE_DOMAIN:-example.com}

branch=${BRANCH:-$(git rev-parse --abbrev-ref HEAD)}
[[ "$branch" == "main" ]] && { echo "main has staging, not a branch environment" >&2; exit 1; }
slug=$(printf '%s' "$branch" | tr '[:upper:]/_ ' '[:lower:]---' | tr -cd 'a-z0-9-' | cut -c1-20); slug=${slug%-}
ENV=${ENV_OVERRIDE:-${slug}-$(printf '%s' "$branch" | shasum -a 256 | cut -c1-6)}   # the sweeper passes the name
APP="myapp-${ENV}"; DB="env_${ENV//-/_}"; HOST="${ENV}-dev.${DOMAIN}"

pg() { fly ssh console -a "$PG_APP" -q -C "psql postgres://postgres:${PG_PASSWORD}@localhost:5433/postgres -At -c \"$1\""; }
exists() { fly apps list --org "$ORG" --json | grep -q "\"Name\": *\"$APP\""; }

up() {
  echo "== $branch -> $ENV"
  exists || fly apps create "$APP" --org "$ORG"                         # idempotent
  [[ "$(pg "SELECT 1 FROM pg_database WHERE datname='$DB'")" == "1" ]] || pg "CREATE DATABASE $DB TEMPLATE seed_template"

  have=$(fly secrets list -a "$APP" --json | grep -o '"Name": *"[^"]*"' | cut -d'"' -f4 || true)
  gen() { grep -qx "$1" <<<"$have" || echo "$1=$(openssl rand -hex 32)"; }   # generated once, never rotated by a later `up`
  { gen SECRET_KEY; gen ENCRYPTION_KEY; gen ENV_TOKEN
    echo "ENV_NAME=$ENV"; echo "ALLOWED_HOSTS=$HOST,$APP.fly.dev"
    echo "DATABASE_URL=postgres://postgres:${PG_PASSWORD}@${PG_APP}.flycast:5432/${DB}"
    # ... plus the shared Redis URL and bucket credentials
  } | fly secrets import -a "$APP" --stage

  fly deploy -a "$APP" --config fly.preview.toml --remote-only --ha=false \
    --build-arg GIT_SHA="$(git rev-parse --short HEAD)" --build-arg GIT_BRANCH="$branch"
  echo "https://$HOST"
}

down() {
  [[ "$APP" =~ ^myapp-[a-z0-9-]+-[0-9a-f]{6}$ ]] || { echo "refusing to destroy $APP" >&2; exit 1; }
  if exists; then
    fly machine start -a "$APP" >/dev/null 2>&1 || true                 # it may be suspended
    fly ssh console -a "$APP" -q -C "python manage.py teardown_env" || echo "warning: cleanup failed; the sweeper will catch it" >&2
    fly apps destroy "$APP" --yes
  fi
  pg "DROP DATABASE IF EXISTS $DB WITH (FORCE)" || true
}

case "${1:-}" in up) up ;; down) down ;; name) echo "$ENV" ;; url) echo "https://$HOST" ;; esac

For the demo I ran it from my laptop rather than pushing, because the three branches were throwaways I didn’t want in the repository’s history and I wanted to watch the output as it went. In the team setup nobody types this: the label workflow further down runs exactly this script, and what follows is what would land in the Actions log. Here is what it printed the first time, on the branch that carries the tooling itself:

$ ./scripts/env up
== feature/branch-environments -> feature-branch-envir-c65cb4
New app created: myapp-feature-branch-envir-c65cb4
CREATE DATABASE
Secrets have been staged, but not set on VMs. Deploy or update machines in this app for the secrets to take effect.
==> Building image
image size: 139 MB
✔ Machine 1a2b3c4d5e6f78 [app] update finished: success
https://feature-branch-envir-c65cb4-dev.example.com

$ curl -s https://feature-branch-envir-c65cb4-dev.example.com/api/version/
{"environment": "feature-branch-envir-c65cb4", "branch": "feature/branch-environments", "commit": "a1b2c3d4", "region": "syd"}

About two and a half minutes from command to URL, most of it the image build. The second up on the same branch, after another commit, printed no “New app created” and no “CREATE DATABASE”, staged the same secret names without regenerating the generated ones, and rebuilt: same URL, new commit.

Some of the choices in there deserve a sentence each.

fly.preview.toml is not fly.toml. Prod runs the API, the WebSocket server and the background worker as separate Fly apps and process groups so each scales on its own metric, which is the whole argument of the stack post. A branch environment doesn’t need to scale; it needs to be one machine. So the preview config has a single process: the container’s entrypoint gets a preview mode that runs migrations and then starts one ASGI server with WebSockets enabled. Channels already routes both HTTP and WebSocket traffic within one ASGI application, so one process serves the whole app. Sync views run thread-sensitive in that mode, roughly one request at a time, which is exactly the trade I moved away from in prod and exactly right here, because nobody testing a branch will notice.

# fly.preview.toml (the app name comes from `fly deploy -a`)
app = 'placeholder'
primary_region = 'syd'

[processes]
  app = "/app/entrypoint.sh preview"   # migrate, then one ASGI server for HTTP and WebSockets

[env]
  DJANGO_SETTINGS_MODULE = 'myapp.settings'
  DEBUG = 'False'

[http_service]
  internal_port = 8000
  force_https = true
  auto_stop_machines = 'suspend'
  auto_start_machines = true
  min_machines_running = 0
  [[http_service.checks]]
    path = '/healthz'
    [http_service.checks.headers]
      Host = '127.0.0.1'               # the app's own hostname isn't known to a static file

[[vm]]
  size = 'shared-cpu-1x'
  memory = '1gb'

auto_stop_machines = "suspend", min_machines_running = 0. A branch environment nobody has opened since lunch should cost nothing. Suspend rather than stop gives the ~3 second resume I ended up with after chasing 504s instead of a 30-second cold start, and here even 30 seconds would be tolerable. Two of the three demo environments were already suspended in fly apps list a minute after their deploy finished.

fly.io/apps/myapp-feature-profile-tagl-fa8832

DEBUG = False, same settings module as prod. A branch environment is a public URL. It should behave like prod in every way that matters for testing (real HTTPS, real cookies, the same health check) and unlike prod in every way that could hurt someone. Rather than a separate preview settings file, the difference is carried by the secrets env up sets: sandbox payment keys, no admin credentials for the auth provider, and the ENV_NAME that switches on every prefix below.

One setting slices everything. ENV_NAME is the only new knob in the settings, and it’s applied in four places:

ENV_NAME = os.getenv("ENV_NAME", "")
_prefix = f"{ENV_NAME}/" if ENV_NAME else ""

CACHES["default"]["KEY_PREFIX"] = ENV_NAME                           # every cache key
CHANNEL_LAYERS["default"]["CONFIG"]["prefix"] = f"{ENV_NAME}:asgi"  # Channels group and channel names
STORAGES["default"]["OPTIONS"]["location"] = f"{_prefix}media"      # uploads
STORAGES["staticfiles"]["OPTIONS"]["location"] = f"{_prefix}static"

Miss the channels one and two branches’ WebSocket servers share group names, and a message sent in one environment turns up in another. That is the kind of bug that costs a day.

There was a fifth place I only found by reading the shared Redis after the fact. The startup lock from the 504s post, the thing that lets one machine per image run migrations while the others wait, was keyed on the image reference. Two branch environments built from the same commit have the same image. The second one would have found the first’s “done” marker and skipped its own migrations against its own, un-migrated database. Now the environment name is part of the key and the key lives under the prefix. It’s the sort of thing a shared-infrastructure design keeps producing: anything that was implicitly “per deployment” has to be made explicitly per environment, and you don’t know what was implicit until you look.

The database is cloned, not migrated from empty. That’s the next section.

The database: a template, cloned in seconds

#

A branch environment’s database can be one of three things. Shared with everyone, which is what got us here. Empty, migrated from zero and seeded with fixtures: deterministic, but fixtures are never as good as real-shaped data and are always eighteen months stale. Or a clone of a prod-shaped snapshot: realistic row counts, realistic nulls in columns the fixtures assumed were always filled, the migration that’s instant on an empty table and takes forty minutes on a real one shows itself.

The third, with the second kept as a CI-only check. And what makes the third fast enough to do per branch is a Postgres feature most people forget exists:

-- Nightly, on the preview cluster. Restore last night's prod backup into seed_raw,
-- run the anonymiser, then freeze it as a template nobody can connect to.
ALTER DATABASE seed_raw RENAME TO seed_template;
UPDATE pg_database SET datistemplate = true, datallowconn = false WHERE datname = 'seed_template';

-- Per branch, in the time it takes to copy the files:
CREATE DATABASE env_feature_payments_retr_3f9a1c TEMPLATE seed_template;

CREATE DATABASE ... TEMPLATE is a file-level copy within the cluster. A few gigabytes is seconds, not the minutes a pg_restore takes. The template can’t have open connections while it’s being cloned, which is what datallowconn = false enforces, and it’s also why the nightly job builds into seed_raw and renames at the end rather than restoring over the live template.

For the demo the template was built by hand rather than from a prod dump: a tunnel to the preview cluster with fly proxy, manage.py migrate, then the repo’s own seed commands for a small but realistic data set. Nineteen megabytes. Three clones of it appeared in the cluster as three branches came up, each 19 MB, each taking under a second:

$ fly ssh console -a myapp-preview-pg -C "psql ... -c '\l'"
             datname             | template | size
---------------------------------+----------+-------
 env_feature_branch_envir_c65cb4 | f        | 19 MB
 env_feature_profile_tagl_fa8832 | f        | 19 MB
 env_feature_profile_webs_dcb2e7 | f        | 19 MB
 seed_template                   | t        | 19 MB

The preview cluster is one unmanaged Fly Postgres node, shared-cpu-1x with a 1 GB volume, about two dollars a month. The branch environments connect to it over .flycast on the private network; nothing about it is public.

The anonymiser is not optional. The moment prod data is reachable on a -dev hostname, it needs to not be prod data: names, emails, phone numbers, addresses, free text, payment references, tokens, all overwritten before the rename. If the anonymiser fails, the previous night’s template stays and someone gets a message. It’s a small amount of work and it’s the thing that makes the whole approach acceptable to whoever answers your privacy questionnaire.

The entrypoint runs migrate on the first boot of every image, so the branch’s own migrations apply on top of the template. That’s the first honest test of “does my migration work on data shaped like prod”, and it happens before anyone looks at the URL.

Hostnames: a wildcard, a Worker, and two things the docs didn’t tell me

#

Every environment needs a public HTTPS hostname, and “public” is the point: a tester on a phone, a designer on another network, a product manager who won’t install a VPN. Fly gives each app myapp-$ENV.fly.dev for free, and that alone works. But *.fly.dev hostnames have three problems: they’re ugly to send around, Cloudflare isn’t in front of them so none of the protections you have on prod apply, and if you want them on your own domain you’re doing a DNS record and a certificate per environment, which is one more thing for the sweeper.

The version that removes all of that is one wildcard and one Cloudflare Worker. I’d planned *.dev.example.com. Two things stopped that, and both are worth knowing before you plan the same.

Universal SSL covers one wildcard level. Cloudflare’s free certificate is issued for example.com and *.example.com. It does not cover *.dev.example.com; a second-level wildcard needs Advanced Certificate Manager at $10 a month. The first request to feature-branch-envir-c65cb4.dev.example.com died with an SSL handshake failure before anything of mine ran. The fix is to keep the environment in the first label: feature-branch-envir-c65cb4-dev.example.com. That’s covered by the certificate that already exists, and by the proxied *.example.com record that already exists, the one I set up as a sinkhole for the scanners walking invented subdomains. So an environment needs no DNS at all.

Redirect Rules run before Workers. With the route in place, every -dev hostname answered 301 to the www homepage without the Worker running. The sinkhole is a zone-level redirect rule, “any other subdomain → www”, and Cloudflare’s redirect phase executes before the Workers phase. The rule needed one clause: and not ends_with(http.host, "-dev.example.com"). The deploy script adds it on up and removes it on down, because a rule someone edited by hand is the thing that gets forgotten.

dash.cloudflare.com/…/example.com/rules/redirect-rules/…/edit

The DNS side is the record that was already there:

dash.cloudflare.com/…/example.com/dns/records

With those two out of the way the routing is a single Worker on *-dev.example.com/*:

// dev-router.js: <env>-dev.example.com -> myapp-<env>.fly.dev
const DEV_SUFFIX = "-dev.example.com";
const ENV_NAME = /^[a-z0-9-]+-[0-9a-f]{6}$/;   // slug-hash, as scripts/env names them

export default {
  async fetch(request) {
    const url = new URL(request.url);
    if (!url.hostname.endsWith(DEV_SUFFIX)) return new Response("not a dev hostname", { status: 404 });
    const env = url.hostname.slice(0, -DEV_SUFFIX.length);
    if (!ENV_NAME.test(env)) return new Response(`no such environment: ${env}`, { status: 404 });

    url.hostname = `myapp-${env}.fly.dev`;
    const upstream = new Request(url, request);
    upstream.headers.set("X-Forwarded-Host", request.headers.get("host") ?? "");

    const res = await fetch(upstream);
    const out = new Response(res.body, res);
    out.headers.set("X-Robots-Tag", "noindex, nofollow");   // a half-finished branch is not for search engines
    out.headers.set("X-Dev-Environment", env);
    return out;
  },
};

Cloudflare terminates TLS on the wildcard, the Worker computes the Fly hostname from the label, and Fly’s own *.fly.dev certificate covers the hop to the machine. WebSocket upgrades pass through fetch unchanged. And because the route is on the zone, everything Cloudflare does for prod is available: WAF rules, rate limits, bot fight mode, and Access.

dash.cloudflare.com/…/example.com/workers

Three branches up at once, one Worker, and what each hostname answered:

Every 200 came back with X-Robots-Tag: noindex, nofollow and X-Dev-Environment: <env> added by the Worker. The 404 is the Worker’s own, before Fly is ever asked.

Cloudflare Access is what turns “public” into “reachable by the people who should reach it”. One application on *-dev.example.com with a policy of “anyone with an @example.com email” and browser visitors get a login page once and never think about it again. That covers designers, PMs and testers on laptops. I didn’t add it for the demo; the environments were up for an hour with noindex and nothing on them but seed data.

It doesn’t cover the mobile app, because a native app can’t complete an Access login redirect. Two ways through. Access supports service tokens (a CF-Access-Client-Id / CF-Access-Client-Secret header pair) that bypass the login, and the mobile app’s dev menu can carry a pair for the dev hostnames. Or, simpler, add an Access bypass for /api/ and have Django reject any request without either a signed-in user or the environment’s ENV_TOKEN, the one env up generated for exactly this environment. Either way, the mobile side is a dev-menu screen with two fields, base URL and token, and the app a tester already has installed is now talking to that branch. No new build. The over-the-air update channel stays pointed at whatever the app was built with; the API base URL is a runtime setting, not a build-time one, and it should have been anyway.

Auth is the one shared service that’s shared on purpose. Identity isn’t environment-specific; a tester wants to log in with the same account on every branch. One dev project at the auth provider with example.com in its authorised domains covers every -dev hostname, and every environment verifies ID tokens against it. The users table in each environment’s database is a copy of the template’s users, so the tester’s account exists everywhere the template did.

Automating preview environments with GitHub Actions: events in, environments out

#

Everything above by hand is fine when you’re the only one doing it. The moment there’s a second developer it has to happen because something happened in git, not because someone remembered. The whole automation is four workflows, one repository setting, one ruleset and one GitHub environment, and every one of them maps a git event to a call of the same scripts/env the developer runs by hand. There is no CI-only path to drift.

The label, push and delete workflow

#

# .github/workflows/branch-env.yml
name: branch environment
on:
  pull_request:
    types: [labeled, unlabeled, synchronize, closed]
  delete:
  workflow_dispatch:
    inputs:
      branch: { description: branch to bring up without a PR, required: true }

concurrency:
  group: env-${{ github.head_ref || github.event.ref || inputs.branch }}
  cancel-in-progress: true

env:
  FLY_API_TOKEN: ${{ secrets.FLY_PREVIEW_TOKEN }}   # org-scoped deploy token; a separate preview org keeps it away from prod
  PG_PASSWORD: ${{ secrets.PREVIEW_PG_PASSWORD }}
  # ... the shared Redis URL and bucket credentials, the same way

jobs:
  up:
    # the gate: a PR carrying the `env` label, or someone asking for a branch by hand
    if: >-
      github.event_name == 'workflow_dispatch' ||
      (github.event_name == 'pull_request' && github.event.action != 'closed'
        && contains(github.event.pull_request.labels.*.name, 'env'))
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with: { ref: "${{ github.head_ref || inputs.branch }}" }
      - uses: superfly/flyctl-actions/setup-flyctl@master
      - id: env
        env: { BRANCH: "${{ github.head_ref || inputs.branch }}" }
        run: scripts/env up && echo "url=$(scripts/env url)" >> "$GITHUB_OUTPUT"
      - if: github.event_name == 'pull_request'
        uses: marocchino/sticky-pull-request-comment@v2
        with:
          header: env
          message: "Branch environment: ${{ steps.env.outputs.url }}"

  down:
    # label removed, labelled PR closed or merged, or the branch itself deleted
    if: >-
      (github.event_name == 'pull_request' &&
        ((github.event.action == 'closed' && contains(github.event.pull_request.labels.*.name, 'env')) ||
         (github.event.action == 'unlabeled' && github.event.label.name == 'env'))) ||
      (github.event_name == 'delete' && github.event.ref_type == 'branch')
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: superfly/flyctl-actions/setup-flyctl@master
      - run: BRANCH="${{ github.head_ref || github.event.ref }}" scripts/env down

The gate is the env label, and it’s the part I’d argue for hardest. Without one, every branch costs an app and a database clone: the branch someone pushes to back up a laptop, the spike that gets squashed tomorrow, the dependency bump a bot opened at 3 a.m. So a push alone does nothing and opening a PR does nothing. Adding env to the PR brings the environment up, every push while the label is on updates it, and removing the label, closing the PR or merging it takes it down. Opting in is still a GitHub event, not a request to a person, and it’s one click made by the one person who knows whether anyone is going to look. The sticky comment is the user interface: label the PR, a comment appears with a URL they can send to anyone, and it updates in place on every push rather than stacking. workflow_dispatch covers the branch you want up before there’s a PR, and ./scripts/env up from a laptop covers the rest.

Two other gates are worth knowing. A branch-name filter (branches: ['feature/**', 'fix/**']) needs no PR but makes every feature branch pay. “Any PR that isn’t a draft” is the most automatic, and the right choice if your team opens PRs late and only when they’re ready. I went with the label because it puts the cost decision with the person who has the context, and because it can be taken back without closing anything. One side effect I like: pull_request runs from forks don’t get secrets, so a fork can’t bring up an environment.

concurrency matters more than it looks. A run of quick pushes to one branch cancels the earlier deploys rather than racing them; two fly deploys to the same app at once end with one of them losing.

closed covers teardown for anything that had a PR, merged or not. The delete event is the backstop for a branch brought up by workflow_dispatch that never got one, and it only fires if branches actually get deleted. That’s a repository setting, Automatically delete head branches, and it’s off by default. With it on, clicking Merge closes the PR and deletes the branch, and either event alone would have run env down. The merge button is the teardown.

gh api -X PATCH repos/org/myapp -f delete_branch_on_merge=true

The FLY_PREVIEW_TOKEN is the dangerous line in the file. It can create and destroy apps, and if $APP were ever anything but a branch environment it could destroy prod. Two guards, and I’d keep both: the token is scoped to a separate Fly org so it literally cannot see the production org (for the demo I used the production org, to skip setting up billing on a new one, and relied on the second guard alone); and down() refuses any app name that doesn’t match the slug-hash pattern.

The check a branch environment can’t do for itself

#

A branch only ever sees its own migrations, so the check that matters runs on the pull request against a merge with main, and it’s the one status check main requires:

# .github/workflows/migrations-check.yml
name: migrations
on:
  pull_request:
    branches: [main]

jobs:
  check:
    runs-on: ubuntu-latest
    services:
      postgres:
        image: postgres:18
        env: { POSTGRES_PASSWORD: ci, POSTGRES_DB: ci }
        ports: ['5432:5432']
        options: --health-cmd pg_isready --health-interval 5s --health-retries 10
    env:
      DATABASE_URL: postgres://postgres:ci@localhost:5432/ci
      SECRET_KEY: ci-only
    steps:
      - uses: actions/checkout@v4
        with: { fetch-depth: 0 }
      - uses: actions/setup-python@v5
        with: { python-version: '3.14' }
      - run: pip install -r requirements.txt
      - name: Merge main into the branch (a conflict here fails the check too)
        run: |
          git config user.email ci@example.com && git config user.name ci
          git merge --no-edit origin/main
      - name: One leaf in the migration graph
        run: python manage.py makemigrations --check --dry-run
      - name: The full chain applies from nothing
        run: python manage.py migrate --noinput

Thirty seconds. It’s what turns the 0042 / 0042 collision from a release-day surprise into a red check on the second developer’s PR; the migrations section below shows it firing.

The rules on main

#

None of this holds up if branches live for a month or if someone pushes to main directly. The rules are less about protection and more about keeping environments small and their migrations current:

• main is protected and always deployable. Merge via PR with the migrations check green against the current main (strict status checks, so a PR that was green last week re-runs after main moves). Linear history, no force-push, no deletion.
• Feature branches are short. A branch open for two weeks has an environment two weeks behind main and a migration two weeks more likely to collide with someone else’s. A month-long feature merges to main behind a flag in pieces. The flag is the long-lived thing, not the branch.
• One branch, one environment, one database. Already enforced by the naming.
• Release tags are the only thing that goes to prod. The image is built once from the tag; staging and prod run that image, not a rebuild.
• A PR that touches migrations/ is reviewed by one of the other two. Below.

As a ruleset, applied once with gh api -X POST repos/org/myapp/rulesets --input ruleset-main.json:

{
  "name": "main", "target": "branch", "enforcement": "active",
  "conditions": { "ref_name": { "include": ["~DEFAULT_BRANCH"], "exclude": [] } },
  "rules": [
    { "type": "deletion" },
    { "type": "non_fast_forward" },
    { "type": "required_linear_history" },
    { "type": "pull_request",
      "parameters": { "required_approving_review_count": 1, "dismiss_stale_reviews_on_push": true,
                      "required_review_thread_resolution": true } },
    { "type": "required_status_checks",
      "parameters": { "strict_required_status_checks_policy": true,
                      "required_status_checks": [ { "context": "check" } ] } }
  ]
}

One honest wrinkle. This repository is private on GitHub’s Free plan, and on Free, branch protection and rulesets are only available for public repositories; the API answers 403 Upgrade to GitHub Pro or make this repository public. The delete_branch_on_merge setting, the environments and the workflows all work on Free. So for a small team on a private Free repo, the ruleset above is a convention: the migrations workflow still runs and still goes red, but nothing greys out the merge button. That’s a $4-a-month decision, and I’d make it the moment a second person can merge.

There’s no develop branch and no release/* branches. Git-flow was designed around integration being expensive and needing to be batched. Branch environments make integration cheap, so the batching goes.

Staging is a tag’s environment, and prod is a promotion

#

Staging reuses the whole machinery. A v* tag on main runs scripts/env up with the branch name staging/v1.42.0, which the naming turns into staging-v1-42-0-<hash>: same clone-from-template database, same suspend-when-idle machine, same hostname scheme. The difference is what comes after:

# .github/workflows/release.yml (abridged)
on:
  push:
    tags: ['v*']
jobs:
  staging:
    steps:
      - run: BRANCH="staging/${GITHUB_REF_NAME}" scripts/env up
      - run: echo "image=$(fly image show -a "myapp-$(BRANCH=staging/$GITHUB_REF_NAME scripts/env name)" --json | jq -r '…')" >> "$GITHUB_OUTPUT"
  production:
    needs: staging
    environment: production        # required reviewer on this environment = the promotion gate
    steps:
      - run: fly deploy --config fly.api.toml --image "${{ needs.staging.outputs.image }}"
      - run: fly deploy --config fly.ws.toml  --image "${{ needs.staging.outputs.image }}"
      - run: BRANCH="staging/${GITHUB_REF_NAME}" scripts/env down

The production job waits on a GitHub environment with a required reviewer. Someone looks at staging, clicks Approve, and prod receives the same image staging ran, by reference, not a rebuild. The last step destroys staging because it’s done its job. If nobody approves, the release candidate was rejected, and the sweeper removes staging when its “branch” (staging/v1.42.0 never exists on origin) fails the liveness check. Nothing gets to exist without a reason the sweeper can check.

The nightly sweeper

#

# .github/workflows/sweep-envs.yml
on:
  schedule:
    - cron: '0 14 * * *'     # 02:00 NZST
  workflow_dispatch:
jobs:
  sweep:
    steps:
      - uses: actions/checkout@v4
        with: { fetch-depth: 0 }
      - uses: superfly/flyctl-actions/setup-flyctl@master
      - run: scripts/sweep-envs

fetch-depth: 0 because the sweeper needs every remote branch and its last commit date, not the one commit Actions checks out by default. workflow_dispatch so it can be run by hand after an incident.

What a developer actually does

#

Push a branch, open a PR, and put the env label on it when they want someone to look. That’s the list. A URL appears on the PR within three minutes, updates on every push, and disappears when the PR is merged or closed. The migration check tells them before anyone else does if they’ve collided with a teammate. Nobody claims staging because there is no staging to claim, and nobody tears anything down because tearing down is what merging does.

And what the person who set it up does per branch, afterwards: nothing. The shared Postgres, Redis and bucket are provisioned once, the secrets go into Actions once, and from then on the recurring work is reading the sweeper’s log on a Monday. No ticket queue, no bot to ask, nobody to prompt.

Teardown that doesn’t trust the teardown

#

env down is the ordinary case, and the interesting part of it is who does the cleaning. The Redis keys and the bucket objects belong to the shared services, and the laptop running down doesn’t need credentials for either, because the environment’s own machine already has them and already knows its prefix. So down starts the machine if it’s suspended, runs a small cleanup command on it, and only then destroys the app and drops the database. The environment cleans up after itself:

$ BRANCH=feature/profile-website ./scripts/env down
redis: deleted 0 keys under feature-profile-webs-dcb2e7:*
bucket: deleted 278 objects under feature-profile-webs-dcb2e7/
Destroyed app myapp-feature-profile-webs-dcb2e7
DROP DATABASE

$ curl -s -o /dev/null -w '%{http_code}\n' https://feature-profile-webs-dcb2e7-dev.example.com/api/version/
530

The 530 is the point about coordination again: the Worker still routes the hostname, Fly has nothing there, and nobody had to tell Cloudflare.

The cleanup command refuses to run without ENV_NAME, because with an empty prefix it would be a wipe of the shared bucket. That guard is the whole reason it exists as a command inside the app rather than three lines in the shell script.

Environments leak anyway. A branch is deleted while GitHub Actions is having an outage. A developer runs env up from a laptop for a spike and never pushes. A fly apps destroy fails transiently and the || true swallows it. Three months later there are forty apps and a bill that makes someone ask what happened.

So there’s a second mechanism that doesn’t trust the first one: the nightly sweeper from the table above. It lists every myapp-* app, works out which branch each one belongs to, and destroys any whose branch is gone or hasn’t been pushed to in seven days. A week-idle branch doesn’t need a live URL; the next push recreates it in two minutes. A hash isn’t reversible, so the sweeper computes the environment name for every live branch and destroys any app that isn’t in that set, then drops any env_* database with no app behind it:

#!/usr/bin/env bash
# scripts/sweep-envs: nightly
set -euo pipefail
ORG=${PREVIEW_ORG:-myapp-preview}; TTL_DAYS=${TTL_DAYS:-7}
cutoff=$(( $(date +%s) - TTL_DAYS*86400 ))

git fetch --prune --quiet origin
live=""
while IFS=$'\t' read -r ts branch; do
  [[ "$branch" == "main" ]] && continue
  (( ts >= cutoff )) || continue
  live+="$(BRANCH="$branch" scripts/env name)"$'\n'
done < <(git for-each-ref --format='%(committerdate:unix)%09%(refname:short)' refs/remotes/origin | sed 's#^\([0-9]*\)\torigin/#\1\t#')

apps=$(fly apps list --org "$ORG" --json | grep -o '"Name": *"myapp-[^"]*"' | cut -d'"' -f4)
for app in $apps; do
  env="${app#myapp-}"
  [[ "$env" =~ ^[a-z0-9-]+-[0-9a-f]{6}$ ]] || continue       # not a branch environment (e.g. myapp-preview-pg)
  grep -qx "$env" <<<"$live" && continue
  echo "sweeping $app: branch gone or idle > ${TTL_DAYS}d"
  ENV_OVERRIDE="$env" scripts/env down || true
done
# ...then any env_* database with no app: DROP DATABASE.

None of the demo branches had ever been pushed (the laptop run above), so from the sweeper’s point of view all of them were gone, which is exactly the state a spike someone brought up from a laptop and never pushed would be in a week later. It took the remaining two apart, including the Redis keys the tagline environment had written:

$ ./scripts/sweep-envs
sweeping myapp-feature-branch-envir-c65cb4: branch gone or idle > 7d
redis: deleted 0 keys under feature-branch-envir-c65cb4:*
bucket: deleted 322 objects under feature-branch-envir-c65cb4/
Destroyed app myapp-feature-branch-envir-c65cb4
DROP DATABASE
sweeping myapp-feature-profile-tagl-fa8832: branch gone or idle > 7d
redis: deleted 3 keys under feature-profile-tagl-fa8832:*
bucket: deleted 278 objects under feature-profile-tagl-fa8832/
Destroyed app myapp-feature-profile-tagl-fa8832
DROP DATABASE

$ fly apps list --org myapp-preview | grep myapp-
 myapp-preview-pg │ myapp-preview │ deployed

(One small thing that bit: macOS ships bash 3, which has no associative arrays. The first version of the sweeper used one and died on declare -A on my laptop. The version above uses a newline-separated list and runs on anything.)

The cost is the pleasant part. A suspended machine costs its rootfs, which is cents a month. A database is 19 MB on a volume that’s already paid for. The Redis is on a pay-as-you-go plan, twenty cents per hundred thousand commands, which at branch-testing volumes rounds to nothing. A bucket prefix costs what it stores. The shared Postgres node is the only fixed line, about two dollars a month. Twenty active branches would cost roughly what one always-on shared-cpu-1x costs, and the sweeper keeps it at twenty rather than two hundred. The whole demo, three environments for an hour plus the shared pieces, cost less than the coffee I drank during it.

Migrations across branches, without the shared database to warn you

#

When each developer owns a feature end to end, every branch has a migration in it. That makes this section the important one.

Here’s the thing people miss when they leave the shared-environment world. When everyone deployed to one dev database, migration conflicts surfaced immediately and painfully, but they surfaced. One migration ran, the next deploy failed, someone fixed it that afternoon. With a database per branch, each migration works perfectly in its own environment, and two of them first meet when the second one merges to main. Done carelessly, this moves the conflict from dev, where it was cheap, to a release, where it isn’t.

Four rules, all enforceable by CI so nobody has to remember them.

1. CI fails on two leaves in the migration graph, against a merge with main. makemigrations --check catches models disagreeing with migrations, but the conflict that matters is two branches both adding 0042_*. Check it against the branch merged into main, not the branch alone:

git fetch origin main
git merge --no-commit --no-ff origin/main || { echo "merge conflict"; exit 1; }
python manage.py makemigrations --check --dry-run
python manage.py migrate --plan 2>&1 | grep -q "Conflicting migrations" && exit 1

This is the one I made happen on purpose. Two of the three demo branches, feature/profile-tagline and feature/profile-website, each added one field to the user profile model with one migration, and each environment applied its own on the way up. The new columns were only where they should be:

env_feature_branch_envir_c65cb4      (no new column)
env_feature_profile_tagl_fa8832      tagline
env_feature_profile_webs_dcb2e7      website

Both migrations are 0042_*. Neither branch can see the other’s, so neither environment can tell you there’s a problem. Merging one into the other, the check does:

$ git merge --no-commit --no-ff feature/profile-tagline
$ python manage.py makemigrations --check --dry-run
CommandError: Conflicting migrations detected; multiple leaf nodes in the migration graph:
  (0042_profile_tagline, 0042_profile_website in accounts).
To fix them run 'python manage.py makemigrations --merge'

$ python manage.py makemigrations --merge --noinput
Created new merge migration accounts/migrations/0043_merge_20260919_0324.py
    dependencies = [
        ('accounts', '0042_profile_tagline'),
        ('accounts', '0042_profile_website'),
    ]

The second author runs that --merge on their branch, in their environment, bothering nobody, and the next env up applies 0043 on top of their clone. (In this case git also flagged a textual conflict first, because both fields had been added at the same line of the model file. Realistic, and worth knowing that the textual conflict hides the migration one until it’s resolved.)

2. Migrations are backwards-compatible with the previous release’s code. During a deploy there’s a window where the new schema and the old code coexist, or the reverse if a machine rolls back. Add columns nullable or with defaults, never rename in one step, never drop what the current release still reads: add, deploy, backfill, switch code, drop in a later release. Expand and contract. The reason it matters more here is that disposable environments make deploying more frequent, so you’re inside that window more often. Staging-per-tag is where it’s rehearsed.

3. CI also migrates from zero into an empty database. The template clone tests “does my migration apply to realistic data”. The empty run tests “does the full chain still work from nothing”, which catches the migration that assumed a previous data migration had populated something. Thirty seconds, and it catches a class of bug that otherwise waits for the next new laptop.

4. Schema changes get a second pair of eyes. In a small team this doesn’t need a CODEOWNERS file, but it does need to be a rule: a PR that touches migrations/ is reviewed by a teammate before merge, and the reviewer’s job isn’t to check the SQL. It’s to know that they, or someone else on the team, are touching the same table this fortnight and say so. Past a dozen developers that knowledge stops fitting in one head, and a CODEOWNERS entry on migrations/ earns its place. No tooling replaces this. With the tooling catching everything mechanical, it’s a two-minute job, and it’s the one part of the old shared-environment world worth keeping: the moment where two people find out they’re about to collide.

Versioning, so a URL tells you what it’s running

#

With one shared environment, “what’s on staging” was a chat question. With many, it has to be a property of the environment or every bug report starts with archaeology.

• Every image carries the commit SHA and the app exposes it. fly deploy --build-arg GIT_SHA=... --build-arg GIT_BRANCH=... stamps them into the image as environment variables, and /api/version/ returns them with the environment name and the region (see the output above). The first thing anyone does with “it’s broken on feature-profile-tagl-fa8832” is check it’s running the commit they think. The first demo environment was deployed from the working tree a commit before the tooling was committed, and the endpoint said so: it reported the previous commit, not the SHA I expected.
• ENVIRONMENT=$ENV is in every log line and the error tracker’s environment tag. Branch errors don’t pollute prod’s error tracking and are filterable in one click.
• Branches get SHAs, releases get tags. feature-payments-retr-3f9a1c never has a version number; it has commits. Staging and prod have v1.42.0.
• The mobile app carries a minimum API version, not a URL. The dev-menu URL field is how a tester points a real phone at a branch, and the version endpoint carrying an API schema version is how the app knows whether it’s talking to something it understands.

What this doesn’t solve

#

It doesn’t solve two features that only make sense together. A branch environment shows one branch. The answer is to merge both behind flags and look at main, which you can do with one standing environment that tracks main (main-dev.example.com, redeployed on every merge). That’s the one long-lived non-prod environment I’d keep, and nobody deploys to it; it’s for seeing what the next release will be.

It doesn’t solve providers that want a registered callback and don’t accept wildcards. My auth provider does accept a parent domain, so example.com covers every -dev branch hostname. Some OAuth and payment providers don’t, and for those you keep one shared hostname that the Worker routes to whichever environment last claimed it. That’s a small lock over a small thing, and it’s a lot better than a lock over everything.

It doesn’t solve the human side of migrations. The tooling catches mechanical conflicts. It doesn’t catch two of you modelling the same concept incompatibly in the same fortnight, and the only fix for that is the migration review where one of you notices.

What I’d check in your setup

#

If you have a shared dev or staging and a chat thread for booking it:

1. How long does a finished branch wait before someone other than its author can see it running? More than an hour and you’re paying for the lock in delivery time, and on a small team that’s a big fraction of it waiting.
2. How often does the shared database get reset, and who loses work when it does?
3. Could a new developer, on day one, get a public URL running their branch without asking anyone for anything? Whatever they’d have to ask for is the thing to automate first.
4. Does a fresh environment start from realistic data? If it starts empty, the first bug it won’t catch is the migration that’s fine on an empty table and takes forty minutes on the real one.
5. Is there anything in your non-prod hosting that would still exist a month after everyone forgot about it? If so, write the sweeper before anything else. The leak starts on day one.

The chat question goes away not because people stop asking it but because the answer is always “you are, on yours.”

Disclosure. Codemagic paid me to do this migration and write about it. They get a review pass for factual accuracy, not for tone, and nothing here was changed at their request. Where I hit a problem and reported it, I say so; if it’s fixed by the time you read this, it’s described as fixed.

Update, 23 September 2026. Codemagic’s developers have worked through the issues I reported. Most are now fixed, including the iOS version-detection bug, the double-counted installs, the silent first run and the noisy debug command, and the docs now cover committed native projects and the crash-rollback budget. I’ve marked each one below where it comes up, and left the few that are still open as they were.

Update, 16 October 2026. The first native upgrade since moving to Patch (Expo SDK 58) built an Android release with no Patch configuration in it, and nothing reported an error. What happened and the one-file fix are in The first native upgrade.

In February I wrote about building my own Expo update server. It still works. maru, the community app I run, has taken every over-the-air fix through it since, on both platforms, for about a dollar a month.

It also means I own every piece of it: the database tables, the API endpoint, an admin command, a publish script, a shared secret, and a set of gotchas I learned one broken update at a time. When I sat down to plan this, I found that the script had been printing “Sent 0 push notifications” after every production publish since the day I wrote it, reading a field this server has never returned. Nobody noticed, least of all me. That’s roughly what running your own thing turns into.

So when Codemagic asked whether I’d try Patch, their open-source, self-hostable take on CodePush, I agreed on one condition: a real app rather than a demo. Real signing, real release process, committed native projects with hand edits in them, the lot. Most people moving off EAS Update are moving an app like that, not a fresh template.

This is how that went, with the rough edges left in.

What Patch is

#

Patch is a server you run yourself and an SDK you put in your app. The server side is an API, a background worker, Postgres, S3-compatible object storage and a web dashboard, shipped as a Docker Compose stack. The client side is @codemagic/react-native-patch, a TurboModule for iOS and Android with an Expo config plugin, plus a CLI, cmpatch, that bundles your JavaScript and publishes it.

Two design decisions stood out once I’d read the source.

The first is that the app never asks the API whether there’s an update. When a release is published, the worker writes plain JSON manifests into object storage, and the app just fetches those. An update check is two static GETs against storage or a CDN; the API only receives metrics. My old endpoint was on the request path of every app launch. This one can’t be.

The second is that every release names the exact binary version it’s for, and the CLI computes a fingerprint of your native project and refuses to publish JavaScript onto a binary whose native code doesn’t match. My server had a version string and a lot of hoping.

Trying it on my laptop first

#

The docs start you on a local evaluation stack, and that’s the right place to start.

npm install -g @codemagic/patch-cli
cmpatch selfhost local-eval up

The CLI installed in two seconds. The stack took 3 minutes 14 seconds the first time, most of it building the server and dashboard images locally, and in non-interactive mode it prints nothing at all while it does that, which looks exactly like a hang. (Since fixed: it now prints one progress line per step.) Then four healthy containers appeared, all bound to localhost: the dashboard on 8080, the API on 3000, Postgres, and MinIO on 9100. It sensibly avoided the ports my own stack was already using.

Sign-in is disabled in this mode, with a warning on every page not to expose it.

localhost:8080

The stack comes seeded with an app called Example Data, which is worth a look before you’ve published anything, because it shows what the dashboard is for. The numbers in the next two screenshots are that seed data, not real users.

localhost:8080

localhost:8080

For comparison, my dashboard was an admin list of updates with an “active” checkbox. I had no idea how many phones had actually taken an update.

Logging the CLI in is a proper browser flow with PKCE and a localhost callback, and the approval page tells you which account is asking and that the code expires in about a minute.

localhost:8080

Pointing it at a real app

#

Patch wants one app per platform, each with a Staging and a Production deployment created for you. I made maru-ios in the dashboard and maru-android with cmpatch app create, mostly to see both. The dialog you get after creating an app is the most useful screen in the product for a migration: both deployment keys, the two URLs the SDK needs, and a note that deployment keys aren’t secrets.

localhost:8080

Two small things. cmpatch init linked a project to apps that already existed, so it couldn’t be the first command, whatever the migration guide implied. That’s since been fixed: in interactive mode it now creates one app per platform itself. And the dialog gives the API URL as http://localhost:8080 while the docs and the CLI say 3000. Both work, because each port serves both, but you’ll wonder.

Then I ran cmpatch doctor against the untouched app, and it found a real problem in three seconds. It reported the iOS binary version as 2.0.0. The app is 2.5.1.

The CLI finds every Info.plist under ios/ and, if none has “test” in its path, takes the first alphabetically. maru has a WidgetKit extension whose folder sorts before the app’s own, so the CLI picked the extension’s Info.plist. That plist was stale at 2.0.0, which is my bug, but the effect is nasty: auto-detection would target every iOS release at a version no phone runs, and nothing would error. Updates would just never arrive, which is exactly the failure I warned about in my self-hosting post. Android read versionName correctly. The workaround is to pass --target-binary-version or --plist-file, and my publish script now passes the version explicitly after checking that app.config.js and both native projects agree. Later, the same bug made doctor’s own manifest check fail against a deployment that was working fine, because it went looking for version 2.0.0’s manifest. Once I’d fixed my stale plist, doctor passed all 24 checks.

Codemagic has since fixed this. The CLI now reads the version from the app target’s Info.plist, not an extension’s, and if more than one target could be the app, cmpatch asks you which.

The JavaScript side

#

Removing expo-updates and adding the SDK is a two-line change. Replacing what the app did with updates took more thought.

maru checks for updates on launch and whenever it comes back to the foreground, at most every fifteen minutes. If it finds one, it downloads it, and offers to restart; say no and it applies on the next cold start. There’s also a manual check in Settings that shows which bundle you’re running, and the error reporter tags every report with the update it came from.

The first thing I learned from the SDK source is that it looks up its native module at import time with TurboModuleRegistry.getEnforcing, which throws if the module isn’t there. maru also ships a web build, and a top-level import would take the web app down. So everything goes through one small module that loads the SDK lazily, with a .web.ts twin that returns nothing, which also keeps the SDK out of the web bundle entirely.

// ota.ts (ota.web.ts exports a patchSdk() that always returns null)
import { Platform } from "react-native";

type PatchSdk = typeof import("@codemagic/react-native-patch");
let sdk: PatchSdk | null | undefined;

export function patchSdk(): PatchSdk | null {
  if (sdk !== undefined) return sdk;
  if (Platform.OS === "web") return (sdk = null);
  try {
    sdk = require("@codemagic/react-native-patch") as PatchSdk;
  } catch {
    sdk = null;
  }
  return sdk;
}

The automatic check itself is mostly sync(), which never throws and resolves to a status:

const status = await patch.sync({
  installMode: "ON_NEXT_RESTART",
  mandatoryInstallMode: "ON_NEXT_RESUME",
});
if (status === "update-installed") offerRestart();

The SDK’s default for mandatory releases is IMMEDIATE, which reloads the JavaScript wherever the user happens to be. I changed it to apply the next time the app comes back from the background, because losing a half-written post is a worse bug than most fixes are good. At the time, InstallMode was a string union, not the enum object the docs’ examples suggested; it’s now exported as a constants object, so InstallMode.ON_NEXT_RESTART works as documented.

The native side, when you commit ios/ and android/

#

The migration guide says to run npx expo prebuild --clean after adding the config plugin. I ran it in a separate checkout and it deleted bundled images, my Android network security config, the iOS privacy manifest and Podfile.lock, and reverted a test-runner fix in build.gradle along with a few hundred lines of Info.plist. Without --clean, Expo cleared both directories anyway.

This isn’t Patch’s fault. maru started as a managed Expo app and grew native customisations, so ios/ and android/ are committed and EAS builds them as they are. But plenty of apps look like that, and the guide didn’t warn you. It does now: the Expo migration guide has two paths, prebuild for generated native projects and the exact hand-applied changes for committed ones.

What the plugin actually changes turns out to be small enough to do by hand. On iOS, one import and one line in AppDelegate.swift:

return CodemagicPatch.bundleURL() ?? Bundle.main.url(forResource: "main", withExtension: "jsbundle")

On Android, one import and one argument in MainApplication.kt (this is the React Native 0.82+ shape; older versions override getJSBundleFile() instead):

ExpoReactHostFactory.getDefaultReactHost(
  jsBundleFilePath = CodemagicPatch.getJSBundleFile(applicationContext),
  // ...

Plus three config values per platform and the removal of the old expo-updates settings: 13 lines added and 20 removed across both projects.

The plugin writes those config values as literals at prebuild time, which, with committed native projects, would mean one hard-coded deployment for every build. I wanted preview builds on Staging and store builds on Production from the same project, so the values come from the build environment instead. In Info.plist, Xcode expands build settings:

<key>CodemagicPatchDeploymentKey</key>
<string>$(PATCH_IOS_DEPLOYMENT_KEY)</string>

And in android/app/build.gradle:

def patchEnv = { name -> System.getenv(name) ?: (findProperty(name) ?: "") }
resValue "string", "CodemagicPatchDeploymentKey", patchEnv("PATCH_ANDROID_DEPLOYMENT_KEY")

The keys then sit in each eas.json build profile next to the EXPO_PUBLIC_ values. An empty value means the SDK quietly does nothing, which is fine for a local build and a disaster for a store build, so app.config.js now throws if a preview or production EAS build is missing them or still points at localhost.

The first update

#

For the first release I changed the onboarding headline, which you can see without signing in, and published it:

./scripts/publish-update.sh ios --notes "Onboarding headline over the air"

That took 30 seconds, most of it Metro and Hermes. The server’s worker published the manifest 0.6 seconds after the upload finished. The update is a 7.3 MB compressed tarball, against the 11 MB Hermes bundle plus assets my old export produced.

One thing to get right here: the JavaScript bundle inlines your EXPO_PUBLIC_ values when it’s built, so an update has to be built with the same values as the binary it lands on, or you ship JavaScript that talks to the wrong backend. My script loads them from the same eas.json profile that built the binary. Patch’s CI docs now spell this out too, along with the fact that release-react doesn’t read EAS profiles itself.

Then I relaunched the app. The device log showed the whole update check as two requests to object storage, meta.json and 2.5.1/manifest.json. The SDK wrote a Downloaded event three seconds after launch and staged the update, and the app offered a restart.

The bug was mine

#

Then I looked at the dashboard. One simulator had installed release v1, and it showed two successful installs and two active users.

localhost:8080

The SDK keeps its state and pending metric events as files in the app’s container, which made this easy to chase. There were two Success events with identical timestamps and different ids. My hook confirmed the running bundle with notifyAppReady() on mount, and then called sync() straight away, and sync() calls notifyAppReady() itself. The two calls overlapped, each saw the release as pending, and each recorded the events. The server de-duplicates on event id, but these had different ids, so both counted, and those duplicates are permanent.

The fix on my side was one shared confirmation per launch that everything else awaits. The SDK could also guard against overlapping calls itself, and I suggested that. It now does: overlapping notifyAppReady() calls, including the one inside sync(), share a single confirmation, so each install is recorded once. The SDK reference also now says that sync() confirms readiness itself, so an app that uses it doesn’t need to call notifyAppReady() separately.

The nice part was shipping the fix. It’s a JavaScript-only change, so it went out as release v2, and because a device on v1 can be sent a binary diff, v2 downloaded as a 543 KB patch instead of the 7.3 MB bundle, about 7% of the size. After the restart: one download, one install, one success, one active device.

localhost:8080

Breaking it on purpose

#

Rollback is the feature I cared about most, because my old setup had none on the device. A bad update would have crashed every launch until I noticed and un-ticked a checkbox.

So I released a v3 that throws before the first render, which means it can never call notifyAppReady(). The app downloaded it, offered a restart, and died to the home screen.

The next launch booted v3 again and crashed again, which surprised me, because the docs say a bundle that crashes before notifyAppReady() is rolled back. The source explains it: the SDK allows three unconfirmed launches before deciding it’s a crash, because one unconfirmed launch can just mean iOS killed a healthy process before any JavaScript ran. The fourth launch came back up on v2.

The device then reported the failure itself, as crash_rollback, and the dashboard showed it against v3. That’s a sensible design, but it’s worth being plain about what it means: a bad release costs each user three crashed launches before they recover, and at the time the docs didn’t mention the number. They do now, and the budget is configurable, with CodemagicPatchMaxLaunchAttempts natively or the Expo plugin’s maxLaunchAttempts. That’s the case for Staging first and a small Production rollout second.

Stopping a bad release for everyone else is one command, and it took under a second:

cmpatch release rollback --app maru-ios --deployment Staging

It doesn’t rewrite history. It publishes the previous release again as a new one (v4, marked as a rollback of v2), so the crashing release stays in the list with its failure count, which is the honest record. The dashboard does the same from a dialog.

localhost:8080

Android

#

Android had two surprises, neither Patch’s. The R8 keep rule I’d added for the SDK lived in app.config.js, which only reaches the native project through prebuild, so my first release build accidentally tested whether the SDK survives R8 without one. It does: R8 renamed all its classes, and updates downloaded, installed and rolled back fine. And the emulator, freshly cold-booted with Gradle still holding memory, was so slow that my first test gave up 45 seconds before the SDK finished the download it had already started.

Release v1 applied, with one success and one active device, and a crashing v2 rolled back after the same three-launch budget.

Past the happy path

#

Updates arriving and rolling back is the core of it, but a team lives in the other features, so I went through them one at a time.

Staged rollouts are deterministic, which I liked. A device is in if the first eight hex digits of md5(deviceId + "-" + releaseLabel), modulo 100, are below the rollout percentage, so the same phone stays in or out for a given release. I worked out that my simulator sat in bucket 91 for the next release, published it at 25%, and the simulator correctly got nothing. Raised to 95% with cmpatch release patch, which took under a second, it updated on the next launch.

Then my next release failed with a 409: “deployment has an active rollout below 100 percent”. Only one partial rollout per deployment is allowed, so you finish, disable or roll back a canary before shipping anything else. That’s reasonable, and documented, but a pipeline that publishes on every merge will run into it in the middle of a canary. Since I reported it, the CLI explains itself here: if an active partial rollout is blocking you, it says waiting won’t help and how to complete or disable the rollout, and if another release job is still running it tells you to wait with release inspect --wait. The docs now have a section on sequential release changes.

Mandatory releases use the SDK’s mandatoryInstallMode. With mine set to apply on the next resume, the app downloaded the update and showed its prompt, and when I ignored the prompt, sent the app to the background and brought it back, the new version was simply there.

Disabling a release does more than stop offering it. The moment I disabled the latest one, the manifests pointed at the newest release still enabled, and devices on the disabled one were offered a patch back to it. Disable everything for a binary version and the manifest’s target becomes null, and the SDK reverts the app to the bundle it shipped with. My update hook didn’t prompt for that case at first, so now it does.

Promoting from Staging to Production took under a second and reused the exact package that was tested, same hash and notes, rather than rebuilding it. It also accepts a rollout percentage, so “promote to 10% of Production” is one command.

Building and publishing can also be separate steps. cmpatch bundle produces a .cmpatch artifact, and the dashboard’s “Bundle upload” option reads it before uploading: platform, target version, fingerprint, signing status, size and bundler. That suits a setup where CI builds the artifact and someone else decides when to publish it.

That’s also where I ran into the fingerprint guard in earnest. After I fixed my stale widget plist, the server refused my next release with a 409, because the native project’s fingerprint no longer matched the one it had recorded for 2.5.1. On Android, editing a comment in app.config.js was enough, since Expo includes the app config in the fingerprint. The dashboard shows the same check as a warning with both fingerprints and an “Upload anyway” button. In both cases nothing native had changed on the installed binary, so overriding was right. In real use, though, the honest answer is nearly always a version bump and a store build, and my publish script now says exactly that when it happens.

Code signing is RS256: the CLI signs the package hash with an RSA private key, and the SDK checks it against a public key compiled into the binary. The key goes in as a single base64 line, which fit my build-setting approach, and I tested it on both platforms. With the key in the app, an unsigned release was refused before it downloaded. With “require code signing” turned on for the app, the server rejected an unsigned publish with an error that named the fix and linked the guide. A signed release installed, and the SDK recorded the signature as verified. It also refused an older unsigned release the server fell back to when I disabled the signed one. The catch is that the public key has to be in the first binary you ship, because an update can’t add it.

localhost:8080

Teams and CI are covered as well. Members get one of four team-wide roles (owner, admin, developer, viewer) and invitations wait until the person first signs in. Personal access tokens worked with an empty home directory, passed either as --token or through environment variables, and the token list shows when each was last used. I’ve also written a manually triggered CI workflow that runs the same publish script on a Linux runner with the signing key from a secret, though I haven’t run it yet, since it needs the hosted server.

One tool I’d have skipped at the time is cmpatch debug ios. It streamed the system log filtered on “OTA” case-insensitively, which matches “Rotation”, so a twenty-second app launch produced a screenful of SpringBoard messages and nothing from the app. That’s fixed: it now shows only lines carrying the SDK’s markers. The SDK doesn’t log much on its own, mostly errors and development-build warnings, so expect a quiet stream. The getpwuid_r warning you may see comes from xcrun simctl, not Patch.

How hard is it, really

#

Codemagic asked for my opinion, so here it is, as plainly as I can put it.

Trying it out is easy. Two commands, a few minutes, and you have the real server, a dashboard with demo data, and a CLI you can point at your own app. I’d rate it the best part of the first hour.

Integrating the SDK is moderate work, and most of the difficulty is in your own project rather than in Patch. On a fresh Expo app with generated native folders, it really is the plugin plus a few lines replacing expo-updates calls. On an app like maru, with committed native projects, a widget extension, a web build, R8 and build profiles, the work was in the edges. I had to apply the native changes by hand to avoid prebuild, make the configuration build-time so Staging and Production builds could differ, and keep the SDK away from web. I also had to make sure the JavaScript in an update was built with the same environment as the binary. None of it was hard once I understood it. When I did it most of that wasn’t in the docs, though the committed-native-projects path and the bundle-time environment now are.

Day-to-day use is good. Publishing is one command and about half a minute. Rollouts, promotion, disabling and rollback are all under a second from the CLI or the dashboard, and the errors say what went wrong and what to do. The server’s guards (the fingerprint check, one canary at a time, signing) are strict in ways that stopped me doing the wrong thing more than once.

Running it in production is the bigger commitment. It’s a Docker Compose stack with Postgres and object storage that you have to host, back up and upgrade, and it needs two domains and an OAuth app. For a team already running servers that’s a quiet afternoon. For a team on EAS precisely because they didn’t want servers, it’s the real cost of switching, more than the SDK work.

In wall-clock terms, I installed the CLI at 12:29 and had the first update applying on a simulator at 13:34, and half an hour of that was an Xcode build that stalled for reasons unrelated to Patch. Everything in this post, on both platforms, was done by mid-afternoon, not counting the hosted server.

The things that cost me the most time, in order: expo prebuild --clean wiping my committed native projects, and then working out how to wire everything by hand; the iOS version-detection bug, which would have sent updates to a version nobody runs; the double-counted installs, which were my own doing but easy for anyone to repeat, since sync() calls notifyAppReady() itself; and the undocumented three-launch crash budget, which surprised me halfway through a test. After those came things that are correct but need planning: the fingerprint guard firing after harmless native-adjacent edits, one canary at a time per deployment, and lifecycle jobs that can’t overlap, which matters as soon as a script does several things in a row. The rest was small: a silent first-run setup, a noisy debug command, and a few docs examples that didn’t match the code.

I wrote all of it up for Codemagic, with the details, and by 23 September most of it was fixed: the version detection, the overlapping confirmations, the prebuild warning, the crash budget, the silent setup, the debug command, the InstallMode examples, and the error you get when a release job or a canary is in the way. What’s still open is smaller: the dashboard showing the API on :8080 while everything else says :3000, the SDK throwing on import on web and in Jest, and a few docs gaps.

Going to production

#

Everything above ran against the evaluation stack. Production is the same app and the same scripts pointed at a real server, and this is the plan for maru.

The supported way to run the server is the Docker Compose install: a Linux machine with ports 80 and 443 open, two domains (one for the API and dashboard, one for downloads), and a GitHub OAuth app for sign-in, which the server won’t start without. The installer does the rest, including TLS:

scripts/selfhost/install.sh \
  --api-domain updates.example.com \
  --storage-domain storage.updates.example.com \
  --email you@example.com \
  --github-oauth-client-id <id> --github-oauth-client-secret <secret>

It writes a .env.selfhost with the database and storage secrets, which belongs in your password manager.

You don’t have to use the bundled database and storage, though, and that’s what makes it fit a stack like mine. maru’s API already runs on Fly.io and its files live on Tigris, and Patch works on both. Underneath the Compose file, the server is one container running in MODE=all (API and release worker together) that needs a Postgres database, S3-compatible storage and a public HTTPS URL for the files. That’s a Fly app built from the repository’s Dockerfile (there’s no published image yet, so fly deploy builds it), a Fly Postgres database, and a Tigris bucket:

MODE=all
DATABASE_URL=postgresql://…            # Fly Postgres
STORAGE_ADAPTER=s3
S3_ENDPOINT=https://fly.storage.tigris.dev
S3_REGION=auto
S3_BUCKET=myapp-patch
S3_FORCE_PATH_STYLE=true
PUBLIC_BASE_URL=https://myapp-patch.fly.storage.tigris.dev/codemagic-patch

Two caveats. Running it outside Compose is documented in the repository as reference material, not a supported path in this first open-source release, so on Fly you’re on your own for upgrades and backups. I haven’t deployed it that way yet either; this is the plan, not a report. And if you’ve read my post on Fly’s scale-to-zero, keep at least one machine running. Phones never wait on the API for an update check, since manifests come straight from storage, but every metrics upload and every release job would otherwise start with a cold boot.

Tigris is the part I’m most keen on. It’s where maru’s files already are, and it puts a CDN in front of the URL every app launch hits. Mind the manifest caching if you do this: with the plain base-URL delivery adapter Patch serves manifests as no-cache, because it can’t purge a CDN, and a manifest cached on its own TTL would keep serving a release you’ve just rolled back. The configuration reference now documents this under MANIFEST_CACHE_CONTROL, and the docs site covers S3-compatible storage.

On the hosted server you create the apps again, put the new keys in the preview and production profiles in eas.json (Staging keys and Production keys), and create a token for publishing:

cmpatch token create --name ci

If you want code signing, decide now: generate the production key pair, keep the private key as a CI secret, put the public key in the build profiles, and turn on “require code signing” for both apps. The first Patch binary also needs a new version number, because the binaries already in the stores run expo-updates. Build it with the preview profile for TestFlight and Play internal testing first, and check the keys actually made it into the built Info.plist and Android resources before handing it to anyone. That’s the one link my local tests didn’t exercise, since I exported the environment by hand. Then publish to Staging, confirm one download and one success per tester, and only then build for the stores.

Production releases start small and grow:

./scripts/publish-update.sh ios --production --rollout 10
cmpatch release patch --app maru-ios --deployment Production --label v1 --rollout-percentage 100

The old server stays up in the meantime. Every copy of 2.5.1 that people haven’t updated still checks it, and hotfixes for them still go through the old script. Once the logs show those requests have stopped, the old server, its publish script and its secret can all go.

The first native upgrade

#

Added 16 October 2026.

Patch has been carrying maru’s production updates since early October. The first native release since then was the move to Expo SDK 58: React Native 0.88, and on Android a new major version of the Gradle plugin. A native change can’t go over the air, so it’s a new binary, 2.6.0, and my publish script refuses to send 2.6.0 updates until that binary exists. Both platforms built cleanly the first time.

The production section above says to check the keys really made it into the built Info.plist and Android resources before handing a build to anyone. This time I did, and that check is the reason this section exists. The iOS build had its four Patch settings. The Android build had none:

bundletool dump resources --bundle=build.aab --values | grep -A1 "string/Codemagic"

That command printed four keys for the 2.5.11 build and nothing for 2.6.0. The environment wasn’t the problem: Gradle’s generateReleaseResValues writes all four. They’re removed later by the release resource shrinker, which in Android Gradle Plugin 9 is much more aggressive. The same app went from 9,240 resources to 7,116. The SDK reads its configuration by name with getIdentifier, which the shrinker can’t see, so it removed the four strings as unused. A missing string reads as empty, and empty means “no updates”. Nothing would have failed. Android 2.6.0 would have installed, run normally and never received an update, and the only sign would have been no Android 2.6.0 devices on the dashboard.

The fix is one resource file that tells the shrinker to keep the four strings:

<!-- android/app/src/main/res/raw/codemagic_patch_keep.xml -->
<resources xmlns:tools="http://schemas.android.com/tools"
    tools:keep="@string/CodemagicPatch*" />

Don’t call it keep.xml: React Native’s bundle task already generates a raw/keep.xml for your JavaScript assets (which is why images and fonts survived), and the two would collide. The better place for this rule is the SDK itself, since a library’s keep file is merged into the app automatically, and a warning when a release build starts with no deployment key would turn the silent case into a visible one. This is the same kind of failure as the iOS version-detection bug earlier: nothing errors, and the updates just never arrive.

The same check turned up a second problem that has nothing to do with Patch, and it’s worth knowing for a different reason. The Android release build crashed on launch, twice, for two separate reasons inside Expo’s own modules. First, Expo’s widgets module failed to register its native classes, so the app died before any JavaScript ran. With that fixed, the app got as far as starting its JavaScript and then stopped, because three modules that had moved to Expo’s new module system were never registered. Patch’s crash rollback can’t help with either, because there is no update to roll back. It protects you from a bad update, not from a bad binary. A native upgrade still has to be installed and opened on a real device or emulator before it goes to the store, Patch or no Patch.

One smaller thing came out of the same testing. When sync() fails, it returns "error" and keeps the reason to itself unless the app is running in development, so a release build can only log that something went wrong. My hook now calls checkForUpdate() once after a failed sync and logs what that throws, which is how I learned that a local simulator build had empty Patch URLs (unsupported URL) rather than a broken server. It would help if the last error were readable in release builds too.

My checklist for a native release now has three lines it didn’t have before: dump the Patch settings from the built .ipa and .aab, compare the deployment keys with the build profile, and launch the release build itself, not a debug build, before submitting.

What I’d tell someone with an Expo bill

#

If you’re paying for EAS Update because building a server looked like too much work, Patch is a real answer: you run a server, but it’s a Compose stack and an installer, not code you maintain. Compared with what I built, you get staged rollouts, per-release install and failure numbers, binary patches, a fingerprint guard, code signing, crash rollback on the device, and a rollback button. I had built none of those.

What it costs: a new binary before anyone gets a Patch update, since the SDK has to be in the app; a server to keep upgraded and backed up; three crashed launches per user by default when a release is bad enough; and some hand work if your native projects are committed. It’s also young. This was version 0.3.0, and I hit a version-detection bug, a noisy debug command and a few gaps in the docs, and made a bug of my own that the SDK could have prevented. The version detection and the debug command have since been fixed, most of the docs gaps filled, and the SDK now guards against the mistake I made. None of it stopped the migration, and all of it was visible, in the CLI, in the dashboard, or in the SDK’s own files on the device, which is more than I can say for my old setup.

Update, 13 September 2026. Since this went up, Codemagic have released Patch, an open-source, self-hostable OTA server in the CodePush mould. If you got here because you’d rather not build the server yourself, it’s now a real option: I’ve since moved an app from the setup below to Patch and written up how that went, and there’s a short note on it under Wrapping up. Disclosure: Codemagic paid me to do that migration and write about it, and asked for a mention here. The code in the rest of this post has since been trimmed to short examples.

My estimating app gets its bug fixes pushed over the air: JavaScript changes land on users’ phones without going anywhere near App Store review. Expo’s hosted service, EAS Update, will happily do this for you. I built my own version anyway, with Django REST Framework and Tigris S3, and it has served every iOS and Android update since.

Expo OTA updates work like this: the app’s expo-updates library asks an update server for a manifest, sending its platform and runtime version in headers. The server answers with the newest compatible update and the URLs of its JavaScript bundle and assets, and the app downloads them and runs the new bundle on its next launch. EAS Update is Expo’s hosted server for that protocol, but any server that speaks it will do, and that’s what I built.

This post walks through the whole thing: the models, the manifest endpoint, the publishing pipeline, and the gotchas that only show up once you’re running it yourself.

Why self-host at all?

#

Mostly cost, in my case. EAS Update pricing scales with usage, so pushing frequent updates to a growing user base adds up, while S3-compatible storage like Tigris stays nearly free. There are other reasons that might weigh more for you: some industries have to keep every application asset inside their own infrastructure, and owning the server means owning the update flow, so you can roll out to specific user segments or A/B test bundles if your product calls for it. It also means nothing about your update pipeline depends on Expo’s availability or their next pricing change.

How the pieces fit

#

Four components: a Django backend that serves update manifests and stores metadata, Tigris holding the actual bundle and asset files, a publishing script that exports, uploads and registers updates, and the mobile app itself pointed at my server instead of Expo’s.

┌─────────────────┐
│   Mobile App    │
│  (expo-updates) │
└────────┬────────┘
         │ 1. Request manifest
         │    (with headers: platform, runtime-version)
         ↓
┌─────────────────┐
│  Django Server  │
│  /api/expo-     │◄─── 2. Query DB for latest update
│   updates/      │
│   manifest/     │
└────────┬────────┘
         │ 3. Generate presigned URLs
         │
         ↓
┌─────────────────┐
│   Tigris S3     │
│  (Asset Files)  │◄─── 4. App downloads bundles directly
└─────────────────┘

Building it

#

The two models

#

Everything hangs off ExpoUpdate and ExpoUpdateAsset. The first stores metadata for each update:

class ExpoUpdate(models.Model):
    id = models.UUIDField(primary_key=True, default=uuid.uuid4)
    runtime_version = models.CharField(max_length=50, db_index=True)
    platform = models.CharField(
        max_length=10,
        choices=[("ios", "iOS"), ("android", "Android")],
        db_index=True
    )
    is_active = models.BooleanField(default=True, db_index=True)
    manifest_data = models.JSONField()
    description = models.TextField(blank=True)
    created_at = models.DateTimeField(auto_now_add=True)

    class Meta:
        indexes = [
            models.Index(fields=["runtime_version", "platform", "is_active", "-created_at"])
        ]

A few deliberate choices in there. runtime_version mirrors the runtimeVersion in app.json, and it does a lot of work: clients only ever download updates matching their own runtime version. iOS and Android get separate rows because the bundles differ. is_active is the rollback mechanism; deactivate a bad update and clients fall back to the previous one. And manifest_data holds the complete Expo Updates v1 protocol manifest as JSON, so serving it later is just a lookup.

ExpoUpdateAsset tracks the individual files:

class ExpoUpdateAsset(models.Model):
    id = models.UUIDField(primary_key=True, default=uuid.uuid4)
    update = models.ForeignKey(ExpoUpdate, on_delete=models.CASCADE, related_name="assets")
    hash = models.CharField(max_length=255, db_index=True)
    key = models.CharField(max_length=255)
    content_type = models.CharField(max_length=100)
    file_extension = models.CharField(max_length=10)
    file_path = models.CharField(max_length=500)
    file_size = models.IntegerField(default=0)

Assets are keyed by SHA-256 hash, which makes them immutable and cacheable. The same asset can be shared across any number of updates.

The manifest endpoint

#

/api/expo-updates/manifest/ is where app and server actually talk. It implements the Expo Updates v1 protocol:

@action(detail=False, methods=["get"], url_path="manifest")
def manifest(self, request):
    # Extract required headers
    protocol_version = request.META.get("HTTP_EXPO_PROTOCOL_VERSION")
    platform = request.META.get("HTTP_EXPO_PLATFORM")
    runtime_version = request.META.get("HTTP_EXPO_RUNTIME_VERSION")

    # Validate protocol version
    if protocol_version != "1":
        return Response(
            {"error": f"Unsupported protocol version: {protocol_version}"},
            status=400
        )

    # Find latest active update for this runtime + platform
    update = ExpoUpdate.objects.filter(
        runtime_version=runtime_version,
        platform=platform,
        is_active=True,
    ).order_by("-created_at").first()

    # No update available - client uses embedded bundle
    if not update:
        response = Response(status=204)
        response["expo-protocol-version"] = "1"
        return response

    # Generate presigned URLs for all assets
    manifest_data = build_manifest(update)  # fills in presigned asset URLs

    return Response(manifest_data, status=200)

Three details matter here. A 204 response means “no update”, and the app carries on with its embedded bundle. The asset URLs in the manifest are presigned, so the app downloads straight from Tigris’s CDN rather than streaming everything through Django. And the expo-protocol-version response header isn’t optional; the client checks for it.

Publishing updates

#

Publishing is a Django management command with a shell script wrapped around it. The command reads the output of expo export, hashes every asset with SHA-256, uploads bundles and assets to Tigris in parallel, writes the database records, and optionally drops an import JSON into the bucket for production sync. Cut down to the part that matters:

from concurrent.futures import ThreadPoolExecutor

from django.db import transaction


def publish(platform, runtime_version, manifest, files):
    # Upload the bundle and assets in parallel (keys are content hashes)
    with ThreadPoolExecutor(max_workers=10) as executor:
        list(executor.map(upload_asset, files))

    # Swap the active update in one transaction
    with transaction.atomic():
        ExpoUpdate.objects.filter(
            runtime_version=runtime_version,
            platform=platform,
            is_active=True,
        ).update(is_active=False)

        return ExpoUpdate.objects.create(
            runtime_version=runtime_version,
            platform=platform,
            manifest_data=manifest,
        )

The wrapper script is what I actually type:

# Publish to local environment
./publish.sh ios

# Publish to production
./publish.sh ios --production

# Dry run to validate
./publish.sh --dry-run

It loads production environment variables before exporting, handles one platform or both, and takes care of the production sync described next.

Syncing to production

#

I didn’t want production credentials sitting on my laptop, so production deploys happen in two steps: publish locally (assets go to Tigris, along with a JSON snapshot of the metadata), then call a production API endpoint with the S3 path so the server imports the metadata itself:

@action(detail=False, methods=["post"], url_path="import-update")
def import_update(self, request):
    # Authenticate via Bearer token
    token = request.META.get("HTTP_AUTHORIZATION", "").removeprefix("Bearer ")
    if not hmac.compare_digest(token, settings.IMPORT_TOKEN):
        return Response({"error": "Invalid token"}, status=401)

    # Download the import JSON from the bucket
    s3_key = request.data.get("s3_key")
    obj = s3_client.get_object(Bucket=bucket_name, Key=s3_key)
    data = json.loads(obj["Body"].read())

    # Import into the production database
    with transaction.atomic():
        update, _ = ExpoUpdate.objects.update_or_create(id=data["id"], defaults={...})
        for asset_data in data["assets"]:
            ExpoUpdateAsset.objects.update_or_create(...)

    # Clean up the import JSON
    s3_client.delete_object(Bucket=bucket_name, Key=s3_key)
    return Response({"update_id": str(update.id)})

Pointing the app at your server

#

In app.json, set the update URL and runtime version:

{
  "expo": {
    "runtimeVersion": "1.0.0",
    "updates": {
      "url": "https://api.example.com/api/expo-updates/manifest/"
    }
  }
}

Be strict about keeping runtimeVersion matched between app and server. When you change native code or upgrade the Expo SDK, bump the runtime version and publish fresh updates for it.

Storage on Tigris

#

Tigris is S3-compatible object storage, considerably cheaper than actual AWS S3, and it includes global edge caching, which matters when users are pulling down multi-megabyte bundles. The Django configuration:

# settings.py
BUCKET_NAME = os.getenv("BUCKET_NAME")
AWS_ENDPOINT_URL_S3 = os.getenv("AWS_ENDPOINT_URL_S3")
AWS_ACCESS_KEY_ID = os.getenv("AWS_ACCESS_KEY_ID")
AWS_SECRET_ACCESS_KEY = os.getenv("AWS_SECRET_ACCESS_KEY")
AWS_REGION = os.getenv("AWS_REGION", "auto")

Creating the client is standard boto3:

import boto3

def create_s3_client(endpoint_url, region, access_key, secret_key):
    return boto3.client(
        "s3",
        endpoint_url=endpoint_url,
        region_name=region,
        aws_access_key_id=access_key,
        aws_secret_access_key=secret_key,
    )

Downloads go through presigned URLs, so the app fetches straight from the CDN:

presigned_url = s3_client.generate_presigned_url(
    "get_object",
    Params={"Bucket": bucket_name, "Key": asset.file_path},
    ExpiresIn=3600,  # 1 hour
)

Security

#

The manifest endpoint is unauthenticated on purpose, because apps need updates before anyone has logged in. The import endpoint is a different story. It can write to the production database, so it wants a shared secret, compared in constant time to rule out timing attacks:

IMPORT_TOKEN = os.getenv("IMPORT_TOKEN")

# Constant-time comparison prevents timing attacks
if not hmac.compare_digest(token, IMPORT_TOKEN):
    return Response({"error": "Invalid token"}, status=401)

Asset integrity comes free with the design: everything is verified against its SHA-256 hash, so a tampered asset simply fails. And presigned URLs expire after an hour, which keeps bundles from being hotlinked indefinitely.

Keeping it fast

#

The composite index on (runtime_version, platform, is_active, -created_at) keeps the manifest query quick no matter how many updates accumulate:

class Meta:
    indexes = [
        models.Index(fields=["runtime_version", "platform", "is_active", "-created_at"])
    ]

The publishing script uploads assets concurrently:

with ThreadPoolExecutor(max_workers=10) as executor:
    futures = {executor.submit(upload_asset, asset): asset for asset in assets}
    for future in as_completed(futures):
        # Track progress

For a typical update with 50 or so assets, that took publishing from about 2 minutes down to 15 seconds. Distribution needs no work at all: Tigris caches assets at edge locations near your users on its own.

Day-to-day workflow

#

During development

#

# 1. Make code changes in the app
git commit -am "Fix bug"

# 2. Publish OTA update to local environment
./publish.sh ios

# 3. Test on device
# App automatically downloads and applies update

Shipping to production

#

# 1. Publish to production
./publish.sh ios --production

# 2. Monitor
# Check Django admin for update records
# Verify assets in Tigris dashboard

Rolling back

#

# Mark problematic update as inactive in Django admin
# or via management shell:
python manage.py shell

>>> from updates.models import ExpoUpdate
>>> bad_update = ExpoUpdate.objects.get(id="uuid-here")
>>> bad_update.is_active = False
>>> bad_update.save()

# Clients will now receive the previous active update

What it costs

#

The app has around 500 active users. On Tigris, the ~200 MB of historical updates costs about $0.02/month, and the ~50 GB of monthly egress from update downloads about $1.00. The Django side rides along on the server my API already runs on, so its incremental cost is zero. Call it a dollar a month, all in. EAS Update at similar usage would land somewhere around $300–500/year, so this paid for itself more or less immediately.

Monitoring and debugging

#

Nothing fancy here. The viewset logs every manifest request:

logger.info(f"Manifest request: platform={platform}, runtime={runtime_version}")

The Django admin is the de facto dashboard. Register the models and you can inspect and filter everything:

@admin.register(ExpoUpdate)
class ExpoUpdateAdmin(admin.ModelAdmin):
    list_display = ["platform", "runtime_version", "is_active", "created_at"]
    list_filter = ["platform", "is_active", "runtime_version"]
    search_fields = ["description"]

And on the client, expo-updates will tell you what it sees:

import * as Updates from 'expo-updates';

Updates.checkForUpdateAsync().then(update => {
  console.log('Update available:', update.isAvailable);
  console.log('Manifest:', update.manifest);
});

Gotchas

#

Runtime version mismatches

#

The most common failure is also the quietest: clients only download updates matching their runtime version. If the installed app is on runtime 1.0.0 and you publish for 1.0.1, nothing arrives and nothing errors. Keep the runtime version in sync with your builds, and only bump it when native code changes.

The createdAt timestamp

#

The expo-updates client compares the manifest’s createdAt against the embedded bundle’s commitTime, and only applies an update whose createdAt is newer. The viewset overrides it with the database timestamp:

manifest_data["createdAt"] = update.created_at.strftime("%Y-%m-%dT%H:%M:%S.%fZ")

For local development, if you build a binary after publishing an OTA, republish the OTA so its timestamp wins.

Manifest asset key values

#

The key field on each manifest asset is what expo-updates uses for caching, and it must be a deterministic hash (the MD5 of the filename works), not a random UUID or arbitrary string. Get this wrong and the client may fail to cache or retrieve assets correctly, and updates silently break after the first successful load. Thanks to reader Raphael Mutschler for flagging this one; he had updates working exactly once before discovering the issue.

App config in expoClient

#

If your app uses Linking, Constants, or anything else that reads the app config at runtime, the manifest’s extra.expoClient field needs to include your app config. Without it the app may launch fine at first, then crash or refuse to reopen after being closed. That’s because expo-updates replaces the embedded manifest with the OTA one, and if expoClient is missing, those APIs lose the config they depend on. Raphael Mutschler caught this one too.

Asset cleanup

#

Old updates pile up in Tigris. For now, cleanup at my end is manual:

# Delete updates older than 30 days
from datetime import timedelta
from django.utils import timezone

cutoff = timezone.now() - timedelta(days=30)
old_updates = ExpoUpdate.objects.filter(created_at__lt=cutoff, is_active=False)

for update in old_updates:
    # Delete assets from S3
    for asset in update.assets.all():
        s3_client.delete_object(Bucket=bucket_name, Key=asset.file_path)
    # Delete DB records
    update.delete()

A scheduled task is the obvious home for this. I haven’t got round to it.

Things I’d add next

#

(September 2026: Patch now does all three of these and more; see the migration write-up.)

Gradual rollouts

#

A rollout_percentage field would let an update go out to a slice of users first:

rollout_percentage = models.IntegerField(default=100)

# In the manifest view:
if update.rollout_percentage < 100:
    # Hash user ID and check if they're in rollout group
    user_hash = int(hashlib.sha256(user_id.encode()).hexdigest(), 16)
    if (user_hash % 100) >= update.rollout_percentage:
        return Response(status=204)  # No update