The general technique is called test impact analysis, and testmon is a free, open-source way to get it in Python. Nothing about it depends on git: it compares against what it saw on the last run, committed or not.

I added it to my marketplace app’s Python tests this week, and the reason wasn’t speed for its own sake.

Why I added it

#

A feature request is usually small: one screen, one endpoint, one rule that should work differently. Running the whole test suite made every one of them bigger than it was. The suite is 2,248 Django tests, about three and a half minutes with four processes and 11 in one, and every edit ran all of it, though almost all of it checks things the request never touched. When something unrelated failed, or the run simply took its time, the feature waited. It was being held up by circumstances outside the feature request, and the thing holding it up was my own test suite.

The alternative wasn’t better. Skip the full run, guess which test file covers the change, and now and then guess wrong. That’s quick, but it isn’t precise, and an untested change I ship is a feature request I’ll be doing twice.

The tests I wait for should be exactly the tests the change can reach, no more and no fewer. testmon gives me that set from a record instead of a guess. The full suite still runs, in CI, where it doesn’t hold up whatever I’m working on next.

How it decides what to run

#

testmon splits each Python file into blocks: the module’s top level, each class, each function, with the bodies of nested functions cut out of the block around them. It stores a checksum of each block, never the code itself. While a test runs, coverage.py records the lines it executes, and testmon notes which blocks those lines belong to. There’s no static analysis and no guessing from imports. A test depends on exactly the blocks it ran.

Take a module like this:

# pricing.py
def subtotal(items):
    return sum(item.price * item.quantity for item in items)

def discount(total, code):
    if code == "WELCOME10":
        return total * 0.9
    return total

A test that only calls subtotal() depends on the module’s top level and on subtotal’s body. Rewrite discount() however you like and testmon won’t rerun that test, because it never executed a line of discount. Change one character inside subtotal and it runs again, even if the change can’t alter the result. testmon doesn’t try to be clever about side effects: if a block you executed changed, you run.

It also tracks what’s around the code. The record includes the Python version and the installed packages with their versions. It doesn’t watch environment variables: if the same code runs under two settings modules, name them with --testmon-env (step 8) or they share one record. Upgrade Django and everything that touched Django is affected again, which in a Django app is everything.

What it can’t see

#

testmon tracks Python that a test executes. It doesn’t track the other files your tests read: templates, JSON, text, anything else on disk. It doesn’t track services reached over the network. And it can’t pin anything that runs once before the tests start on a particular test.

In a Django project that means migrations, which run once when the test database is created, so no test’s record includes them (the wrapper below rebuilds the test database when there’s a new one); settings and start-up code, which run while pytest starts up, outside any one test; and templates and data files, since a template isn’t Python, so editing one reruns nothing that renders it.

My suite has a few more of its own. Some tests read a JSON file of fixtures that the mobile app’s tests share. Some check the built web bundle, and some check the output of a static site generated in the same repository. Change any of those and testmon will happily tell you nothing needs to run. That’s why the how-to below wraps testmon in a script that notices these files and runs everything when one has changed.

How to add it to a Django project

#

These steps assume pytest and pytest-django, and uv for the dependencies. With pip, swap uv add --dev for pip install and drop uv run.

1. Install it.

uv add --dev pytest-testmon

2. Make sure pytest can run the suite. testmon is a pytest plugin, so it does nothing under python manage.py test. pytest-django needs the settings module and the file patterns your tests use:

# pytest.ini
[pytest]
DJANGO_SETTINGS_MODULE = myproject.settings
python_files = tests.py test_*.py *_tests.py
python_classes = Test* *Tests *TestCase

If the suite has only ever run under Django’s runner, run it under pytest once and fix whatever differs before adding testmon. Otherwise you’ll be debugging two things at once.

3. Keep the record out of git. testmon writes .testmondata in the pytest root directory. It’s a SQLite database, so you’ll also see -shm and -wal files next to it, and the script below adds a stamp file. One pattern covers all of them:

# .gitignore
.testmondata*

4. Record the whole suite once. testmon can’t select anything until it has seen every test run:

uv run pytest --testmon

In a fresh checkout that first run is the whole suite in one process. Here it takes about 12 minutes for 2,248 tests. From then on, the same command runs only the affected tests. With nothing changed, it runs nothing.

Two catches. On testmon 2.2.0 and earlier, it needs pytest’s cache plugin: my first attempt ran with -p no:cacheprovider and crashed with KeyError: 'lf' (2.3.0 treats a missing --lf as unset). And on Python 3.14 or later, set COVERAGE_CORE=ctrace before you record. “The bug that looked like a green run”, further down, explains why. It’s the most important line in the script below.

Update, 7 November: testmon 2.3.0, released on 2 November, forces coverage’s C tracer itself, so on that version the export is belt and braces. Keep it if you’re on 2.2.0 or earlier, and keep the break-something check whatever the version.

5. Wrap it for the files it can’t see. This is the part that makes it safe to use every day. The script keeps a stamp file from the start of the last run. If any file testmon can’t trace is newer than the stamp, it runs everything and records afresh. A new migration only rebuilds the test database. Otherwise it lets testmon choose. A cut-down example:

#!/usr/bin/env bash
# scripts/test-changed.sh: the tests a change can affect, or all when in doubt.
#   scripts/test-changed.sh            # affected tests
#   scripts/test-changed.sh --all      # everything, re-recorded
#   scripts/test-changed.sh -k orders  # anything else goes to pytest
set -euo pipefail
cd "$(dirname "$0")/.."

# coverage's C tracer: from Python 3.14 coverage defaults to sys.monitoring,
# which reports a line only the first time a process runs it, so testmon
# would credit each line to one test. Changing this means --all.
export COVERAGE_CORE=ctrace

STAMP=.testmondata.started
full=false
rebuild=false
if [ "${1:-}" = "--all" ]; then
    full=true; shift
elif [ ! -f .testmondata ] || [ ! -f "$STAMP" ]; then
    echo "No record yet: running everything once."
    full=true
else
    # Changed since the last run started, and not traceable per test:
    # anything that isn't Python, plus settings and conftest.
    newer="$(find myapp myproject fixtures pytest.ini \
        -type f -newer "$STAMP" ! -path '*/migrations/*' \
        \( ! -name '*.py' -o -path 'myproject/settings*' -o -name 'conftest.py' \) \
        ! -path '*/__pycache__/*' | head -5)"
    if [ -n "$newer" ]; then
        echo "Changed, and not traceable per test:"; printf '  %s\n' $newer
        full=true
    fi
    # A new migration needs a fresh test database, not a full run.
    [ -n "$(find myapp -path '*/migrations/*.py' -newer "$STAMP" | head -1)" ] && rebuild=true
fi

# Stamp the start, so an edit made while the tests run counts next time.
started="$(mktemp .testmondata.run.XXXXXX)"
args=(--testmon --reuse-db)
# --create-db alone deletes the database when the run ends; with
# --reuse-db as well, the rebuilt one is kept for next time.
[ "$rebuild" = true ] && args=(--testmon --reuse-db --create-db)
[ "$full" = true ] && args=(--testmon-noselect --reuse-db --create-db)

status=0
uv run pytest "${args[@]}" -q "$@" || status=$?
# 0: passed, or nothing selected. 1: some failed (still recorded).
# 5: nothing collected at all.
case "$status" in 0|1|5) mv "$started" "$STAMP" ;; *) rm -f "$started" ;; esac
[ "$status" = 5 ] && exit 0
exit "$status"

Two details in there are worth explaining. --reuse-db is pytest-django’s flag for keeping the test database between runs, which matters once a run takes seconds: creating it again each time can take longer than the tests themselves. A full run, or one after a new migration, adds --create-db so the database has the current schema. Keep --reuse-db next to it: --create-db on its own makes pytest-django delete the database when the run ends, so the next run would build it all over again.

And when testmon deselects everything, the run exits 0, so “nothing changed” counts as a pass. Exit code 5 is pytest’s “no tests collected”, which the script treats as fine too.

Notice what’s not in the list: pyproject.toml and the lock file. My first version ran everything whenever the lock file changed, which made the slow fallback fire on most days and the whole thing worse than before. testmon already tracks installed package versions itself, so it doesn’t need the help.

6. Give it its own database and cache. If other things run tests on the same machine, like a second terminal, a coding agent or manage.py test, point the script at its own test database and its own cache number so a kept database never meets someone else’s run:

export DATABASE_URL="postgresql://myapp:myapp@localhost:5432/myapp_changed"
export REDIS_URL="redis://localhost:6379/8"

7. Empty that cache before each run. I learnt this one the hard way. Subset runs started failing with KeyErrors that full runs didn’t. Some category labels are cached with no expiry, and a test database kept from one run met a cache still warm from an earlier one. Clearing the cache at the top of the script fixed it:

uv run python -c "import os, redis; redis.Redis.from_url(os.environ['REDIS_URL']).flushdb()"

That’s testmon doing what its docs say it will: running a different subset each time exposes tests that only passed because of what ran before them. The fix belongs in the test setup, not in testmon.

8. Know the flags for when selection is wrong.

• --testmon-noselect runs every test but puts the likely failures first. testmon switches to it by itself when you also pass -k, -m, --lf or a test’s node ID.
• --testmon-forceselect runs only affected tests that also match your -k or -m.
• --testmon-nocollect uses the existing record but doesn’t add to it. It’s forced under a debugger or when coverage is already running.
• --no-testmon turns it off for one run, for when it’s in addopts.
• --testmon-env=<name> keeps separate records in one file, for a suite you run under two settings modules.

If the project has a JavaScript side, Jest has its own version built in. jest --changedSince=main runs the tests that import anything changed since main, following imports rather than coverage. I added it to the app’s package.json as npm run test:changed so both halves have the same habit. An edit to one utility module selected 7 of the app’s 121 test files.

What it saves

#

I timed the wrapper once it was in place, start-up included, after fixing the bug in the next section:

For an everyday edit that’s about 25 times faster than the parallel suite, and when nothing has changed it’s about two hundred times faster.

The record says the same thing more broadly. Of the 307 non-test Python files the suite runs, a change to the median one reaches 22 tests, about 8 seconds of test time. A few core modules sit at the other end: almost every test runs them, so a change there reaches around 1,500 tests, and testmon reruns most of the suite. That’s the right answer, not a failure. Those figures are upper bounds per file: testmon works per block, so changing one function selects only the tests that ran that function.

The cost shows up when the wrapper falls back. A full run is the same single-process recording as the first one, about 12 minutes, which is slower than the parallel suite it replaces. It’s single-process only because I haven’t tried testmon under pytest-xdist yet, which it has supported since version 1.4; that’s the obvious next thing to measure. That’s why the list of files that trigger it should be as short as it can honestly be.

The bug that looked like a green run

#

My first numbers were better than these: one edit, 6 tests, 3.3 seconds. They were wrong, and nothing on screen said so.

From Python 3.14, coverage.py uses a new core by default, built on the interpreter’s sys.monitoring. That core reports a line only the first time a process runs it. testmon builds its record of which test ran which line from coverage, so each line was credited to the first test that happened to run it, and to no other. Edit a line that eight tests share, and testmon selects one of them.

It was caught on a second, smaller project of mine that copied this setup. Part of adopting it there was breaking a line on purpose and comparing the results, and the numbers didn’t match. Back in this suite I ran the same check. I broke one line in the home-screen banner code, the title of a slide. Plain pytest reported 8 failing tests. testmon selected 4. The other 4 broken tests never ran, and nothing in the output said they’d been left out.

The fix is one line, export COVERAGE_CORE=ctrace, which puts coverage back on its classic C tracer. A different tracer means a different record, so I re-recorded: 12 minutes 3 seconds for 2,248 tests. Afterwards, the same kind of breaking edit ran all 11 tests it broke.

Update, 7 November: testmon 2.3.0, released on 2 November, now sets the C tracer itself, in process and for subprocesses, so a fresh install no longer needs the export. The check in the next paragraph still does.

That’s the lesson I’d pass on before anything else here. A test selector that selects too little looks exactly like success: fewer tests, a faster run, all green. So when you set one up, break something shared on purpose, run the plain suite and the selector, and check they fail the same tests. Then do it again whenever Python or coverage.py changes version.

Where a coding agent comes in

#

I don’t make every change in that repository myself any more. A coding agent, Claude Code in my case, makes a lot of the small ones, and a good agent runs the tests after each edit. Faced with an 11-minute suite, it has the same two bad options I had: wait, or guess which test file matters.

So the repository’s agent instructions now say the wrapper script is the default way to run tests while working, with --all for a full run. The agent gets the same precise set of tests I do, after every edit, in seconds. It’s also why step 6 gives the script its own database and cache: on a busy day there’s more than one session running tests on the same machine, and a kept database must never meet somebody else’s run.

None of this needs anything AI-specific from testmon. It’s the same record and the same selection. It just matters more when whatever is making the changes makes a great many of them.

Why CI got something coarser

#

testmon has no CI mode. It behaves the same on a runner as on a laptop, and its author writes that sharing or caching .testmondata across CI runners is far less effective than their hosted service, testmon.net. A cached record would also be blind in all the ways above, and my CI runs the suite with Django’s runner in four processes, so I left testmon out.

What CI got instead is the same idea at file level. The checks workflow now starts with a small job that diffs the push and decides which half of the repository it touched: the Python API, the mobile app, both or neither. Before this, CI ran both the Python and the app jobs on every push, docs-only ones included. Now a docs-only push runs nothing, and a push to the app alone skips the Django suite. I dry-ran the job against five real commits: the docs-only one ran nothing, the app-only one skipped the Python job, and the backend-only one skipped the app job. A cut-down example:

jobs:
  changes:
    runs-on: ubuntu-latest
    outputs:
      python: ${{ steps.diff.outputs.python }}
    steps:
      - uses: actions/checkout@v7
        with:
          fetch-depth: 0
      - id: diff
        env:
          BEFORE: ${{ github.event.before }}
          HEAD: ${{ github.sha }}
        run: |
          FILES="$(git diff --name-only "$BEFORE" "$HEAD")"
          # Python's inputs include the non-Python files its tests read.
          PY='^(myapp/|myproject/|fixtures/shared\.json$|web/dist/|pyproject\.toml$|uv\.lock$)'
          if printf '%s\n' "$FILES" | grep -qE "$PY"; then
            echo "python=true" >> "$GITHUB_OUTPUT"
          else
            echo "python=false" >> "$GITHUB_OUTPUT"
          fi

  python:
    needs: changes
    if: needs.changes.outputs.python == 'true'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - run: uv run python manage.py test --parallel 4 --noinput

The real one also runs everything when there’s no range to compare: a new branch, a force push or a manual run.

The blind spot is the same one in a different form. testmon can’t see a JSON file, and neither can a path filter unless you list it. So the pattern names the shared fixtures, the built bundle and the generated site explicitly, with a comment asking whoever edits it to keep the list honest. The file-level version is cruder, but it fails in the safe direction: if anything on the Python side changed, the whole suite runs.

Is it worth it?

#

For a suite that takes minutes, yes. The real cost is what it can’t see, and that’s what the wrapper script is for. Even so, the laptop loop is where testmon belongs. CI still runs the whole suite whenever the Python side changes, and that’s the run I trust before a deploy.

If you came here to run Docker Compose in GitHub Actions: I did that for two years and stopped. CI now declares the same Postgres and Valkey images as services: and runs the same uv run … and npm run … commands straight on the runner; Compose stays what a developer runs. The rest is why, with the files.

In 2024 I wrote Use Docker Compose for CI/CD Instead of Platform-Specific YAML, which said to keep the build logic in Compose and let the CI tool do nothing but call it. Two years on, I went back through every repository I ship from. Four have workflows, each with one committer, me, and none of those workflows calls docker compose any more. The principle held; the mechanism didn’t. Here’s what replaced it, with cut-down example versions of the files, the step timings GitHub recorded, and the parts that are less automated than a CI/CD diagram would like.

Every pipeline I run, in one table

#

In textbook terms the Hugo sites have continuous deployment, and the app (a marketplace and community app I build) has continuous integration plus one-command releases, which is continuous delivery: the checks tell me something is wrong, but they don’t stop me deploying. That’s deliberate. An OTA update reaches every phone on the binary version it targets, so publishing one should be a decision rather than a side effect of a merge, and the API is where the database migrations run. My other two Django apps have no workflows at all.

The CI half: one workflow, two jobs

#

The app’s CI is one file, .github/workflows/checks.yml. Until 11 October there wasn’t any. This is a trimmed example of its shape:

# Lint, types and tests on every push and pull request to main.
# Nothing here deploys.
name: Checks

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

concurrency:
  group: checks-${{ github.ref }}
  cancel-in-progress: true

jobs:
  python:
    runs-on: ubuntu-latest
    timeout-minutes: 30
    services:
      postgres:
        image: postgres:17
        env:
          POSTGRES_USER: myapp
          POSTGRES_PASSWORD: myapp
          POSTGRES_DB: myapp
        ports: ["5432:5432"]
        options: >-
          --health-cmd "pg_isready -U myapp"
          --health-interval 5s --health-timeout 5s --health-retries 20
      valkey:
        image: valkey/valkey:8-alpine
        ports: ["6379:6379"]
    env:
      DEBUG: "True"
      DATABASE_URL: postgresql://myapp:myapp@localhost:5432/myapp
      REDIS_URL: redis://localhost:6379
      PYTHON_JIT: "0"   # match production
    steps:
      - uses: actions/checkout@v7
      - uses: astral-sh/setup-uv@v10
        with:
          enable-cache: true
          prune-cache: true
      - run: uv sync --locked
      - run: uv run ruff check .
      - run: uv run mypy myapp/
      - run: uv run python manage.py test --parallel 4 --noinput

  app:
    runs-on: ubuntu-latest
    timeout-minutes: 20
    defaults:
      run:
        working-directory: frontend
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-node@v7
        with:
          node-version: 24
          cache: npm
          cache-dependency-path: frontend/package-lock.json
      - run: npm ci
      # Expo writes this on `expo start` and it is gitignored,
      # so a fresh checkout doesn't have it.
      - run: printf '/// <reference types="expo/types" />\n' > expo-env.d.ts
      - run: npm run typecheck
      - run: npx jest --ci

The two jobs run side by side. Across the 35 green runs so far, the app job’s median is about seven minutes and the Python job’s about twelve and a half.

The services: block is where the Compose file went, which is the next section. The environment is a few variables and a flag; everything else falls back to the development default docker compose up gives a developer, so the tests need no secrets. I checked by running part of the suite with exactly that environment and nothing else.

CI runs the same interpreter as my laptop and the production image, uv’s own build of Python 3.15.0, because uv sync reads .python-version and downloads exactly that build. (My real file also passes setup-uv a python-version-file input, which does nothing: setup-uv v6 has no such input, and every run carries a warning saying so. I only noticed while writing this.)

Matching interpreters earned their keep eight days after the workflow went in. On 19 October I switched on 3.15’s JIT compiler, and the first run with it failed a test: a plain yield from, on a code path the app takes on every save, raised AttributeError: 'list' object has no attribute 'send'. The same commit with the JIT off passed. My Mac is arm64 and never hit it; the runner is x86_64, like the production machines. The JIT is off again, in CI and in the Dockerfile, and the bug never reached production. It’s python/cpython#155823, fixed upstream on 13 October, after 3.15.0 shipped.

The very first run failed too, for a duller reason. Expo writes expo-env.d.ts when you run expo start, and the file is gitignored, so every working copy I’d used had one and a fresh checkout didn’t. TypeScript reported five errors that had never appeared locally. The step that writes the file is the fix.

concurrency cancels a run when a newer push to the same branch arrives, which is how all twelve cancelled runs so far ended. A pre-commit hook runs the linter, plus the type checker when Python or TypeScript is staged, so most mistakes never reach a push.

Where the Docker Compose file went

#

Here is the part of the app’s docker-compose.yml that CI cares about, as an example. The real one has more services and a lot more environment.

services:
  valkey:
    image: valkey/valkey:8-alpine
    ports:
      - "6379:6379"

  # Host port 55432, not 5432, so it can't collide with a local Postgres.
  postgres:
    image: postgres:17-alpine
    environment:
      - POSTGRES_USER=myapp
      - POSTGRES_PASSWORD=myapp
      - POSTGRES_DB=myapp
    ports:
      - "55432:5432"
    volumes:
      - postgres_data:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U myapp -d myapp"]
      interval: 5s
      retries: 10

  api:
    build: .
    command: python manage.py runserver 0.0.0.0:8000
    volumes:
      - .:/app
    ports:
      - "8000:8000"
    environment:
      - DATABASE_URL=postgresql://myapp:myapp@postgres:5432/myapp
      - REDIS_URL=redis://valkey:6379
    depends_on:
      postgres:
        condition: service_healthy
      valkey:
        condition: service_started

volumes:
  postgres_data:

Put it next to the services: block in checks.yml and it’s the same two images: Postgres 17 (the Alpine build here, the Debian one in CI) and valkey/valkey:8-alpine. That’s the relationship now. The Compose file is what a developer runs; CI declares the same images as service containers and runs the same commands a developer runs, uv run … and npm run …, straight on the runner.

Against the 2024 post, the part I cared about most held: the CI tool is thin, and the checks and deploys are commands I can run myself. What didn’t survive is CI calling Compose.

The app’s CI doesn’t build the Docker image at all; Fly’s builder does that once, at deploy time, and CI only has to say whether the code is good, which with a cached toolchain takes a six-second install. The Compose file also bind-mounts the working copy and seeds data, which a test run shouldn’t, so CI would have needed a Compose file of its own: a second description of the same services, kept in step by hand.

And the 2024 example had a bug I didn’t notice at the time. docker compose build builds the image and never runs the service’s command, so the npm test in that file never ran in CI. The workflow went green whether the tests passed or not.

The risk that post worried about most, switching CI vendors, never came up. The move that did happen was at the other end: this blog and the client’s site moved hosts to Cloudflare Pages, and for the blog that took one new 45-line workflow, because the build was already one command. And the three-hundred-line workflow I warned about? I have one, pyhtml2text’s. It’s long because the wheels have to be built on Linux, macOS and Windows runners, for several architectures each, which makes it the one job where platform-specific YAML is the right tool.

Caching: read the step timings first

#

GitHub keeps the duration of every step, and gh api repos/<owner>/<repo>/actions/runs/<run-id>/jobs hands it over. These are from the app’s latest green run, on 20 October:

Both dependency caches are on, and both are the cheap part. The uv cache GitHub keeps for the repository is about 200 KB, because setup-uv prunes whatever uv can simply download again; the npm one is nearly 400 MB. The time is in the tests, which already run four ways in parallel. If I wanted the run faster, I’d change the suite, not the cache.

The blog is the opposite case. Hugo’s processed images live in the gitignored resources/_gen, so every fresh runner re-encoded every cover, body image and placeholder. The workflow further down restores them with actions/cache, keyed on a hash of the pictures and falling back to the newest older cache, since Hugo only processes what’s missing. In the last eight runs before the cache, the build step took 33 to 76 seconds; in the 26 since, 6 to 20, with one exception at 94, even though the same commit added placeholders and a second card size.

The app’s image builds on Fly’s builder, with the dependency files copied before the code and uv’s download cache in a BuildKit cache mount (RUN --mount=type=cache,target=/root/.cache/uv). The bigger fix wasn’t a cache. The old .dockerignore was a blocklist, and it missed a folder of extra git worktrees, each with its own dependencies and build output, so 7.7 GB across 277,000 files went to the builder on every deploy. Rewritten as an allowlist, the context is about 40 MB. An example of the pattern:

# Exclude everything, then let back in only what the Dockerfile COPYs.
*

!pyproject.toml
!uv.lock
!manage.py
!myapp
!static

# Rules match the whole relative path and the last match wins,
# so these re-excludes apply inside the allowed directories too.
**/__pycache__/
**/*.pyc
**/.DS_Store
**/*.log

Secrets, in as few places as possible

#

Secrets live in three places. GitHub Actions secrets are for what workflows do, and the two Hugo repositories have exactly one each, CLOUDFLARE_API_TOKEN. Fly secrets are the app’s runtime configuration, set with flyctl secrets set, and they never pass through Actions. And a gitignored .env on my laptop holds the token the deploy script uses to purge Cloudflare’s cache.

PyPI needs no secret at all. pyhtml2text uses trusted publishing: PyPI accepts the short-lived OIDC token GitHub issues to the workflow, so the release job asks for id-token: write and the repository’s secrets page is empty. There’s nothing to leak and nothing to rotate.

Tokens get the least they need. The app’s Fly token in GitHub serves a scheduled check that only lists machines, so it’s created with fly tokens create readonly. The Pages deploys ask for contents: read and deployments: write, and nothing more.

Inputs go through env:, never straight into a script. The OTA workflow takes free-text release notes. Written as ${{ inputs.notes }} inside run:, the text would be pasted into the script before bash reads it, so a note with a quote and a semicolon in it would become code. Passed as an environment variable, it stays data. An example of the pattern:

      - name: Publish
        env:
          OTA_API_TOKEN: ${{ secrets.OTA_API_TOKEN }}
          INPUT_PLATFORM: ${{ inputs.platform }}
          INPUT_ROLLOUT: ${{ inputs.rollout }}
          INPUT_NOTES: ${{ inputs.notes }}
        run: |
          args=("$INPUT_PLATFORM")
          [ -n "$INPUT_ROLLOUT" ] && args+=(--rollout "$INPUT_ROLLOUT")
          [ -n "$INPUT_NOTES" ] && args+=(--notes "$INPUT_NOTES")
          ./scripts/publish-ota.sh "${args[@]}"

Some keys stay off GitHub altogether. One scheduled report needs a key with admin rights over the app’s sign-in backend, so it moved from a scheduled workflow to a scheduled job on my Mac. The workflow is still there as a manual fallback, and leaves out the sections that need that key.

Deploys: on every push, by script, and on a tag

#

The Hugo sites deploy on every push

#

An example version of the blog’s workflow, with the first-run step that creates the Pages project left out:

name: Deploy to Cloudflare Pages

on:
  push:
  workflow_dispatch:

env:
  HUGO_VERSION: "0.165.0"
  PAGES_PROJECT: my-site

concurrency:
  group: deploy-${{ github.ref }}
  cancel-in-progress: true

jobs:
  deploy:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      deployments: write
    steps:
      - uses: actions/checkout@v7
        with:
          submodules: true

      - uses: peaceiris/actions-hugo@v3
        with:
          hugo-version: ${{ env.HUGO_VERSION }}
          extended: true

      # Processed images are gitignored; restore them so Hugo only
      # re-encodes what is new. An older cache still helps.
      - uses: actions/cache@v6
        with:
          path: resources/_gen
          key: hugo-resources-${{ hashFiles('content/**/*.jpg', 'content/**/*.png', 'assets/img/**') }}
          restore-keys: |
            hugo-resources-

      # A multilingual site has no root 404.html; without one, Pages
      # answers unknown paths with index.html and a 200.
      - name: Build
        run: |
          hugo --gc
          cp public/en/404.html public/404.html

      # ...

      - name: Deploy
        uses: cloudflare/wrangler-action@v4
        with:
          apiToken: ${{ secrets.CLOUDFLARE_API_TOKEN }}
          accountId: ${{ vars.CLOUDFLARE_ACCOUNT_ID }}
          command: pages deploy public --project-name=${{ env.PAGES_PROJECT }} --branch=${{ github.ref_name }}

--branch=${{ github.ref_name }} makes main production and gives every other branch a preview URL. The client site’s workflow mirrors this one, plus an rm -rf public before the build, because its public/ is committed and removed pages would otherwise linger in the upload.

Hugo is pinned here and nowhere else. In October a package-manager upgrade moved my laptop to 0.168, one past the newest version the theme supports. The deploys stayed on 0.165 and never noticed.

And the build is the test. On 14 September the preview deploy for one branch failed three times running: a post on it linked, through Hugo’s ref, to one that was still a draft. CI builds what production builds, drafts excluded, so the link pointed at nothing and Hugo stopped with REF_NOT_FOUND. main carried on deploying in the meantime.

The app’s API deploys from a script

#

The core of the deploy script, as an example:

IMAGE_LABEL="deploy-$(git rev-parse --short HEAD)-$(date +%Y%m%d%H%M%S)"

# Build first, then wake a spare machine, then roll out exactly that image.
flyctl deploy --config fly.toml --build-only --push --image-label "$IMAGE_LABEL"
curl -fsS --retry 5 https://myapp.fly.dev/health > /dev/null
flyctl deploy --config fly.toml --image "registry.fly.io/myapp:$IMAGE_LABEL"

And the matching lines in fly.toml:

[deploy]
  # max_unavailable only counts machines that are running,
  # which is why the script wakes one first.
  strategy = "rolling"
  max_unavailable = 1

The image is built once, pushed to Fly’s registry under a label made of the commit and the time, and deployed by that label, so what rolls out is exactly what was built.

Waking a spare first is what makes it zero-downtime. max_unavailable = 1 only counts running machines, and the app keeps none running when it’s idle (a suspended machine resumes in about a second), so a lone awake machine used to be replaced in the same batch as the sleeping ones, with nothing warm behind it. On 28 September a probe against the origin saw 7 timeouts and a 7.3-second gap during a plain flyctl deploy, and no failures in 2,514 requests during the scripted deploy, which builds first and has a machine awake before the rollout starts.

The rest of the script marks the deploy in the team chat channel the error alerts go to, so a burst of errors can be read against what just changed, purges Cloudflare’s cache and warms the machines. The web front end skips the image entirely; that’s in Deploy Frontend and Backend Separately, Without Redeploying the API.

Migrations aren’t a pipeline step. Fly’s release_command hung for me, so the first machine to boot on a new image takes a lock and migrates while the others wait. Every migration therefore has to work with the code that’s still running: add a column in one deploy, stop using it in the next, drop it in a third.

Nothing in the script needs my laptop as such, but a couple of local-only files keep it there: the Cloudflare credentials, and a record of what was last deployed, which a fresh runner wouldn’t have. It will also deploy uncommitted changes, and says so in the chat channel, where the web release script refuses a dirty tree.

In Preview Environments per Branch on Fly.io I wrote that doing this by hand is fine while you’re the only one doing it, and that from the second developer on it has to happen because something happened in git. The app is still at one developer; the branch-environment workflows from that post live on the branch I built them on.

The OTA workflow is manual-only because publishing should be a decision, and because the update service refuses a release while a partial rollout is open, which an on-push trigger would hit during every canary. It has never run. Every OTA update so far has gone out from my laptop through the same script the workflow calls, which is the case for thin workflows in one line: however a release starts, it’s the same script.

pyhtml2text releases on a tag

#

The release job, with the package name made generic:

  release:
    runs-on: ubuntu-latest
    if: ${{ startsWith(github.ref, 'refs/tags/') || github.event_name == 'workflow_dispatch' }}
    needs: [linux, macos, windows, sdist]
    environment:
      name: pypi
      url: https://pypi.org/p/mypackage
    permissions:
      id-token: write       # trusted publishing and signing
      contents: write
      attestations: write
    steps:
      - uses: actions/download-artifact@v8
      - uses: actions/attest@v4
        with:
          subject-path: 'wheels-*/*'
      - uses: astral-sh/setup-uv@v10
        if: ${{ startsWith(github.ref, 'refs/tags/') }}
        with:
          prune-cache: true
      - name: Publish to PyPI
        if: ${{ startsWith(github.ref, 'refs/tags/') }}
        run: uv publish --trusted-publishing always 'wheels-*/*'

The release job waits for every build job, and only a tag publishes; a manual run builds and attests but uploads nothing. actions/attest records which workflow built each wheel, and uv publish --trusted-publishing always uploads them with the OIDC token rather than a password. The package itself is in A Closed Pull Request, a Fork and a PyPI Package.

Scheduled workflows are best effort

#

The app has one scheduled workflow, a check that Fly hasn’t started more machines than the baseline, which would otherwise first show up on the bill. It alerts by failing, since GitHub emails the owner about a failed scheduled run.

It’s set to cron: '*/15 * * * *', 96 runs a day. Over the past two weeks it ran between three and eight times a day, with a median gap of about three hours forty. The only explanation GitHub’s documentation offers is load: scheduled runs can be delayed when Actions is busy, and dropped when it’s busy enough. For a sustained scale-out, a few hours is fine. For anything that has to fire within minutes, a GitHub cron isn’t a monitor.

What I’d ask of a pipeline

#

If you’re building one, or looking at one you already have:

1. If GitHub Actions were down for a day, could you still release? If not, your release logic lives in YAML, and the 2024 post still applies.
2. Does CI run the interpreter, the CPU architecture and the database major version that production runs? Mine caught a compiler bug only because the runner and production are both x86_64 and my laptop isn’t.
3. Does a fresh checkout pass? Generated, gitignored files are the usual reason it doesn’t.
4. Where does a run’s time actually go? Read the step timings before you add a cache.
5. For each secret, how many places does it live, and does it need to be in GitHub at all?

pip install pyhtml2text

from pyhtml2text import html2markdown

html = """<h2>Before you start</h2>
<p>Read the <a href="setup.html">setup guide</a> and <b>back up</b> your data.</p>
<img src="https://jared.lynskey.co.nz/img/diagram.png" alt="How it fits together">
<table><tr><th>Plan</th><th>Price</th></tr><tr><td>Basic</td><td>Free</td></tr></table>"""

print(html2markdown(html, base_url="https://example.com/docs/intro.html"))

## Before you start

Read the [setup guide](https://example.com/docs/setup.html) and **back up** your data.

![How it fits together](https://example.com/img/diagram.png)

| Plan | Price |
| --- | --- |
| Basic | Free |

It started as a pull request to rust-html2text. The maintainer closed it because it was written with an LLM, so pyhtml2text runs on a fork instead. This post covers how that happened, what’s in the fork and the package, and how I checked that the Markdown is right.

Why I needed it

#

A marketplace and community app I build answers Accept: text/markdown with Markdown instead of HTML, for AI agents and LLM clients that would rather not wade through nav bars and footers. I covered the edge side of that in the post on crawlers and the cache. Pages that come from app data build their Markdown directly, because their HTML is an empty JavaScript shell. Everything else is static HTML (the marketing, legal and blog pages), and that gets converted.

The problem with converting it is the links. A static site is full of links like /en/posts/... and ../setup/. They’re fine in a browser, which knows the page they came from. An agent reading the Markdown has no such context, so it can’t follow /login. It needs https://example.com/login.

The pure-Python html2text has a baseurl option for this, and that’s what I used at first. But it’s slow, and its Markdown has problems I’ll get to below. rust-html2text was fast and well tested, but it had no way to resolve links.

Asking upstream first

#

On 29 August I opened issue #287 with the use case and the behaviour I wanted: RFC 3986 resolution, with absolute URLs and mailto:, data: and #fragment links left alone. A few minutes later I opened pull request #288. It added a Config::base_url(...) option, and I said in the issue that it was a starting point and that I’d understand if the maintainer wanted to take a different approach.

On 11 October the maintainer closed the PR:

Thanks for the contribution, but I’m unable to accept this as it appears to be LLM generated. (I have now added a brief policy to the README.md - it wasn’t previously stated)

and replied on the issue:

Hi - this is a reasonable feature request, thanks! I have closed the PR as it appears to be LLM-written which I can’t accept, but thanks for the attempt.

A short HTML sample and expected output would definitely help get the feature implemented!

The README now has this policy:

At this time, this project is not able to accept code contributions produced with generative AI/LLMs due to the uncertain copyright status.

That was a correct read: I wrote that code with an AI coding assistant. I don’t think the policy is unreasonable, either. The copyright status of LLM output isn’t settled, and it’s the maintainer who would carry the risk of merging it into an MIT crate that other projects depend on. It’s their project, and the reply was polite. The feature request is still open.

I still needed the feature, though, and arguing with a maintainer over their own contribution policy isn’t a good way to get it. So I kept the change on a fork and built the Python package on top of that, and both say clearly that they aren’t upstream.

What’s in the fork

#

The fork is the pyhtml2text-perf branch of my rust-html2text fork. I’ve kept it to three small commits so that it rebases cleanly onto upstream. The main one is the Config::base_url option from the PR: relative links are resolved per RFC 3986 §5.3, and without a base URL the output is unchanged. Another stops the crate rebuilding its style rules on every call, which took a short snippet from 7.3 µs to 5.6 µs. That’s nothing on a big page, but it adds up when you call it a few thousand times. The third just exposes what the Markdown converter below needs from the crate.

The Python package

#

pyhtml2text is built with PyO3 and maturin, and it has two functions:

from pyhtml2text import html2text, html2markdown

html2text(html, width=80, *, base_url=None, decorate=True, link_footnotes=True,
          table_borders=True, wrap_links=True, max_wrap_width=None, raw=False)

html2markdown(html, *, base_url=None)

html2text is rust-html2text’s plain-text renderer, which is built for terminals and the text/plain part of an email:

print(html2text('<p>Please <a href="https://jared.lynskey.co.nz/login">sign in</a> first.</p>',
                base_url="https://example.com/docs/page.html"))

Please [sign in][1] first.

[1]: https://example.com/login

Some things about the bindings that may not be obvious:

• Both functions take str or UTF-8 bytes, so you can pass response.content without decoding it first. The input is borrowed, not copied.
• The GIL is released while converting. It’s pure Rust from start to finish, so threads scale across cores. The module also says it doesn’t need the GIL, so free-threaded Python 3.14t doesn’t turn it back on when the module is imported.
• A panic comes back as ValueError. By default PyO3 raises a PanicException, which except Exception doesn’t catch. A web worker that thinks it has handled conversion errors would crash on that, so the bindings catch it and re-raise it as a ValueError.
• There are 30 wheels: abi3 wheels for CPython 3.10 and later, plus free-threaded 3.14t wheels, across Linux (glibc and musl), macOS and Windows on every architecture maturin’s CI can build. So nobody needs a Rust toolchain to install it.

Making it fast

#

Version 0.2.0 was 1.5 to 1.7 times faster than 0.1.0 on real pages, and twice as fast on short snippets, with byte-for-byte identical output. Most of that came from profile-guided optimisation: CI builds an instrumented wheel, runs it over a training set of documentation pages and RFCs, and uses the profile to build the real wheel. That’s about 25% faster on pages outside the training set, on the platforms where CI can run the instrumented build natively. The rest came from switching to the mimalloc allocator, which copes far better with lots of small allocations, and from the style-rule change in the fork.

To check the speed-ups didn’t change anything, I hashed the output for a corpus of real pages under every set of options, plus edge cases like empty input and very deep nesting. Each change had to leave every hash the same.

Here are the median times on an Apple Silicon Mac, converting to Markdown:

And converting to plain text:

Deeply nested HTML, and a patch to html5ever

#

Nobody writes 40,000 nested lists by hand. But if you convert pages you didn’t write, sooner or later something hostile or broken will send you one. With the stock parser, 40,000 nested <ul><li> took 6 seconds to parse, and that’s a worker stuck on a single request.

The cause was in html5ever, the HTML5 parser rust-html2text uses. Most block-level start tags have to check whether there’s an open <p> element “in button scope”, and html5ever answered that by walking the whole stack of open elements. The deeper the nesting, the longer each tag took, so the total time grew with the square of the depth.

The package ships html5ever 0.39.0 with one patch that keeps the answer up to date as elements are opened and closed, so the check no longer depends on depth. The same document now parses in 0.14 seconds. To be sure the patch changed nothing else, I compared the DOMs from the stock and patched parsers on every html5lib tree-construction test, about 20 real pages and 300,000 random tag-soup documents, and they were all identical.

The Markdown converter has its own limits for the same reason. No input can overflow the thread’s stack, blockquotes and lists stop indenting past 32 levels so the output can’t grow quadratically, and a table whose expanded grid would pass 100,000 cells is written as its cells alone.

html2markdown

#

Version 0.3.0 added html2markdown, and most of the work in the package went into it. rust-html2text’s renderer makes text that looks like Markdown, but it’s wrapped for a terminal and isn’t meant to be parsed. So html2markdown only borrows the parser and writes its own CommonMark, plus GitHub-style pipe tables and ~~strikethrough~~:

• # headings, - bullets, _emphasis_ and **strong**.
• Inline links and images, made absolute when base_url is given. A link whose only content is an icon uses its aria-label or title as its text.
• Fenced code blocks, tagged with the language when the page names one (class="language-python").
• Pipe tables. Cells that span rows or columns become empty cells, a table without a header row gets an empty one, and tables used for page layout are written as paragraphs.
• No line wrapping. Hidden content (hidden, aria-hidden, display: none), scripts, styles and form controls are left out.

Escaping only what needs it

#

Some HTML-to-Markdown converters escape every character that could be Markdown syntax. markdownify, for example, escapes every _ and * by default. The output is correct, but it’s full of snake\_case and 5 \* 3, which is harder for people to read and a waste of tokens for a model. pyhtml2text escapes a character only where a CommonMark parser would otherwise read it as syntax:

html2markdown('<p>Use <code>snake_case</code>, 5 * 3, $5 and '
              '<a href="../setup.html">setup</a></p>',
              base_url="https://example.com/docs/intro.html")

Use `snake_case`, 5 * 3, $5 and [setup](https://example.com/setup.html)

Bold before a Korean particle

#

This one came from my app’s own pages. In Korean the grammatical particle goes straight after the noun with no space, so a bold term with brackets in it looks like this:

<p><b>타이타닉(Titanic)</b>의 감독</p>

The obvious Markdown is **타이타닉(Titanic)**의, which is what markdownify writes, and it doesn’t work. CommonMark’s flanking rules don’t let ** close after punctuation unless a space or more punctuation follows it, and 의 is a letter, so the asterisks stay as literal text. The pure-Python html2text avoids this by adding a space:

**타이타닉(Titanic)** 의 감독

That renders as bold, but it changes the Korean. The particle is now a separate word, which is a spelling mistake. pyhtml2text moves the punctuation outside the markers instead:

**타이타닉(Titanic**)의 감독

The bracket is no longer bold, but the emphasis still parses and every character of the original is kept. The same applies to Japanese and Chinese, and to any language that doesn’t put a space before what follows.

Checking the output by reading it back

#

I don’t trust a converter because its unit tests pass, because the bugs in Markdown output are in combinations nobody thought to write a test for. So the output gets checked by reading it back.

While converting, each paragraph is parsed again as CommonMark to confirm the emphasis reads as intended. If it doesn’t, the paragraph is written again with the other emphasis style (* instead of _, or the other way round), and if neither works the text, links and code are kept without the formatting, so no words are lost. The test suite does a round trip through a different Markdown parser and compares every visible word, link target and image source with the original HTML. Text eaten by syntax that wasn’t escaped, a link that didn’t parse or a stray * all show up as differences. It runs over real pages and tens of thousands of random documents, and shrinks any failure down to a minimal case.

In production

#

My app’s Markdown negotiation switched from the pure-Python html2text to pyhtml2text the day 0.3.0 came out. On one of its guide pages the conversion went from 4.2 ms to 0.36 ms, about 12 times faster. The links are absolute, the tables are real pipe tables, and the Korean bold keeps its particles attached. The abi3 wheels already cover Python 3.14 on x86_64 and arm64, so the container image needed nothing new to install it.

Credits, and where to report bugs

#

The plain-text rendering is rust-html2text, by Chris Emerson, under the MIT licence. pyhtml2text adds a fork of it, a patched copy of html5ever (MIT/Apache-2.0) and the Markdown converter. The package isn’t affiliated with or endorsed by either project, so please report problems to pyhtml2text’s issue tracker, not upstream. Any bug in this code is mine to fix, and the upstream maintainers shouldn’t get reports about code they declined to take.

If you’re converting HTML in Python, especially for a model to read, give it a try:

pip install pyhtml2text

The source is on GitHub, and the changelog lists everything in 0.1.0 to 0.3.0.

I’m currently shipping two products as a solo developer: a field estimation app for tradespeople who quote jobs on site, and a marketplace and community app with listings and real-time messaging.

Different products, different users, nearly identical stack — and both run entirely on shared-cpu-1x machines, the cheapest tier Fly.io offers. That constraint is deliberate. Instead of sizing servers for peak load, I squeeze as much as possible out of one small machine, measure exactly where it collapses, and let Fly’s autoscaler add and remove machines around that measured limit. This post covers the optimisations that make that possible, the load-test numbers behind the thresholds, and how scaling up and down actually works on Fly.io.

The Philosophy: Measure One Machine, Then Multiply

#

The scaling strategy for both apps is the same three steps:

1. Make a single small machine as fast as it can reasonably be.
2. Load-test it until it breaks, and note the concurrency level where latency collapses.
3. Configure Fly’s autoscaler to start a second machine just before that point, and stop idle machines when traffic drops.

Capacity then scales roughly linearly with machine count, and the baseline cost is one small VM per app. Everything below serves one of those three steps.

Making One Small Machine Fast

#

Granian: WSGI for the API, uvloop ASGI for WebSockets

#

Both backends are Django 6 + Django REST Framework. The API is served by Granian, a Rust-based server — but under WSGI with a blocking thread pool, not ASGI:

granian --interface wsgi myapp.wsgi:application \
    --workers 1 --blocking-threads 16 \
    --workers-max-rss 1400 \
    --workers-kill-timeout 15 --respawn-failed-workers

I started on --interface asgi --loop uvloop, and it was the wrong call. Every HTTP view in these apps is sync, and Django’s ASGI handler runs sync views thread_sensitive — one at a time per worker. So ASGI gave me roughly one concurrent request per worker and loaded the whole asyncio/uvloop/Channels stack into RAM for nothing. WSGI with --blocking-threads runs those same sync views in a real thread pool: one worker × 16 threads = 16 concurrent I/O-bound requests (DB reads, S3 presigns) at lower resident memory than the async build.

Why a single worker? The VM is shared-cpu-1x — one core. A second worker can’t run Python in parallel on one core (GIL + one CPU), but it does cost a whole extra resident Django process; Granian warns about exactly this. Sixteen threads on one worker is the honest ceiling of a 1-vCPU box for I/O-bound work. --workers-max-rss 1400 respawns the worker if it drifts past its memory cap, so a runaway PDF or image job gets recycled instead of walking the machine toward OOM.

WebSockets moved out to their own Granian process — an ASGI app with uvloop and --ws, one worker for long-lived connections. The sync API needs no ASGI path at all: when a view wants to push to a socket it calls async_to_sync(channel_layer.group_send) over Redis, so the message still crosses to the WebSocket process. Two servers, each running the interface it’s actually good at. Local Docker runs the same WSGI thread pool, so my laptop approximates one production machine when testing.

Rust in the Hot Paths

#

Wherever a Rust-backed drop-in exists for a Python hot path, I take it:

• orjson via drf-orjson-renderer — every API response serialises through Rust instead of the stdlib JSON encoder.
• blurhash-rs — image placeholders computed at upload time.
• uuid-utils — Rust UUID generation.
• psycopg 3 (binary) — C-accelerated Postgres driver.

None of these change the architecture; they just raise the ceiling of what one vCPU can do before you need a second machine.

Kill the Hidden Latency Taxes

#

The single biggest win in either codebase wasn’t an algorithm — it was connection reuse. Under Granian’s thread pool, every new thread was building a fresh boto3 S3 client, and every endpoint that returns a photo or PDF paid that cost. A small custom storage class now caches the client once per process instead:

Signed URLs went from ~150ms to ~1ms on new threads.

The same lesson applies to the database: Fly’s Postgres sits behind PgBouncer, so the app runs conn_max_age=0 and lets the pooler own connection reuse, with a 5-second connect_timeout to fail fast on stale SSL instead of hanging a worker.

Cache Where the Bottleneck Actually Is

#

Load testing the marketplace revealed something worth internalising: the ceiling was Postgres connections, not CPU. Browse traffic is overwhelmingly anonymous and identical across users, and under load those identical list queries saturate the connection pool while the CPU sits idle.

The fix is a Redis cache for anonymous list pages, with per-namespace version counters for invalidation — a write bumps one counter (O(1), no key bookkeeping) and every cached page for that resource is instantly stale. A short TTL backstops anything missed. Serving browse pages straight from Redis removes the real bottleneck and lets the small machine spend its connections on logged-in traffic.

Background Work Without a Broker

#

Both apps use django-tasks-db, Django’s database-backed task framework, instead of Celery. A separate db_worker process polls the database for jobs — image processing, PDF generation, machine translation, reminders. No broker, no extra service to size and monitor, and the API process never blocks on heavy work. The worker even connects straight to Postgres, bypassing PgBouncer, since long-running jobs don’t play well with transaction pooling.

Memory: jemalloc and a Swap Cushion

#

Image and PDF processing spike memory in ways steady-state sizing doesn’t predict. Two cheap mitigations: jemalloc preloaded in the Docker image to reduce fragmentation, and swap on every machine (2GB on the API machines, 1GB on the WebSocket machine) so a spike swaps to disk instead of getting the process OOM-killed. Swap is not a performance feature — it’s an insurance policy that lets me keep memory allocations small.

Django Load Testing: Finding the Knee

#

Thresholds are only defensible if they’re measured. Each project has its own harness:

• The estimation app has a small Python stress script that sweeps concurrency levels and reports RPS, p50/p95/p99 and error rates. It runs against a few hundred generated sample records, so the list endpoints are queried at realistic data volumes.
• The marketplace uses Locust with a step-load shape that ramps 50 → 500 users in 30-second steps to find the saturation point, with endpoint weights matching real browse-heavy traffic, plus a second, smaller suite that ramps 5 → 50 virtual users against production with a p95 < 2s threshold.

Representative numbers from the estimation app’s sweep (one Granian worker × 16 blocking threads, Docker on my dev machine):

And the number that drives everything else comes from load-testing the marketplace on a single shared-cpu-1x machine. On the old ASGI build this was a hard wall: sync views serialised to roughly three usable slots, so p95 held under 300ms up to ~5 concurrent requests and then collapsed past ~8–10 as everything queued behind the thread-sensitive lock. Moving the API to a WSGI thread pool is what pushed that knee out — 16 real concurrent slots per machine instead of ~3 — and that new ceiling is the whole basis of the autoscaling config.

How Scaling Works on Fly.io

#

Fly’s model is simple: your app is a fleet of identical machines behind a proxy, and the proxy counts load per machine. You declare two numbers:

[http_service.concurrency]
  type = 'requests'
  soft_limit = 12   # past this, start another machine
  hard_limit = 24   # past this, shed load instead of queueing

auto_stop_machines = 'stop'
auto_start_machines = true
min_machines_running = 1

Scaling up: when in-flight requests on a machine exceed the soft limit, the proxy starts a stopped machine and routes new traffic there. The WSGI thread pool gives each machine 16 real concurrent slots, so the soft limit sits at 12 — about 75% of the pool, so machine #2 starts while threads still have headroom rather than after users are seeing multi-second latency. The hard limit of 24 (1.5× the pool) is the circuit breaker: short bursts queue on threads, and beyond it the proxy sheds load rather than letting one machine melt down. (The old ASGI build ran 8/20, sized for its ~3 real slots — widening the pool is what let both numbers move up.)

Scaling down: when traffic drops, Fly stops idle machines automatically, down to min_machines_running = 1. A stopped machine costs nothing for compute, only its rootfs storage, so the baseline cost of each app is literally one small VM with burst capacity always ready.

I originally wrote here that a stopped machine “cold-starts in well under a second.” That was wrong, and expensively so. When I finally measured it instead of assuming it, a cold start took 25 to 33 seconds, and a request that lands on a machine still booting has to wait out all of it. I first blamed those cold starts for a 22% failure rate on my CDN dashboard, while every health check stayed green. That number turned out to be the CDN’s own Early Hints probes failing against the origin; no real client got a 504. The cold starts were real all the same.

The fix is not in the scaling numbers above; those are fine. It is in what the machine does on the way up, and in asking Fly to suspend machines rather than stop them, which turns a wake into a ~3 second memory restore. I wrote that up separately: Scale to Zero, and Every Restart Is a 30-Second Outage covers the diagnosis and what to change. If you are running anything on Fly with auto_stop_machines, read that one before you trust this paragraph.

One subtlety: the concurrency type is requests, not connections. Modern clients hold keep-alive connections open, so counting connections barely moves under real load and the autoscaler never triggers. Counting in-flight requests tracks actual work.

WebSockets Scale on a Different Axis

#

The marketplace splits its WebSocket server into a second Fly app sharing the same Docker image. The API app runs one worker × 16 blocking threads under WSGI and counts requests (soft limit 12). The WebSocket app runs 1 worker under ASGI with uvloop on a 512MB machine and counts connections:

[http_service.concurrency]
  type = 'connections'
  soft_limit = 1500
  hard_limit = 2000

A long-lived socket is nearly free to hold open but occupies a slot for hours, while an HTTP request occupies a slot for milliseconds — the same autoscaling signal can’t serve both. Splitting them means each fleet scales on its own honest metric, and idle WebSocket capacity costs 512MB, not 2GB. On the Channels side, the Redis layer runs with capacity: 1500 and 60-second message expiry, plus socket keepalives every 30 seconds because Fly’s Redis proxy idle-closes the long-lived BRPOP sockets that Channels depends on.

Offloading Compute to the Edge and the Client

#

The cheapest server work is the work the server never does:

• Static and media files live in Fly’s S3-compatible object storage, served through a CDN with signed URLs for private media — zero image bytes transit the Django machines.
• AI inference runs on the user’s phone. Both apps embed a small local LLM (a quantised 4B model, around 2.7GB) that runs on the device. Zero inference cost server-side, and it works offline.
• Offline-first mobile (SQLite on the device and a sync queue with retries) means the apps batch writes and tolerate dead zones instead of hammering the API with chatty per-keystroke requests.

The Rest of the Stack, Briefly

#

Auth is a hosted identity provider plus WebAuthn passkeys, and WebSocket connections are authenticated before the upgrade. Postgres in production, SQLite in development, and a couple of off-the-shelf Django packages for the admin and audit history. Mobile is React Native and TypeScript on Expo, with Expo Router and Tailwind, over-the-air updates from a self-hosted Expo Updates server, and a subscriptions service for in-app purchases. Quality gates: pytest and mypy on the backend, unit and end-to-end tests on the client.

What I’d Tell You to Steal

#

The transferable idea isn’t any single library — it’s the loop. Make one cheap machine genuinely fast (Rust hot paths, connection reuse, cache in front of the real bottleneck, offload what you can to CDN and client). Load-test until you find the exact concurrency where it breaks. Set the autoscaler’s soft limit at that knee, let idle machines stop, and keep the minimum fleet at one. Two production apps, real-time features, AI assistants — and the steady-state bill is a couple of the cheapest VMs on the platform.

I spent the last few weeks on two related projects. The first was reproducing a 2024 Scientific Reports paper by Lee et al., “Social signals predict contemporary art prices better than visual features, particularly in emerging markets”. It trains an XGBoost model on 34,200 auction records from 590 living artists. The second was building an art platform with a knowledge graph of artists, owners, and auction records. It’s basically the data plumbing you’d need around a model like that one if you wanted to do anything useful with it.

Both projects kept circling back to the same question: can you actually predict what an artwork sells for? The answer is yes, but not from looking at the artwork. The features that matter are almost entirely social.

Why Art Is Hard to Value

#

Before writing any code, I spent a couple of days reading about how appraisers and auction houses price work. The list of factors is long and most of them don’t quantify cleanly. Provenance — who’s owned it, where it’s been shown, which catalogues reference it. Attribution: a piece confirmed by the artist’s foundation is worth orders of magnitude more than one that’s only “attributed to” them. The artist’s reputation, built from solo shows, biennial inclusions, museum acquisitions, and critical reception. Then condition and restoration history, rarity within the artist’s body of work, and medium — oil on canvas tends to outperform works on paper, which outperform prints. A piece held by a major museum carries a stamp the secondary market trusts. And timing matters: five-year-old comparables are stale.

Auction-house specialists also talk about “emotional value” — collector desire, bidding-war dynamics, personal resonance. It can push hammer prices well past any rational estimate. You can’t model that directly, but you can pick up its shape indirectly through auction-house estimates.

The dominant valuation method in practice is comparable sales analysis: find recent auction results for similar works by the same artist and adjust for differences. An XGBoost model is doing the same thing, just at scale and with more features than any human appraiser would track by hand.

The Paper’s Central Claim

#

Lee et al. make a counterintuitive argument: the visual content of an artwork explains very little of its price. Their visual-only model, using color, composition, edge density, and ResNet18 embeddings, gets an R² of about 0.055. That’s barely better than guessing the mean.

A model using only artist-level “social” features (career stage, exhibition history, past auction prices, ranking on ArtFacts) hits R² ≈ 0.73. Add the auction house’s pre-sale estimate as a feature and it climbs to 0.92.

In other words, the artwork barely matters. What matters is who made it and what the market has already said about them.

The paper’s secondary claim is that this gap widens further in emerging markets, by which they mean anything outside the established USA, UK, France, and Germany core. In those markets, social signals matter more and expert estimates are less accurate, leaving more room for algorithmic prediction to add value.

Reproducing the Paper

#

The brief was straightforward: reproduce the paper as a single-developer MVP. No production deployment, no live data pipeline. Faithful reimplementation, prototype-quality code, and a written report on findings.

The Data

#

The paper provides cleaned CSVs as supplementary material:

The whole thing is about 5 GB. It fits comfortably in RAM, and XGBoost training on a dataset this size takes under five minutes on a laptop CPU. No GPU needed for the core model. Gradient-boosted trees on tabular data at this scale are nearly free to train, which is easy to forget if you’ve spent the last few years in neural-net land.

Features

#

The model uses 38 features split across three categories.

About the artist (30 features):

• Demographics: age, gender, education level, top-school flag.
• Career: ArtFacts ranking, solo shows, group shows, biennial inclusions, awards.
• Collections: counts of private and public collections.
• Price history: mean, median, max, and min from the artist’s last 5 and last 10 sales.
• Size-adjusted price history (price per square inch).
• Geography: where the artist works and lives, encoded as country flags.
• Match flags: does the artwork’s genre match the artist’s typical genre? Does the sale country match?

About the market (8 features):

• Auction house tier (1–4).
• Country and region price levels for that year (min, mean, median, max).

About the artwork itself (non-visual):

• Width, height, area in square inches.
• Medium category (painting, print, photo, sculpture, other).

Notice what’s absent: anything about the artwork’s actual visual content. That comes in only as a separate ablation study.

The Train/Test Split

#

Time-based, not random. Everything before May 2011 goes into training (~80%, ~27,360 records); everything after goes into the test set (~20%, ~6,840 records). Random splits would leak future information through the artist’s price-history features. If you scatter one artist’s records randomly across train and test, the test rows have already “seen” each other’s prices through the rolling windows, and your R² gets artificially inflated.

It’s the kind of thing that’s obvious once you see it but easy to miss on the first build.

The Model

#

XGBoost regressor, predicting log10(price_usd). Hyperparameter search over max_depth and learning_rate against a held-out validation slice. Two model variants:

1. Without expert estimates. Target R² ≈ 0.73.
2. With expert estimates. Target R² ≈ 0.92.

The “with estimates” variant uses the auction house’s pre-sale low and high estimates as additional features. Those two estimates alone, used by themselves, get R² ≈ 0.90. Adding the social features on top picks up another 0.02. That’s small in absolute terms, but it’s the part of the model doing real predictive work beyond what the experts already know.

What I Aimed For

#

I wasn’t trying to beat the paper. The goal was to land within ±0.03 of their R² values across all conditions. For the headline numbers (~0.73 metadata-only, ~0.92 with estimates), reproducibility is good. XGBoost is deterministic with a fixed seed, and the features are well-specified. I expected drift in two places: the visual-feature ablation, where small differences in PCA dimensions or image preprocessing can move things, and the emerging-market split, where there are only 4,346 records and the noise floor is higher.

Why Visual Features Barely Help

#

This was the most interesting result for me. The paper’s visual pipeline extracts 8,971 numbers per image:

• GIST descriptor (960 dims): overall spatial layout.
• Histogram of Oriented Gradients (2,915 dims): edge structure.
• Color histogram (4,096 dims): palette breakdown.
• ResNet18 features (1,000 dims): high-level pretrained embedding.
• Colorfulness (1 dim): vividness scalar.
• Complexity (1 dim): edge-density scalar.

Compress all that with PCA and feed it to XGBoost, and you get R² ≈ 0.055. Color alone gets 0.056. Edge structure gets 0.029. The neural-net embedding gets 0.009, basically zero.

The clean explanation is that auction prices are set at the level of the artist, not the individual work. Two paintings by the same artist sell for similar amounts regardless of what’s on the canvas, because what’s being priced is the signature.

You can see why this would be true if you’ve ever read auction catalogue notes. They spend almost all their words on provenance, exhibition history, and comparable sales. They barely describe the work itself.

If you take that 5% number seriously, the aesthetic content of a work is almost decoupled from its market value. Whatever the market rewards, it isn’t what’s on the canvas. A model that understood art at the pixel level would predict prices worse than one that only knew the artist’s CV.

Phase 3 Was Conditional

#

The paper’s image-link column points to URLs that were valid in 2012. By 2026, most are dead. The plan I wrote up flagged this risk on day 8: pull a hundred random URLs and see how many resolve. If more than half were gone, skip the visual phase entirely and document why.

Visual features only explain about 5% of price variance anyway. Spending two weeks fighting bit rot to confirm a number that small isn’t a good trade. Better to write up the dead-link finding as part of the report and move on.

Established vs. Emerging Markets

#

The paper trains separate models for established markets (USA, UK, France, Germany) versus emerging markets (19 other countries). The headline finding:

Two things stand out. First, social signals matter more in emerging markets, not less. I’d assumed established markets, with their richer data, would show better social-feature performance, but it’s the opposite.

Second, expert estimates are less accurate in emerging markets — that’s the 0.916 vs 0.859 gap once estimates are added in. In an established market, an algorithmic prediction has to compete with a specialist at a major auction house who has decades of comparable-sales data behind them. In an emerging market, that specialist has thinner data and the algorithm has more room to add value.

If you ever wanted to commercialise a model like this, emerging markets are where it would earn its keep.

Limitations Worth Stating

#

This is a paper reproduction, not a production system. It has no live data pipeline — the model trains on 1996–2012 auction data and never updates. There’s no deployment; the outputs are notebooks and a report, not a prediction API. There’s no drift detection, so once trained, the model has no idea when the market has changed. And the dataset is frozen: crypto art, NFTs, and post-COVID market shifts simply don’t exist in it. If you wanted to predict prices at a Korean auction house, you’d want auction-specific features and possibly a separate model for the Korean market — nothing here specialises that far.

The model also reflects historical patterns. If the contemporary market changed structurally around 2015 (and there are arguments it did, with art fairs becoming a major primary-market venue and a new generation of collectors entering), then the 1996–2012 patterns may not generalise.

From Paper to Platform

#

Reading the paper changed how I thought about the second project. The platform was originally framed as a crowdfunding marketplace for art exhibitions. Artists secure funding for upcoming shows, and investors get exposure to emerging artists earlier than the auction market typically allows.

The product surface is deliberately simple. Artists post upcoming shows that need funding (venue, dates, expected works). Investors back specific shows in tiered amounts ($100, $500, $1k, $5k) and share in the upside when the work sells. AI agents handle most of the artist-investor chat so that neither side has to be online for the conversation to keep moving. That’s particularly useful when a Korean gallery is talking to a U.S. collector across a 14-hour gap.

Once it sinks in that the artist is the asset and not the artwork, the data model has to change to match. What you need to capture is who has owned each work, where it has been shown, what comparable works have sold for, and who else owns work by the same artist. Ownership chains, institutional recognition, auction comps, collector networks — that’s what a knowledge graph is for.

The data model is small on purpose: artists, artworks, auction sales, ownership periods, and a provenance log for the data itself, so every record can be traced back to where it was scraped from. That last part sounds like bookkeeping, but with scraped data it’s what lets you trust the graph at all.

The mobile app renders all of this as a force-directed graph. A time-range slider lets you scrub through years and watch the network evolve: which collectors entered when, which artists got picked up by which museums, which works changed hands during the 2008 dip.

Each kind of entity gets its own shape and colour. Artists are diamonds, artworks are rounded rectangles, auction houses are hexagons, and the different kinds of owner (collectors, dealers, museums, estates) are circles in different colours.

The shapes aren’t decorative. When you zoom out to a few hundred nodes the silhouette tells you what type of entity you’re looking at without waiting for labels to render. A diamond surrounded by circles is an artist with a collector network. A hexagon connected to many diamonds is an auction house that handles a roster of artists. You can read the structure of a market at a glance.

A second tab shows the same data on a timeline: ownership periods as horizontal bars, sales as point events. Same underlying graph, different lens. Useful when you want to see the sequence of who held what, rather than the network shape at a single moment.

The platform isn’t itself a price-prediction tool. It’s the data layer underneath one. If you wanted to feed something like the Lee et al. model with live, current data instead of frozen 2012 CSVs, this is the structure you’d want.

Crowdsourced Valuation as a Game

#

The piece I haven’t built yet, and the part I’m most interested in, is turning art valuation into a game.

The Lee et al. paper takes auction-house pre-sale estimates as a feature and gets a big R² lift from them. Those estimates are valuable because they’re the aggregated judgment of trained specialists who have seen thousands of comparable works and have skin in the game. But auction-house specialists are a narrow, expensive resource. There are a few hundred of them globally, they work on specific consignments, and they don’t touch the long tail of artists nobody is consigning yet.

The question I keep coming back to is whether you can synthesise a comparable signal from a much wider, cheaper pool of judgments.

The rough design:

• Show a user an artwork: image, dimensions, year, artist name, brief CV.
• Ask them what it would sell for at auction.
• Let them pick a price bracket, or guess a hammer figure directly.
• Reveal the actual sale price after they commit.
• Score them with a Brier-score-style accuracy rating over time.
• Surface a leaderboard, daily streaks, and badges for hot streaks.
• Weight each user’s future predictions by their historical accuracy.

It’s a prediction market for art prices in the form of a guessing game. The loop is the same as a geography guessing game or online chess puzzle ratings: repeatable, scorable, with feedback you can build skill against. People who guess well rise on the leaderboard, and their future guesses count for more in the aggregated signal.

The model side is where this gets useful. A weighted aggregation of those guesses becomes a feature you can feed into XGBoost alongside the social signals. If the crowd’s accuracy-weighted median guess correlates well with hammer prices, you have something close to a synthetic auction-house estimate for works that don’t have a real one. That’s exactly the gap on the long tail of artists where the Lee et al. model needs the most help.

There’s also a cold-start angle. The model trains well on artists with rich price histories and badly on artists with three sales to their name. Three sales plus a thousand crowd guesses is at least something to work with. Crowdsourced valuation is a way of generating training signal for the cases the underlying data doesn’t cover.

Gamification matters here because the only way to make the crowd reliable is to make participating fun enough that people do it a lot. Single-question forms get you noisy data from people who clicked through once. A daily streak with a leaderboard and a calibration badge gets you the same person making 500 carefully considered predictions over a year. Volume plus self-selection (good predictors stay, bad predictors get bored) is what gives you a signal rather than noise.

There are design questions I’d need to answer before building any of it. How much do you show the user — just image and CV, or comparable sales too? More context means better-informed guesses, but past a point the user is mostly regurgitating what you’ve already shown them. How much price feedback do you give? Reveal the exact hammer, a range, or just whether they landed within ±20%? Too granular and you train people to anchor on specific numbers; too vague and they can’t calibrate. Then there’s leaderboard gaming: players will gravitate toward artists they already know, and a Banksy guess is mostly trivia rather than skill. Probably the fix is forced random sampling, the way language-learning apps pick lessons, plus a separate ranking for “blind” rounds where the artist’s name is hidden. And hindsight bias is unavoidable — once a user knows a Basquiat sold for $110M, they can’t un-know it. The supply of fresh, unseen artworks is what keeps the loop honest, and that supply is finite per week.

The gamification framing also pulls the platform’s two halves together. The crowdfunding side rewards investors when shows perform well financially. The guessing-game side rewards anyone, paying investor or not, for being good at calling prices. Both are versions of the same idea: let the crowd’s collective judgment turn into a tradeable signal. One trades capital, the other trades attention. Players who develop a real edge could eventually graduate to the investor side, bringing a verifiable track record of price-calling accuracy with them. That’s a better filter than “knew the right gallerist.”

What I Took Away

#

Pre-sale estimates are very good. Auction houses get a lot of grief for opaque pricing, but their estimates capture something close to 90% of price variance on their own. Whatever specialists are doing, they’re doing it well. And yet the artwork itself barely matters statistically — which is uncomfortable if you care about art for aesthetic reasons. The market isn’t pricing what you see. It’s pricing what other people have already paid for adjacent work.

On the modelling side, two habits earned their keep. Time-based splits are the only honest way to evaluate price models; random splits give you flattering R² values that don’t survive contact with production. And gradient-boosted trees are still the right tool here — XGBoost beats every neural baseline in the paper, and for tabular data at this scale, with this much engineered feature structure, there’s no reason to reach for a transformer.

The commercial observations: the interesting opportunity is in emerging markets, where expert pricing is thinnest and algorithmic prediction has the biggest gap to fill. And the crowd is an underused data source. The single biggest feature in the Lee et al. model was an auction house’s pre-sale estimate; if a calibrated crowd can produce a comparable signal for the long tail of artists, that’s a real opening, both as a model input and as a product in its own right.

The two projects ended up more complementary than I planned. The paper reproduction taught me what predicts prices. The platform work taught me what data structure you’d need to put that knowledge to use. The gamification idea ties the two together: the knowledge graph supplies the artworks and historical sales, the guessing game generates a crowd-derived estimate feature, and the model consumes that feature alongside the social signals to produce predictions for artists who would otherwise be invisible to it.

None of this is finished. The paper reproduction is a working prototype with a written report, the platform is a demo with seeded data, and the crowdsourced game only exists as the design notes you just read. But the connecting idea is consistent enough that I think the right shape is reasonably clear: price the artist rather than the artwork, aggregate the crowd rather than relying only on experts, and treat valuation as something the market produces collectively rather than something specialists hand down.

That’s the version I’d ship next, given the time to build it.