I use both every day. Ollama runs on my laptop and on the GPU workstation in the office, for development and for my own tools. vLLM serves a chat model from a single 96 GB GPU for a freelance project, first under Docker Compose and now in a small Kubernetes cluster.

vLLM vs Ollama at a glance

#

What Ollama is

#

Ollama is the easiest way I know to get a model answering on your own machine. It’s built on llama.cpp and runs quantised GGUF models, which is why an 8B model fits in about 5 GB and runs happily on a MacBook. You pull a model by name, it loads the first time you call it, and it drops out of memory after five minutes with nothing to do, so it shares a machine politely with everything else you have running.

That politeness is the trade-off. Ollama is built around one person at a time. By default it handles one request at a time; OLLAMA_NUM_PARALLEL adds slots, but each one costs a full context of memory, and it was never meant to batch dozens of users onto one GPU.

What vLLM is

#

vLLM is a serving engine, and it’s built for exactly that batching. Two ideas carry most of the weight. PagedAttention stores each request’s KV cache (the attention state a model builds up as it reads and writes tokens) in small fixed-size blocks, so GPU memory isn’t lost to padding and fragmentation. Continuous batching lets new requests join the batch between generation steps instead of waiting for the current batch to finish. Between them, one GPU running vLLM serves far more concurrent users than the same GPU working through requests one at a time. I haven’t benchmarked the two against each other myself, so this post has no throughput numbers of its own. Red Hat’s Ollama vs. vLLM benchmark, one A100 running Llama 3.1 8B at 1 to 256 concurrent users, is the measured version of this claim: vLLM peaked at about 793 output tokens a second against Ollama’s 41, even with OLLAMA_NUM_PARALLEL raised to 32.

The cost is that vLLM behaves as if it owns the card. It claims its share of GPU memory when it starts, fills whatever the weights don’t need with KV cache, and holds all of it whether anyone is calling or not. It also wants full Hugging Face weights or a GPU-friendly quantisation like FP8 or AWQ. It will load a GGUF file, but vLLM’s own docs call that support highly experimental and under-optimised, so don’t plan on reusing Ollama’s downloads.

How much GPU memory you need

#

Rough sizes for the weights alone:

That’s about 2 bytes per parameter at 16-bit, 1 at 8-bit, and a little over half a byte at Q4_K_M. On top of the weights goes the KV cache, which grows with context length and with how many requests are in flight. That second part is why the same model needs much more memory on vLLM serving fifty users than on Ollama serving you.

One API, two servers

#

Because both expose the OpenAI chat completions API, my code doesn’t know which one it’s talking to. The URL and model name come from the environment:

import os
from openai import OpenAI

# Laptop:     LLM_URL=http://localhost:11434/v1  LLM_MODEL=qwen3:8b
# GPU server: LLM_URL=http://localhost:8000/v1   LLM_MODEL=qwen3-8b
client = OpenAI(base_url=os.environ["LLM_URL"], api_key="not-needed")

reply = client.chat.completions.create(
    model=os.environ["LLM_MODEL"],
    messages=[{"role": "user", "content": "Say hello in Korean."}],
)
print(reply.choices[0].message.content)

On my laptop, which has no big GPU, the app points at Ollama. On the server it points at vLLM. Nothing else changes, and the app can be developed without the GPU box switched on at all. On the vLLM side, --served-model-name gives the model a short name, so clients don’t have to send the full Hugging Face path.

One caution, because it’s easy to forget: the same model on both doesn’t give the same answers. Ollama’s qwen3:8b is a 4-bit quantisation, while on vLLM you’d usually run 16-bit or FP8 weights. A prompt tuned against one can behave differently on the other, so anything you’re measuring, whether that’s answer quality, refusals or whether the JSON parses, has to be checked against the server production will actually use.

How to install Ollama

#

On a Mac or Windows, download the app from ollama.com. On Linux:

curl -fsSL https://ollama.com/install.sh | sh
ollama run qwen3:8b

That’s the whole job. The API is now at http://localhost:11434, and it only listens on localhost unless you set OLLAMA_HOST, which is what you want. Mine runs on the office workstation bound to 127.0.0.1, and I reach it from anywhere through a tunnel (how I set that up with Cloudflare WARP).

How to install vLLM

#

On Linux with an NVIDIA GPU and a recent driver:

python -m venv .venv && source .venv/bin/activate
pip install vllm
vllm serve Qwen/Qwen3-8B --served-model-name qwen3-8b --max-model-len 16384

The first run downloads the weights from Hugging Face. Once the log says Application startup complete, curl localhost:8000/v1/models lists the model, and the Python above works against it.

Running vLLM in Docker

#

For anything that has to stay up, I use the official image rather than a pip install, mostly so CUDA, PyTorch and vLLM come pinned together. A generic Compose service:

services:
  vllm:
    image: vllm/vllm-openai:v0.30.0   # the tag I ran; pin whatever is current. About 30 GB
    restart: unless-stopped
    ipc: host                          # PyTorch needs shared memory
    command: >
      --model Qwen/Qwen3-8B
      --served-model-name qwen3-8b
      --gpu-memory-utilization 0.85
      --max-model-len 16384
    volumes:
      - ${HOME}/.cache/huggingface:/root/.cache/huggingface
    ports:
      - "127.0.0.1:8000:8000"          # reach it over ssh -L, not the LAN
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    healthcheck:
      test: ["CMD", "python3", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:8000/health')"]
      interval: 10s
      timeout: 5s
      retries: 90
      start_period: 120s

Pin the tag. vLLM moves quickly, flags and defaults change between releases, and you want to be the one who decides when that happens. ipc: host (or a generous shm_size) is in vLLM’s own Docker instructions. The long start_period is the first of the things below.

What bit me running vLLM

#

Startup takes minutes

#

Loading the weights, compiling and capturing CUDA graphs took 190 seconds on my server, and three to five minutes is normal for a large model. A health check written for a web app will kill it before it’s ready. In Compose that means a long start_period; in Kubernetes it means a startup probe. It also means every restart is a few minutes with no model, so restarts shouldn’t be casual.

It takes the GPU memory you give it, at startup

#

--gpu-memory-utilization is the share of the card vLLM may use, 0.92 by default in recent releases, and whatever the weights don’t need becomes KV cache. That’s fine when vLLM is alone on the card. Mine isn’t, because smaller models run beside it, so I set 0.70 on the 96 GB card. The model, a 35B mixture-of-experts in FP8, loaded at 47.7 GiB and left 15 GiB for KV cache, which is plenty at a 16k context. If vLLM refuses to start because the maximum sequence length won’t fit in the KV cache, lower --max-model-len before you go looking for a bigger GPU.

LoRA slots cost memory up front

#

vLLM can serve one base model with several LoRA adapters and switch between them per request, by model name, which Ollama can’t really do. But it reserves room for --max-loras adapters at --max-lora-rank when it starts, and on a mixture-of-experts model that reservation covers the experts too, so it adds up fast. Four adapters at rank 64 didn’t fit at 0.70. Two did.

The default sequence limit didn’t suit a hybrid model

#

The model I serve mixes Mamba-style layers in with attention, and each sequence needs its own cache block for those layers. vLLM’s default --max-num-seqs is 1024 on a card with 70 GB or more (256 below that), so on that card it failed to start; --max-num-seqs 256 fixed it. If a newer architecture won’t start, check this flag before anything more exotic.

A Kubernetes Service named vllm crashes vLLM

#

This one’s my favourite. Kubernetes injects environment variables for every Service in the namespace, named after the Service, so a Service called vllm hands every pod VLLM_PORT=tcp://<cluster-ip>:8000. vLLM reads VLLM_PORT as one of its own settings, couldn’t make sense of a URL there, and died. After loading the whole model, naturally, so every attempt cost three minutes. vLLM’s docs warn about this, which I found afterwards. The fix is enableServiceLinks: false on the pod, or a Service with a different name.

Here’s the shape of a Deployment that works, cut down to the parts that matter:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm
spec:
  replicas: 1
  strategy:
    type: Recreate               # the old pod has to free the GPU first
  selector:
    matchLabels: {app: vllm}
  template:
    metadata:
      labels: {app: vllm}
    spec:
      enableServiceLinks: false  # no VLLM_PORT=tcp://... from the Service
      containers:
        - name: vllm
          image: vllm/vllm-openai:v0.30.0
          args: ["--model=Qwen/Qwen3-8B", "--served-model-name=qwen3-8b"]
          resources:
            limits:
              nvidia.com/gpu: "1"
              memory: 32Gi       # loading needs far more host RAM than serving
          volumeMounts:
            - {name: shm, mountPath: /dev/shm}
          startupProbe:          # don't judge it until the model is loaded
            httpGet: {path: /health, port: 8000}
            initialDelaySeconds: 120
            periodSeconds: 10
            failureThreshold: 90
          livenessProbe:         # a restart costs a full reload, so be patient
            httpGet: {path: /health, port: 8000}
            periodSeconds: 30
            timeoutSeconds: 5
            failureThreshold: 10
      volumes:
        - name: shm
          emptyDir: {medium: Memory, sizeLimit: 16Gi}

The probes are the part worth copying. The startup probe waits two minutes, then gives the model another fifteen to come up before Kubernetes calls it dead. The liveness probe only restarts it after about five minutes of failures, because /health can be slow on a busy engine and a restart costs a full reload anyway. Recreate matters on a single GPU: a rolling update starts the new pod first, and it sits there waiting for a GPU the old pod still holds. The memory limit looks generous because it is; mine settles at about 7 GiB of host RAM once the weights are on the GPU, but the load itself needs a lot more.

What to know about Ollama

#

Fewer surprises, mostly because it does less.

The first request after a quiet spell is slow, because the model was unloaded and has to load again. OLLAMA_KEEP_ALIVE sets how long a model stays in memory after its last request; raise it if that pause annoys you, or set it to -1 on a machine that does nothing else.

The context window depends on how much GPU memory you have. Under 24 GiB, Ollama defaults to 4k tokens, far less than most models support, and a prompt that doesn’t fit is trimmed quietly rather than refused. If answers seem to ignore the start of a long prompt, set OLLAMA_CONTEXT_LENGTH (or num_ctx per request) to what you actually need.

If you forward a remote Ollama to your laptop, don’t put it on 11434. That’s Ollama’s default port, and if a local Ollama ever starts, the two race for it and you can end up talking to the wrong one without noticing. I forward mine to 11435.

And it isn’t a multi-user server. It’s fine for an internal tool a handful of people use; once real concurrency turns up, that’s the point to move to vLLM.

Which one to use

#

If you want a model on your own machine, install Ollama and stop there; you’ll have something answering in ten minutes. If you have a GPU server and users, or an app, that will call it all day, use vLLM, accept the slow start and the memory it takes, and get the probes right. Plenty of setups end up with both, mine included: Ollama where the code gets written, vLLM where the requests land.

There are other options. SGLang is the usual alternative to vLLM, but I haven’t run it in production, so I’ve left it out rather than guess. And if the model has to run on a phone, that’s llama.cpp again, which I wrote about in running an LLM on-device in React Native.

If you came here to run Docker Compose in GitHub Actions: I did that for two years and stopped. CI now declares the same Postgres and Valkey images as services: and runs the same uv run … and npm run … commands straight on the runner; Compose stays what a developer runs. The rest is why, with the files.

In 2024 I wrote Use Docker Compose for CI/CD Instead of Platform-Specific YAML, which said to keep the build logic in Compose and let the CI tool do nothing but call it. Two years on, I went back through every repository I ship from. Four have workflows, each with one committer, me, and none of those workflows calls docker compose any more. The principle held; the mechanism didn’t. Here’s what replaced it, with cut-down example versions of the files, the step timings GitHub recorded, and the parts that are less automated than a CI/CD diagram would like.

Every pipeline I run, in one table

#

In textbook terms the Hugo sites have continuous deployment, and the app (a marketplace and community app I build) has continuous integration plus one-command releases, which is continuous delivery: the checks tell me something is wrong, but they don’t stop me deploying. That’s deliberate. An OTA update reaches every phone on the binary version it targets, so publishing one should be a decision rather than a side effect of a merge, and the API is where the database migrations run. My other two Django apps have no workflows at all.

The CI half: one workflow, two jobs

#

The app’s CI is one file, .github/workflows/checks.yml. Until 11 October there wasn’t any. This is a trimmed example of its shape:

# Lint, types and tests on every push and pull request to main.
# Nothing here deploys.
name: Checks

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

concurrency:
  group: checks-${{ github.ref }}
  cancel-in-progress: true

jobs:
  python:
    runs-on: ubuntu-latest
    timeout-minutes: 30
    services:
      postgres:
        image: postgres:17
        env:
          POSTGRES_USER: myapp
          POSTGRES_PASSWORD: myapp
          POSTGRES_DB: myapp
        ports: ["5432:5432"]
        options: >-
          --health-cmd "pg_isready -U myapp"
          --health-interval 5s --health-timeout 5s --health-retries 20
      valkey:
        image: valkey/valkey:8-alpine
        ports: ["6379:6379"]
    env:
      DEBUG: "True"
      DATABASE_URL: postgresql://myapp:myapp@localhost:5432/myapp
      REDIS_URL: redis://localhost:6379
      PYTHON_JIT: "0"   # match production
    steps:
      - uses: actions/checkout@v7
      - uses: astral-sh/setup-uv@v10
        with:
          enable-cache: true
          prune-cache: true
      - run: uv sync --locked
      - run: uv run ruff check .
      - run: uv run mypy myapp/
      - run: uv run python manage.py test --parallel 4 --noinput

  app:
    runs-on: ubuntu-latest
    timeout-minutes: 20
    defaults:
      run:
        working-directory: frontend
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-node@v7
        with:
          node-version: 24
          cache: npm
          cache-dependency-path: frontend/package-lock.json
      - run: npm ci
      # Expo writes this on `expo start` and it is gitignored,
      # so a fresh checkout doesn't have it.
      - run: printf '/// <reference types="expo/types" />\n' > expo-env.d.ts
      - run: npm run typecheck
      - run: npx jest --ci

The two jobs run side by side. Across the 35 green runs so far, the app job’s median is about seven minutes and the Python job’s about twelve and a half.

The services: block is where the Compose file went, which is the next section. The environment is a few variables and a flag; everything else falls back to the development default docker compose up gives a developer, so the tests need no secrets. I checked by running part of the suite with exactly that environment and nothing else.

CI runs the same interpreter as my laptop and the production image, uv’s own build of Python 3.15.0, because uv sync reads .python-version and downloads exactly that build. (My real file also passes setup-uv a python-version-file input, which does nothing: setup-uv v6 has no such input, and every run carries a warning saying so. I only noticed while writing this.)

Matching interpreters earned their keep eight days after the workflow went in. On 19 October I switched on 3.15’s JIT compiler, and the first run with it failed a test: a plain yield from, on a code path the app takes on every save, raised AttributeError: 'list' object has no attribute 'send'. The same commit with the JIT off passed. My Mac is arm64 and never hit it; the runner is x86_64, like the production machines. The JIT is off again, in CI and in the Dockerfile, and the bug never reached production. It’s python/cpython#155823, fixed upstream on 13 October, after 3.15.0 shipped.

The very first run failed too, for a duller reason. Expo writes expo-env.d.ts when you run expo start, and the file is gitignored, so every working copy I’d used had one and a fresh checkout didn’t. TypeScript reported five errors that had never appeared locally. The step that writes the file is the fix.

concurrency cancels a run when a newer push to the same branch arrives, which is how all twelve cancelled runs so far ended. A pre-commit hook runs the linter, plus the type checker when Python or TypeScript is staged, so most mistakes never reach a push.

Where the Docker Compose file went

#

Here is the part of the app’s docker-compose.yml that CI cares about, as an example. The real one has more services and a lot more environment.

services:
  valkey:
    image: valkey/valkey:8-alpine
    ports:
      - "6379:6379"

  # Host port 55432, not 5432, so it can't collide with a local Postgres.
  postgres:
    image: postgres:17-alpine
    environment:
      - POSTGRES_USER=myapp
      - POSTGRES_PASSWORD=myapp
      - POSTGRES_DB=myapp
    ports:
      - "55432:5432"
    volumes:
      - postgres_data:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U myapp -d myapp"]
      interval: 5s
      retries: 10

  api:
    build: .
    command: python manage.py runserver 0.0.0.0:8000
    volumes:
      - .:/app
    ports:
      - "8000:8000"
    environment:
      - DATABASE_URL=postgresql://myapp:myapp@postgres:5432/myapp
      - REDIS_URL=redis://valkey:6379
    depends_on:
      postgres:
        condition: service_healthy
      valkey:
        condition: service_started

volumes:
  postgres_data:

Put it next to the services: block in checks.yml and it’s the same two images: Postgres 17 (the Alpine build here, the Debian one in CI) and valkey/valkey:8-alpine. That’s the relationship now. The Compose file is what a developer runs; CI declares the same images as service containers and runs the same commands a developer runs, uv run … and npm run …, straight on the runner.

Against the 2024 post, the part I cared about most held: the CI tool is thin, and the checks and deploys are commands I can run myself. What didn’t survive is CI calling Compose.

The app’s CI doesn’t build the Docker image at all; Fly’s builder does that once, at deploy time, and CI only has to say whether the code is good, which with a cached toolchain takes a six-second install. The Compose file also bind-mounts the working copy and seeds data, which a test run shouldn’t, so CI would have needed a Compose file of its own: a second description of the same services, kept in step by hand.

And the 2024 example had a bug I didn’t notice at the time. docker compose build builds the image and never runs the service’s command, so the npm test in that file never ran in CI. The workflow went green whether the tests passed or not.

The risk that post worried about most, switching CI vendors, never came up. The move that did happen was at the other end: this blog and the client’s site moved hosts to Cloudflare Pages, and for the blog that took one new 45-line workflow, because the build was already one command. And the three-hundred-line workflow I warned about? I have one, pyhtml2text’s. It’s long because the wheels have to be built on Linux, macOS and Windows runners, for several architectures each, which makes it the one job where platform-specific YAML is the right tool.

Caching: read the step timings first

#

GitHub keeps the duration of every step, and gh api repos/<owner>/<repo>/actions/runs/<run-id>/jobs hands it over. These are from the app’s latest green run, on 20 October:

Both dependency caches are on, and both are the cheap part. The uv cache GitHub keeps for the repository is about 200 KB, because setup-uv prunes whatever uv can simply download again; the npm one is nearly 400 MB. The time is in the tests, which already run four ways in parallel. If I wanted the run faster, I’d change the suite, not the cache.

The blog is the opposite case. Hugo’s processed images live in the gitignored resources/_gen, so every fresh runner re-encoded every cover, body image and placeholder. The workflow further down restores them with actions/cache, keyed on a hash of the pictures and falling back to the newest older cache, since Hugo only processes what’s missing. In the last eight runs before the cache, the build step took 33 to 76 seconds; in the 26 since, 6 to 20, with one exception at 94, even though the same commit added placeholders and a second card size.

The app’s image builds on Fly’s builder, with the dependency files copied before the code and uv’s download cache in a BuildKit cache mount (RUN --mount=type=cache,target=/root/.cache/uv). The bigger fix wasn’t a cache. The old .dockerignore was a blocklist, and it missed a folder of extra git worktrees, each with its own dependencies and build output, so 7.7 GB across 277,000 files went to the builder on every deploy. Rewritten as an allowlist, the context is about 40 MB. An example of the pattern:

# Exclude everything, then let back in only what the Dockerfile COPYs.
*

!pyproject.toml
!uv.lock
!manage.py
!myapp
!static

# Rules match the whole relative path and the last match wins,
# so these re-excludes apply inside the allowed directories too.
**/__pycache__/
**/*.pyc
**/.DS_Store
**/*.log

Secrets, in as few places as possible

#

Secrets live in three places. GitHub Actions secrets are for what workflows do, and the two Hugo repositories have exactly one each, CLOUDFLARE_API_TOKEN. Fly secrets are the app’s runtime configuration, set with flyctl secrets set, and they never pass through Actions. And a gitignored .env on my laptop holds the token the deploy script uses to purge Cloudflare’s cache.

PyPI needs no secret at all. pyhtml2text uses trusted publishing: PyPI accepts the short-lived OIDC token GitHub issues to the workflow, so the release job asks for id-token: write and the repository’s secrets page is empty. There’s nothing to leak and nothing to rotate.

Tokens get the least they need. The app’s Fly token in GitHub serves a scheduled check that only lists machines, so it’s created with fly tokens create readonly. The Pages deploys ask for contents: read and deployments: write, and nothing more.

Inputs go through env:, never straight into a script. The OTA workflow takes free-text release notes. Written as ${{ inputs.notes }} inside run:, the text would be pasted into the script before bash reads it, so a note with a quote and a semicolon in it would become code. Passed as an environment variable, it stays data. An example of the pattern:

      - name: Publish
        env:
          OTA_API_TOKEN: ${{ secrets.OTA_API_TOKEN }}
          INPUT_PLATFORM: ${{ inputs.platform }}
          INPUT_ROLLOUT: ${{ inputs.rollout }}
          INPUT_NOTES: ${{ inputs.notes }}
        run: |
          args=("$INPUT_PLATFORM")
          [ -n "$INPUT_ROLLOUT" ] && args+=(--rollout "$INPUT_ROLLOUT")
          [ -n "$INPUT_NOTES" ] && args+=(--notes "$INPUT_NOTES")
          ./scripts/publish-ota.sh "${args[@]}"

Some keys stay off GitHub altogether. One scheduled report needs a key with admin rights over the app’s sign-in backend, so it moved from a scheduled workflow to a scheduled job on my Mac. The workflow is still there as a manual fallback, and leaves out the sections that need that key.

Deploys: on every push, by script, and on a tag

#

The Hugo sites deploy on every push

#

An example version of the blog’s workflow, with the first-run step that creates the Pages project left out:

name: Deploy to Cloudflare Pages

on:
  push:
  workflow_dispatch:

env:
  HUGO_VERSION: "0.165.0"
  PAGES_PROJECT: my-site

concurrency:
  group: deploy-${{ github.ref }}
  cancel-in-progress: true

jobs:
  deploy:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      deployments: write
    steps:
      - uses: actions/checkout@v7
        with:
          submodules: true

      - uses: peaceiris/actions-hugo@v3
        with:
          hugo-version: ${{ env.HUGO_VERSION }}
          extended: true

      # Processed images are gitignored; restore them so Hugo only
      # re-encodes what is new. An older cache still helps.
      - uses: actions/cache@v6
        with:
          path: resources/_gen
          key: hugo-resources-${{ hashFiles('content/**/*.jpg', 'content/**/*.png', 'assets/img/**') }}
          restore-keys: |
            hugo-resources-

      # A multilingual site has no root 404.html; without one, Pages
      # answers unknown paths with index.html and a 200.
      - name: Build
        run: |
          hugo --gc
          cp public/en/404.html public/404.html

      # ...

      - name: Deploy
        uses: cloudflare/wrangler-action@v4
        with:
          apiToken: ${{ secrets.CLOUDFLARE_API_TOKEN }}
          accountId: ${{ vars.CLOUDFLARE_ACCOUNT_ID }}
          command: pages deploy public --project-name=${{ env.PAGES_PROJECT }} --branch=${{ github.ref_name }}

--branch=${{ github.ref_name }} makes main production and gives every other branch a preview URL. The client site’s workflow mirrors this one, plus an rm -rf public before the build, because its public/ is committed and removed pages would otherwise linger in the upload.

Hugo is pinned here and nowhere else. In October a package-manager upgrade moved my laptop to 0.168, one past the newest version the theme supports. The deploys stayed on 0.165 and never noticed.

And the build is the test. On 14 September the preview deploy for one branch failed three times running: a post on it linked, through Hugo’s ref, to one that was still a draft. CI builds what production builds, drafts excluded, so the link pointed at nothing and Hugo stopped with REF_NOT_FOUND. main carried on deploying in the meantime.

The app’s API deploys from a script

#

The core of the deploy script, as an example:

IMAGE_LABEL="deploy-$(git rev-parse --short HEAD)-$(date +%Y%m%d%H%M%S)"

# Build first, then wake a spare machine, then roll out exactly that image.
flyctl deploy --config fly.toml --build-only --push --image-label "$IMAGE_LABEL"
curl -fsS --retry 5 https://myapp.fly.dev/health > /dev/null
flyctl deploy --config fly.toml --image "registry.fly.io/myapp:$IMAGE_LABEL"

And the matching lines in fly.toml:

[deploy]
  # max_unavailable only counts machines that are running,
  # which is why the script wakes one first.
  strategy = "rolling"
  max_unavailable = 1

The image is built once, pushed to Fly’s registry under a label made of the commit and the time, and deployed by that label, so what rolls out is exactly what was built.

Waking a spare first is what makes it zero-downtime. max_unavailable = 1 only counts running machines, and the app keeps none running when it’s idle (a suspended machine resumes in about a second), so a lone awake machine used to be replaced in the same batch as the sleeping ones, with nothing warm behind it. On 28 September a probe against the origin saw 7 timeouts and a 7.3-second gap during a plain flyctl deploy, and no failures in 2,514 requests during the scripted deploy, which builds first and has a machine awake before the rollout starts.

The rest of the script marks the deploy in the team chat channel the error alerts go to, so a burst of errors can be read against what just changed, purges Cloudflare’s cache and warms the machines. The web front end skips the image entirely; that’s in Deploy Frontend and Backend Separately, Without Redeploying the API.

Migrations aren’t a pipeline step. Fly’s release_command hung for me, so the first machine to boot on a new image takes a lock and migrates while the others wait. Every migration therefore has to work with the code that’s still running: add a column in one deploy, stop using it in the next, drop it in a third.

Nothing in the script needs my laptop as such, but a couple of local-only files keep it there: the Cloudflare credentials, and a record of what was last deployed, which a fresh runner wouldn’t have. It will also deploy uncommitted changes, and says so in the chat channel, where the web release script refuses a dirty tree.

In Preview Environments per Branch on Fly.io I wrote that doing this by hand is fine while you’re the only one doing it, and that from the second developer on it has to happen because something happened in git. The app is still at one developer; the branch-environment workflows from that post live on the branch I built them on.

The OTA workflow is manual-only because publishing should be a decision, and because the update service refuses a release while a partial rollout is open, which an on-push trigger would hit during every canary. It has never run. Every OTA update so far has gone out from my laptop through the same script the workflow calls, which is the case for thin workflows in one line: however a release starts, it’s the same script.

pyhtml2text releases on a tag

#

The release job, with the package name made generic:

  release:
    runs-on: ubuntu-latest
    if: ${{ startsWith(github.ref, 'refs/tags/') || github.event_name == 'workflow_dispatch' }}
    needs: [linux, macos, windows, sdist]
    environment:
      name: pypi
      url: https://pypi.org/p/mypackage
    permissions:
      id-token: write       # trusted publishing and signing
      contents: write
      attestations: write
    steps:
      - uses: actions/download-artifact@v8
      - uses: actions/attest@v4
        with:
          subject-path: 'wheels-*/*'
      - uses: astral-sh/setup-uv@v10
        if: ${{ startsWith(github.ref, 'refs/tags/') }}
        with:
          prune-cache: true
      - name: Publish to PyPI
        if: ${{ startsWith(github.ref, 'refs/tags/') }}
        run: uv publish --trusted-publishing always 'wheels-*/*'

The release job waits for every build job, and only a tag publishes; a manual run builds and attests but uploads nothing. actions/attest records which workflow built each wheel, and uv publish --trusted-publishing always uploads them with the OIDC token rather than a password. The package itself is in A Closed Pull Request, a Fork and a PyPI Package.

Scheduled workflows are best effort

#

The app has one scheduled workflow, a check that Fly hasn’t started more machines than the baseline, which would otherwise first show up on the bill. It alerts by failing, since GitHub emails the owner about a failed scheduled run.

It’s set to cron: '*/15 * * * *', 96 runs a day. Over the past two weeks it ran between three and eight times a day, with a median gap of about three hours forty. The only explanation GitHub’s documentation offers is load: scheduled runs can be delayed when Actions is busy, and dropped when it’s busy enough. For a sustained scale-out, a few hours is fine. For anything that has to fire within minutes, a GitHub cron isn’t a monitor.

What I’d ask of a pipeline

#

If you’re building one, or looking at one you already have:

1. If GitHub Actions were down for a day, could you still release? If not, your release logic lives in YAML, and the 2024 post still applies.
2. Does CI run the interpreter, the CPU architecture and the database major version that production runs? Mine caught a compiler bug only because the runner and production are both x86_64 and my laptop isn’t.
3. Does a fresh checkout pass? Generated, gitignored files are the usual reason it doesn’t.
4. Where does a run’s time actually go? Read the step timings before you add a cache.
5. For each secret, how many places does it live, and does it need to be in GitHub at all?

Most teams write their build and test steps straight into GitHub Actions YAML. It works, right up until you want to run the same build on your laptop, or someone suggests moving to GitLab and you realise the whole pipeline is welded to one vendor’s config format. My preference is fairly boring: keep the real logic in Docker Compose, and let the CI tool do nothing except call it.

Why I keep build logic out of Actions YAML

#

The obvious problem is lock-in. An Actions workflow means nothing to GitLab, Bitbucket or Jenkins, so if your build steps live in the workflow file, changing providers means rewriting all of them. Nobody ever budgets time for that.

The quieter problem is that you can’t run the pipeline locally. When the only way to test a change to the build is pushing a commit and watching the Actions tab, you end up with a git history full of “fix ci”, “fix ci again”, “please”. A Compose file runs the same on my laptop as on the runner, which also kills most “works on my machine” arguments before they start.

And the YAML grows. Twenty tidy lines turn into three hundred once caching tricks and conditionals show up, and now every developer has to learn a CI dialect on top of Git and everything else. A script should do one thing; a big workflow file ends up doing all of them at once.

Compose splits the responsibilities cleanly: the Compose file owns building and testing, and the CI platform is just the thing that runs it on every push.

The setup

#

1. Start with a Dockerfile, so building the image has exactly one home:

# Use an official Node.js runtime as the base image
FROM node:14

# Set the working directory in the container
WORKDIR /app

# Copy the package.json and package-lock.json files to the container
COPY package*.json ./

# Install the dependencies
RUN npm install

# Copy the rest of the application code to the container
COPY . .

# Expose the port the app runs on
EXPOSE 3000

# Define the command to run the application
CMD ["npm", "start"]

2. Then a docker-compose.build.yaml that defines the services needed to build and test:

   services:
     app:
       image: myprivaterepo.com/image:latest
       build: .
       volumes:
         - .:/app
       command: sh -c "npm install && npm test"

3. Run it locally first. This is the whole point of the exercise:

   docker compose -f docker-compose.build.yaml build

4. With that working, the GitHub Actions workflow (or its equivalent on any other platform) shrinks down to an orchestrator:

   name: CI/CD Pipeline
   on: [push, pull_request]
   jobs:
     build:
       runs-on: ubuntu-latest
       steps:
         - uses: actions/checkout@v2
         - name: Set up Docker Buildx
           uses: docker/setup-buildx-action@v1
         - name: Build and Test with Docker Compose
           run: docker compose -f docker-compose.build.yaml build
         - name: Push the image referenced in the docker compose file
           run: docker compose -f docker-compose.build.yaml push

The CI tool now does two things, build and push, and everything worth understanding lives in files that any platform (or any new team member) can run directly.

Pushing the image

#

A nice side effect of the Compose file is that pushing is trivial. Docker works out where the image goes from its name: image: myprivaterepo.com/image:latest means it gets pushed to myprivaterepo.com. For a private registry you’ll need an access token — check your registry’s docs for that part.

If you’re on GitHub Actions and happy, none of this is urgent. But the day you switch providers, the migration is copying two commands instead of porting three hundred lines of YAML. And until then, you get to debug your builds locally like a civilised person.