Until this week, the only way I found out our app had crashed was when someone told me. Most people don’t. They reopen the app, and if it happens again they stop opening it. The API side was covered (every ERROR log already lands in a team chat channel), but a crash on a phone left no trace anywhere I could see.

The app did have one reporter already. When it puts an error message in front of someone, it posts that message to a small endpoint on the API, which logs it at ERROR, so it reaches the chat channel too. I added it after a delete that had worked left the row in the list, and tapping it again showed “No Item matches the given query”. The server had logged two ordinary 404s and said nothing. That covers errors the app catches and shows. A crash is different: the JavaScript that would send the report is the thing that just died.

The obvious fix is Sentry. I like Sentry’s SDK a lot; it’s the reason crash reporting on React Native is a solved problem. What I didn’t want was another vendor holding user data, another privacy-policy line, and another bill for something that, at our size, is a few reports a day at most. So I kept the SDK and replaced the service: the app’s DSN points at our own API, and a small piece of the Django app there receives what it sends, groups it, alerts on it, and keeps it.

This post is how that works, and how I checked that a real native crash actually makes it through.

The DSN trick

#

A Sentry DSN looks like https://<public-key>@<host>/<project-id>. The SDK turns that into a POST to https://<host>/api/<project-id>/envelope/, with the key in an X-Sentry-Auth header. Nothing about it is specific to sentry.io. So the DSN baked into the app is:

https://<CRASH_REPORT_KEY>@api.example.com/1

and Django has a route that matches what the SDK will call:

path("api/<int:project_id>/envelope/", crash_envelope, name="crash_envelope"),

The key is in every binary, so it isn’t a secret. It’s there so that if the endpoint ever starts attracting junk I can change it, and so that an unset key on the server refuses everything rather than accepting the world:

def crash_envelope(request, project_id):
    expected = settings.CRASH_REPORT_KEY
    if not expected:
        return Response({"detail": "Crash reporting is not configured"}, status=503)
    if not hmac.compare_digest(_key(request).encode(), expected.encode()):
        return Response({"detail": "Unknown key"}, status=401)
    ...

_key reads sentry_key= out of the X-Sentry-Auth header, falling back to the query string, which some SDKs use instead.

What the SDK sends: envelopes

#

Sentry’s wire format is the envelope: one JSON header line, then any number of items, each one a JSON header line followed by a payload. If the item header has a length, the payload is exactly that many bytes; otherwise it runs to the next newline.

{"event_id":"9ec79c33...","dsn":"https://...@api.example.com/1"}
{"type":"event","length":4123}
{ ...the event JSON, 4123 bytes... }
{"type":"client_report"}
{"timestamp":1791400000.5,"discarded_events":[]}

Two details caught me out:

• The native SDKs gzip the body; the JavaScript one doesn’t. A JS error captured by @sentry/react-native arrives as plain text, while a native crash from sentry-cocoa or sentry-android arrives with Content-Encoding: gzip. The decoder handles gzip, zlib-wrapped deflate and raw deflate, and stops decompressing at 1MB, because a crash event is a few kilobytes and anything bigger isn’t one.
• Length-prefixed payloads can contain newlines. If you split on \n you’ll mangle every event the native SDKs send. You have to honour length when it’s there.

The parser is short. It only keeps event items; sessions, transactions, attachments and client reports are accepted and dropped.

Answering in a way the SDK understands

#

This matters more than I expected. The SDK keeps a queue on disk and retries anything it isn’t sure was delivered, so what the server answers decides whether it keeps trying:

• 200 once stored, or deliberately dropped. If I answered a session item with a 4xx, the SDK would keep it and retry forever.
• 429 with Retry-After when throttled. The endpoint has a per-IP throttle (120 an hour), because a phone in a crash loop sends one report per launch, and the SDK backs off on a 429 the way you’d hope.
• 400 for a malformed envelope, logged as a warning, not an error: a broken report says a client is broken, and the next good one will say what crashed.

The SDK also resends the same event if it isn’t sure the first send arrived, so event_id has a unique constraint and a duplicate is a quiet no-op.

Grouping without source maps

#

This is where a self-built receiver has to make a choice Sentry makes for you. Sentry groups crashes by stack trace. But a release Hermes bundle is minified, so frame names are single letters and line/column positions change with every build. Group by frames, and the same bug opens a new issue every release.

Without source maps (more on that at the end), the most stable thing in a JavaScript error is its type and message. So the fingerprint is the exception type plus the message, with the parts that differ between sightings masked out:

_MASKS = [
    (re.compile(r"0x[0-9a-fA-F]+"), "<addr>"),
    (re.compile(r"\b[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}\b"), "<uuid>"),
    (re.compile(r"\b[0-9a-fA-F]{16,}\b"), "<hex>"),
    (re.compile(r"\d+"), "<n>"),
]

Cannot read property 'title' of undefined (reading index 3) and (reading index 17) become the same issue. If the SDK supplies its own fingerprint without {{ default }} in it, that wins, so I can force grouping from the client when the default is wrong. An event with no exception falls back to its masked log message, and one with neither falls back to the top in-app frame. For a native signal crash the message is something like Signal 6, Code 0, so those group by signal, which is coarse but honest until there’s symbolication.

Alerting without flooding the channel

#

Each fingerprint becomes one issue, and each report is stored as an event under it. The issue only logs at ERROR (which is what the chat alerts listen for) in exactly these cases:

• a new issue
• a resolved issue that comes back, which reopens it
• the 10th, 100th, 1,000th and 10,000th sighting

An ordinary repeat only increments a counter. An issue marked ignored keeps counting and never alerts. With the chat handler’s own rate limit on top, a crash loop on a bad release reads as a handful of messages rather than a wall.

Storage is bounded the same way: each issue keeps its 50 most recent events and deletes older ones as new ones arrive. That broke my first chart, because 50 events can’t tell you how often something happened last week, so each issue also keeps a running count per day for 90 days, updated in the same transaction as the counter. The chart draws from the counts; the events are only for reading.

Two small races are worth handling if you build one of these. When a brand-new crash hits several phones at once, two reports can both try to create the issue, so make the fingerprint unique and have the loser re-read the existing row. And measure the 90-day window from the server’s today, not from the newest report, because a phone with its clock set wrong can date a report anywhere, including next year.

Payloads are capped at 128KB. An event over that loses its breadcrumbs and debug metadata first, and if it’s still too big only the exception, message, release and platform are kept, with a flag so I know it was cut.

Scrubbing

#

The SDK runs with sendDefaultPii: false, so in principle no IP address, cookies or request bodies are sent. The server doesn’t rely on that: before storing anything it keeps only id and username from user and drops request entirely, in case a later SDK version or setting starts sending them.

Testing against a real device turned up two more things to strip:

• Breadcrumbs record full URLs, query strings included. The app’s search requests carry the search term in the query string, so a crash right after a search would have stored what the person searched for. The server now cuts every breadcrumb URL at the ?.
• Console breadcrumbs carry the raw arguments that were passed to console.log, which can be any object the app happened to log. The message text stays (it’s useful), and the arguments are dropped.

I only found those by looking at an actual report, which is the main argument for the next section.

The client side

#

On the app, everything goes through one small module, for the same reason I wrap the over-the-air update SDK: an OTA bundle also runs on binaries built before the native module existed. A top-level import * as Sentry from "@sentry/react-native" on a binary without RNSentry throws on import, and that’s a blank screen. So the SDK is required lazily, and only after checking the native module is actually there:

import { NativeModules, Platform, TurboModuleRegistry } from "react-native";
import Constants from "expo-constants";

const DSN = process.env.EXPO_PUBLIC_CRASH_DSN;

function nativeModulePresent(): boolean {
  try {
    return Boolean(TurboModuleRegistry.get("RNSentry") ?? NativeModules.RNSentry);
  } catch {
    return false;
  }
}

export function initCrashReporting(): boolean {
  if (!DSN || Platform.OS === "web" || !nativeModulePresent()) return false;
  const Sentry = require("@sentry/react-native");
  Sentry.init({
    dsn: DSN,
    release: Constants.expoConfig?.version ?? "unknown",
    sendDefaultPii: false,
    tracesSampleRate: 0,
    enableAutoSessionTracking: false,
    // ...turn off the other tracing options the same way
    maxBreadcrumbs: 50,
    beforeSend(event) { /* tag the OTA bundle */ return event; },
  });
  return true;
}

Errors only. The server doesn’t store traces, sessions, profiles or replays, so none of them are switched on. No DSN (development, E2E builds) means the SDK never starts. The build profiles set the DSN and an environment (staging or production), so both kinds of build report to the same endpoint and can be told apart. The signed-in user is set by account ID, never a name or an email.

The error boundaries needed an explicit call. Expo Router’s root ErrorBoundary catches a render error and shows its own screen, so that error never reaches the global handler the SDK hooks into. The fix is a small wrapper that calls captureException in a useEffect and then renders the router’s boundary as before, used for the root and per-route boundaries alike. It’s in an effect rather than in the render so it reports once rather than on every re-render. The root component is also wrapped with Sentry.wrap, and if that throws it falls back to the bare component.

For the web there’s a .web.ts version of the module with every function switched off. Metro picks the platform file, so the SDK never ends up in the web bundle at all. The web app has no native module to run on.

Proving a native crash arrives

#

The unit tests post hand-written envelopes, which proves the parser handles what I think the SDK sends. Two days after release, production had zero crash reports. Only three people had updated yet, so zero was plausible, but “nobody has crashed” and “crashes don’t arrive” look identical from the server. I wanted to see a real one.

So I built a Release simulator binary with the DSN pointed at a local Django on port 8000 and a throwaway key:

EXPO_PUBLIC_CRASH_DSN=http://localcrashtestkey0000@localhost:8000/1 \
EXPO_PUBLIC_CRASH_ENVIRONMENT=local-verify \
xcodebuild -workspace ios/MyApp.xcworkspace -scheme MyApp -configuration Release \
  -destination 'platform=iOS Simulator,id=<udid>' -derivedDataPath /tmp/iosbuild

(Release, not Debug: Debug builds link the dev launcher and behave differently, and Release is what people run.) Then installed it, launched it, waited a few seconds, and killed it the way a real fault would:

xcrun simctl install booted /tmp/iosbuild/Build/Products/Release-iphonesimulator/MyApp.app
xcrun simctl launch booted com.example.myapp      # prints the pid
kill -ABRT <pid>

Simulator apps are ordinary macOS processes, so a signal reaches SentryCrash’s handler exactly as it would on a device. The handler writes the report to disk; nothing is sent while the process is dying. On the next launch the SDK finds it and sends it. I relaunched, and the simulator’s network log showed a connection to :8000 a second later.

Then nothing in the Django log, which had me confused for a minute: the dev server only logs non-2xx responses at the level I had it on. The database had it:

SIGABRT: Signal 6, Code 0 | cocoa 1.4.0 iOS 27.1 iPhone19,4 handled=False
received 09:19:53, crashed 09:19:38
contexts: app, device, os, ota_updates, trace · 24 breadcrumbs

Fifteen seconds from crash to stored, most of which was me relaunching. And looking at the real payload is what taught me things the tests couldn’t:

• The user ID wasn’t the account ID. The test build isn’t really signed in, so sentry-cocoa fell back to its own per-install ID. Worth knowing, and worth saying in the privacy note: a report carries the account’s ID if you’re signed in and a random install ID if you’re not.
• There’s no time zone. I’d assumed the culture context would be there. The device context has locale, memory, screen size, low-power mode and the kernel version, but no time zone.
• Native crashes skip beforeSend. My JS beforeSend adds an ota tag with the running OTA bundle, but a native crash is assembled by the native SDK on the next launch and never passes through the JS hook. To tag native crashes, the tag has to be set on the scope with setTag once the bundle is known, so it syncs down to the native layer.
• The breadcrumbs showed what the app does. Console warnings, view-controller lifecycle, and HTTP calls, including the analytics SDK’s own requests to its collection endpoint, which is a good reminder of what else the app talks to. And the URLs had query strings, which is how I found the search-term problem above.

Reading them

#

The reports are read in the Django admin, on an issue page loosely modelled on Sentry’s: counts and Resolve / Ignore / Reopen at the top, a bar per day, the split by release, OS, device and OTA bundle, then one event at a time with its exception, its frames (app frames highlighted, library frames folded away), the breadcrumbs and the device context. It didn’t need a new dependency.

One fix came out of building it: for a native crash, the “culprit” frame was the binary’s path on the device, which includes a per-install UUID, so the same crash looked different on every phone. It now shows only the address and the binary name, like 0x1000a1b2c (MyApp).

The privacy policy, all of it

#

Crash reports are personal information, near enough: an account ID, a phone model, and what someone did in the minute before the app died. So the privacy policy needed a paragraph. Writing it properly meant saying exactly what a report carries, so I listed it from the payload I’d just looked at: the error, app version and OTA update, phone model, OS and language, free memory and screen size, the last few dozen steps (screens, web addresses with anything after a ? removed, the app’s own log lines), and the account ID or a random install ID. The policy also says the reports come to our server and not to Sentry or any other crash-reporting company. Retention gets its own line in the table: the 50 most recent reports of each problem, plus a daily count for 90 days.

Once I was checking one paragraph against the code, it was hard not to check the rest. The existing policy didn’t hold up. The delete-account page said push-notification tokens and passkeys were deleted immediately when you close your account, and that server logs were gone within 90 days. None of that was true. Closing an account unlinks the sign-in methods and anonymises the profile, but the push tokens and passkeys stay until someone deletes them by hand, and the log archive has no expiry yet. So the whole policy was rewritten from an inventory of what the code actually does, and those lines now say “not yet removed automatically; email us”. That’s less reassuring, and it’s accurate.

The repo now has a rule for it: the policy describes what the code does, not what we intend, and a change that collects, sends or keeps personal data somewhere new updates it in the same commit. That includes the summary screen in the app, the delete-account page, and the app-store privacy forms, which only live in the store consoles. Crash reporting was the change that made me write that down.

What’s missing

#

Symbolication. This is the real gap, and the one thing sentry.io does that I haven’t replaced. Stacks are minified for JavaScript and unsymbolicated for native code:

• JavaScript: frames point into the Hermes bundle. Fixing it means keeping the composed Hermes source map for every store build and every OTA bundle, keyed by release and bundle, and applying it to JS frames when the page renders. That’s the next piece, because most of our crashes will be JavaScript.
• iOS native: instruction addresses need the build’s dSYMs.
• Android: Java frames are obfuscated by R8 when minification is on (they need mapping.txt), and C++ frames need NDK symbols.

There’s also a smaller gap I found above: native crashes don’t carry the OTA tag yet, because it’s only set in beforeSend. Setting it on the scope once the running bundle is confirmed is a few lines I haven’t written yet.

For now, an issue tells me what crashed, on which release, OTA bundle and device, and what the person was doing just before. It doesn’t tell me where in the source. For a small app that ships a few times a week, the type, the message and the breadcrumbs have been enough to find things so far.

The receiver, the admin page and the tests aren’t a lot of code, and it costs nothing on top of the API that was already running. The SDK does the hard part, which is catching a crash on a phone and getting the report off it on the next launch.

Disclosure. Codemagic paid me to do this migration and write about it. They get a review pass for factual accuracy, not for tone, and nothing here was changed at their request. Where I hit a problem and reported it, I say so; if it’s fixed by the time you read this, it’s described as fixed.

Update, 23 September 2026. Codemagic’s developers have worked through the issues I reported. Most are now fixed, including the iOS version-detection bug, the double-counted installs, the silent first run and the noisy debug command, and the docs now cover committed native projects and the crash-rollback budget. I’ve marked each one below where it comes up, and left the few that are still open as they were.

Update, 16 October 2026. The first native upgrade since moving to Patch (Expo SDK 58) built an Android release with no Patch configuration in it, and nothing reported an error. What happened and the one-file fix are in The first native upgrade.

In February I wrote about building my own Expo update server. It still works. maru, the community app I run, has taken every over-the-air fix through it since, on both platforms, for about a dollar a month.

It also means I own every piece of it: the database tables, the API endpoint, an admin command, a publish script, a shared secret, and a set of gotchas I learned one broken update at a time. When I sat down to plan this, I found that the script had been printing “Sent 0 push notifications” after every production publish since the day I wrote it, reading a field this server has never returned. Nobody noticed, least of all me. That’s roughly what running your own thing turns into.

So when Codemagic asked whether I’d try Patch, their open-source, self-hostable take on CodePush, I agreed on one condition: a real app rather than a demo. Real signing, real release process, committed native projects with hand edits in them, the lot. Most people moving off EAS Update are moving an app like that, not a fresh template.

This is how that went, with the rough edges left in.

What Patch is

#

Patch is a server you run yourself and an SDK you put in your app. The server side is an API, a background worker, Postgres, S3-compatible object storage and a web dashboard, shipped as a Docker Compose stack. The client side is @codemagic/react-native-patch, a TurboModule for iOS and Android with an Expo config plugin, plus a CLI, cmpatch, that bundles your JavaScript and publishes it.

Two design decisions stood out once I’d read the source.

The first is that the app never asks the API whether there’s an update. When a release is published, the worker writes plain JSON manifests into object storage, and the app just fetches those. An update check is two static GETs against storage or a CDN; the API only receives metrics. My old endpoint was on the request path of every app launch. This one can’t be.

The second is that every release names the exact binary version it’s for, and the CLI computes a fingerprint of your native project and refuses to publish JavaScript onto a binary whose native code doesn’t match. My server had a version string and a lot of hoping.

Trying it on my laptop first

#

The docs start you on a local evaluation stack, and that’s the right place to start.

npm install -g @codemagic/patch-cli
cmpatch selfhost local-eval up

The CLI installed in two seconds. The stack took 3 minutes 14 seconds the first time, most of it building the server and dashboard images locally, and in non-interactive mode it prints nothing at all while it does that, which looks exactly like a hang. (Since fixed: it now prints one progress line per step.) Then four healthy containers appeared, all bound to localhost: the dashboard on 8080, the API on 3000, Postgres, and MinIO on 9100. It sensibly avoided the ports my own stack was already using.

Sign-in is disabled in this mode, with a warning on every page not to expose it.

localhost:8080

The stack comes seeded with an app called Example Data, which is worth a look before you’ve published anything, because it shows what the dashboard is for. The numbers in the next two screenshots are that seed data, not real users.

localhost:8080

localhost:8080

For comparison, my dashboard was an admin list of updates with an “active” checkbox. I had no idea how many phones had actually taken an update.

Logging the CLI in is a proper browser flow with PKCE and a localhost callback, and the approval page tells you which account is asking and that the code expires in about a minute.

localhost:8080

Pointing it at a real app

#

Patch wants one app per platform, each with a Staging and a Production deployment created for you. I made maru-ios in the dashboard and maru-android with cmpatch app create, mostly to see both. The dialog you get after creating an app is the most useful screen in the product for a migration: both deployment keys, the two URLs the SDK needs, and a note that deployment keys aren’t secrets.

localhost:8080

Two small things. cmpatch init linked a project to apps that already existed, so it couldn’t be the first command, whatever the migration guide implied. That’s since been fixed: in interactive mode it now creates one app per platform itself. And the dialog gives the API URL as http://localhost:8080 while the docs and the CLI say 3000. Both work, because each port serves both, but you’ll wonder.

Then I ran cmpatch doctor against the untouched app, and it found a real problem in three seconds. It reported the iOS binary version as 2.0.0. The app is 2.5.1.

The CLI finds every Info.plist under ios/ and, if none has “test” in its path, takes the first alphabetically. maru has a WidgetKit extension whose folder sorts before the app’s own, so the CLI picked the extension’s Info.plist. That plist was stale at 2.0.0, which is my bug, but the effect is nasty: auto-detection would target every iOS release at a version no phone runs, and nothing would error. Updates would just never arrive, which is exactly the failure I warned about in my self-hosting post. Android read versionName correctly. The workaround is to pass --target-binary-version or --plist-file, and my publish script now passes the version explicitly after checking that app.config.js and both native projects agree. Later, the same bug made doctor’s own manifest check fail against a deployment that was working fine, because it went looking for version 2.0.0’s manifest. Once I’d fixed my stale plist, doctor passed all 24 checks.

Codemagic has since fixed this. The CLI now reads the version from the app target’s Info.plist, not an extension’s, and if more than one target could be the app, cmpatch asks you which.

The JavaScript side

#

Removing expo-updates and adding the SDK is a two-line change. Replacing what the app did with updates took more thought.

maru checks for updates on launch and whenever it comes back to the foreground, at most every fifteen minutes. If it finds one, it downloads it, and offers to restart; say no and it applies on the next cold start. There’s also a manual check in Settings that shows which bundle you’re running, and the error reporter tags every report with the update it came from.

The first thing I learned from the SDK source is that it looks up its native module at import time with TurboModuleRegistry.getEnforcing, which throws if the module isn’t there. maru also ships a web build, and a top-level import would take the web app down. So everything goes through one small module that loads the SDK lazily, with a .web.ts twin that returns nothing, which also keeps the SDK out of the web bundle entirely.

// ota.ts (ota.web.ts exports a patchSdk() that always returns null)
import { Platform } from "react-native";

type PatchSdk = typeof import("@codemagic/react-native-patch");
let sdk: PatchSdk | null | undefined;

export function patchSdk(): PatchSdk | null {
  if (sdk !== undefined) return sdk;
  if (Platform.OS === "web") return (sdk = null);
  try {
    sdk = require("@codemagic/react-native-patch") as PatchSdk;
  } catch {
    sdk = null;
  }
  return sdk;
}

The automatic check itself is mostly sync(), which never throws and resolves to a status:

const status = await patch.sync({
  installMode: "ON_NEXT_RESTART",
  mandatoryInstallMode: "ON_NEXT_RESUME",
});
if (status === "update-installed") offerRestart();

The SDK’s default for mandatory releases is IMMEDIATE, which reloads the JavaScript wherever the user happens to be. I changed it to apply the next time the app comes back from the background, because losing a half-written post is a worse bug than most fixes are good. At the time, InstallMode was a string union, not the enum object the docs’ examples suggested; it’s now exported as a constants object, so InstallMode.ON_NEXT_RESTART works as documented.

The native side, when you commit ios/ and android/

#

The migration guide says to run npx expo prebuild --clean after adding the config plugin. I ran it in a separate checkout and it deleted bundled images, my Android network security config, the iOS privacy manifest and Podfile.lock, and reverted a test-runner fix in build.gradle along with a few hundred lines of Info.plist. Without --clean, Expo cleared both directories anyway.

This isn’t Patch’s fault. maru started as a managed Expo app and grew native customisations, so ios/ and android/ are committed and EAS builds them as they are. But plenty of apps look like that, and the guide didn’t warn you. It does now: the Expo migration guide has two paths, prebuild for generated native projects and the exact hand-applied changes for committed ones.

What the plugin actually changes turns out to be small enough to do by hand. On iOS, one import and one line in AppDelegate.swift:

return CodemagicPatch.bundleURL() ?? Bundle.main.url(forResource: "main", withExtension: "jsbundle")

On Android, one import and one argument in MainApplication.kt (this is the React Native 0.82+ shape; older versions override getJSBundleFile() instead):

ExpoReactHostFactory.getDefaultReactHost(
  jsBundleFilePath = CodemagicPatch.getJSBundleFile(applicationContext),
  // ...

Plus three config values per platform and the removal of the old expo-updates settings: 13 lines added and 20 removed across both projects.

The plugin writes those config values as literals at prebuild time, which, with committed native projects, would mean one hard-coded deployment for every build. I wanted preview builds on Staging and store builds on Production from the same project, so the values come from the build environment instead. In Info.plist, Xcode expands build settings:

<key>CodemagicPatchDeploymentKey</key>
<string>$(PATCH_IOS_DEPLOYMENT_KEY)</string>

And in android/app/build.gradle:

def patchEnv = { name -> System.getenv(name) ?: (findProperty(name) ?: "") }
resValue "string", "CodemagicPatchDeploymentKey", patchEnv("PATCH_ANDROID_DEPLOYMENT_KEY")

The keys then sit in each eas.json build profile next to the EXPO_PUBLIC_ values. An empty value means the SDK quietly does nothing, which is fine for a local build and a disaster for a store build, so app.config.js now throws if a preview or production EAS build is missing them or still points at localhost.

The first update

#

For the first release I changed the onboarding headline, which you can see without signing in, and published it:

./scripts/publish-update.sh ios --notes "Onboarding headline over the air"

That took 30 seconds, most of it Metro and Hermes. The server’s worker published the manifest 0.6 seconds after the upload finished. The update is a 7.3 MB compressed tarball, against the 11 MB Hermes bundle plus assets my old export produced.

One thing to get right here: the JavaScript bundle inlines your EXPO_PUBLIC_ values when it’s built, so an update has to be built with the same values as the binary it lands on, or you ship JavaScript that talks to the wrong backend. My script loads them from the same eas.json profile that built the binary. Patch’s CI docs now spell this out too, along with the fact that release-react doesn’t read EAS profiles itself.

Then I relaunched the app. The device log showed the whole update check as two requests to object storage, meta.json and 2.5.1/manifest.json. The SDK wrote a Downloaded event three seconds after launch and staged the update, and the app offered a restart.

The bug was mine

#

Then I looked at the dashboard. One simulator had installed release v1, and it showed two successful installs and two active users.

localhost:8080

The SDK keeps its state and pending metric events as files in the app’s container, which made this easy to chase. There were two Success events with identical timestamps and different ids. My hook confirmed the running bundle with notifyAppReady() on mount, and then called sync() straight away, and sync() calls notifyAppReady() itself. The two calls overlapped, each saw the release as pending, and each recorded the events. The server de-duplicates on event id, but these had different ids, so both counted, and those duplicates are permanent.

The fix on my side was one shared confirmation per launch that everything else awaits. The SDK could also guard against overlapping calls itself, and I suggested that. It now does: overlapping notifyAppReady() calls, including the one inside sync(), share a single confirmation, so each install is recorded once. The SDK reference also now says that sync() confirms readiness itself, so an app that uses it doesn’t need to call notifyAppReady() separately.

The nice part was shipping the fix. It’s a JavaScript-only change, so it went out as release v2, and because a device on v1 can be sent a binary diff, v2 downloaded as a 543 KB patch instead of the 7.3 MB bundle, about 7% of the size. After the restart: one download, one install, one success, one active device.

localhost:8080

Breaking it on purpose

#

Rollback is the feature I cared about most, because my old setup had none on the device. A bad update would have crashed every launch until I noticed and un-ticked a checkbox.

So I released a v3 that throws before the first render, which means it can never call notifyAppReady(). The app downloaded it, offered a restart, and died to the home screen.

The next launch booted v3 again and crashed again, which surprised me, because the docs say a bundle that crashes before notifyAppReady() is rolled back. The source explains it: the SDK allows three unconfirmed launches before deciding it’s a crash, because one unconfirmed launch can just mean iOS killed a healthy process before any JavaScript ran. The fourth launch came back up on v2.

The device then reported the failure itself, as crash_rollback, and the dashboard showed it against v3. That’s a sensible design, but it’s worth being plain about what it means: a bad release costs each user three crashed launches before they recover, and at the time the docs didn’t mention the number. They do now, and the budget is configurable, with CodemagicPatchMaxLaunchAttempts natively or the Expo plugin’s maxLaunchAttempts. That’s the case for Staging first and a small Production rollout second.

Stopping a bad release for everyone else is one command, and it took under a second:

cmpatch release rollback --app maru-ios --deployment Staging

It doesn’t rewrite history. It publishes the previous release again as a new one (v4, marked as a rollback of v2), so the crashing release stays in the list with its failure count, which is the honest record. The dashboard does the same from a dialog.

localhost:8080

Android

#

Android had two surprises, neither Patch’s. The R8 keep rule I’d added for the SDK lived in app.config.js, which only reaches the native project through prebuild, so my first release build accidentally tested whether the SDK survives R8 without one. It does: R8 renamed all its classes, and updates downloaded, installed and rolled back fine. And the emulator, freshly cold-booted with Gradle still holding memory, was so slow that my first test gave up 45 seconds before the SDK finished the download it had already started.

Release v1 applied, with one success and one active device, and a crashing v2 rolled back after the same three-launch budget.

Past the happy path

#

Updates arriving and rolling back is the core of it, but a team lives in the other features, so I went through them one at a time.

Staged rollouts are deterministic, which I liked. A device is in if the first eight hex digits of md5(deviceId + "-" + releaseLabel), modulo 100, are below the rollout percentage, so the same phone stays in or out for a given release. I worked out that my simulator sat in bucket 91 for the next release, published it at 25%, and the simulator correctly got nothing. Raised to 95% with cmpatch release patch, which took under a second, it updated on the next launch.

Then my next release failed with a 409: “deployment has an active rollout below 100 percent”. Only one partial rollout per deployment is allowed, so you finish, disable or roll back a canary before shipping anything else. That’s reasonable, and documented, but a pipeline that publishes on every merge will run into it in the middle of a canary. Since I reported it, the CLI explains itself here: if an active partial rollout is blocking you, it says waiting won’t help and how to complete or disable the rollout, and if another release job is still running it tells you to wait with release inspect --wait. The docs now have a section on sequential release changes.

Mandatory releases use the SDK’s mandatoryInstallMode. With mine set to apply on the next resume, the app downloaded the update and showed its prompt, and when I ignored the prompt, sent the app to the background and brought it back, the new version was simply there.

Disabling a release does more than stop offering it. The moment I disabled the latest one, the manifests pointed at the newest release still enabled, and devices on the disabled one were offered a patch back to it. Disable everything for a binary version and the manifest’s target becomes null, and the SDK reverts the app to the bundle it shipped with. My update hook didn’t prompt for that case at first, so now it does.

Promoting from Staging to Production took under a second and reused the exact package that was tested, same hash and notes, rather than rebuilding it. It also accepts a rollout percentage, so “promote to 10% of Production” is one command.

Building and publishing can also be separate steps. cmpatch bundle produces a .cmpatch artifact, and the dashboard’s “Bundle upload” option reads it before uploading: platform, target version, fingerprint, signing status, size and bundler. That suits a setup where CI builds the artifact and someone else decides when to publish it.

That’s also where I ran into the fingerprint guard in earnest. After I fixed my stale widget plist, the server refused my next release with a 409, because the native project’s fingerprint no longer matched the one it had recorded for 2.5.1. On Android, editing a comment in app.config.js was enough, since Expo includes the app config in the fingerprint. The dashboard shows the same check as a warning with both fingerprints and an “Upload anyway” button. In both cases nothing native had changed on the installed binary, so overriding was right. In real use, though, the honest answer is nearly always a version bump and a store build, and my publish script now says exactly that when it happens.

Code signing is RS256: the CLI signs the package hash with an RSA private key, and the SDK checks it against a public key compiled into the binary. The key goes in as a single base64 line, which fit my build-setting approach, and I tested it on both platforms. With the key in the app, an unsigned release was refused before it downloaded. With “require code signing” turned on for the app, the server rejected an unsigned publish with an error that named the fix and linked the guide. A signed release installed, and the SDK recorded the signature as verified. It also refused an older unsigned release the server fell back to when I disabled the signed one. The catch is that the public key has to be in the first binary you ship, because an update can’t add it.

localhost:8080

Teams and CI are covered as well. Members get one of four team-wide roles (owner, admin, developer, viewer) and invitations wait until the person first signs in. Personal access tokens worked with an empty home directory, passed either as --token or through environment variables, and the token list shows when each was last used. I’ve also written a manually triggered CI workflow that runs the same publish script on a Linux runner with the signing key from a secret, though I haven’t run it yet, since it needs the hosted server.

One tool I’d have skipped at the time is cmpatch debug ios. It streamed the system log filtered on “OTA” case-insensitively, which matches “Rotation”, so a twenty-second app launch produced a screenful of SpringBoard messages and nothing from the app. That’s fixed: it now shows only lines carrying the SDK’s markers. The SDK doesn’t log much on its own, mostly errors and development-build warnings, so expect a quiet stream. The getpwuid_r warning you may see comes from xcrun simctl, not Patch.

How hard is it, really

#

Codemagic asked for my opinion, so here it is, as plainly as I can put it.

Trying it out is easy. Two commands, a few minutes, and you have the real server, a dashboard with demo data, and a CLI you can point at your own app. I’d rate it the best part of the first hour.

Integrating the SDK is moderate work, and most of the difficulty is in your own project rather than in Patch. On a fresh Expo app with generated native folders, it really is the plugin plus a few lines replacing expo-updates calls. On an app like maru, with committed native projects, a widget extension, a web build, R8 and build profiles, the work was in the edges. I had to apply the native changes by hand to avoid prebuild, make the configuration build-time so Staging and Production builds could differ, and keep the SDK away from web. I also had to make sure the JavaScript in an update was built with the same environment as the binary. None of it was hard once I understood it. When I did it most of that wasn’t in the docs, though the committed-native-projects path and the bundle-time environment now are.

Day-to-day use is good. Publishing is one command and about half a minute. Rollouts, promotion, disabling and rollback are all under a second from the CLI or the dashboard, and the errors say what went wrong and what to do. The server’s guards (the fingerprint check, one canary at a time, signing) are strict in ways that stopped me doing the wrong thing more than once.

Running it in production is the bigger commitment. It’s a Docker Compose stack with Postgres and object storage that you have to host, back up and upgrade, and it needs two domains and an OAuth app. For a team already running servers that’s a quiet afternoon. For a team on EAS precisely because they didn’t want servers, it’s the real cost of switching, more than the SDK work.

In wall-clock terms, I installed the CLI at 12:29 and had the first update applying on a simulator at 13:34, and half an hour of that was an Xcode build that stalled for reasons unrelated to Patch. Everything in this post, on both platforms, was done by mid-afternoon, not counting the hosted server.

The things that cost me the most time, in order: expo prebuild --clean wiping my committed native projects, and then working out how to wire everything by hand; the iOS version-detection bug, which would have sent updates to a version nobody runs; the double-counted installs, which were my own doing but easy for anyone to repeat, since sync() calls notifyAppReady() itself; and the undocumented three-launch crash budget, which surprised me halfway through a test. After those came things that are correct but need planning: the fingerprint guard firing after harmless native-adjacent edits, one canary at a time per deployment, and lifecycle jobs that can’t overlap, which matters as soon as a script does several things in a row. The rest was small: a silent first-run setup, a noisy debug command, and a few docs examples that didn’t match the code.

I wrote all of it up for Codemagic, with the details, and by 23 September most of it was fixed: the version detection, the overlapping confirmations, the prebuild warning, the crash budget, the silent setup, the debug command, the InstallMode examples, and the error you get when a release job or a canary is in the way. What’s still open is smaller: the dashboard showing the API on :8080 while everything else says :3000, the SDK throwing on import on web and in Jest, and a few docs gaps.

Going to production

#

Everything above ran against the evaluation stack. Production is the same app and the same scripts pointed at a real server, and this is the plan for maru.

The supported way to run the server is the Docker Compose install: a Linux machine with ports 80 and 443 open, two domains (one for the API and dashboard, one for downloads), and a GitHub OAuth app for sign-in, which the server won’t start without. The installer does the rest, including TLS:

scripts/selfhost/install.sh \
  --api-domain updates.example.com \
  --storage-domain storage.updates.example.com \
  --email you@example.com \
  --github-oauth-client-id <id> --github-oauth-client-secret <secret>

It writes a .env.selfhost with the database and storage secrets, which belongs in your password manager.

You don’t have to use the bundled database and storage, though, and that’s what makes it fit a stack like mine. maru’s API already runs on Fly.io and its files live on Tigris, and Patch works on both. Underneath the Compose file, the server is one container running in MODE=all (API and release worker together) that needs a Postgres database, S3-compatible storage and a public HTTPS URL for the files. That’s a Fly app built from the repository’s Dockerfile (there’s no published image yet, so fly deploy builds it), a Fly Postgres database, and a Tigris bucket:

MODE=all
DATABASE_URL=postgresql://…            # Fly Postgres
STORAGE_ADAPTER=s3
S3_ENDPOINT=https://fly.storage.tigris.dev
S3_REGION=auto
S3_BUCKET=myapp-patch
S3_FORCE_PATH_STYLE=true
PUBLIC_BASE_URL=https://myapp-patch.fly.storage.tigris.dev/codemagic-patch

Two caveats. Running it outside Compose is documented in the repository as reference material, not a supported path in this first open-source release, so on Fly you’re on your own for upgrades and backups. I haven’t deployed it that way yet either; this is the plan, not a report. And if you’ve read my post on Fly’s scale-to-zero, keep at least one machine running. Phones never wait on the API for an update check, since manifests come straight from storage, but every metrics upload and every release job would otherwise start with a cold boot.

Tigris is the part I’m most keen on. It’s where maru’s files already are, and it puts a CDN in front of the URL every app launch hits. Mind the manifest caching if you do this: with the plain base-URL delivery adapter Patch serves manifests as no-cache, because it can’t purge a CDN, and a manifest cached on its own TTL would keep serving a release you’ve just rolled back. The configuration reference now documents this under MANIFEST_CACHE_CONTROL, and the docs site covers S3-compatible storage.

On the hosted server you create the apps again, put the new keys in the preview and production profiles in eas.json (Staging keys and Production keys), and create a token for publishing:

cmpatch token create --name ci

If you want code signing, decide now: generate the production key pair, keep the private key as a CI secret, put the public key in the build profiles, and turn on “require code signing” for both apps. The first Patch binary also needs a new version number, because the binaries already in the stores run expo-updates. Build it with the preview profile for TestFlight and Play internal testing first, and check the keys actually made it into the built Info.plist and Android resources before handing it to anyone. That’s the one link my local tests didn’t exercise, since I exported the environment by hand. Then publish to Staging, confirm one download and one success per tester, and only then build for the stores.

Production releases start small and grow:

./scripts/publish-update.sh ios --production --rollout 10
cmpatch release patch --app maru-ios --deployment Production --label v1 --rollout-percentage 100

The old server stays up in the meantime. Every copy of 2.5.1 that people haven’t updated still checks it, and hotfixes for them still go through the old script. Once the logs show those requests have stopped, the old server, its publish script and its secret can all go.

The first native upgrade

#

Added 16 October 2026.

Patch has been carrying maru’s production updates since early October. The first native release since then was the move to Expo SDK 58: React Native 0.88, and on Android a new major version of the Gradle plugin. A native change can’t go over the air, so it’s a new binary, 2.6.0, and my publish script refuses to send 2.6.0 updates until that binary exists. Both platforms built cleanly the first time.

The production section above says to check the keys really made it into the built Info.plist and Android resources before handing a build to anyone. This time I did, and that check is the reason this section exists. The iOS build had its four Patch settings. The Android build had none:

bundletool dump resources --bundle=build.aab --values | grep -A1 "string/Codemagic"

That command printed four keys for the 2.5.11 build and nothing for 2.6.0. The environment wasn’t the problem: Gradle’s generateReleaseResValues writes all four. They’re removed later by the release resource shrinker, which in Android Gradle Plugin 9 is much more aggressive. The same app went from 9,240 resources to 7,116. The SDK reads its configuration by name with getIdentifier, which the shrinker can’t see, so it removed the four strings as unused. A missing string reads as empty, and empty means “no updates”. Nothing would have failed. Android 2.6.0 would have installed, run normally and never received an update, and the only sign would have been no Android 2.6.0 devices on the dashboard.

The fix is one resource file that tells the shrinker to keep the four strings:

<!-- android/app/src/main/res/raw/codemagic_patch_keep.xml -->
<resources xmlns:tools="http://schemas.android.com/tools"
    tools:keep="@string/CodemagicPatch*" />

Don’t call it keep.xml: React Native’s bundle task already generates a raw/keep.xml for your JavaScript assets (which is why images and fonts survived), and the two would collide. The better place for this rule is the SDK itself, since a library’s keep file is merged into the app automatically, and a warning when a release build starts with no deployment key would turn the silent case into a visible one. This is the same kind of failure as the iOS version-detection bug earlier: nothing errors, and the updates just never arrive.

The same check turned up a second problem that has nothing to do with Patch, and it’s worth knowing for a different reason. The Android release build crashed on launch, twice, for two separate reasons inside Expo’s own modules. First, Expo’s widgets module failed to register its native classes, so the app died before any JavaScript ran. With that fixed, the app got as far as starting its JavaScript and then stopped, because three modules that had moved to Expo’s new module system were never registered. Patch’s crash rollback can’t help with either, because there is no update to roll back. It protects you from a bad update, not from a bad binary. A native upgrade still has to be installed and opened on a real device or emulator before it goes to the store, Patch or no Patch.

One smaller thing came out of the same testing. When sync() fails, it returns "error" and keeps the reason to itself unless the app is running in development, so a release build can only log that something went wrong. My hook now calls checkForUpdate() once after a failed sync and logs what that throws, which is how I learned that a local simulator build had empty Patch URLs (unsupported URL) rather than a broken server. It would help if the last error were readable in release builds too.

My checklist for a native release now has three lines it didn’t have before: dump the Patch settings from the built .ipa and .aab, compare the deployment keys with the build profile, and launch the release build itself, not a debug build, before submitting.

What I’d tell someone with an Expo bill

#

If you’re paying for EAS Update because building a server looked like too much work, Patch is a real answer: you run a server, but it’s a Compose stack and an installer, not code you maintain. Compared with what I built, you get staged rollouts, per-release install and failure numbers, binary patches, a fingerprint guard, code signing, crash rollback on the device, and a rollback button. I had built none of those.

What it costs: a new binary before anyone gets a Patch update, since the SDK has to be in the app; a server to keep upgraded and backed up; three crashed launches per user by default when a release is bad enough; and some hand work if your native projects are committed. It’s also young. This was version 0.3.0, and I hit a version-detection bug, a noisy debug command and a few gaps in the docs, and made a bug of my own that the SDK could have prevented. The version detection and the debug command have since been fixed, most of the docs gaps filled, and the SDK now guards against the mistake I made. None of it stopped the migration, and all of it was visible, in the CLI, in the dashboard, or in the SDK’s own files on the device, which is more than I can say for my old setup.

My marketplace app is a single-page app built with React Native Web. Every listing page is one script tag and an empty div until JavaScript runs. That is fine for people and fatal for search: Google can sometimes execute the JS, Bing will not, and Naver’s crawler — the one that matters most when nearly all your listings are in Korean — certainly will not.

This is the story of making those pages visible, and then of the caching rule that undid it in a way I did not see coming.

Dynamic rendering, which is not cloaking

#

The fix is old and Google-sanctioned: detect a crawler, and serve it server-rendered HTML containing the same content the SPA would have painted. Not different content. The same content, earlier.

So a listing URL like /listings/view?id=... branches. A browser gets the SPA shell. A crawler gets a real document: a heading, the listing’s key facts, an image per photo with alt text, a body excerpt, and links to a handful of related listings so the page is not an orphan.

That last part mattered more than I expected. Before those internal links existed, roughly a thousand listing pages sat with nothing pointing at them, and Search Console showed the crawl settling to seven to eleven requests a day. A crawler that cannot find your pages does not care how good they are.

The trap: a vendor’s name is not their bot’s name

#

Crawler detection is a user-agent match, and here is where it bit.

Korean platforms crawl under names that look nothing like the company. Naver’s crawler is Yeti. Daum’s is Daumoa. Kakao’s link scraper is kakaotalk-scrap. None of them is the vendor name.

Meanwhile those same vendors ship apps with embedded browsers, and those send a real browser user-agent with the app’s name appended. KakaoTalk’s in-app browser says KAKAOTALK/10.5.1. Kakao’s scraper says kakaotalk-scrap/1.0.

Match on kakao and you match both. Which is exactly what I did, and so every KakaoTalk, Naver-app and Daum-app user who opened a link got the crawler page: no JavaScript, no navigation, no login. A perfectly good document, and completely useless to a human.

The fix is an ordering rule, and it is worth stating as one:

import re

# In-app browsers append the app's name to an ordinary browser UA.
INAPP_BROWSER_RE = re.compile(r"kakaotalk/|naver\(inapp|daumapps", re.I)

# Name the bot, never the vendor: Naver crawls as Yeti, Daum as Daumoa,
# Kakao's link-preview scraper as kakaotalk-scrap.
CRAWLER_RE = re.compile(r"googlebot|bingbot|yeti/|daumoa|kakaotalk-scrap", re.I)


def is_crawler(user_agent: str) -> bool:
    # An in-app browser is never a crawler, so that check comes first.
    if not user_agent or INAPP_BROWSER_RE.search(user_agent):
        return False
    return bool(CRAWLER_RE.search(user_agent))

In-app browsers are checked first and win. And every crawler token has to name a bot, never a vendor.

Then I put Cloudflare in front of it

#

The origin is a single shared-cpu machine in Sydney that sustains twenty to thirty requests a second. Googlebot has about 995 URLs to walk. Every one of them was reaching Sydney, because cf-cache-status read DYNAMIC on all of them: Cloudflare does not cache HTML unless a rule tells it to. About 16% of the zone was cached, and none of the part that mattered.

So: a cache rule on the crawler pages. Straightforward. Except that a listing URL does not have one body. It has three.

• A crawler asking for HTML gets the server-rendered page.
• A human gets the SPA shell.
• An agent sending Accept: text/markdown gets markdown, built from the database, because converting an empty HTML shell would produce an empty document.

The origin says Vary: Accept and Vary on the user-agent would be hopeless anyway. And here is the thing I did not know:

Cloudflare ignores Vary on HTML. All three bodies share one cache key.

Whichever body arrives first wins that URL for every later visitor in that edge location, for the hour the origin’s s-maxage allows, plus a day of stale-while-revalidate.

I got to observe this in production in both directions:

• A Naver-app user warmed a listing URL with the crawler page, and plain iPhone Safari then got that page on a HIT. A human staring at a JavaScript-less document.
• A human warmed a URL with the SPA shell, and Googlebot got the empty shell. The exact failure the whole SEO effort existed to prevent, now served faster and from the edge.

The rule that fixes it by narrowing

#

You cannot make Cloudflare respect Vary on HTML. What you can do is make sure only one of those three bodies is ever eligible to enter the cache:

(starts_with(http.request.uri.path, "/listings/")
 and not any(http.request.headers["accept"][*] contains "text/markdown")
 and (lower(http.user_agent) contains "googlebot" or ...)
 and not (lower(http.user_agent) contains "kakaotalk/" or ...))

Match crawler user-agents only, and exclude markdown requests. Now a human request matches no rule at all, so it reads DYNAMIC and goes to the origin as before. Only crawler requests read or write the cache, and the only body that can ever be in there is the crawler one.

Edge TTL is respect_origin, so the lifetime stays owned by the code that renders the page rather than being pinned in a dashboard.

One operational detail that is easy to miss: narrowing a rule does not evict what is already in the cache. You have to purge, or you keep serving the poisoned entries you just stopped creating.

The two lists, and why their errors are not symmetric

#

The rule needs a list of crawler user-agents, and Django already has one. Two copies of the same knowledge in two systems is usually a smell, but here the interesting part is what happens when they drift, because it is not symmetric:

• The bot list in Cloudflare must stay a subset of the one in Django. A token Cloudflare thinks is a bot but Django does not would cache an SPA shell under the crawler-only key, and then serve that to a real crawler.
• The in-app browser exclusion list must stay a superset of Django’s. Excluding too much only costs you a cache hit.

Get either one wrong in the safe direction and you lose some caching. Get the first one wrong in the other direction and you are back to serving Googlebot an empty page. That asymmetry is why the lists use contains rather than a regex, incidentally: matches needs a Business plan, and contains is expressive enough for a list that is allowed to be approximate in a known-safe direction.

An invariant that fails the build instead of the page

#

The crawler pages embed presigned image URLs, valid for seven days. The pages themselves are cached at the edge for an hour, plus a day of stale-while-revalidate.

Those two numbers are related, and if the cache lifetime ever exceeds the image lifetime, the edge starts serving pages whose every photo 403s. Nothing would alert on it; the HTML is fine.

So the code that defines those numbers refuses to import if the relationship breaks:

CRAWLER_IMAGE_TTL = 7 * 24 * 3600  # presigned image URLs
EDGE_S_MAXAGE = 3600
EDGE_STALE_WHILE_REVALIDATE = 24 * 3600

# Fail at import time if the edge cache could outlive the image URLs.
if EDGE_S_MAXAGE + EDGE_STALE_WHILE_REVALIDATE >= CRAWLER_IMAGE_TTL:
    raise RuntimeError("edge cache lifetime must stay below the image URL lifetime")

A constraint between two constants in different files is invisible in code review and obvious at process start. Put it where the process starts.

Two things Cloudflare did that I had to undo

#

Vary: Origin kept the bundle out of the cache. django-cors-headers stamps Vary: Origin on every response it touches, and Cloudflare declines to cache anything that varies on more than Accept-Encoding. That quietly excluded the 1.6 MB JavaScript bundle — the single biggest thing on the critical path — from the edge entirely. The fix is a middleware placed first in the list, so it runs last on the way out, rewriting Vary on the static asset paths.

Compression at the edge was worse than compression at the origin. Letting Cloudflare do Brotli on the fly produced 2,041,742 bytes where the origin’s own gzip produced 1,658,115. Twenty-three percent larger, for the privilege of not thinking about it. On-the-fly compression optimises for CPU, not for ratio; precompressed assets from the origin win.

Telling search engines without waiting to be crawled

#

Two protocols, and they cover disappointingly little between them.

IndexNow takes a ping and covers Bing, Yandex and Seznam. Google and Naver do not participate. I ping it in the background whenever a listing is saved, with a short timeout, and skip listings that are sold or closed. A network failure logs a warning and never fails the save that triggered it, which is the only acceptable behaviour for a best-effort side channel.

Google’s Indexing API accepts exactly two schema types, and one of them is JobPosting, which I have. Live jobs send URL_UPDATED, closed ones send URL_DELETED. That deletion is only honest because closed listings also emit noindex, and the code says so, so that whoever removes the noindex finds out what else they broke.

Two things cost me time there. The quota is 200 URLs a day, and a bulk import of listings would fire it hundreds of times — so imported listings are skipped, or one import spends the entire day’s budget. And the endpoint is urlNotifications:publish, with a colon. Use a slash and Google returns a plain HTML 404, which looks exactly like a disabled API.

What the structured data taught me

#

Three rules, each learned by having Search Console complain:

1. Omit what you cannot parse. Salaries arrive as free text. I used to emit the raw string as baseSalary.value, which is invalid, and flagged every job for a missing unit. Now an unparseable salary is simply absent.
2. Never fabricate a field to satisfy a schema. Job ads rarely carry a street address. The temptation is to invent something plausible for PostalAddress. A fabricated address on a job ad is worse than an incomplete one, so only locality and country go in.
3. Expiry is not optional. Google drops a JobPosting with no validThrough, so one is derived, with a floor so a stale listing does not claim to have expired yesterday.

And a fourth, about sitemaps: do not claim changefreq: daily for everything. Asking Googlebot to re-crawl a thousand listings every day reads as an overloaded host, and it responds by cutting the crawl rate for the whole site. Derive it from how recently the thing actually changed.

The sitemap also has a quality floor: forty characters of title and body combined. Search Console had reported around a thousand listings as “Crawled – currently not indexed,” and one used-goods page’s entire unique content was a one-word title and a phone number. Forty characters is deliberately low, because 471 of 484 listings are in Korean and Korean says more per character; an English-tuned threshold would have dropped half the marketplace. Thin listings stay crawlable through the index pages, and re-enter the sitemap automatically when someone edits them.

What I would take from this

#

• Serving different bodies at one URL is a cache-key problem before it is anything else. Work out what your CDN keys on, and assume it ignores Vary until you have proven otherwise on that exact content type.
• When two systems must agree on a list, work out which direction of error is safe and write that down next to both copies. “These can drift, and here is the only way they are allowed to” beats pretending they cannot.
• Match bots by bot name, never by vendor name. Especially outside the Anglosphere, where the vendor also ships the browser your users are holding.
• Put invariants between distant constants somewhere that fails loudly. Import time is free and nobody can skip it.

I’m currently shipping two products as a solo developer: a field estimation app for tradespeople who quote jobs on site, and a marketplace and community app with listings and real-time messaging.

Different products, different users, nearly identical stack — and both run entirely on shared-cpu-1x machines, the cheapest tier Fly.io offers. That constraint is deliberate. Instead of sizing servers for peak load, I squeeze as much as possible out of one small machine, measure exactly where it collapses, and let Fly’s autoscaler add and remove machines around that measured limit. This post covers the optimisations that make that possible, the load-test numbers behind the thresholds, and how scaling up and down actually works on Fly.io.

The Philosophy: Measure One Machine, Then Multiply

#

The scaling strategy for both apps is the same three steps:

1. Make a single small machine as fast as it can reasonably be.
2. Load-test it until it breaks, and note the concurrency level where latency collapses.
3. Configure Fly’s autoscaler to start a second machine just before that point, and stop idle machines when traffic drops.

Capacity then scales roughly linearly with machine count, and the baseline cost is one small VM per app. Everything below serves one of those three steps.

Making One Small Machine Fast

#

Granian: WSGI for the API, uvloop ASGI for WebSockets

#

Both backends are Django 6 + Django REST Framework. The API is served by Granian, a Rust-based server — but under WSGI with a blocking thread pool, not ASGI:

granian --interface wsgi myapp.wsgi:application \
    --workers 1 --blocking-threads 16 \
    --workers-max-rss 1400 \
    --workers-kill-timeout 15 --respawn-failed-workers

I started on --interface asgi --loop uvloop, and it was the wrong call. Every HTTP view in these apps is sync, and Django’s ASGI handler runs sync views thread_sensitive — one at a time per worker. So ASGI gave me roughly one concurrent request per worker and loaded the whole asyncio/uvloop/Channels stack into RAM for nothing. WSGI with --blocking-threads runs those same sync views in a real thread pool: one worker × 16 threads = 16 concurrent I/O-bound requests (DB reads, S3 presigns) at lower resident memory than the async build.

Why a single worker? The VM is shared-cpu-1x — one core. A second worker can’t run Python in parallel on one core (GIL + one CPU), but it does cost a whole extra resident Django process; Granian warns about exactly this. Sixteen threads on one worker is the honest ceiling of a 1-vCPU box for I/O-bound work. --workers-max-rss 1400 respawns the worker if it drifts past its memory cap, so a runaway PDF or image job gets recycled instead of walking the machine toward OOM.

WebSockets moved out to their own Granian process — an ASGI app with uvloop and --ws, one worker for long-lived connections. The sync API needs no ASGI path at all: when a view wants to push to a socket it calls async_to_sync(channel_layer.group_send) over Redis, so the message still crosses to the WebSocket process. Two servers, each running the interface it’s actually good at. Local Docker runs the same WSGI thread pool, so my laptop approximates one production machine when testing.

Rust in the Hot Paths

#

Wherever a Rust-backed drop-in exists for a Python hot path, I take it:

• orjson via drf-orjson-renderer — every API response serialises through Rust instead of the stdlib JSON encoder.
• blurhash-rs — image placeholders computed at upload time.
• uuid-utils — Rust UUID generation.
• psycopg 3 (binary) — C-accelerated Postgres driver.

None of these change the architecture; they just raise the ceiling of what one vCPU can do before you need a second machine.

Kill the Hidden Latency Taxes

#

The single biggest win in either codebase wasn’t an algorithm — it was connection reuse. Under Granian’s thread pool, every new thread was building a fresh boto3 S3 client, and every endpoint that returns a photo or PDF paid that cost. A small custom storage class now caches the client once per process instead:

Signed URLs went from ~150ms to ~1ms on new threads.

The same lesson applies to the database: Fly’s Postgres sits behind PgBouncer, so the app runs conn_max_age=0 and lets the pooler own connection reuse, with a 5-second connect_timeout to fail fast on stale SSL instead of hanging a worker.

Cache Where the Bottleneck Actually Is

#

Load testing the marketplace revealed something worth internalising: the ceiling was Postgres connections, not CPU. Browse traffic is overwhelmingly anonymous and identical across users, and under load those identical list queries saturate the connection pool while the CPU sits idle.

The fix is a Redis cache for anonymous list pages, with per-namespace version counters for invalidation — a write bumps one counter (O(1), no key bookkeeping) and every cached page for that resource is instantly stale. A short TTL backstops anything missed. Serving browse pages straight from Redis removes the real bottleneck and lets the small machine spend its connections on logged-in traffic.

Background Work Without a Broker

#

Both apps use django-tasks-db, Django’s database-backed task framework, instead of Celery. A separate db_worker process polls the database for jobs — image processing, PDF generation, machine translation, reminders. No broker, no extra service to size and monitor, and the API process never blocks on heavy work. The worker even connects straight to Postgres, bypassing PgBouncer, since long-running jobs don’t play well with transaction pooling.

Memory: jemalloc and a Swap Cushion

#

Image and PDF processing spike memory in ways steady-state sizing doesn’t predict. Two cheap mitigations: jemalloc preloaded in the Docker image to reduce fragmentation, and swap on every machine (2GB on the API machines, 1GB on the WebSocket machine) so a spike swaps to disk instead of getting the process OOM-killed. Swap is not a performance feature — it’s an insurance policy that lets me keep memory allocations small.

Django Load Testing: Finding the Knee

#

Thresholds are only defensible if they’re measured. Each project has its own harness:

• The estimation app has a small Python stress script that sweeps concurrency levels and reports RPS, p50/p95/p99 and error rates. It runs against a few hundred generated sample records, so the list endpoints are queried at realistic data volumes.
• The marketplace uses Locust with a step-load shape that ramps 50 → 500 users in 30-second steps to find the saturation point, with endpoint weights matching real browse-heavy traffic, plus a second, smaller suite that ramps 5 → 50 virtual users against production with a p95 < 2s threshold.

Representative numbers from the estimation app’s sweep (one Granian worker × 16 blocking threads, Docker on my dev machine):

And the number that drives everything else comes from load-testing the marketplace on a single shared-cpu-1x machine. On the old ASGI build this was a hard wall: sync views serialised to roughly three usable slots, so p95 held under 300ms up to ~5 concurrent requests and then collapsed past ~8–10 as everything queued behind the thread-sensitive lock. Moving the API to a WSGI thread pool is what pushed that knee out — 16 real concurrent slots per machine instead of ~3 — and that new ceiling is the whole basis of the autoscaling config.

How Scaling Works on Fly.io

#

Fly’s model is simple: your app is a fleet of identical machines behind a proxy, and the proxy counts load per machine. You declare two numbers:

[http_service.concurrency]
  type = 'requests'
  soft_limit = 12   # past this, start another machine
  hard_limit = 24   # past this, shed load instead of queueing

auto_stop_machines = 'stop'
auto_start_machines = true
min_machines_running = 1

Scaling up: when in-flight requests on a machine exceed the soft limit, the proxy starts a stopped machine and routes new traffic there. The WSGI thread pool gives each machine 16 real concurrent slots, so the soft limit sits at 12 — about 75% of the pool, so machine #2 starts while threads still have headroom rather than after users are seeing multi-second latency. The hard limit of 24 (1.5× the pool) is the circuit breaker: short bursts queue on threads, and beyond it the proxy sheds load rather than letting one machine melt down. (The old ASGI build ran 8/20, sized for its ~3 real slots — widening the pool is what let both numbers move up.)

Scaling down: when traffic drops, Fly stops idle machines automatically, down to min_machines_running = 1. A stopped machine costs nothing for compute, only its rootfs storage, so the baseline cost of each app is literally one small VM with burst capacity always ready.

I originally wrote here that a stopped machine “cold-starts in well under a second.” That was wrong, and expensively so. When I finally measured it instead of assuming it, a cold start took 25 to 33 seconds, and a request that lands on a machine still booting has to wait out all of it. I first blamed those cold starts for a 22% failure rate on my CDN dashboard, while every health check stayed green. That number turned out to be the CDN’s own Early Hints probes failing against the origin; no real client got a 504. The cold starts were real all the same.

The fix is not in the scaling numbers above; those are fine. It is in what the machine does on the way up, and in asking Fly to suspend machines rather than stop them, which turns a wake into a ~3 second memory restore. I wrote that up separately: Scale to Zero, and Every Restart Is a 30-Second Outage covers the diagnosis and what to change. If you are running anything on Fly with auto_stop_machines, read that one before you trust this paragraph.

One subtlety: the concurrency type is requests, not connections. Modern clients hold keep-alive connections open, so counting connections barely moves under real load and the autoscaler never triggers. Counting in-flight requests tracks actual work.

WebSockets Scale on a Different Axis

#

The marketplace splits its WebSocket server into a second Fly app sharing the same Docker image. The API app runs one worker × 16 blocking threads under WSGI and counts requests (soft limit 12). The WebSocket app runs 1 worker under ASGI with uvloop on a 512MB machine and counts connections:

[http_service.concurrency]
  type = 'connections'
  soft_limit = 1500
  hard_limit = 2000

A long-lived socket is nearly free to hold open but occupies a slot for hours, while an HTTP request occupies a slot for milliseconds — the same autoscaling signal can’t serve both. Splitting them means each fleet scales on its own honest metric, and idle WebSocket capacity costs 512MB, not 2GB. On the Channels side, the Redis layer runs with capacity: 1500 and 60-second message expiry, plus socket keepalives every 30 seconds because Fly’s Redis proxy idle-closes the long-lived BRPOP sockets that Channels depends on.

Offloading Compute to the Edge and the Client

#

The cheapest server work is the work the server never does:

• Static and media files live in Fly’s S3-compatible object storage, served through a CDN with signed URLs for private media — zero image bytes transit the Django machines.
• AI inference runs on the user’s phone. Both apps embed a small local LLM (a quantised 4B model, around 2.7GB) that runs on the device. Zero inference cost server-side, and it works offline.
• Offline-first mobile (SQLite on the device and a sync queue with retries) means the apps batch writes and tolerate dead zones instead of hammering the API with chatty per-keystroke requests.

The Rest of the Stack, Briefly

#

Auth is a hosted identity provider plus WebAuthn passkeys, and WebSocket connections are authenticated before the upgrade. Postgres in production, SQLite in development, and a couple of off-the-shelf Django packages for the admin and audit history. Mobile is React Native and TypeScript on Expo, with Expo Router and Tailwind, over-the-air updates from a self-hosted Expo Updates server, and a subscriptions service for in-app purchases. Quality gates: pytest and mypy on the backend, unit and end-to-end tests on the client.

What I’d Tell You to Steal

#

The transferable idea isn’t any single library — it’s the loop. Make one cheap machine genuinely fast (Rust hot paths, connection reuse, cache in front of the real bottleneck, offload what you can to CDN and client). Load-test until you find the exact concurrency where it breaks. Set the autoscaler’s soft limit at that knee, let idle machines stop, and keep the minimum fleet at one. Two production apps, real-time features, AI assistants — and the steady-state bill is a couple of the cheapest VMs on the platform.

The estimating app I’m building has one hard requirement: it has to keep working with no signal. Quotes get written up on site visits, and site visits happen in new builds and basements where reception is a rumour. That constraint forced me to get honest about a question most React projects fudge: which kind of state lives where?

The usual failure mode is one store for everything. API caching, user preferences, form state, auth tokens, all shoved into the same Redux blob. But those are different kinds of state with different lifecycles, and they want different tools. Zustand, React Query and AsyncStorage each solve exactly one of these problems well. Draw the boundaries between them and the architecture mostly sorts itself out.

Three kinds of state

#

Before arguing about libraries, it helps to name what you’re actually managing.

Client state exists only in your app’s runtime: UI toggles, the selected tab, form inputs, whether a modal is open. Nothing upstream owns it, and it dies with the process.

Server state is different. The server owns it; your app just holds a local copy. User profiles, product listings, notifications, feed data. Your copy goes stale, needs refetching, and other clients may be looking at a different version of it right now.

Persistent state is whatever has to survive a restart: auth tokens, the onboarding-complete flag, cached preferences, offline data. It lives on the device itself.

Most apps have all three. Most messes come from handling all three with one tool.

Zustand: client state

#

Zustand is a small state library for React with none of the ceremony. No providers, no boilerplate, no context wrappers stacked five deep. You create a store, you use it in components, and that’s the whole lesson.

It’s the right tool when several components need the same UI state (sidebar open or closed, active filters, selected items), when app-level state doesn’t come from an API (theme, language, feature flags read at startup), or when there’s real client-side logic to manage, like cart calculations or a multi-step wizard. Anything that should update synchronously and predictably.

A basic store:

import { create } from 'zustand'

interface AppState {
  theme: 'light' | 'dark'
  sidebarOpen: boolean
  selectedFilters: string[]
  setTheme: (theme: 'light' | 'dark') => void
  toggleSidebar: () => void
  setFilters: (filters: string[]) => void
}

const useAppStore = create<AppState>((set) => ({
  theme: 'light',
  sidebarOpen: false,
  selectedFilters: [],
  setTheme: (theme) => set({ theme }),
  toggleSidebar: () => set((state) => ({ sidebarOpen: !state.sidebarOpen })),
  setFilters: (filters) => set({ selectedFilters: filters }),
}))

And in a component:

function Sidebar() {
  const { sidebarOpen, toggleSidebar } = useAppStore()

  if (!sidebarOpen) return null

  return (
    <div className="sidebar">
      <button onClick={toggleSidebar}>Close</button>
      {/* sidebar content */}
    </div>
  )
}

What I don’t put in Zustand

#

API responses, mainly. I’ve seen projects where every API call writes into a Zustand store and components read from the store instead of querying. You end up reimplementing cache invalidation, loading states, error handling, refetching and pagination by hand, which is exactly the work React Query exists to do for you.

The other thing is anything that must survive a restart. Zustand state lives in memory; when the app closes, it’s gone. The persist middleware can bridge into AsyncStorage (more on that later), but that should be a deliberate decision rather than a habit.

React Query: server state

#

React Query (TanStack Query these days) manages the whole lifecycle of remote data: fetching, caching, synchronising, updating, garbage collecting. The mental shift is that it treats server data as a cache you keep fresh, not as state you own. The server owns it. You’re holding a copy.

I use it for anything that comes from an API. That covers data multiple components need from the same endpoint (requests get deduplicated automatically), paginated and infinite-scroll lists, data that should refetch in the background when someone returns to the app, and optimistic updates where the UI changes immediately and rolls back if the server says no.

import { useQuery, useMutation, useQueryClient } from '@tanstack/react-query'

function useUser(userId: string) {
  return useQuery({
    queryKey: ['user', userId],
    queryFn: () => fetch(`/api/users/${userId}`).then(res => res.json()),
    staleTime: 5 * 60 * 1000, // Consider fresh for 5 minutes
  })
}

function useUpdateUser() {
  const queryClient = useQueryClient()

  return useMutation({
    mutationFn: (data: { id: string; name: string }) =>
      fetch(`/api/users/${data.id}`, {
        method: 'PATCH',
        body: JSON.stringify(data),
      }),
    onSuccess: (_, variables) => {
      queryClient.invalidateQueries({ queryKey: ['user', variables.id] })
    },
  })
}

In a component:

function UserProfile({ userId }: { userId: string }) {
  const { data: user, isLoading, error } = useUser(userId)
  const updateUser = useUpdateUser()

  if (isLoading) return <Spinner />
  if (error) return <ErrorMessage error={error} />

  return (
    <div>
      <h1>{user.name}</h1>
      <button
        onClick={() => updateUser.mutate({ id: userId, name: 'New Name' })}
        disabled={updateUser.isPending}
      >
        Update Name
      </button>
    </div>
  )
}

What you get without writing it

#

The reason I’m firm about putting server data here is the sheer amount of machinery that comes free. Five components requesting the same user produces one network request. Data refetches when the window regains focus or the network reconnects. Old cache entries get garbage collected on their own. Every query hands you isLoading, isError, data and friends, so loading and error UI stops being bespoke work. Failed requests retry with exponential backoff. And optimistic updates with rollback are a built-in pattern rather than an afternoon of custom plumbing.

What it’s not for

#

Client-only state. If the data never touches a server, React Query just adds ceremony; a modal’s open state doesn’t need cache invalidation or a refetch interval.

It’s also not a persistence layer. The cache lives in memory, so a restart means an empty cache and a round of refetching. You can persist the cache (and on React Native you should, see the offline section below), but the source of truth is always the server.

AsyncStorage: what survives a restart

#

AsyncStorage is React Native’s key-value storage. On the web the equivalent is localStorage (synchronous) or IndexedDB (async, more capable). Same idea either way: data that outlives the process because it’s written to the device.

What it actually is on each platform

#

One API, three quite different backends, assuming you’re using @react-native-async-storage/async-storage.

On Android it’s SQLite under the hood, via RKStorage, in a database inside the app’s internal storage directory. Fast, reliable, and sandboxed so other apps can’t touch it. Mind the default size cap in the table below.

On iOS it’s NSUserDefaults for small values and serialised files for larger ones, again sandboxed inside the app container. Apple doesn’t set a hard limit on NSUserDefaults, but the sensible convention is to keep individual values under a few hundred KB. If your values are bigger than that, you probably want a proper database like WatermelonDB or Realm anyway.

On web (React Native Web or Expo Web) it falls back to localStorage, which caps out around 5-10 MB depending on the browser. Web-only React apps can use localStorage directly, or IndexedDB through a wrapper like idb-keyval for larger datasets.

What I keep in it

#

Auth tokens and session data, preferences that should stick around (language, theme, notification settings), the onboarding-completed flag, cached data for offline use. Small key-value things that need to survive a restart; that’s the whole category.

The API is about as plain as storage gets:

import AsyncStorage from '@react-native-async-storage/async-storage'

// Store a value
await AsyncStorage.setItem('auth_token', token)

// Read a value
const token = await AsyncStorage.getItem('auth_token')

// Store an object (must serialize)
await AsyncStorage.setItem('user_preferences', JSON.stringify({
  theme: 'dark',
  language: 'en',
  notifications: true,
}))

// Read an object
const prefs = JSON.parse(await AsyncStorage.getItem('user_preferences') ?? '{}')

// Remove a value
await AsyncStorage.removeItem('auth_token')

// Clear everything (careful with this)
await AsyncStorage.clear()

The web versions

#

Web-only React apps don’t need AsyncStorage at all. localStorage covers the simple cases:

// Synchronous  - blocks the main thread, but fine for small data
localStorage.setItem('theme', 'dark')
const theme = localStorage.getItem('theme')

// For structured data
localStorage.setItem('user', JSON.stringify({ name: 'Jared', role: 'admin' }))
const user = JSON.parse(localStorage.getItem('user') ?? '{}')

And IndexedDB covers the bigger ones:

import { get, set, del } from 'idb-keyval'

await set('large-dataset', hugeArray)
const data = await get('large-dataset')
await del('large-dataset')

What it’s not for

#

It’s a key-value store, not a database. Relational data, arrays with thousands of items, anything that needs indexing or querying: that’s SQLite (via expo-sqlite), WatermelonDB or Realm territory.

It’s also not secure storage. On a rooted or jailbroken device, AsyncStorage contents are readable. Sensitive tokens belong in expo-secure-store or react-native-keychain instead.

How the three fit together

#

Real apps use all three at once. A few patterns I lean on constantly.

The auth flow

#

// 1. AsyncStorage: persist the auth token
import AsyncStorage from '@react-native-async-storage/async-storage'

async function saveToken(token: string) {
  await AsyncStorage.setItem('auth_token', token)
}

async function getToken(): Promise<string | null> {
  return AsyncStorage.getItem('auth_token')
}

// 2. Zustand: track auth state in memory
import { create } from 'zustand'

interface AuthState {
  isAuthenticated: boolean
  token: string | null
  setAuth: (token: string) => void
  clearAuth: () => void
}

const useAuthStore = create<AuthState>((set) => ({
  isAuthenticated: false,
  token: null,
  setAuth: (token) => set({ isAuthenticated: true, token }),
  clearAuth: () => set({ isAuthenticated: false, token: null }),
}))

// 3. React Query: fetch user profile using the token
function useCurrentUser() {
  const token = useAuthStore((s) => s.token)

  return useQuery({
    queryKey: ['currentUser'],
    queryFn: () =>
      fetch('/api/me', {
        headers: { Authorization: `Bearer ${token}` },
      }).then(res => res.json()),
    enabled: !!token, // Only fetch when we have a token
  })
}

And on startup:

// App initialization
async function initializeApp() {
  const token = await getToken() // Read from AsyncStorage
  if (token) {
    useAuthStore.getState().setAuth(token) // Put in Zustand for quick access
    // React Query will automatically fetch the user profile
  }
}

AsyncStorage keeps the token across restarts, Zustand makes it cheap to read from anywhere, and React Query fetches the profile the moment a token exists. Each tool does the one thing it’s good at.

A theme that survives restarts

#

import { create } from 'zustand'
import { persist, createJSONStorage } from 'zustand/middleware'
import AsyncStorage from '@react-native-async-storage/async-storage'

// Zustand with persistence middleware bridges the gap
const useThemeStore = create(
  persist(
    (set) => ({
      theme: 'light' as 'light' | 'dark',
      toggleTheme: () =>
        set((state) => ({
          theme: state.theme === 'light' ? 'dark' : 'light',
        })),
    }),
    {
      name: 'theme-storage',
      storage: createJSONStorage(() => AsyncStorage), // React Native
      // storage: createJSONStorage(() => localStorage), // Web
    }
  )
)

Zustand manages the runtime state and the persist middleware syncs it to AsyncStorage (or localStorage on web) behind the scenes. Your components neither know nor care that a persistence layer exists.

Offline-first reads

#

import { useQuery } from '@tanstack/react-query'
import AsyncStorage from '@react-native-async-storage/async-storage'

function useProducts() {
  return useQuery({
    queryKey: ['products'],
    queryFn: async () => {
      try {
        const res = await fetch('/api/products')
        const data = await res.json()

        // Cache in AsyncStorage for offline use
        await AsyncStorage.setItem('cached_products', JSON.stringify(data))

        return data
      } catch (error) {
        // Network failed  - try cached data
        const cached = await AsyncStorage.getItem('cached_products')
        if (cached) return JSON.parse(cached)
        throw error
      }
    },
    staleTime: 10 * 60 * 1000,
  })
}

React Query does the fetching and in-memory caching; AsyncStorage is the fallback when the network isn’t there.

Offline mode, step by step

#

This is the part I actually care about. The general wisdom is that people open apps on the subway, in lifts, on planes, and that’s true. In my case the requirement is blunter: quotes get written up on site, and I can’t assume the site has reception. A spinner in a basement is a lost quote.

The nice surprise is that these three libraries get you a genuinely solid offline setup without reaching for a heavy framework.

Step 1: know when you’re offline

#

The device needs to tell the app about the network, and the app needs somewhere to keep that answer. @react-native-community/netinfo provides the events; a tiny Zustand store makes the state readable everywhere.

import { create } from 'zustand'
import NetInfo from '@react-native-community/netinfo'

interface NetworkState {
  isOnline: boolean
  setOnline: (online: boolean) => void
}

const useNetworkStore = create<NetworkState>((set) => ({
  isOnline: true,
  setOnline: (online) => set({ isOnline: online }),
}))

// Subscribe once at app startup
NetInfo.addEventListener((state) => {
  useNetworkStore.getState().setOnline(state.isConnected ?? false)
})

Now any component can check useNetworkStore((s) => s.isOnline) to show an offline banner, disable a submit button, or explain that changes will sync later.

Step 2: tell React Query about offline

#

React Query has offline behaviour built in through networkMode, which controls what queries and mutations do when the network is out.

import { QueryClient } from '@tanstack/react-query'

const queryClient = new QueryClient({
  defaultOptions: {
    queries: {
      networkMode: 'offlineFirst',
      // Return cached data immediately, then refetch in background when online
      staleTime: 5 * 60 * 1000,
      gcTime: 24 * 60 * 60 * 1000, // Keep cache for 24 hours
      retry: (failureCount, error) => {
        // Don't retry if we're offline  - it'll just fail again
        if (!useNetworkStore.getState().isOnline) return false
        return failureCount < 3
      },
    },
    mutations: {
      networkMode: 'offlineFirst',
    },
  },
})

The three modes:

For most mobile apps offlineFirst is the one you want: cached data appears immediately, and fresh data arrives when the network allows.

Step 3: persist the query cache

#

Out of the box the cache is memory-only, so a restart means spinners everywhere. For offline mode you persist it to AsyncStorage:

import { QueryClient } from '@tanstack/react-query'
import { PersistQueryClientProvider } from '@tanstack/react-query-persist-client'
import { createAsyncStoragePersister } from '@tanstack/query-async-storage-persister'
import AsyncStorage from '@react-native-async-storage/async-storage'

const queryClient = new QueryClient({
  defaultOptions: {
    queries: {
      gcTime: 24 * 60 * 60 * 1000, // 24 hours  - must be >= maxAge
    },
  },
})

const asyncStoragePersister = createAsyncStoragePersister({
  storage: AsyncStorage,
  key: 'react-query-cache',
})

// In your App component
function App() {
  return (
    <PersistQueryClientProvider
      client={queryClient}
      persistOptions={{
        persister: asyncStoragePersister,
        maxAge: 24 * 60 * 60 * 1000, // Don't restore data older than 24 hours
        dehydrateOptions: {
          shouldDehydrateQuery: (query) => {
            // Only persist successful queries
            return query.state.status === 'success'
          },
        },
      }}
    >
      <YourApp />
    </PersistQueryClientProvider>
  )
}

Now someone opening the app in a dead zone sees the last-fetched data straight away instead of a blank screen, and React Query quietly refetches once the network returns.

Step 4: queue writes while offline

#

Reads are the easy half. Writes are where offline gets interesting: someone adds a comment, submits an order or edits their profile with no network, and that change has to be queued and replayed later.

React Query’s answer is useMutation with an optimistic update in onMutate:

import { useMutation, useQueryClient } from '@tanstack/react-query'
import AsyncStorage from '@react-native-async-storage/async-storage'

interface Comment {
  id: string
  text: string
  postId: string
  createdAt: string
  pending?: boolean
}

function useAddComment(postId: string) {
  const queryClient = useQueryClient()

  return useMutation({
    mutationFn: async (text: string) => {
      const res = await fetch(`/api/posts/${postId}/comments`, {
        method: 'POST',
        headers: { 'Content-Type': 'application/json' },
        body: JSON.stringify({ text }),
      })
      return res.json()
    },

    // Optimistic update  - show the comment immediately
    onMutate: async (text) => {
      await queryClient.cancelQueries({ queryKey: ['comments', postId] })

      const previous = queryClient.getQueryData<Comment[]>(['comments', postId])

      const optimisticComment: Comment = {
        id: `temp-${Date.now()}`,
        text,
        postId,
        createdAt: new Date().toISOString(),
        pending: true, // Show a "sending..." indicator in the UI
      }

      queryClient.setQueryData<Comment[]>(
        ['comments', postId],
        (old) => [...(old ?? []), optimisticComment]
      )

      return { previous }
    },

    // Roll back on failure
    onError: (err, text, context) => {
      if (context?.previous) {
        queryClient.setQueryData(['comments', postId], context.previous)
      }
    },

    // Refetch to get the real data from the server
    onSettled: () => {
      queryClient.invalidateQueries({ queryKey: ['comments', postId] })
    },
  })
}

With networkMode: 'offlineFirst' on mutations, the mutationFn pauses until the network is back. The optimistic update puts the comment on screen immediately, and the real API call fires on reconnect.

If you need a more serious queue, say dozens of pending changes that have to replay in order, you can persist the mutation queue as well:

import { MutationCache } from '@tanstack/react-query'

// Save pending mutations to AsyncStorage
const mutationCache = new MutationCache({
  onError: async (error, variables, context, mutation) => {
    // Log failed mutations for debugging
    const pending = JSON.parse(
      await AsyncStorage.getItem('pending_mutations') ?? '[]'
    )
    pending.push({
      key: mutation.options.mutationKey,
      variables,
      timestamp: Date.now(),
    })
    await AsyncStorage.setItem('pending_mutations', JSON.stringify(pending))
  },
})

Step 5: reconcile when the network returns

#

React Query does most of the reconnection work itself: paused mutations resume, stale queries refetch. You just have to wire its managers up to React Native’s events.

import NetInfo from '@react-native-community/netinfo'
import { onlineManager, focusManager } from '@tanstack/react-query'
import { AppState } from 'react-native'

// Tell React Query about network state changes
onlineManager.setEventListener((setOnline) => {
  return NetInfo.addEventListener((state) => {
    setOnline(!!state.isConnected)
  })
})

// Refetch when the app comes back to foreground
focusManager.setEventListener((setFocused) => {
  const subscription = AppState.addEventListener('change', (status) => {
    setFocused(status === 'active')
  })
  return () => subscription.remove()
})

When someone backgrounds the app for an hour and comes back, focusManager triggers refetches so they see fresh data. When they walk out of a tunnel and regain signal, onlineManager resumes whatever was paused.

Step 6: say so in the UI

#

Whatever else you do, don’t swallow network errors silently. Tell people they’re offline and show which changes are still pending.

import { useNetworkStore } from './stores/network'

function OfflineBanner() {
  const isOnline = useNetworkStore((s) => s.isOnline)

  if (isOnline) return null

  return (
    <View style={styles.banner}>
      <Text>You're offline. Changes will sync when you reconnect.</Text>
    </View>
  )
}

function CommentItem({ comment }: { comment: Comment }) {
  return (
    <View style={[styles.comment, comment.pending && styles.pending]}>
      <Text>{comment.text}</Text>
      {comment.pending && (
        <Text style={styles.pendingLabel}>Sending...</Text>
      )}
    </View>
  )
}

The whole picture

#

┌─────────────────────────────────────────────────┐
│                   Components                     │
│  useQuery() for reads    useMutation() for writes│
└──────────┬──────────────────────┬────────────────┘
           │                      │
     ┌─────▼──────┐        ┌─────▼──────┐
     │ React Query │        │ React Query │
     │   Cache     │        │  Mutation   │
     │ (in-memory) │        │   Queue     │
     └─────┬──────┘        └─────┬──────┘
           │                      │
     ┌─────▼──────────────────────▼──────┐
     │     AsyncStorage Persister         │
     │  (survives app restart)            │
     └─────┬──────────────────────┬──────┘
           │                      │
     ┌─────▼──────┐        ┌─────▼──────┐
     │   Zustand   │        │   Network  │
     │ (isOnline,  │◄───────│   NetInfo  │
     │  UI state)  │        │            │
     └────────────┘        └────────────┘

When you need a real sync engine

#

All of the above assumes the server is the source of truth and offline is temporary. If you need true offline-first with conflict resolution (think a notes app where two devices edit the same document offline), this stack won’t cut it and you want a proper sync engine. WatermelonDB is built for React Native, runs on SQLite and ships a sync protocol for resolving conflicts. Realm with Atlas Device Sync is a full offline-first database with automatic conflict resolution through MongoDB Atlas. PowerSync is a SQLite-based sync layer that works with your existing Postgres backend. And if you need total control, Expo SQLite plus your own sync logic is always an option.

Zustand + React Query + AsyncStorage covers about 80% of mobile apps. The other 20%, the collaborative editors and multi-device offline-heavy workflows, need a dedicated sync database.

Deciding where a piece of state goes

#

When I’m not sure where something belongs, I run it through these questions:

Common cases

#

Setup by platform

#

React Native (Android + iOS)

#

All three at once:

npm install zustand @tanstack/react-query @react-native-async-storage/async-storage

Plus secure storage for anything sensitive:

npx expo install expo-secure-store
# or
npm install react-native-keychain

Expo

#

AsyncStorage works out of the box with Expo, no native linking required.

npx expo install @react-native-async-storage/async-storage

Web-only React

#

Skip AsyncStorage entirely and use localStorage or IndexedDB directly.

npm install zustand @tanstack/react-query
# Optional for IndexedDB
npm install idb-keyval

Zustand’s persist middleware already defaults to localStorage on web:

persist(storeConfig, {
  name: 'my-store',
  // localStorage is the default on web  - no extra config needed
})

Mistakes I keep seeing

#

Putting API data in Zustand. If there’s a setUsers(apiResponse.users) in one of your actions, stop. That’s React Query’s job, and you’re one sprint away from reinventing cache invalidation badly.

Using React Query for client state. If the queryFn doesn’t make a network request, wrong tool.

Treating AsyncStorage like a database. Serialising an array of 10,000 items into a key-value store ends in tears. Use SQLite or a real database.

Leaving tokens unencrypted. AsyncStorage is not secure storage. Auth tokens, API keys and credentials go in expo-secure-store or the platform keychain.

Hand-rolling persistence. If you’re reading AsyncStorage on mount and writing on every state change, Zustand’s persist middleware already does that, plus hydration and serialisation, with less code and fewer bugs.

Server data goes through React Query. Client state lives in Zustand. Anything that must survive a restart goes to AsyncStorage, and anything sensitive goes to secure storage.

The worst architectures I’ve worked on all had one giant store that everything flowed through. The best ones had clear lines between server state, client state and what needs to persist. Draw those lines early and everything after gets easier.

Update, 13 September 2026. Since this went up, Codemagic have released Patch, an open-source, self-hostable OTA server in the CodePush mould. If you got here because you’d rather not build the server yourself, it’s now a real option: I’ve since moved an app from the setup below to Patch and written up how that went, and there’s a short note on it under Wrapping up. Disclosure: Codemagic paid me to do that migration and write about it, and asked for a mention here. The code in the rest of this post has since been trimmed to short examples.

My estimating app gets its bug fixes pushed over the air: JavaScript changes land on users’ phones without going anywhere near App Store review. Expo’s hosted service, EAS Update, will happily do this for you. I built my own version anyway, with Django REST Framework and Tigris S3, and it has served every iOS and Android update since.

Expo OTA updates work like this: the app’s expo-updates library asks an update server for a manifest, sending its platform and runtime version in headers. The server answers with the newest compatible update and the URLs of its JavaScript bundle and assets, and the app downloads them and runs the new bundle on its next launch. EAS Update is Expo’s hosted server for that protocol, but any server that speaks it will do, and that’s what I built.

This post walks through the whole thing: the models, the manifest endpoint, the publishing pipeline, and the gotchas that only show up once you’re running it yourself.

Why self-host at all?

#

Mostly cost, in my case. EAS Update pricing scales with usage, so pushing frequent updates to a growing user base adds up, while S3-compatible storage like Tigris stays nearly free. There are other reasons that might weigh more for you: some industries have to keep every application asset inside their own infrastructure, and owning the server means owning the update flow, so you can roll out to specific user segments or A/B test bundles if your product calls for it. It also means nothing about your update pipeline depends on Expo’s availability or their next pricing change.

How the pieces fit

#

Four components: a Django backend that serves update manifests and stores metadata, Tigris holding the actual bundle and asset files, a publishing script that exports, uploads and registers updates, and the mobile app itself pointed at my server instead of Expo’s.

┌─────────────────┐
│   Mobile App    │
│  (expo-updates) │
└────────┬────────┘
         │ 1. Request manifest
         │    (with headers: platform, runtime-version)
         ↓
┌─────────────────┐
│  Django Server  │
│  /api/expo-     │◄─── 2. Query DB for latest update
│   updates/      │
│   manifest/     │
└────────┬────────┘
         │ 3. Generate presigned URLs
         │
         ↓
┌─────────────────┐
│   Tigris S3     │
│  (Asset Files)  │◄─── 4. App downloads bundles directly
└─────────────────┘

Building it

#

The two models

#

Everything hangs off ExpoUpdate and ExpoUpdateAsset. The first stores metadata for each update:

class ExpoUpdate(models.Model):
    id = models.UUIDField(primary_key=True, default=uuid.uuid4)
    runtime_version = models.CharField(max_length=50, db_index=True)
    platform = models.CharField(
        max_length=10,
        choices=[("ios", "iOS"), ("android", "Android")],
        db_index=True
    )
    is_active = models.BooleanField(default=True, db_index=True)
    manifest_data = models.JSONField()
    description = models.TextField(blank=True)
    created_at = models.DateTimeField(auto_now_add=True)

    class Meta:
        indexes = [
            models.Index(fields=["runtime_version", "platform", "is_active", "-created_at"])
        ]

A few deliberate choices in there. runtime_version mirrors the runtimeVersion in app.json, and it does a lot of work: clients only ever download updates matching their own runtime version. iOS and Android get separate rows because the bundles differ. is_active is the rollback mechanism; deactivate a bad update and clients fall back to the previous one. And manifest_data holds the complete Expo Updates v1 protocol manifest as JSON, so serving it later is just a lookup.

ExpoUpdateAsset tracks the individual files:

class ExpoUpdateAsset(models.Model):
    id = models.UUIDField(primary_key=True, default=uuid.uuid4)
    update = models.ForeignKey(ExpoUpdate, on_delete=models.CASCADE, related_name="assets")
    hash = models.CharField(max_length=255, db_index=True)
    key = models.CharField(max_length=255)
    content_type = models.CharField(max_length=100)
    file_extension = models.CharField(max_length=10)
    file_path = models.CharField(max_length=500)
    file_size = models.IntegerField(default=0)

Assets are keyed by SHA-256 hash, which makes them immutable and cacheable. The same asset can be shared across any number of updates.

The manifest endpoint

#

/api/expo-updates/manifest/ is where app and server actually talk. It implements the Expo Updates v1 protocol:

@action(detail=False, methods=["get"], url_path="manifest")
def manifest(self, request):
    # Extract required headers
    protocol_version = request.META.get("HTTP_EXPO_PROTOCOL_VERSION")
    platform = request.META.get("HTTP_EXPO_PLATFORM")
    runtime_version = request.META.get("HTTP_EXPO_RUNTIME_VERSION")

    # Validate protocol version
    if protocol_version != "1":
        return Response(
            {"error": f"Unsupported protocol version: {protocol_version}"},
            status=400
        )

    # Find latest active update for this runtime + platform
    update = ExpoUpdate.objects.filter(
        runtime_version=runtime_version,
        platform=platform,
        is_active=True,
    ).order_by("-created_at").first()

    # No update available - client uses embedded bundle
    if not update:
        response = Response(status=204)
        response["expo-protocol-version"] = "1"
        return response

    # Generate presigned URLs for all assets
    manifest_data = build_manifest(update)  # fills in presigned asset URLs

    return Response(manifest_data, status=200)

Three details matter here. A 204 response means “no update”, and the app carries on with its embedded bundle. The asset URLs in the manifest are presigned, so the app downloads straight from Tigris’s CDN rather than streaming everything through Django. And the expo-protocol-version response header isn’t optional; the client checks for it.

Publishing updates

#

Publishing is a Django management command with a shell script wrapped around it. The command reads the output of expo export, hashes every asset with SHA-256, uploads bundles and assets to Tigris in parallel, writes the database records, and optionally drops an import JSON into the bucket for production sync. Cut down to the part that matters:

from concurrent.futures import ThreadPoolExecutor

from django.db import transaction


def publish(platform, runtime_version, manifest, files):
    # Upload the bundle and assets in parallel (keys are content hashes)
    with ThreadPoolExecutor(max_workers=10) as executor:
        list(executor.map(upload_asset, files))

    # Swap the active update in one transaction
    with transaction.atomic():
        ExpoUpdate.objects.filter(
            runtime_version=runtime_version,
            platform=platform,
            is_active=True,
        ).update(is_active=False)

        return ExpoUpdate.objects.create(
            runtime_version=runtime_version,
            platform=platform,
            manifest_data=manifest,
        )

The wrapper script is what I actually type:

# Publish to local environment
./publish.sh ios

# Publish to production
./publish.sh ios --production

# Dry run to validate
./publish.sh --dry-run

It loads production environment variables before exporting, handles one platform or both, and takes care of the production sync described next.

Syncing to production

#

I didn’t want production credentials sitting on my laptop, so production deploys happen in two steps: publish locally (assets go to Tigris, along with a JSON snapshot of the metadata), then call a production API endpoint with the S3 path so the server imports the metadata itself:

@action(detail=False, methods=["post"], url_path="import-update")
def import_update(self, request):
    # Authenticate via Bearer token
    token = request.META.get("HTTP_AUTHORIZATION", "").removeprefix("Bearer ")
    if not hmac.compare_digest(token, settings.IMPORT_TOKEN):
        return Response({"error": "Invalid token"}, status=401)

    # Download the import JSON from the bucket
    s3_key = request.data.get("s3_key")
    obj = s3_client.get_object(Bucket=bucket_name, Key=s3_key)
    data = json.loads(obj["Body"].read())

    # Import into the production database
    with transaction.atomic():
        update, _ = ExpoUpdate.objects.update_or_create(id=data["id"], defaults={...})
        for asset_data in data["assets"]:
            ExpoUpdateAsset.objects.update_or_create(...)

    # Clean up the import JSON
    s3_client.delete_object(Bucket=bucket_name, Key=s3_key)
    return Response({"update_id": str(update.id)})

Pointing the app at your server

#

In app.json, set the update URL and runtime version:

{
  "expo": {
    "runtimeVersion": "1.0.0",
    "updates": {
      "url": "https://api.example.com/api/expo-updates/manifest/"
    }
  }
}

Be strict about keeping runtimeVersion matched between app and server. When you change native code or upgrade the Expo SDK, bump the runtime version and publish fresh updates for it.

Storage on Tigris

#

Tigris is S3-compatible object storage, considerably cheaper than actual AWS S3, and it includes global edge caching, which matters when users are pulling down multi-megabyte bundles. The Django configuration:

# settings.py
BUCKET_NAME = os.getenv("BUCKET_NAME")
AWS_ENDPOINT_URL_S3 = os.getenv("AWS_ENDPOINT_URL_S3")
AWS_ACCESS_KEY_ID = os.getenv("AWS_ACCESS_KEY_ID")
AWS_SECRET_ACCESS_KEY = os.getenv("AWS_SECRET_ACCESS_KEY")
AWS_REGION = os.getenv("AWS_REGION", "auto")

Creating the client is standard boto3:

import boto3

def create_s3_client(endpoint_url, region, access_key, secret_key):
    return boto3.client(
        "s3",
        endpoint_url=endpoint_url,
        region_name=region,
        aws_access_key_id=access_key,
        aws_secret_access_key=secret_key,
    )

Downloads go through presigned URLs, so the app fetches straight from the CDN:

presigned_url = s3_client.generate_presigned_url(
    "get_object",
    Params={"Bucket": bucket_name, "Key": asset.file_path},
    ExpiresIn=3600,  # 1 hour
)

Security

#

The manifest endpoint is unauthenticated on purpose, because apps need updates before anyone has logged in. The import endpoint is a different story. It can write to the production database, so it wants a shared secret, compared in constant time to rule out timing attacks:

IMPORT_TOKEN = os.getenv("IMPORT_TOKEN")

# Constant-time comparison prevents timing attacks
if not hmac.compare_digest(token, IMPORT_TOKEN):
    return Response({"error": "Invalid token"}, status=401)

Asset integrity comes free with the design: everything is verified against its SHA-256 hash, so a tampered asset simply fails. And presigned URLs expire after an hour, which keeps bundles from being hotlinked indefinitely.

Keeping it fast

#

The composite index on (runtime_version, platform, is_active, -created_at) keeps the manifest query quick no matter how many updates accumulate:

class Meta:
    indexes = [
        models.Index(fields=["runtime_version", "platform", "is_active", "-created_at"])
    ]

The publishing script uploads assets concurrently:

with ThreadPoolExecutor(max_workers=10) as executor:
    futures = {executor.submit(upload_asset, asset): asset for asset in assets}
    for future in as_completed(futures):
        # Track progress

For a typical update with 50 or so assets, that took publishing from about 2 minutes down to 15 seconds. Distribution needs no work at all: Tigris caches assets at edge locations near your users on its own.

Day-to-day workflow

#

During development

#

# 1. Make code changes in the app
git commit -am "Fix bug"

# 2. Publish OTA update to local environment
./publish.sh ios

# 3. Test on device
# App automatically downloads and applies update

Shipping to production

#

# 1. Publish to production
./publish.sh ios --production

# 2. Monitor
# Check Django admin for update records
# Verify assets in Tigris dashboard

Rolling back

#

# Mark problematic update as inactive in Django admin
# or via management shell:
python manage.py shell

>>> from updates.models import ExpoUpdate
>>> bad_update = ExpoUpdate.objects.get(id="uuid-here")
>>> bad_update.is_active = False
>>> bad_update.save()

# Clients will now receive the previous active update

What it costs

#

The app has around 500 active users. On Tigris, the ~200 MB of historical updates costs about $0.02/month, and the ~50 GB of monthly egress from update downloads about $1.00. The Django side rides along on the server my API already runs on, so its incremental cost is zero. Call it a dollar a month, all in. EAS Update at similar usage would land somewhere around $300–500/year, so this paid for itself more or less immediately.

Monitoring and debugging

#

Nothing fancy here. The viewset logs every manifest request:

logger.info(f"Manifest request: platform={platform}, runtime={runtime_version}")

The Django admin is the de facto dashboard. Register the models and you can inspect and filter everything:

@admin.register(ExpoUpdate)
class ExpoUpdateAdmin(admin.ModelAdmin):
    list_display = ["platform", "runtime_version", "is_active", "created_at"]
    list_filter = ["platform", "is_active", "runtime_version"]
    search_fields = ["description"]

And on the client, expo-updates will tell you what it sees:

import * as Updates from 'expo-updates';

Updates.checkForUpdateAsync().then(update => {
  console.log('Update available:', update.isAvailable);
  console.log('Manifest:', update.manifest);
});

Gotchas

#

Runtime version mismatches

#

The most common failure is also the quietest: clients only download updates matching their runtime version. If the installed app is on runtime 1.0.0 and you publish for 1.0.1, nothing arrives and nothing errors. Keep the runtime version in sync with your builds, and only bump it when native code changes.

The createdAt timestamp

#

The expo-updates client compares the manifest’s createdAt against the embedded bundle’s commitTime, and only applies an update whose createdAt is newer. The viewset overrides it with the database timestamp:

manifest_data["createdAt"] = update.created_at.strftime("%Y-%m-%dT%H:%M:%S.%fZ")

For local development, if you build a binary after publishing an OTA, republish the OTA so its timestamp wins.

Manifest asset key values

#

The key field on each manifest asset is what expo-updates uses for caching, and it must be a deterministic hash (the MD5 of the filename works), not a random UUID or arbitrary string. Get this wrong and the client may fail to cache or retrieve assets correctly, and updates silently break after the first successful load. Thanks to reader Raphael Mutschler for flagging this one; he had updates working exactly once before discovering the issue.

App config in expoClient

#

If your app uses Linking, Constants, or anything else that reads the app config at runtime, the manifest’s extra.expoClient field needs to include your app config. Without it the app may launch fine at first, then crash or refuse to reopen after being closed. That’s because expo-updates replaces the embedded manifest with the OTA one, and if expoClient is missing, those APIs lose the config they depend on. Raphael Mutschler caught this one too.

Asset cleanup

#

Old updates pile up in Tigris. For now, cleanup at my end is manual:

# Delete updates older than 30 days
from datetime import timedelta
from django.utils import timezone

cutoff = timezone.now() - timedelta(days=30)
old_updates = ExpoUpdate.objects.filter(created_at__lt=cutoff, is_active=False)

for update in old_updates:
    # Delete assets from S3
    for asset in update.assets.all():
        s3_client.delete_object(Bucket=bucket_name, Key=asset.file_path)
    # Delete DB records
    update.delete()

A scheduled task is the obvious home for this. I haven’t got round to it.

Things I’d add next

#

(September 2026: Patch now does all three of these and more; see the migration write-up.)

Gradual rollouts

#

A rollout_percentage field would let an update go out to a slice of users first:

rollout_percentage = models.IntegerField(default=100)

# In the manifest view:
if update.rollout_percentage < 100:
    # Hash user ID and check if they're in rollout group
    user_hash = int(hashlib.sha256(user_id.encode()).hexdigest(), 16)
    if (user_hash % 100) >= update.rollout_percentage:
        return Response(status=204)  # No update

Separate staging updates

#

An environment field would let staging builds receive different updates from production:

environment = models.CharField(max_length=20, default="production")

# Client sends environment in custom header
environment = request.META.get("HTTP_X_UPDATE_ENVIRONMENT", "production")
update = ExpoUpdate.objects.filter(environment=environment, ...).first()

Download analytics

#

And a small model would answer the “did anyone actually get this?” question properly:

class ExpoUpdateDownload(models.Model):
    update = models.ForeignKey(ExpoUpdate, on_delete=models.CASCADE)
    user_id = models.CharField(max_length=255, null=True)
    platform = models.CharField(max_length=10)
    downloaded_at = models.DateTimeField(auto_now_add=True)

Wrapping up

#

The whole system is two Django models, a couple of views, a publishing script and a dollar a month in infrastructure. That’s less than I expected going in, and it has quietly done its job for the app on both platforms since.

The snippets above are simplified examples of what runs in that production app, so adapt them to whatever you’re building. Reader Raphael Mutschler also published a standalone implementation at expo-ota-server if you’d rather start from something that already runs.

If you’d rather not write the server

#

Two things have appeared since I wrote this that cover the same ground without the Django. The first is Raphael’s expo-ota-server above. The second is Codemagic Patch: open source, runs on your own box from a Docker Compose install, and speaks CodePush rather than the Expo Updates protocol, so you swap expo-updates for its SDK and eas update for cmpatch release-react. It can use S3-compatible storage, so a Tigris bucket like the one in this post works too.

Out of the box it has the things I listed under “Things I’d add next” and never built: staging and production deployments, percentage rollouts, per-release download and install numbers, mandatory updates, binary patches, code signing, and a rollback on the device when a new bundle crashes before the app reports ready. The trade is a new native binary before anyone gets a Patch update, since the SDK has to be in the app, and a server to run instead of a management command.

I moved one of my own apps, with its committed native projects and real release pipeline, from the setup in this post to Patch. The write-up covers what worked, what didn’t, and how hard it really is. Codemagic sponsored that work and had a factual review pass on it, not an editorial one.

For the full spec, see the Expo Updates Protocol Specification.

This post walks through how I built it with llama.rn (React Native bindings for llama.cpp) and Qwen 1.7B, a small model that turned out to be more capable than I expected.

The short version: install llama.rn, download a 4-bit GGUF model (about 1 GB) the first time the app opens, load it with initLlama, and stream completions from it. On an iPhone 14 Pro that runs at about 15 tokens a second with no network at all. The rest of this post is what it took to make that useful: picking the model, getting tool calls out of it reliably, and keeping memory in check.

📝 Update: This implementation now uses Qwen3.5-2B-GGUF instead of the original Qwen3-1.7B. I’ve also stopped using the /no_think message hack and now disable thinking mode properly via the chat_template_kwargs parameter:

chat_response = client.chat.completions.create(
    model="Qwen/Qwen3.5-27B",
    messages=messages,
    max_tokens=32768,
    temperature=0.7,
    top_p=0.8,
    presence_penalty=1.5,
    extra_body={
        "top_k": 20,
        "chat_template_kwargs": {"enable_thinking": False},
    },
)

Much cleaner than smuggling control flags into the prompt.

Why I didn’t want cloud AI for this

#

The standard recipe for an AI feature in a mobile app is: user types a message, app sends it to a cloud model provider, response comes back, bill ticks up.

For an app that tradespeople use to run their business, that recipe has problems. Customer names, addresses, and phone numbers go to a third party. Project details, pricing, and notes sit on someone else’s servers. There are GDPR and data-residency questions I’d rather not spend time on. And the API bill grows with every user.

The alternative is to run the model on the user’s phone, so that’s what I did.

Why this is finally practical

#

A year earlier I wouldn’t have bothered, but three things changed roughly at once. Quantised models got small — Qwen 1.7B at Q4_K_M is 1.1 GB, smaller than most games, and it downloads once and lives in app storage. Mobile GPUs got fast — llama.cpp uses Metal on iOS and OpenCL on recent Qualcomm Android GPUs, and on an iPhone 14 Pro I see about 15 tokens a second, plenty for real-time streaming. And small models got genuinely useful: Qwen 1.7B follows structured instructions, parses JSON, and chains multi-step reasoning. That’s exactly the profile business logic needs. Nobody’s asking it to write poetry.

Picking a model

#

I tested four small models before settling on Qwen3-1.7B.

TinyLlama 1.1B (637 MB) was the fastest, but it kept hallucinating customer IDs and dropping required fields — structured output just wasn’t reliable. Phi-3-mini (1.8 GB) reasons well but wouldn’t shut up; simple queries came back as 200-word essays when I needed 20. Gemma-2B (1.2 GB) was quick and accurate at classification but weak at function calling — it couldn’t consistently produce the <action> tags my tool system needs. Qwen3-1.7B (1.1 GB) hit the sweet spot: reliable structured output, precise instruction following, and chain-of-thought support via <think> tags.

Q4_K_M is one of llama.cpp’s “k-quant” formats: weights are stored as 4-bit values in blocks that each carry their own scale, and the M mix keeps a few sensitive tensors at 6-bit. That makes the file about 70% smaller than 16-bit weights for a small quality loss.

Setting up llama.rn

#

llama.rn wraps llama.cpp for React Native, and installation is the easy part:

npm install llama.rn
cd ios && pod install

On Expo, llama.rn is a native module, so it won’t run in Expo Go. Add its config plugin and use a development build:

// app.config.js
export default {
  expo: {
    plugins: ["llama.rn"],
  },
};

npx expo install llama.rn
npx expo run:ios

For production builds, the plugin also adds the two iOS entitlements llama.rn recommends for big models (extended virtual addressing and the increased memory limit).

The model itself downloads on first use:

const MODEL_URL = "https://huggingface.co/unsloth/Qwen3-1.7B-GGUF/resolve/main/Qwen3-1.7B-Q4_K_M.gguf";
const MODEL_PATH = FileSystem.documentDirectory + "llama-models/Qwen3-1.7B-Q4_K_M.gguf";

const downloadModel = async () => {
  const downloadResumable = FileSystem.createDownloadResumable(
    MODEL_URL,
    MODEL_PATH,
    {},
    (progress) => {
      const pct = progress.totalBytesWritten / progress.totalBytesExpectedToWrite;
      setDownloadProgress(pct);
    }
  );

  await downloadResumable.downloadAsync();
};

On WiFi that’s 2-3 minutes; on LTE more like 5-8. Once it’s on disk, load it with GPU acceleration:

const ctx = await initLlama({
  model: MODEL_PATH,
  n_ctx: 8192,      // 8K context window
  n_gpu_layers: 99, // Use GPU for all layers
});

Don’t skip n_gpu_layers: 99. Offloading everything to Metal or OpenCL instead of the CPU is worth about a 5x speedup.

Streaming inference

#

Nobody wants to stare at a spinner for ten seconds, so responses stream token by token:

let fullResponse = "";

await llamaContext.completion(
  {
    messages: [
      { role: "system", content: systemPrompt },
      { role: "user", content: "Create a job for John Smith" }
    ],
    n_predict: 512,
    temperature: 0.7,
    top_p: 0.8,
  },
  (data) => {
    // Called for each token
    fullResponse += data.token;
    setStreamingText(fullResponse);
  }
);

On modern phones the first token lands in about 200ms (that’s prompt processing) and each token after that takes 50-80ms. Compared to anything network-bound, it feels instant.

Function calling that small models can actually do

#

The function calling that big cloud models offer leans on JSON schemas, and small models fall over on it — malformed JSON, missing fields, the lot. So I use a simpler XML-based protocol:

const systemPrompt = `You are an AI assistant. When you need to take an action, output:
<action>{"type":"search_customers","query":"Smith"}</action>

Available actions:
- search_customers: {"type":"search_customers","query":"John"}
- create_job: {"type":"create_job","customer_id":5,"alias":"Living Room"}
- update_job: {"type":"update_job","job_id":"J-0042","status":"quoting"}

Rules:
1. ONE action tag per response, at the very end
2. When searching/creating → end with <action>
3. When just chatting → no action tag
`;

XML tags won for boring reasons: <action> and </action> are unambiguous delimiters, a regex is all the parsing you need, the examples in the prompt double as documentation, and it still works when the model wraps the tag in extra chatter.

Parsing is exactly as dumb as it looks:

function parseAction(text: string): Action | null {
  const match = text.match(/<action>([\s\S]*?)<\/action>/);
  if (!match) return null;

  try {
    return JSON.parse(match[1].trim());
  } catch {
    return null;
  }
}

When an action comes out, I execute it and feed the result back into the conversation:

const action = parseAction(modelResponse);
if (action) {
  const result = await executeAction(action);

  // Inject result as a system message
  conversationHistory.push({
    role: "user",
    content: `[TOOL_RESULT] ${result.message}\n\n→ NEXT: Tell the user what happened.`
  });

  // Continue generation
  await generateNextResponse();
}

That’s enough for proper multi-step workflows:

User: “Create a job for Smith”

1. Model: <action>{"type":"search_customers","query":"Smith"}</action>
2. System: [TOOL_RESULT] Found: Jane Smith (id:42), Bob Smith (id:89)
3. Model: “I found two Smiths—Jane and Bob. Which one?”
4. User: “Jane”
5. Model: <action>{"type":"create_job","customer_id":42}</action>
6. System: [TOOL_RESULT] Job J-0073 created
7. Model: “Done! Created job J-0073 for Jane Smith.”

Chain-of-thought with <think> tags

#

Small models do noticeably better when you make them reason out loud. Qwen has a thinking mode built in:

const systemPrompt = `Before each response, wrap your reasoning in <think> tags:

<think>
INTENT: what does the user want?
HAVE: what data do I already have?
NEED: what's still missing?
DECISION: call tool | ask user | just respond
</think>

Then output your actual response.`;

What the model produces internally looks like this:

<think>
INTENT: create a job
HAVE: customer query "Smith"
NEED: exact customer_id
DECISION: search first
</think>
<action>{"type":"search_customers","query":"Smith"}</action>

I strip the <think> tags before anything hits the UI, but leave them visible during development. The reliability difference is dramatic — the model talks itself through the logic before committing to an action.

Keeping long conversations inside 8K

#

The 8K context window fills up faster than you’d think once tool results start piling in. Around 30 turns, I hit the wall.

My fix is conversation compaction — I ask the model to summarise itself:

const compactPrompt = `Summarize this conversation in under 200 words.
Include: user's goal, customers/jobs created (with IDs), and any unfinished tasks.

${conversationHistory.map(m => `${m.role}: ${m.content}`).join('\n\n')}`;

const summary = await chat([{ role: "user", content: compactPrompt }]);

// Replace history with summary
conversationHistory = [
  { role: "user", content: `[CONTEXT SUMMARY]\n${summary}\n[END SUMMARY]` }
];

A 50-message conversation squeezes down to about 150 tokens, and the model carries on without losing track of which jobs and customer IDs it created.

Saving sessions

#

I also save chat sessions to the app’s backend. The one idea worth copying is to store every assistant message twice: once cleaned up for display, and once raw, with the <think> and <action> tags left in. History then replays cleanly in the UI, while a resumed session gets the full context back, actions and all. Users switch between past conversations from a history dropdown, the way most chat apps work.

Benchmarks from production

#

iPhone 14 Pro (A16 Bionic, 6GB RAM):

• Model load: ~3 seconds
• First token: 180-220ms
• Streaming: 14-16 tokens/sec
• Memory: ~1.8 GB

Samsung Galaxy S23 (Snapdragon 8 Gen 2, 8GB RAM):

• Model load: ~4 seconds
• First token: 250-300ms
• Streaming: 10-12 tokens/sec
• Memory: ~2.1 GB

iPhone 11 (A13, 4GB RAM):

• Model load: ~6 seconds
• First token: 400-500ms
• Streaming: 6-8 tokens/sec
• Memory: ~2.2 GB (occasionally crashes on low memory)

My rule of thumb: 6GB+ RAM for a smooth experience. It runs on 4GB, but you’ll probably need to unload the model when the app backgrounds.

Memory management

#

iOS will happily kill your app for hoarding memory, so I release the model whenever the app leaves the foreground:

useEffect(() => {
  const subscription = AppState.addEventListener("change", (nextAppState) => {
    if (nextAppState === "background") {
      llamaContext?.release(); // Free ~2 GB
    } else if (nextAppState === "active") {
      loadModel(); // Reload when foregrounded
    }
  });

  return () => subscription.remove();
}, []);