I first met them in September, when a Cloudflare dashboard said 22% of my requests were failing while every health check stayed green; I switched Early Hints off and moved on. In November the same zone showed another 2,600 504s in a day, with Early Hints on again, and this time I wanted to know exactly what they are and whether the right fix was to turn the feature off a second time or to make it do its job. The zone fronts one Fly.io origin serving a Django API, an Expo web app and a Hugo marketing site.

The short answer

#

• Those 504s are written by Cloudflare’s Early Hints cache, not by your visitors. In Cloudflare’s logs they carry ClientRequestSource = earlyHintsCache.
• A 504 there means a cache miss: Cloudflare had no hints stored for that URL. A hit is logged as a 2xx.
• They never reach your origin. OriginResponseStatus is 0.
• Early Hints runs on the Link HTTP response header. <link rel="preload"> tags in your HTML don’t feed it.
• Send the header, and those misses turn into hits and real 103 Early Hints responses on the pages that have something to preload.

Three more things worth knowing up front. Hint only the render-critical CSS, scripts and fonts, and keep crossorigin on the fonts. Verify with curl -sv --http2: the 103 should arrive before the 200. And keep earlyHintsCache out of your alerting whatever else you decide; if you’ll never send the header, turn the feature off rather than paying for it in log noise.

How to tell a fake Cloudflare 504 from a real one

#

If you use Logpush or Instant Logs, look at two fields before anything else. ClientRequestSource is eyeball for a real visitor’s request and earlyHintsCache for the Early Hints lookups. OriginResponseStatus is 0 when the origin was never contacted.

In the GraphQL Analytics API the same split is the requestSource dimension. This is the query I now run before believing any 5xx spike:

query ($zone: String!, $since: Time!) {
  viewer {
    zones(filter: { zoneTag: $zone }) {
      httpRequestsAdaptiveGroups(
        limit: 20
        filter: { datetime_geq: $since, edgeResponseStatus: 504 }
        orderBy: [count_DESC]
      ) {
        count
        dimensions { requestSource userAgent clientRequestPath }
      }
    }
  }
}

Grouping by user agent tells the same story. In September the Early Hints lookups showed up as bastion early hints and nginx-ssl early hints. In November they were fl2 early hints and nginx-ssl early hints (FL2 is the newer generation of Cloudflare’s proxy). Across 24 hours, all 2,631 of the zone’s 504s came from those user agents. None came from a browser, the app, curl or a search crawler.

A real 504 that a visitor actually saw almost always came from your side of Cloudflare. A proxy or load balancer in front of your application gave up waiting: nginx, a cloud load balancer, or a platform proxy waiting for a cold machine to boot. Cloudflare’s own timeouts use different codes. 522 means it couldn’t connect to your origin, and 524 means it connected but got no response within 125 seconds (the default proxy read timeout on plans below Enterprise).

So if you landed here searching for how to raise Cloudflare’s timeout because of 504s: the timeout behind a 504 is almost never Cloudflare’s. Check proxy_read_timeout in nginx, your load balancer’s idle timeout, or how long your platform takes to wake a machine. A useful clue: if requests fail at a round number well under 125 seconds, such as 30 or 60, it’s your own proxy’s limit, not Cloudflare’s. (Only Enterprise plans can raise Cloudflare’s 125-second limit, up to 6,000 seconds, and Cloudflare itself treats that as a stopgap.)

What those 504s actually are

#

On a zone with Early Hints enabled, every request for an HTML page or an extensionless URL (which, on an API-heavy zone, is most API calls) gets an extra internal lookup: Cloudflare checking its own cache for a set of hints to send. That lookup is logged as its own line, with ClientRequestSource = earlyHintsCache. It isn’t an end-user request, and it is never sent to your origin.

The status on that line is the result of the cache lookup, not an error:

• 2xx is a hit. Cloudflare had Link headers stored for that URL and sent them to the browser as a 103. On my zone, hits show up as both 200 and 204.
• 504 is a miss. Nothing was stored, so there was nothing to send early.

If your origin never sends the header Early Hints runs on, every lookup is a miss and every request writes a 504 line. In the three hours before I changed anything, the zone logged 173 Early Hints lookups and all 173 were misses. Since the feature had been switched on, it had not served a single hint.

One correction to my September post: I described these as probes that “get no response from the Fly origin”. That isn’t quite right. They never go to the origin at all. The 504 records a cache miss, not a timeout.

Cloudflare documents this, briefly, on its Early Hints page: an Early Hints lookup logged with ClientRequestSource: earlyHintsCache is “expected and benign”, a 200 is a cache hit and a 504 a miss, and Log Explorer hides them with ClientRequestSource != "earlyHintsCache". Cache Analytics already leaves them out. The HTTP traffic chart that scared me doesn’t.

These fake 504s talked me into a server upgrade

#

This is why the split matters. On 3 November, the day before I started on this post, Cloudflare logged 3,292 504s against about 13,300 visitor requests: an apparent error rate of 25%. The API runs on Fly.io’s shared-cpu-1x machines, which are burst-credited. They get a small guaranteed share of a CPU and spend credits to run faster, and a load test in August had shown them throttling to roughly 20 to 30 requests a second under sustained load. “A quarter of requests timing out at peak” looked exactly like that throttling, so I moved the API to shared-cpu-2x.

Every one of those 504s was an Early Hints cache miss. Here is what the requestSource split and the origin’s own logs showed:

• Every Cloudflare 5xx from 28 October to 4 November was an earlyHintsCache lookup, apart from about one 525 a day (the one I traced came from a security scanner).
• The origin’s access logs for 24 October to 4 November had no 5xx responses at all, and one request slower than 10 seconds.
• Fly’s metrics never showed more than about 7 requests a second, and the machine never used more than a fifth of one CPU core. It never got near the throttling ceiling.

The bigger machines fixed nothing, because nothing was broken. The API is back on shared-cpu-1x.

Don’t make capacity decisions from a 5xx chart you haven’t split by request source. Check it against your origin’s own logs as well: if Cloudflare reports a quarter of requests failing and your server logged no errors, find out what Cloudflare is counting before you change the server.

Updated 5 November 2026: the first version of this section said the API had really fallen over on shared-cpu-1x that week. It hadn’t. The numbers above are what I found when I checked.

Is HTTP 103 an error?

#

No. Status codes in the 1xx range are informational: interim responses a server can send before the real one. 103 Early Hints, defined in RFC 8297, carries Link headers so the browser can start fetching critical CSS, JavaScript and fonts, or open connections to other origins, while the server is still building the page. The final 200 follows on the same request.

Support is uneven but safe. Chromium-based browsers (Chrome since version 103, and Edge) and current Firefox act on both preload and preconnect hints. Safari (17 and later) uses only preconnect. Browsers process 103 only over HTTP/2 and HTTP/3. A browser that ignores it simply waits for the 200 as it always did, so a 103 is a head start for most visitors and a no-op for the rest.

Why mine was empty: the Link header vs <link rel=preload>

#

This is the part that cost me the most time, because it looks like it should already work.

Early Hints caches exactly one thing: the HTTP Link response header with a rel=preload or rel=preconnect relation. Cloudflare reads it from your normal responses (status 200, 301 or 302, on HTML or extensionless URLs), stores it, and replays it to the next visitor as a 103 before your full response is ready.

My pages already had preloads, but as tags in the HTML <head>:

<link rel="preload" href="/_expo/static/css/global-7d77bd.css" as="style">
<link rel="preload" href="/_expo/static/js/web/entry-fd7fa4.js" as="script">
<link rel="preload" href="/fonts/manrope-latin.woff2" as="font" type="font/woff2" crossorigin>

Those tags still do something: the browser acts on them. But it only sees them after it has received the HTML and started parsing it, which is exactly the moment Early Hints exists to beat. The Expo export and my Hugo theme both write these tags for you, and neither sets the header. So the feature was enabled, the preloads were declared, and the two never met.

The fix: send the header you already have the data for

#

The cleanest fix doesn’t involve maintaining a hand-written list of critical assets that goes stale every time a bundle hash changes. Each page already declares what it wants preloaded, in those tags. So mirror them into a Link header when you serve the page. The standard library’s html.parser copes with quoted and unquoted attributes (Hugo’s minifier strips the quotes), so there’s no regex to get wrong:

from html.parser import HTMLParser


class PreloadLinks(HTMLParser):
    def __init__(self):
        super().__init__()
        self.hints = []

    def handle_starttag(self, tag, attrs):
        a = dict(attrs)
        if tag != "link" or "preload" not in (a.get("rel") or "").split() or not a.get("href"):
            return
        hint = f"<{a['href']}>; rel=preload"
        if a.get("as"):
            hint += f"; as={a['as']}"
        if a.get("as") == "font" or "crossorigin" in a:
            hint += "; crossorigin"
        self.hints.append(hint)


def link_header(html: str, limit: int = 12) -> str | None:
    parser = PreloadLinks()
    parser.feed(html)
    return ", ".join(list(dict.fromkeys(parser.hints))[:limit]) or None

And on the HTML response, in Django:

response = HttpResponse(html, content_type="text/html")
if (link := link_header(html)) and "Link" not in response:
    response["Link"] = link

One detail bit me, and it’s worth keeping: keep crossorigin on fonts. A font preload without it is treated as a different request, so the font gets downloaded twice and the page gets slower, not faster.

You don’t write the 103 yourself. There’s no ASGI, Granian or uvicorn interim-response code to add. Cloudflare builds the 103 from your header; your only job is to send the header.

On other stacks

#

The rule is the same everywhere: put a Link header on the HTML response and let Cloudflare do the rest.

• nginx, on the location that serves your HTML:

  location = / {
      add_header Link "</css/app.css>; rel=preload; as=style" always;
      add_header Link "</fonts/inter.woff2>; rel=preload; as=font; crossorigin" always;
  }

• Laravel with Vite already has middleware for this, AddLinkHeadersForPreloadedAssets, which turns Vite’s preloads into Link headers. Two issues reported against it are the same traps as above: its modulepreload hints have gone out without crossorigin, so browsers can discard them and download the script twice, and on large pages the header has grown past Apache’s roughly 8 KB header limit. Check both on your version.

• Anything else: set one Link header, comma-separated, or several Link headers. Either works.

What to hint is where people overdo it. Hint the render-critical CSS, the entry script and the one or two fonts above the fold, not every chunk. A long list makes the browser fetch things it doesn’t need yet, ahead of things it does. I cap mine at twelve, and only because the Expo export already curates its preload list.

If you’re not behind Cloudflare, or your CDN doesn’t do Early Hints, the Link header still helps a little: the browser can start those fetches as soon as the response headers arrive, before it parses the body. But no 103 is sent unless your server or CDN emits one.

How to check it’s working

#

curl shows the 103 directly. Ask for HTTP/2 and keep only the status lines and Link headers:

$ curl -sv --http2 -o /dev/null https://example.com/register 2>&1 | grep -E '^< (HTTP|link)'
< HTTP/2 103
< link: </_expo/static/css/reset-f5d396.css>; as=style; rel=preload, …
< HTTP/2 200
< link: </_expo/static/css/reset-f5d396.css>; rel=preload; as=style, …

The 103 arrives first, then the 200. Cloudflare reorders the attributes in the hint, which is harmless. Don’t expect every request to get a 103. Two of my first three did. The first visit after a change only stores the header, and Cloudflare skips the 103 when the final response headers are already available.

Then go back to the analytics query above and filter on requestSource: "earlyHintsCache", grouped by status and path. On the pages you fixed, misses should give way to hits.

What changed, and the caveat

#

Minutes after the header went live, the zone logged its first Early Hints hits ever, and curl showed real 103 responses on the web pages. One header fixed both problems: the feature started working, and a slice of the “504s” went away.

The caveat is that a JSON API has nothing to preload. The API endpoints keep missing and keep writing 504 lines, and on an API-heavy zone that’s still most of the volume. You have three sensible options. Send the header on the pages that benefit and accept that API paths keep logging misses; the feature works, and the charts are quieter but not silent. Exclude earlyHintsCache from your 5xx alerting and dashboards whatever else you decide, because an internal cache lookup should never page anyone. Or turn Early Hints off (Speed → Settings → Content Optimization in the dashboard, or the zone’s early_hints setting in the API) if you’ll never send Link headers; a single-page app whose HTML is tiny may get little from it, and then every lookup stops and the noise goes with it.

I kept it on. The web pages now get a head start, the API’s misses are filtered out of everything that alerts, and that’s what I should have done in September instead of switching the feature off.

So the error is rarely about speed. Usually one step was waiting for something that was never going to happen. Here are the causes in the order I’d check them, with the fix for each:

1. A locator matches nothing, so a click() or fill() waits until the test is out of time. A renamed test ID, changed text or a dialog in the way is the usual reason. Fix the locator.
2. A navigation or waitForURL() is waiting for a page that never comes, because a redirect, a login screen or a first-run screen took the app somewhere else. Fix what you’re waiting for.
3. Setup really is slow: a dev server compiling on first request, a backend waking from scale-to-zero, a full UI login in every beforeEach. Fix the environment, or do the setup once.
4. The test is simply long. Then, and only then, give that test more time with test.slow() or test.setTimeout().

Whichever it is, the first job is finding the step that was still running when the time ran out. The second is setting actionTimeout, so the next failure names the step for you.

Two lines in use do it:

import { defineConfig } from "@playwright/test";

export default defineConfig({
  timeout: 30_000,
  expect: { timeout: 5_000 },
  use: {
    actionTimeout: 10_000,
    navigationTimeout: 15_000,
    trace: "retain-on-failure",
  },
});

What the 30 seconds covers

#

The test timeout is one budget, shared by fixture setup, beforeEach hooks and the test function. When the test finishes, afterEach hooks and fixture teardown get a second budget of the same size. beforeAll and afterAll get the same value as well, but they don’t share it with any test, and they fail with their own message.

Inside that budget, only assertions have a clock of their own. await expect(locator).toBeVisible() gives up after 5 seconds by default and tells you exactly what it was looking for. Actions and navigations don’t: actionTimeout and navigationTimeout default to no limit at all. That is the whole reason this error is so vague. A click on an element that will never exist doesn’t fail. It waits, politely, until something else ends the test.

Here is every timeout that can end a run, and what each one prints. The messages are copied from Playwright 1.58 and 1.63, the oldest and newest versions in my projects.

The action, navigation and whole-run rows have no default to print, so those numbers are whatever you set.

Five tests and a renamed test ID

#

In February, five tests in the suite for my estimating app failed in one run, all the same way. The suite drives the web version of the app, served by its dev server, on one worker, against a freshly seeded local backend. Here’s the first of the five, trimmed:

  1) [default] › tests/example.spec.ts:50:7 › Templates › should edit a template ───

    Test timeout of 30000ms exceeded.

    Error: locator.click: Target page, context or browser has been closed
    Call log:
      - waiting for locator('[data-testid="template-item"]').filter({ hasText: 'E2E Edit Template' }).locator('[data-testid="edit-template"]')


      57 |       .locator('[data-testid="template-item"]')
      58 |       .filter({ hasText: "E2E Edit Template" });
    > 59 |     await item.locator('[data-testid="edit-template"]').click();
         |                                                         ^
      60 |     await page.waitForTimeout(500);

The Target page, context or browser has been closed line reads like the browser falling over. It isn’t. It’s what an action says when the page is closed underneath it: the 30 seconds ran out, Playwright tore the test down, and the click that was still waiting reported the page it had lost. With the built-in page fixture, that line would have repeated the timeout instead, as Error: locator.click: Test timeout of 30000ms exceeded. These tests open their own browser context in beforeAll, so the click got the generic version.

The useful line is the call log. The click was waiting for [data-testid="template-item"]. Ten hours earlier I had changed the row component’s test ID from template-item to template-item-${name}, so the end-to-end tests on the native app could pick out one row by name. An attribute selector with = matches the whole value, so the web tests were now asking for an element that existed nowhere on the page, and with no action timeout that click was always going to wait out the test. Five tests at 30 seconds each is two and a half minutes, in a run that took 3.8.

What I added first was a retry. Within half an hour the config had retries: process.env.CI ? 2 : 1, with a comment that still says it’s there “for flaky browser crashes”. On the next run, four of the same five failed again, retry included, and the run took 6.4 minutes instead of 3.8. These describe blocks run in serial mode, where a failure skips the rest of the group and a retry starts the group again from the top, so every bad click now waited out its 30 seconds twice. A retry can’t rescue a test that fails the same way every time. It just bills you twice.

The fix was a prefix match:

const item = page
  .locator('[data-testid^="template-item-"]')
  .filter({ hasText: "E2E Edit Template" });
await item.locator('[data-testid="edit-template"]').click();

The wider lesson is that a test ID is a contract, and I had two test suites depending on this one without either knowing about the other. Playwright’s best-practice guide recommends user-facing locators such as getByRole() over CSS selectors like mine, so that tests survive DOM changes a user would never notice. If you’re setting up Playwright for the first time, Visual Regression Testing with Playwright for Small Teams is my case for starting with screenshot comparisons.

Find the step that hung

#

Read the call log first. It belongs to the step that was still running when the time ran out, and a waiting for locator(...) line with nothing after it means the locator never matched anything.

If that isn’t enough, open the trace. My two app suites, one for the estimating app and one for a marketplace and community app I build, both use trace: "on-first-retry", which only records on a retry, so it needs retries set to at least 1. Without retries, use "retain-on-failure"; for a one-off run, pass --trace on. Then open the trace directly, or from the HTML report:

npx playwright show-trace test-results/<test-folder>/trace.zip
npx playwright show-report

The action that never finished is the one carrying the error, near the end of the list. Select it and look at the snapshot: that’s the page as it was while Playwright waited, and you can inspect it for the element you expected. In a case like mine, the row is sitting right there with an ID the test isn’t asking for.

Recent versions also write an error-context.md beside the failure screenshot, with a text snapshot of the page at the moment it failed. For a missing element, that’s often enough on its own.

One trap: --debug opens the Inspector and also sets --timeout=0, along with --max-failures=1, --headed and --workers=1. A test that passes under --debug hasn’t told you anything about the 30 seconds.

Give every action a ceiling

#

None of my configs set an action timeout, which is how one missing element got to spend 30 seconds, five times over. With the config at the top of this post, the same broken locator fails like this, while the test still has time left:

    TimeoutError: locator.click: Timeout 10000ms exceeded.
    Call log:
      - waiting for locator('[data-testid="template-item"]').filter({ hasText: 'E2E Edit Template' }).locator('[data-testid="edit-template"]')

That’s the step, the action and how long it waited. I’d keep both numbers well under the test timeout, so a single stuck step can’t use up the whole budget and hide behind the vaguer message. In February the 30 seconds was never the problem. It was the only clock on that click, so it was the one that went off.

Raise the timeout for long tests, not stuck ones

#

Raising the timeout is the right fix when a test does real work that takes a while. It’s the wrong one when a step is waiting for something that won’t happen, because a bigger number only makes the failure slower. The trace usually settles which you have. A steady run of actions that each finished, just a lot of them, is a long test. One action sitting on waiting for until the end is a stuck one.

For the long ones, add time where the length is rather than across the whole suite. The marketplace app’s suite has one smoke test that walks through the chat screen taking screenshots as it goes, and it gets three minutes:

test("chat: open, send a message, go back", async ({ page }) => {
  test.setTimeout(180_000);
  // ...
});

Called inside a describe, test.setTimeout() applies to every test in the group (the documented form is now test.describe.configure({ timeout: 60_000 })), which is how the job-cost test in the estimating app’s suite gets its minute. test.slow() triples whatever the timeout is, and can take a condition. In a beforeEach, testInfo.setTimeout() extends every test that runs the hook:

test("slow in WebKit", async ({ page, browserName }) => {
  test.slow(browserName === "webkit", "This page is slow in WebKit");
  // ...
});

test.beforeEach(async ({ page }, testInfo) => {
  testInfo.setTimeout(testInfo.timeout + 30_000);
});

Fixed sleeps are the other thing that eats the budget, quietly, on every run. The estimating app’s suite has more than a hundred page.waitForTimeout() calls, in nearly every spec file. The edit test above sleeps for three seconds before it even reaches the click, and no amount of sleeping would have saved it. Playwright’s API docs are blunt about this: “Never wait for timeout in production. Tests that wait for time are inherently flaky.” A web-first assertion already waits as long as it needs to, and no longer:

// before
await page.locator('[data-testid="save-template"]').click();
await page.waitForTimeout(1500);
await expect(page.getByText("E2E Edited Template")).toBeVisible();

// after
await page.locator('[data-testid="save-template"]').click();
await expect(page.getByText("E2E Edited Template")).toBeVisible();

Assertion timeouts need the same care. Both app suites raise expect.timeout to 15 seconds, and both run against a dev server rather than a production build. That buys slack, but two slow assertions in a row can spend the whole 30 seconds between them, and then the test timeout fires first and you’re back to the vague message.

When the clock isn’t the test’s

#

If the message mentions config.webServer, no test has run yet. Playwright starts your webServer command and waits for its url to answer before running anything, on a separate timeout of 60 seconds by default. Both app suites give the dev server two minutes:

webServer: {
  command: "npm run web -- --port 8081",
  url: "http://localhost:8081",
  reuseExistingServer: false,
  timeout: 120_000,
},

If your tests run against a deployed environment instead, find out how long its first request takes. When I finally measured mine, a machine starting from scale-to-zero on my host took 25 to 33 seconds to serve anything, which is most or all of a test’s budget before the first click. That story is in Fly.io Cold Starts with Scale to Zero: From 33 Seconds to 3.

If it says “No tests found” instead

#

No tests found sounds related, but it isn’t a timeout: Playwright looked and found nothing to run, usually because the spec sits outside testDir or the command-line argument (a regular expression matched against file paths) doesn’t match it. npx playwright test --list shows what it collected without running anything; if your spec isn’t there, it was never going to time out.

Update, 4 November 2026: one more correction. Those 504s aren’t probes the origin fails to answer. They’re Cloudflare’s own Early Hints cache lookups, which never reach the origin at all, and a 504 there only means nothing was cached for that URL. Turning Early Hints off wasn’t the best fix either: sending a Link response header made the feature work and turned those misses into hits. The full story is in Cloudflare 504 errors that aren’t real: Early Hints and HTTP 103.

With auto_stop_machines on, any request that lands on a stopped Fly.io machine waits for a full boot, and mine took 25 to 33 seconds because the entrypoint ran Django migrations and collectstatic on every machine start, and nothing had ever cached the bytecode. Running that release work once per image (a Redis lock keyed on FLY_IMAGE_REF) and adding compileall to the build got a cold boot down to about nine seconds. Switching auto_stop_machines from 'stop' to 'suspend', which meant dropping a swap file Fly won’t suspend with, got wake-ups to about three.

I went looking for a traffic spike on my Cloudflare dashboard and found something much worse underneath it: 22% of every request my app served in a day had failed. No alert had fired. Every health check was green. The logs looked perfectly healthy, because the failing requests never reached the application at all.

The cause turned out to be my own container startup script, doing sensible work at exactly the wrong time.

The spike was a red herring

#

The thing I set out to explain was one hour that took about 4,900 requests against a median of 121. Forty-two unique visitors. That is 110 requests per visitor, which is not how humans browse.

It was a scanner. A single bot walking through invented subdomains, asking each one for secrets:

api-stage-asia.example.com/.env.local
api-us-east-1-demo.example.com/.env
api-eu-uat.example.com/wp-config.php~
app-test-us-east-1.example.com/actuator/configprops
prod-api-use2.example.com/.env.local
relay.example.com/.env.production

Thirty or so hostnames, every .env variant you can think of, plus a Java framework’s actuator endpoints on a Django app. All of them got a 301 to my canonical host and nothing was served. Boring, and genuinely harmless.

But while I had the analytics open, I grouped the same day by status code. That is where the real problem was.

Twenty-two percent. And not spread evenly across the site, but concentrated exactly where it hurts:

Those are the endpoints my mobile app polls every 30 seconds. So this was not an abstract error-rate number. It was every user’s phone, quietly failing to load their unread badge, all day, for weeks probably.

Why the logs looked fine

#

My first instinct was that the app had become slow. It had not. I pulled the application’s own request lines for one of the worst hours out of the log archive:

[2026-09-08 01:00:29] "GET /api/notifications/unread/" 304 61.615
[2026-09-08 01:00:29] "GET /api/notifications/"        304 63.568
[2026-09-08 01:00:59] "GET /api/notifications/unread/" 304 58.742
[2026-09-08 01:00:59] "GET /api/notifications/"        304 105.438

Forty-four to a hundred milliseconds, 304s all the way down. The origin was fast.

That is the important clue, and it is worth stating as a rule: if your edge reports errors your origin has never heard of, the requests are dying before they arrive. Stop reading application logs and start looking at what sits in front of them.

In my case that is Fly.io’s proxy, and the answer was in how my machines are scheduled.

Scale to zero is a promise about money, not latency

#

My config looks like this, and on the face of it it is reasonable:

[http_service]
  auto_stop_machines = 'stop'
  auto_start_machines = true
  min_machines_running = 1

Idle machines stop. Traffic starts them again. You pay for what you use. For a small app with bursty traffic this is exactly what you want, and it is why I had ten machines defined but almost always one running.

The bill it hides is latency. When a request arrives for a stopped machine, somebody has to wait for that machine to boot. If booting takes longer than the proxy’s patience, that somebody gets a 504.

So: how long did my machines take to boot? I had never measured. The log archive knows, because the startup script announces each phase, and every line carries the machine id:

WITH s AS (
  SELECT instance, timestamp,
         CASE WHEN message ILIKE '%Starting app server%' THEN 'begin'
              WHEN message ILIKE '%Migrations complete%' THEN 'migrated'
              WHEN message LIKE  '%"GET /healthz%'       THEN 'serving' END AS phase
  FROM logs
  WHERE app = 'myapp' AND timestamp::date = '2026-09-08'
)
SELECT instance,
       min(timestamp) FILTER (WHERE phase = 'begin')    AS started,
       min(timestamp) FILTER (WHERE phase = 'migrated') AS migrated,
       min(timestamp) FILTER (WHERE phase = 'serving')  AS first_serve
FROM s GROUP BY instance

The answer, consistently, across every machine:

Then I counted how often that happened:

SELECT count(*) AS starts, count(DISTINCT instance) AS machines
FROM logs
WHERE app = 'myapp' AND timestamp::date = '2026-09-08'
  AND message ILIKE '%Starting app server%'
-- starts: 92, machines: 10

Ninety-two cold starts in one day. Each one a half-minute window in which anything routed to that machine timed out. Twenty-two percent stopped looking mysterious.

The cause was in my own entrypoint

#

Here is what every single machine ran before it would answer a request:

# entrypoint.sh, run on every machine start
echo "Running migrations..."
python manage.py migrate --noinput

echo "Running collectstatic in background..."
python manage.py collectstatic --noinput &

exec "$@"    # then start the app server

Read that again with “92 times a day” in mind.

migrate is a full Django boot: import every model, connect to Postgres, query the migrations table, decide there is nothing to do. On a shared CPU that is ten seconds to accomplish nothing. collectstatic is a second full Django boot that then uploads static files to object storage, competing for the same single core that the web server is trying to start on. The app server is the third boot.

Three Python interpreters, two of them doing work that had already been done, every time a machine woke up.

The reason it was written that way is good, and it is probably why yours looks similar: migrations have to run somewhere, Fly’s release_command had hung on me before, and the entrypoint is the one place guaranteed to execute on deploy. It is a correct place to put deploy work. It is just not a correct place to put it per machine start, and once your machines stop and start on their own, those are no longer the same event.

That is the whole bug, and I think it is a common one:

Work that belongs to a release was being run once per machine start. Scale-to-zero turned one event into ninety-two.

The fix: claim the release once

#

Migrations do not need to run on the eighth machine to boot a given image. They need to run once for that image, and everyone else needs to know it has happened.

That is a lock with a name that changes per deploy. Fly hands you the name for free in FLY_IMAGE_REF, and I already had Redis. The whole thing is a small script that runs before Django exists, so the common answer costs milliseconds rather than a framework boot:

# release_gate.py: runs before Django exists. Exit code 0 means "prepare this release".
import os, time

PREPARE, SKIP = 0, 1
LOCK_TTL, WAIT, POLL = 300, 240, 2
_sleep, _monotonic = time.sleep, time.monotonic    # aliases, so tests can replace them


def release_id() -> str:
    """Something that changes exactly when the deployed image does."""
    return os.getenv("FLY_IMAGE_REF") or os.getenv("FLY_MACHINE_VERSION") or ""


def claim(client, release: str) -> int:
    if client is None or not release:   # no Redis, or no idea which release: behave as before
        return PREPARE
    done_key, lock_key = f"release:{release}:done", f"release:{release}:lock"
    if client.get(done_key):            # somebody already did it for this image
        return SKIP

    owner = os.getenv("FLY_MACHINE_ID", "unknown")
    if client.set(lock_key, owner, nx=True, ex=LOCK_TTL):
        return PREPARE                  # we won the race; we do the work

    # Someone else is preparing. Wait for them, because serving requests
    # against a half-migrated schema is worse than starting slowly.
    deadline = _monotonic() + WAIT
    while _monotonic() < deadline:
        _sleep(POLL)
        if client.get(done_key):
            return SKIP
        if client.set(lock_key, owner, nx=True, ex=LOCK_TTL):
            return PREPARE              # the owner died and its lock expired; take over
    return PREPARE                      # waited long enough; do it ourselves

# ... plus `done` (sets done_key) and a main() that treats any Redis error as PREPARE

The entrypoint becomes an if:

if python3 release_gate.py claim; then
    python manage.py migrate --noinput
    python manage.py collectstatic --noinput &
    python3 release_gate.py done
fi

exec "$@"

Every failure has to end in “prepare it anyway”

#

This is the part worth copying, more than the code.

A gate like this sits between your users and a schema migration. If it is wrong in the direction of skipping, you serve requests against a database that has not been migrated, which is a genuinely bad outage. If it is wrong in the direction of preparing, you run an idempotent no-op migration twice and waste ten seconds.

Those outcomes are not symmetrical, so the code must not treat them as such. Every uncertain path returns PREPARE:

• No Redis, or Redis unreachable. Prepare. This is exactly the old behaviour, so a cache outage degrades to “slow”, not “broken”.
• No image reference in the environment. Prepare. I do not know what release this is, so I cannot know whether it is ready.
• The lock’s owner died mid-migration. Its lock expires, the next waiter takes it over and prepares.
• Waited longer than the timeout. Prepare anyway. A machine that never serves is worse than a duplicated migration.

There is one deliberate asymmetry in the other direction: a machine that finds another machine actively holding the lock waits rather than serving. Starting slowly during a deploy is fine. Answering queries against a half-applied schema is not.

Watch out for one trap if you build this on a cache wrapper that swallows connection errors, as I first did. A wrapper like that returns False from add() when Redis is unreachable, which reads identically to “someone else holds the lock” and would have made every machine skip its migrations. That is precisely the dangerous direction. Distinguish the two by reading the lock back: a lock nobody holds means the cache is broken, not that you lost a race.

Testing it without a Redis

#

The logic is worth unit testing, because the interesting paths are the ones you cannot reproduce by hand. A fake with two methods covers it:

class FakeRedis:
    """Enough of a Redis client for the gate: get, and set with nx/ex."""

    def __init__(self) -> None:
        self.store: dict[str, str] = {}

    def get(self, key): return self.store.get(key)

    def set(self, key, value, nx=False, ex=None):
        if nx and key in self.store:
            return None
        self.store[key] = value
        return True

Then drive the awkward cases by making time pass on your terms. This is why the gate calls module-level _sleep and _monotonic aliases rather than time.sleep directly, so a test can replace them without patching the time module for every other thread in the process:

def test_a_waiter_takes_over_when_the_owner_disappears(self):
    redis = FakeRedis()
    self.assertEqual(release_gate.claim(redis, "img-1"), release_gate.PREPARE)   # the owner takes the lock

    def owner_dies(_seconds):
        redis.store.pop("release:img-1:lock", None)                              # its lock expired

    with patch.object(release_gate, "_sleep", owner_dies):
        self.assertEqual(release_gate.claim(redis, "img-1"), release_gate.PREPARE)

Did it work? Partly

#

Deploy, stop a machine, start it again, watch the clock. The startup now announces its decision and gets out of the way:

15:23:34  Starting app server...
15:23:36  release gate: this release is already prepared, starting straight away
15:23:44  "GET /healthz HTTP/1.1" 200

Ten seconds, and a second machine measured eleven. Against 25 to 33 before, two thirds of the window is gone, and the gate itself costs two seconds because it never imports Django.

But ten seconds is still ten seconds. A request landing in the first of them still fails. So I went looking for the rest.

The bytecode nobody was caching

#

The image sets a line that appears in more or less every Python Dockerfile ever written:

ENV PYTHONDONTWRITEBYTECODE=1

It is good advice. A container should not be scribbling .pyc files into a layer at runtime. What I had never thought through is the other half of the bargain: if the runtime never writes bytecode, and the build never writes it either, then nothing is ever cached, and every process start compiles the entire dependency tree from source.

I counted:

site-packages .py  files : 5,707
site-packages .pyc files : 0

Django, the REST framework, every library, recompiled on each of those 92 daily starts. Measured on a production machine:

The fix is one line, and compileall writes explicitly so it works despite the env var:

RUN python -m compileall -q /app/.venv/lib /app/myapp || true

Sixteen seconds of build time, once, in a cached layer.

While I was there I found something quieter. .dockerignore said:

__pycache__/
*.pyc

Docker matches those against the whole relative path, not each path component, so an unanchored pattern only ever matches at the root of the build context. Every nested __pycache__ was shipping. The image contained 218 .pyc files compiled by Python 3.15 on a laptop, riding along inside an image whose interpreter is 3.14 and quietly ignoring all of them. The patterns want to be **/__pycache__/ and **/*.pyc.

Honest result: end to end this bought about a second, not two. Ten seconds became nine. The isolated benchmark oversold it, as isolated benchmarks do.

Suspend, and the one-line setting that was blocking it

#

At nine seconds I had run out of work to remove. What was left was one unavoidable boot: Python starting, Django importing, the WSGI app coming up on a shared core.

So stop booting. Fly can snapshot a machine’s memory instead of shutting it down:

[http_service]
  auto_stop_machines = 'suspend'   # was 'stop'

I went to test it and Fly refused, with the most useful error message of the whole exercise:

failed to suspend VM: failed_precondition:
Machines with swap cannot be suspended

My config had this near the top, with a comment I had written myself and believed:

# Swap cushion so a memory spike swaps to disk instead of getting OOM-killed.
swap_size_mb = 2048

A reasonable precaution. Was it doing anything? Every machine said:

MemTotal:   985220 kB
SwapTotal: 2097148 kB
SwapFree:  2097148 kB

Not one page had ever been swapped. And the day’s logs held no OOM kills — the eleven lines that matched “oom” turned out to be requests for a JavaScript bundle whose content hash happens to contain those three letters. Meanwhile the actual memory guard was somewhere else entirely, in the app server’s own settings, which respawn any worker past 800MB long before it can reach the 1GB ceiling.

So the cushion was insurance against an event that had never happened, already covered by a different mechanism, and its premium was the ability to suspend. Out it went.

Three trials, all within half a second. Every endpoint answered 200 in 250 to 435 milliseconds immediately after waking.

What happens to your sockets while the machine sleeps

#

Nothing good, and there is no hook to fix it. Fly freezes the VM; the process is not signalled and cannot close anything on the way down. Whatever connections it held are still in its memory when it wakes, pointing at sockets the other end abandoned minutes ago.

You do not solve that going down. You solve it coming up, and mostly by having already made the right decisions:

• Database connections are opened per request (conn_max_age = 0), so there is nothing long-lived to go stale.
• The Redis cache degrades instead of raising. A wrapper catches connection errors and returns a miss.

I got to watch the second one earn its place. One resume logged exactly the failure you would expect:

redis.exceptions.ConnectionError: Error while reading from fly-...

and the request it happened inside returned 200 in 3.6 seconds, because a dead cache is a slow page, not an error page. Later resumes logged nothing at all.

If your cache client raises on a dead connection, suspend will convert every wake into a burst of 500s. Check that before you flip the setting, not after.

The test that lied to me

#

My first suspend measurement said the feature was a disaster: 502, after 30 seconds, twice.

I had pinned the request to a specific machine with fly-force-instance-id, which is a lovely header for load testing one instance and a terrible one for this. Forcing an instance bypasses the proxy logic that would have woken it. Fly told me so, if I had read it as an answer rather than an error:

machine was recently stopped and is unavailable to service request

The number that matters comes from the path real traffic takes. Forty concurrent requests through the edge, well past the soft limit of 12, so the proxy has to scale out onto suspended machines:

status codes:  40 × 200
slowest:       2.1s

Two lessons, and the second one is the expensive one. Measure the path your users take, not the path that is convenient to instrument. And when your tooling refuses to do something, read the refusal: “Machines with swap cannot be suspended” was the sentence that unlocked this entire afternoon.

What I would check in your app

#

If you run anything on a platform that stops idle instances, these questions took me an afternoon and would have saved me weeks of quiet failure:

1. How long does a cold start take? Not how long the container takes to run, how long until it answers a real request. If you cannot answer in seconds, measure it before you need to.
2. What does your entrypoint do per start that belongs to a deploy? Migrations, static file collection, cache warming, index building. All fine once. All expensive ninety-two times.
3. Is anything caching your bytecode? PYTHONDONTWRITEBYTECODE with no compileall in the build means the answer is no.
4. Can you suspend instead of stop? And if not, what is stopping you? Mine was a swap file that had never been touched.
5. Does your edge see errors your origin does not? Compare the two. A gap between them is not a reporting quirk, it is requests dying in the space between, and your application logs will never show it.

The health checks were green the whole time, incidentally. They always are: a health check runs against a machine that is already up. It cannot report on the thirty seconds before it existed.

I spend way too much time debugging. Everyone does. These days most of mine happens in Xcode and the simulator, chasing bugs in my own app, and the thing that keeps striking me is how much debugging machinery Xcode ships with that hardly anyone touches. This is the workflow I’ve settled into over the years, the parts that actually save time when something breaks.

Breakpoints Are More Powerful Than You Think

#

Most people click the gutter to set a breakpoint and leave it at that. Right-click one, though, and there’s a whole menu of options that changes how you debug.

Conditional Breakpoints

#

Say you’ve got a loop processing 1000 items and only item 347 misbehaves. Hitting the breakpoint 346 times first is a waste of an afternoon.

for i in 0..<users.count {
    processUser(users[i])  // Set breakpoint here with condition: i == 347
}

Right-click the breakpoint, add the condition i == 347, and Xcode only stops when it’s true. Works just as well with things like userId == "abc123" or error != nil.

Symbolic Breakpoints

#

These are for when you don’t know where a method is being called from. Open the breakpoint navigator (Cmd+8), click the + button, and choose “Symbolic Breakpoint”.

Want to see every time any view controller loads? Set the symbol to viewDidLoad and Xcode pauses on every single one. Too noisy? Narrow it down: MyViewController.viewDidLoad.

The ones I keep around permanently:

• UIViewAlertForUnsatisfiableConstraints catches Auto Layout conflicts the moment they happen
• objc_exception_throw breaks on any Objective-C exception before the crash
• malloc_error_break finds memory allocation problems

Exception Breakpoints

#

This should be the first thing you set up in any new project. Add one (breakpoint navigator, + → Exception Breakpoint) and Xcode pauses the moment something throws, instead of after the app has already died.

It’s caught more nil-unwrapping bugs for me than I’d like to admit, the kind that otherwise just show up as mysterious crashes in the console.

Breakpoint Actions

#

Breakpoints don’t have to stop execution at all. Edit one, click “Add Action”, and it can:

• Print variables without stopping: "User count: @(users.count)@"
• Run LLDB commands automatically
• Execute shell scripts
• Play a sound (I use this for rare code paths I want to know about)

Tick “Automatically continue after evaluating actions” and the breakpoint becomes a logger that never interrupts you.

LLDB: The Console Commands You’ll Actually Use

#

When you hit a breakpoint, the console at the bottom isn’t just for reading crash logs. It’s LLDB, and it’s genuinely useful.

The Basics

#

(lldb) po user
▿ User
  - id: "123"
  - name: "John Doe"
  - email: "john@example.com"

po prints objects in a readable format. That’s 90% of my LLDB usage right there.

Need more detail? Try p instead:

(lldb) p user.name
(String) $R0 = "John Doe"

And to see every local variable at once:

(lldb) frame variable

Changing Things at Runtime

#

This is where LLDB earns its keep: you can modify variables without recompiling.

(lldb) expr user.name = "Jane Doe"
(lldb) expr index = 0

Found a bug and want to try a fix without rebuilding? Change the variable and continue. I use this constantly when narrowing down edge cases.

You can call methods too:

(lldb) po self.refreshUI()
(lldb) expr navigationController?.popViewController(animated: true)

Navigation

#

(lldb) bt              # Show the call stack
(lldb) frame select 3  # Jump to a different stack frame
(lldb) continue        # Keep running (or just 'c')
(lldb) n               # Step over (next line)
(lldb) s               # Step into function

Watchpoints

#

Want to know when a variable changes? Set a watchpoint:

(lldb) watchpoint set variable user.isLoggedIn

Execution now pauses whenever that value changes, no matter where in your code it happens. Gold for tracking down unexpected state mutations.

Console.app: The Hidden Debugging Tool

#

Xcode’s console is fine until you need to see everything: system logs, crash reports, the lot. That’s when I open Console.app.

If you’re using OSLog (you should be), Console.app makes those logs searchable and filterable:

import os.log

let logger = Logger(subsystem: "com.example.app", category: "networking")

logger.info("Starting API request")
logger.debug("Request URL: \(url.absoluteString)")
logger.error("Failed to decode: \(error.localizedDescription)")

Then in Console.app:

1. Select your simulator or connected device
2. Filter by your subsystem: subsystem:com.example.app
3. Filter by level: subsystem:com.example.app AND level:error

You can save filter predicates for the sessions you run often. I’ve got one for network requests, one for database operations, and one that shows nothing but errors and warnings.

Why OSLog Over print()?

#

OSLog beats scattering print() statements everywhere. It’s structured, so you can filter by level and category. It’s fast enough that logging won’t slow the app down. It redacts sensitive data in production automatically, and the logs survive your app crashing.

print() is still fine for quick throwaway checks. Anything you might want to look at later belongs in OSLog.

Advanced Console.app Filtering

#

Console.app supports proper predicate queries. The ones I use regularly:

# All errors and faults in your app
subsystem:com.example.app AND (level:error OR level:fault)

# Network requests that failed
subsystem:com.example.app AND category:networking AND eventMessage CONTAINS "failed"

# Everything from the last 5 minutes
subsystem:com.example.app AND timestamp >= now(-5m)

You can also export logs to attach to bug reports, which beats copy-pasting out of Xcode’s console.

View Debugging

#

When the UI is broken and you can’t see why, the view debugger is the fastest way out.

Run the app, navigate to the broken screen, and click the “Debug View Hierarchy” button in Xcode’s debug bar (or press Cmd+Shift+D). You get a 3D exploded view of the whole hierarchy.

Rotate it around and things jump out:

• Views rendering behind other views
• Views sitting way off-screen
• Views with zero size
• The full constraint chain for whatever you select

I’ve used it to find invisible buttons swallowing tap events, labels rendering at 0x0, and an image that was accidentally 10,000 pixels wide.

Auto Layout Debugging

#

Purple warning icons in the view hierarchy mean constraint conflicts. Click one and Xcode shows exactly which constraints are fighting.

Give your constraints identifiers:

heightConstraint.identifier = "ProfileImageHeight"

When a constraint breaks, you’ll see "ProfileImageHeight" in the error instead of a memory address, which makes the log actually readable.

From LLDB you can also print the constraint tree:

(lldb) po view.hasAmbiguousLayout
(lldb) po view._autolayoutTrace()

Simulator Features That Save Time

#

Slow Animations

#

Debug → Slow Animations runs everything at 1/10 speed. Perfect for seeing what’s actually happening during a transition, or why an animation looks off.

Simulate Memory Warnings

#

Debug → Simulate Memory Warning tests how your app copes with low memory. I’ve found plenty of image caching bugs this way: everything works fine until the warning fires and suddenly all your images vanish.

Network Link Conditioner

#

Xcode → Open Developer Tools → Network Link Conditioner

Simulate 3G, LTE, high packet loss, or complete offline mode. If an API request has ever worked fine on WiFi and timed out on cellular, this is how you reproduce it.

I keep a “Bad Network” profile with 500ms latency and 10% packet loss. It surfaces timeout bugs and loading-state problems you’d never see on fast office WiFi.

Status Bar Overrides

#

Right-click the status bar in the simulator and you can override the time, battery level, signal strength, and carrier. Handy for consistent screenshots, or for checking how the UI looks at different battery levels.

Location Simulation

#

Debug → Location simulates different locations without leaving your desk: custom coordinates, city walks, freeway drives. Very useful for location-based features, or bugs that only show up in certain regions.

Memory Debugging

#

Debug Memory Graph

#

Click the Debug Memory Graph button (Cmd+Shift+M) and Xcode shows every object in memory, their relationships, and the part you actually care about: which ones are leaking.

Purple exclamation marks mean leaks. Click one and you can see the retain cycle.

The most common leak I see:

class ViewController: UIViewController {
    var onComplete: (() -> Void)?

    func setupHandler() {
        onComplete = {
            self.dismiss(animated: true)  // ❌ Captures self strongly
        }
    }
}

Fix it with weak self:

onComplete = { [weak self] in
    self?.dismiss(animated: true)  // ✓ No retain cycle
}

Delegates need to be weak too:

weak var delegate: ManagerDelegate?  // Not just 'var'

Instruments

#

For serious memory investigation, use Instruments (Product → Profile, or Cmd+I).

The Allocations instrument shows every object allocation, memory growth over time, which classes are using the most memory, and stack traces for where each allocation happened.

What I’m usually looking for: memory that grows linearly over time (probably a leak), unexpectedly large allocations (did I just load all 1000 images at once?), and objects that should have been deallocated but weren’t.

Mark generations (the little flag button) before and after an action to see what’s persisting when it shouldn’t.

Performance Profiling

#

Time Profiler

#

Product → Profile → Time Profiler. Record while doing the slow thing in your app, then read the call tree.

Sort by “Self Weight” to find the bottlenecks. If a function shows 40% self weight, that’s where your time is going.

I once found a JSON parsing function being called 1000 times a second. Moved it to a background queue and the UI stopped stuttering.

Main Thread Checker

#

Xcode automatically catches UI updates on background threads. If you see this:

Main Thread Checker: UI API called on a background thread: -[UILabel setText:]

you did something like this:

URLSession.shared.dataTask(with: url) { data, response, error in
    self.label.text = "Loaded"  // ❌ Crash!
}

Fix it:

URLSession.shared.dataTask(with: url) { data, response, error in
    DispatchQueue.main.async {
        self.label.text = "Loaded"  // ✓
    }
}

Runtime Diagnostics

#

Edit Scheme → Run → Diagnostics. There’s a row of checkboxes in here that catch bugs you’d never find by hand.

Address Sanitizer

#

Finds memory corruption: use-after-free errors, buffer overflows, memory leaks. It makes the app 2-3x slower, but it catches bugs that are near impossible to track down any other way. I turn it on when chasing crashes that only happen sometimes.

Thread Sanitizer

#

Catches data races and threading bugs. Enable it, run the app, and use it normally; if there’s a race condition in there, Thread Sanitizer will find it.

You can’t run Address Sanitizer and Thread Sanitizer at the same time, so I alternate between them when a crash is being weird.

Zombie Objects

#

Catches messages sent to deallocated objects. With it enabled, Xcode tells you exactly when you’re touching freed memory:

*** -[MyViewController viewDidLoad]: message sent to deallocated instance 0x600001234000

Don’t leave it on permanently. It stops objects from being deallocated, so memory climbs forever. But for pinning down a specific use-after-free bug, it’s great.

Common Debugging Scenarios

#

App Crashes on Launch

#

Add an exception breakpoint first; that usually catches it.

If not, the usual suspects are missing keys in Info.plist, force-unwrapped optionals in initialisation code, or dependency injection failing.

UI Not Updating

#

Every. Single. Time. It’s one of:

1. The update isn’t on the main thread
2. The outlet isn’t connected
3. The view isn’t actually visible (check with po view.window in LLDB; if it returns nil, the view isn’t in the hierarchy)

And for table or collection views: you forgot to call reloadData().

Memory Warnings Crashing the App

#

Use Instruments Allocations to see what’s holding memory. It’s usually images never released from a cache, view controllers that won’t deallocate (retain cycle), or some array or dictionary growing forever.

Simulate memory warnings in the simulator to test your cleanup code.

Mysterious Crashes

#

Turn on the sanitizers. Intermittent usually means threading, so run with Thread Sanitizer. Crashing inside system frameworks usually means memory corruption, so run with Address Sanitizer.

Crashes You Can’t Reproduce

#

When the crash happened on someone else’s phone, the report usually still exists. For TestFlight and App Store builds, Xcode’s Organizer (Window → Organizer → Crashes) collects crash reports from users who share analytics, symbolicated if the dSYMs went up with the build. For a tester’s phone, the reports are on the device itself under Settings → Privacy & Security → Analytics & Improvements → Analytics Data, as .ips files starting with the app’s name, and they can AirDrop one over. Start from the crashed thread’s stack trace, then work through the scenarios above.

Debugging OTA Updates in the Simulator

#

My app ships over-the-air updates, and the simulator is the easiest place to debug that whole flow.

I start by watching the network traffic. Console.app filtered to the app’s subsystem shows the manifest requests and responses in real time, so you can see exactly which headers go out and what the server sends back.

Then I make the network bad on purpose. Network Link Conditioner shows you how the update flow copes with slow connections and packet loss. Does the app hang? Show a loading indicator? Fall back gracefully?

Around the update check itself, I log everything: when the check starts, when an update is available, when the download begins, and when the new bundle loads.

import os.log

let logger = Logger(subsystem: "com.example.app", category: "updates")

logger.info("Checking for updates...")
let update = try await Updates.checkForUpdateAsync()
logger.info("Update available: \(update.isAvailable)")

Finally, it’s worth testing the manifest endpoint on its own. You can curl it with the same headers the app sends:

curl -H "expo-protocol-version: 1" \
     -H "expo-platform: ios" \
     -H "expo-runtime-version: 1.0.0" \
     https://updates.example.com/manifest

That confirms the server is returning the right data before you go digging into the client.

Third-Party Tools

#

Charles Proxy / Proxyman

#

See all your network traffic, modify requests and responses, test error conditions. Essential for API debugging.

Proxyman has a nicer UI than Charles and feels more at home on macOS. I switched last year and haven’t looked back.

Reveal

#

Like Xcode’s view debugger, but more capable. It works on physical devices without being attached to Xcode, which makes it great for chasing layout issues on testers’ devices.

Simulator vs Real Device

#

Most of my debugging happens in the simulator because it’s quick: no cable, no signing, no waiting for an install. But the simulator is an iOS build running on your Mac’s processor, with your Mac’s memory and your Mac’s network connection, and some bugs only exist on a phone.

So I build and debug logic in the simulator, and before a release I run the app on the oldest iPhone I still support. Performance, memory pressure and anything that touches hardware haven’t really been tested until they’ve run on a device.

Running on a Device

#

Plug the iPhone in, pick it as the run destination, and set a team under Signing & Capabilities (a free Apple ID is enough for your own phone). Since iOS 16 the phone also needs Developer Mode turned on under Settings → Privacy & Security, which restarts it the first time. Once it’s on, the phone’s own Settings → Developer menu has a Network Link Conditioner with the same bad-network profiles as the Mac’s.

Wireless Debugging

#

Plug the device in over USB once, go to Window → Devices and Simulators, tick “Connect via network”, then unplug. The device stays available in Xcode as long as you’re on the same WiFi.

Device Console

#

Window → Devices and Simulators → Open Console. This shows everything: system logs, app logs, crash reports. Far more detail than Xcode’s own console.

When a tester tells me “the app crashed”, I always ask them to plug their device in so I can grab the console logs. There’s often a system-level error sitting right there that explains the whole thing.

What I Actually Do When Debugging

#

1. Add an exception breakpoint if there isn’t one already
2. Reproduce the bug
3. Set a breakpoint near where I think it’s going wrong
4. Check values with po in LLDB
5. Modify variables to test fixes without recompiling
6. Check the view hierarchy if it’s UI-related
7. Profile with Instruments if it’s performance-related
8. Turn on the sanitizers if it’s memory or threading

The trick is working backwards from the symptom. Crash? Exception breakpoint. Slow? Time Profiler. Memory? Instruments. Weird UI? View debugger.

Tips I Wish I’d Known Earlier

#

Use assertions. They catch logic errors before they turn into bugs:

assert(users.count > 0, "Users array should never be empty here")

Commit before you start debugging. Once you’re changing things to test theories, you’ll want a clean point to revert to.

Delete your print statements when you’re done, or at least wrap them in #if DEBUG. Old debug logs make the next bug harder to see.

Learn LLDB properly. It’s faster than rebuilding your app twenty times.

And remember the simulator is not a real phone. Bugs that only appear on device are almost always threading or memory. Or performance, because the simulator runs on your Mac’s CPU rather than an actual iPhone chip.

Tools I Keep Open

#

• Xcode, obviously
• Console.app filtered to my app’s subsystem
• Proxyman for network debugging
• Instruments when things get serious

No single tool covers everything. The view debugger won’t find a memory leak, and Instruments won’t untangle an Auto Layout conflict, so most of the skill is matching the tool to the symptom. That part just comes with repetition: after enough retain cycles you start to smell them from the stack trace alone. Debugging never becomes fun exactly, but it does stop being miserable.