I first met them in September, when a Cloudflare dashboard said 22% of my requests were failing while every health check stayed green; I switched Early Hints off and moved on. In November the same zone showed another 2,600 504s in a day, with Early Hints on again, and this time I wanted to know exactly what they are and whether the right fix was to turn the feature off a second time or to make it do its job. The zone fronts one Fly.io origin serving a Django API, an Expo web app and a Hugo marketing site.

The short answer

#

• Those 504s are written by Cloudflare’s Early Hints cache, not by your visitors. In Cloudflare’s logs they carry ClientRequestSource = earlyHintsCache.
• A 504 there means a cache miss: Cloudflare had no hints stored for that URL. A hit is logged as a 2xx.
• They never reach your origin. OriginResponseStatus is 0.
• Early Hints runs on the Link HTTP response header. <link rel="preload"> tags in your HTML don’t feed it.
• Send the header, and those misses turn into hits and real 103 Early Hints responses on the pages that have something to preload.

Three more things worth knowing up front. Hint only the render-critical CSS, scripts and fonts, and keep crossorigin on the fonts. Verify with curl -sv --http2: the 103 should arrive before the 200. And keep earlyHintsCache out of your alerting whatever else you decide; if you’ll never send the header, turn the feature off rather than paying for it in log noise.

How to tell a fake Cloudflare 504 from a real one

#

If you use Logpush or Instant Logs, look at two fields before anything else. ClientRequestSource is eyeball for a real visitor’s request and earlyHintsCache for the Early Hints lookups. OriginResponseStatus is 0 when the origin was never contacted.

In the GraphQL Analytics API the same split is the requestSource dimension. This is the query I now run before believing any 5xx spike:

query ($zone: String!, $since: Time!) {
  viewer {
    zones(filter: { zoneTag: $zone }) {
      httpRequestsAdaptiveGroups(
        limit: 20
        filter: { datetime_geq: $since, edgeResponseStatus: 504 }
        orderBy: [count_DESC]
      ) {
        count
        dimensions { requestSource userAgent clientRequestPath }
      }
    }
  }
}

Grouping by user agent tells the same story. In September the Early Hints lookups showed up as bastion early hints and nginx-ssl early hints. In November they were fl2 early hints and nginx-ssl early hints (FL2 is the newer generation of Cloudflare’s proxy). Across 24 hours, all 2,631 of the zone’s 504s came from those user agents. None came from a browser, the app, curl or a search crawler.

A real 504 that a visitor actually saw almost always came from your side of Cloudflare. A proxy or load balancer in front of your application gave up waiting: nginx, a cloud load balancer, or a platform proxy waiting for a cold machine to boot. Cloudflare’s own timeouts use different codes. 522 means it couldn’t connect to your origin, and 524 means it connected but got no response within 125 seconds (the default proxy read timeout on plans below Enterprise).

So if you landed here searching for how to raise Cloudflare’s timeout because of 504s: the timeout behind a 504 is almost never Cloudflare’s. Check proxy_read_timeout in nginx, your load balancer’s idle timeout, or how long your platform takes to wake a machine. A useful clue: if requests fail at a round number well under 125 seconds, such as 30 or 60, it’s your own proxy’s limit, not Cloudflare’s. (Only Enterprise plans can raise Cloudflare’s 125-second limit, up to 6,000 seconds, and Cloudflare itself treats that as a stopgap.)

What those 504s actually are

#

On a zone with Early Hints enabled, every request for an HTML page or an extensionless URL (which, on an API-heavy zone, is most API calls) gets an extra internal lookup: Cloudflare checking its own cache for a set of hints to send. That lookup is logged as its own line, with ClientRequestSource = earlyHintsCache. It isn’t an end-user request, and it is never sent to your origin.

The status on that line is the result of the cache lookup, not an error:

• 2xx is a hit. Cloudflare had Link headers stored for that URL and sent them to the browser as a 103. On my zone, hits show up as both 200 and 204.
• 504 is a miss. Nothing was stored, so there was nothing to send early.

If your origin never sends the header Early Hints runs on, every lookup is a miss and every request writes a 504 line. In the three hours before I changed anything, the zone logged 173 Early Hints lookups and all 173 were misses. Since the feature had been switched on, it had not served a single hint.

One correction to my September post: I described these as probes that “get no response from the Fly origin”. That isn’t quite right. They never go to the origin at all. The 504 records a cache miss, not a timeout.

Cloudflare documents this, briefly, on its Early Hints page: an Early Hints lookup logged with ClientRequestSource: earlyHintsCache is “expected and benign”, a 200 is a cache hit and a 504 a miss, and Log Explorer hides them with ClientRequestSource != "earlyHintsCache". Cache Analytics already leaves them out. The HTTP traffic chart that scared me doesn’t.

These fake 504s talked me into a server upgrade

#

This is why the split matters. On 3 November, the day before I started on this post, Cloudflare logged 3,292 504s against about 13,300 visitor requests: an apparent error rate of 25%. The API runs on Fly.io’s shared-cpu-1x machines, which are burst-credited. They get a small guaranteed share of a CPU and spend credits to run faster, and a load test in August had shown them throttling to roughly 20 to 30 requests a second under sustained load. “A quarter of requests timing out at peak” looked exactly like that throttling, so I moved the API to shared-cpu-2x.

Every one of those 504s was an Early Hints cache miss. Here is what the requestSource split and the origin’s own logs showed:

• Every Cloudflare 5xx from 28 October to 4 November was an earlyHintsCache lookup, apart from about one 525 a day (the one I traced came from a security scanner).
• The origin’s access logs for 24 October to 4 November had no 5xx responses at all, and one request slower than 10 seconds.
• Fly’s metrics never showed more than about 7 requests a second, and the machine never used more than a fifth of one CPU core. It never got near the throttling ceiling.

The bigger machines fixed nothing, because nothing was broken. The API is back on shared-cpu-1x.

Don’t make capacity decisions from a 5xx chart you haven’t split by request source. Check it against your origin’s own logs as well: if Cloudflare reports a quarter of requests failing and your server logged no errors, find out what Cloudflare is counting before you change the server.

Updated 5 November 2026: the first version of this section said the API had really fallen over on shared-cpu-1x that week. It hadn’t. The numbers above are what I found when I checked.

Is HTTP 103 an error?

#

No. Status codes in the 1xx range are informational: interim responses a server can send before the real one. 103 Early Hints, defined in RFC 8297, carries Link headers so the browser can start fetching critical CSS, JavaScript and fonts, or open connections to other origins, while the server is still building the page. The final 200 follows on the same request.

Support is uneven but safe. Chromium-based browsers (Chrome since version 103, and Edge) and current Firefox act on both preload and preconnect hints. Safari (17 and later) uses only preconnect. Browsers process 103 only over HTTP/2 and HTTP/3. A browser that ignores it simply waits for the 200 as it always did, so a 103 is a head start for most visitors and a no-op for the rest.

Why mine was empty: the Link header vs <link rel=preload>

#

This is the part that cost me the most time, because it looks like it should already work.

Early Hints caches exactly one thing: the HTTP Link response header with a rel=preload or rel=preconnect relation. Cloudflare reads it from your normal responses (status 200, 301 or 302, on HTML or extensionless URLs), stores it, and replays it to the next visitor as a 103 before your full response is ready.

My pages already had preloads, but as tags in the HTML <head>:

<link rel="preload" href="/_expo/static/css/global-7d77bd.css" as="style">
<link rel="preload" href="/_expo/static/js/web/entry-fd7fa4.js" as="script">
<link rel="preload" href="/fonts/manrope-latin.woff2" as="font" type="font/woff2" crossorigin>

Those tags still do something: the browser acts on them. But it only sees them after it has received the HTML and started parsing it, which is exactly the moment Early Hints exists to beat. The Expo export and my Hugo theme both write these tags for you, and neither sets the header. So the feature was enabled, the preloads were declared, and the two never met.

The fix: send the header you already have the data for

#

The cleanest fix doesn’t involve maintaining a hand-written list of critical assets that goes stale every time a bundle hash changes. Each page already declares what it wants preloaded, in those tags. So mirror them into a Link header when you serve the page. The standard library’s html.parser copes with quoted and unquoted attributes (Hugo’s minifier strips the quotes), so there’s no regex to get wrong:

from html.parser import HTMLParser


class PreloadLinks(HTMLParser):
    def __init__(self):
        super().__init__()
        self.hints = []

    def handle_starttag(self, tag, attrs):
        a = dict(attrs)
        if tag != "link" or "preload" not in (a.get("rel") or "").split() or not a.get("href"):
            return
        hint = f"<{a['href']}>; rel=preload"
        if a.get("as"):
            hint += f"; as={a['as']}"
        if a.get("as") == "font" or "crossorigin" in a:
            hint += "; crossorigin"
        self.hints.append(hint)


def link_header(html: str, limit: int = 12) -> str | None:
    parser = PreloadLinks()
    parser.feed(html)
    return ", ".join(list(dict.fromkeys(parser.hints))[:limit]) or None

And on the HTML response, in Django:

response = HttpResponse(html, content_type="text/html")
if (link := link_header(html)) and "Link" not in response:
    response["Link"] = link

One detail bit me, and it’s worth keeping: keep crossorigin on fonts. A font preload without it is treated as a different request, so the font gets downloaded twice and the page gets slower, not faster.

You don’t write the 103 yourself. There’s no ASGI, Granian or uvicorn interim-response code to add. Cloudflare builds the 103 from your header; your only job is to send the header.

On other stacks

#

The rule is the same everywhere: put a Link header on the HTML response and let Cloudflare do the rest.

• nginx, on the location that serves your HTML:

  location = / {
      add_header Link "</css/app.css>; rel=preload; as=style" always;
      add_header Link "</fonts/inter.woff2>; rel=preload; as=font; crossorigin" always;
  }

• Laravel with Vite already has middleware for this, AddLinkHeadersForPreloadedAssets, which turns Vite’s preloads into Link headers. Two issues reported against it are the same traps as above: its modulepreload hints have gone out without crossorigin, so browsers can discard them and download the script twice, and on large pages the header has grown past Apache’s roughly 8 KB header limit. Check both on your version.

• Anything else: set one Link header, comma-separated, or several Link headers. Either works.

What to hint is where people overdo it. Hint the render-critical CSS, the entry script and the one or two fonts above the fold, not every chunk. A long list makes the browser fetch things it doesn’t need yet, ahead of things it does. I cap mine at twelve, and only because the Expo export already curates its preload list.

If you’re not behind Cloudflare, or your CDN doesn’t do Early Hints, the Link header still helps a little: the browser can start those fetches as soon as the response headers arrive, before it parses the body. But no 103 is sent unless your server or CDN emits one.

How to check it’s working

#

curl shows the 103 directly. Ask for HTTP/2 and keep only the status lines and Link headers:

$ curl -sv --http2 -o /dev/null https://example.com/register 2>&1 | grep -E '^< (HTTP|link)'
< HTTP/2 103
< link: </_expo/static/css/reset-f5d396.css>; as=style; rel=preload, …
< HTTP/2 200
< link: </_expo/static/css/reset-f5d396.css>; rel=preload; as=style, …

The 103 arrives first, then the 200. Cloudflare reorders the attributes in the hint, which is harmless. Don’t expect every request to get a 103. Two of my first three did. The first visit after a change only stores the header, and Cloudflare skips the 103 when the final response headers are already available.

Then go back to the analytics query above and filter on requestSource: "earlyHintsCache", grouped by status and path. On the pages you fixed, misses should give way to hits.

What changed, and the caveat

#

Minutes after the header went live, the zone logged its first Early Hints hits ever, and curl showed real 103 responses on the web pages. One header fixed both problems: the feature started working, and a slice of the “504s” went away.

The caveat is that a JSON API has nothing to preload. The API endpoints keep missing and keep writing 504 lines, and on an API-heavy zone that’s still most of the volume. You have three sensible options. Send the header on the pages that benefit and accept that API paths keep logging misses; the feature works, and the charts are quieter but not silent. Exclude earlyHintsCache from your 5xx alerting and dashboards whatever else you decide, because an internal cache lookup should never page anyone. Or turn Early Hints off (Speed → Settings → Content Optimization in the dashboard, or the zone’s early_hints setting in the API) if you’ll never send Link headers; a single-page app whose HTML is tiny may get little from it, and then every lookup stops and the noise goes with it.

I kept it on. The web pages now get a head start, the API’s misses are filtered out of everything that alerts, and that’s what I should have done in September instead of switching the feature off.

There’s no hosted APM, no tracing in production and no dashboard I have to remember to open, and the only part that costs extra is the log archive, at about US$2 a month. Here’s the whole list, and the things I dropped or stopped trusting, one of which had me chasing an outage no user ever saw.

What I mean by monitoring, and how small this is

#

By DevOps monitoring I mean finding out that something’s broken before somebody tells me, and being able to work out why without guessing. Every tool below is judged on those two things.

Last year I wrote about setting up monitoring that developers actually use, mostly with teams in mind: a cloud provider’s built-in monitoring for a small team, a hosted APM once there are hundreds of services. This is the small end of the same idea. Both products are Django on Fly’s shared-cpu-1x machines, as described in My Current Stack, and I’m the only developer on either. If monitoring is a product and developers are its users, mine has a user base of one. A baseline I pulled from Fly’s metrics in September had the marketplace’s API averaging 1% CPU. This isn’t an enterprise stack, and I won’t dress it up as one.

Of the three pillars, metrics, logs and traces, I have two. Neither product has distributed tracing: each is one Django codebase and one database, with no service boundary for a trace to cross. The only traces I read are the reverse proxy’s, on my workstation.

The list

#

Everything in the table is the marketplace unless it says both apps. The estimating app has less: Fly’s health check, logs and metrics, app analytics on iOS, and a daily Parquet export of its database that I query with DuckDB. It has no crash reporting and no alerts channel, which is the biggest gap here. (A disclosure for the update-service row: the company behind it pays me to write for their blog.)

Errors: a logging handler posting to chat

#

Until September, flyctl logs was the only place an unhandled 500, a failed push notification or a broken WebSocket frame showed up, and nobody reads that stream unprompted. Now a logging handler sends every record at ERROR or above to a chat channel, from the API, the WebSocket server and the background worker. Warnings deliberately don’t alert.

The handler posts on a background thread, so it never slows a request. Nothing in it logs, so it can’t recurse. And the same error goes out at most once every ten minutes, with a cap of 30 an hour, so an outage reads as one message rather than a thousand.

The channel also carries what I want to read errors against: deploys post when they start, when they’re up and where they failed, with the commit and who ran it, and so do over-the-air updates and web releases. A burst of wrong sign-in codes on one account logs at ERROR too, so someone guessing shows up next to the outages.

In October an alert arrived that matched no line in any log. Now each alert carries a short random id, the hostname and the process’s start command, and writes the same line to that process’s log, so an id I can’t find means a process that doesn’t log there, like an SSH shell. Everything in the channel also goes into a feed that a coding agent I work with can read, so the agent working on the code sees the errors when I do.

Logs: Fly’s shipper to S3, queried with DuckDB

#

Fly’s own buffer holds a few minutes of output, roughly the last hundred lines, which is fine for watching a deploy and no use for yesterday. So a third Fly app runs Fly’s log shipper, which is Vector subscribed to the organisation’s log stream, with one change to its S3 sink:

[sinks.aws_s3]
  type = "aws_s3"
  bucket = "myapp-logs"
  compression = "gzip"
  key_prefix = "logs/{{fly.app.name}}/%F/"   # one folder per app per day
  batch.timeout_secs = 300                  # a quiet app still lands its lines

Every line from the marketplace’s apps lands in an S3-compatible bucket as gzipped JSON. The shipper is one shared-cpu-1x machine with 256 MB, about US$2 a month plus pennies of storage. DuckDB reads the files where they sit through one macro, and a day comes back in about a second:

CREATE MACRO logs(app, day := '*') AS TABLE
  SELECT * FROM read_json_auto('s3://myapp-logs/logs/' || app || '/' || day || '/*.log.gz',
                               union_by_name := true);

SELECT timestamp, message FROM logs('myapp-api', '2026-09-06') WHERE message LIKE '% 500 %';

That macro is how I measured the cold starts in the scale-to-zero post: the startup script announces each phase, every line carries the machine’s id, and the rest is a GROUP BY.

So it’s three levels of one stream, by who reads them: the archive keeps INFO and up, which is why the apps log at INFO in production, a live-tail script shows warnings and errors, and the chat channel gets ERROR. None of them knows the caller, though. The access log has no client IP or user agent, and for that I go to Cloudflare.

Metrics: Fly’s built-in Prometheus

#

Fly runs a managed Prometheus for each organisation, so I don’t run a metrics stack of my own. I query its API when I have a question: memory, CPU and burst credit per machine (fly_instance_cpu_balance matters on a shared CPU), responses and concurrency per app. Two things I wish I’d known first. A token from fly tokens create goes in as Authorization: FlyV1 <token>; Bearer only works with the personal token from flyctl auth token, and a read-only token gets “something went wrong resolving organization”. And it keeps about two weeks (Fly says roughly 15 days).

It earned its place in September, when the logs had “Could not open a database connection” several times a day, which reads exactly like an app bug. The managed Postgres cluster is in the same Prometheus, and at every one of those minutes the primary’s scrape time jumped from about 80 milliseconds to between one and ten seconds, while the replica was fine and Postgres’s start time never moved. Postgres hadn’t restarted; the host under it was stalling. I sent Fly’s support the timestamps, and after a maintenance restart in early October moved the primary to another host, the daily stalls stopped.

Crash reports: Sentry’s SDK, pointed at my own API

#

A crash on a phone used to leave no trace anywhere I could see. The obvious answer was Sentry, and I took half of it: the app carries Sentry’s open-source SDK, but its DSN points at an endpoint in my own Django API that accepts Sentry’s envelope format, groups the crashes, and alerts through the same chat handler. Tracing, sessions and replays are off. It’s errors only.

I didn’t sign up for sentry.io because, at a few reports a day, it would have been another vendor holding user data, another line in the privacy policy and another bill. The price is symbolication, which I haven’t replaced. Stacks are minified JavaScript or raw native addresses, so an issue tells me what crashed, on which release and device, and what the person did just before, but not where in the source.

The only dashboard is a message

#

I haven’t built a dashboard for either app. Last year’s post had a test for them: if nobody opens one unprompted, it isn’t earning its place. A message that arrives on its own doesn’t need opening, so the closest thing I have is a post in the team chat every morning.

It starts with the last 24 hours of production errors, read from the log archive rather than from the chat channel, because throttled alerts can’t say how often something happened. Numbers, ids and addresses are masked before grouping, so a hundred copies of one failure are one row with a count and no email reaches the channel.

Then failed health checks, yesterday’s usage against the same weekday a week earlier (a Sunday is never like a Monday), the last week of search traffic, chats waiting on a reply, the task board and what I’ve spent. A section that can’t load says so, and the post still goes out.

Reading app analytics through an API teaches you to distrust them a little. In August, 121 Android first-opens from Azerbaijan were followed by 119 uninstalls, and I read that month’s Android totals with them taken out.

Releases have their own screen. When over-the-air updates came from my own server, my view of a release was an admin list with an “is active” checkbox, and I had no idea how many phones had taken one. The hosted service’s dashboard shows which release devices run, how installs went, and a crash rollback when a bad bundle crashes and the SDK restores the previous one. I wrote up the move to a hosted update service separately.

What I dropped, or stopped believing

#

None of this was a paid tool. Mostly it was things I’d been treating as signals that weren’t, and one job running in the wrong place.

Cloudflare’s 504 rate, until I grouped it by user agent

#

In September the Cloudflare dashboard showed what looked like 22% of a day’s requests failing with 504s, concentrated on the endpoints the app polls every 30 seconds, while every health check was green and the app’s own logs said those requests were fast. I spent days on it, found that machines took 25 to 33 seconds to cold-start and did it 92 times a day, fixed that, and wrote it up in the scale-to-zero post as the cause.

The next morning I grouped the same 504s by user agent. All 7,344 of them between 5 and 9 September came from Cloudflare’s own Early Hints cache lookups (user agents bastion early hints and nginx-ssl early hints), which never reach the origin: a 504 on one of those lines is a cache miss, not a timeout. They copy real request paths, so they landed on the busiest endpoints and looked exactly like users failing. The app, browsers, curl and search crawlers had none. Early Hints wasn’t doing anything for me because the origin sent no Link header, so I turned it off, and later made it work.

The cold-start work still paid off, since deploys and wakes are faster for it, but the outage I thought I was fixing never reached a user.

So this is the rule I’d most like to pass on: before you believe a CDN’s error rate, group it by user agent and check it isn’t the CDN’s own machinery. On Cloudflare’s free plan that’s a GraphQL query on httpRequestsAdaptiveGroups, a day at a time. The same query settled the next scare, when one polling endpoint went from 70–260 requests a day to 2,880. It wasn’t a scraper. It was my own Mac, with the app left running in the iOS simulator after testing.

A green health check

#

Fly checks /api/health/ on both apps every 30 seconds and uses the answer to decide which machines get traffic, which it’s good at. But a health check runs against a machine that’s already up, so it can’t see what happens before one is: a cold start, a queue at the proxy, a request that never arrives. Routing is all I ask of it now.

Deploys get the probe script instead, which hits the origin throughout and exits 1 if anything failed. It caught a plain flyctl deploy leaving a 7.3-second gap, then counted no failures in 2,514 requests through the deploy script, which wakes a spare machine first.

The health check has also caused two outages of its own. In July, Django’s SECURE_SSL_REDIRECT answered Fly’s plain-HTTP probe with a 301, every machine failed its check, and everyone got a 503. In August, narrowing an except Exception in the health view let a “connection already closed” error escape as a 500, and Fly pulled the machine in exactly the case the view’s retry was there for. Now the check only asks whether the process can serve, and a database stall no longer takes both apps out of routing.

The morning summary had to learn the same lesson about counting. The marketplace suspends idle machines, so a machine woken by a request fails its first check a few hundred milliseconds after starting and passes the next. Counting those made every morning read like an outage, so a failure that passes again within a minute is now listed separately, as machines waking up and deploys.

A scheduled job in CI

#

The morning summary started as a CI workflow on a cron schedule and moved within a day. Once it read the analytics and search data, the key it needed was the analytics project’s service-account key, which can do anything in that project, and I didn’t want that sitting in CI secrets. Now a scheduler on my Mac runs it every 20 minutes, and it posts once it’s past eight in Auckland and today’s post hasn’t gone out, so a Mac asleep at eight posts when it wakes. The workflow stays as a manual fallback, minus the analytics and search sections.

If I were starting again

#

I’d set up three things before the first user rather than after the first incident: errors to a chat channel the team already reads, with a rate limit; a log archive in object storage I can query with SQL; and whatever metrics the platform already collects. That’s a couple of dollars a month. Crash reporting comes the week an app goes onto phones, because otherwise a crash on a phone tells nobody anything. A morning summary comes once there are enough places to check that I’d stop checking them.

I’d leave tracing until there’s a second service for a request to cross, and a hosted APM until there’s a team that needs shared dashboards, an on-call rota and a budget for someone else’s bill. That’s when the tools in last year’s post start paying for themselves. Before then the most useful habit is free: when a number looks alarming, find out what produced it before deciding what broke. Several of the scares in this post came from the monitoring, not the product.

There’s a gap between “works on my machine” and “works in production”, and monitoring is supposed to bridge it. Most of the time it doesn’t, because it gets built as an ops tool rather than something a developer reaches for when things break.

I’ve just gone freelance doing DevOps and infrastructure work, and the monitoring setup is one of the first things I look at anywhere I land — it tells you more about a team than the README does. The failure mode is nearly always the same: the tools exist, but they don’t connect to how developers actually work. Someone burns three hours on a bug that production logs would’ve solved in five minutes. Failures arrive as support tickets instead of alerts. Or the opposite: 200 alerts a day, most of them noise, so everyone ignores all of them, and the first half hour of every incident is spent working out what broke rather than fixing it.

This is the general version. For the specifics, I’ve since written up what I actually run today on two small products.

Metrics, logs, and traces — linked, or useless

#

Metrics tell you something is wrong (error rate spiked, latency jumped). Logs tell you what happened (stack traces, request payloads, error messages). Traces tell you where (which service, which endpoint, which database call). None of that is news. What matters is whether they’re linked: when an alert fires, you should be able to click from the metric to the relevant logs to the trace. If developers are manually correlating timestamps across three tools, the setup has failed, no matter how good each tool looks on paper.

Dashboards deserve the same test. CPU, memory and disk I/O matter for capacity planning, but they don’t help anyone debug a 500. The dashboards people open voluntarily show business metrics next to technical ones (signups per hour beside error rate), recent deployments overlaid on the timeline, the top five errors in the last hour with links to logs, and latency percentiles — p50, p95, p99 — rather than averages. If nobody opens a dashboard unprompted, it isn’t earning its place.

Alerts, same idea. Every alert should answer two questions: what’s broken, and where do I start looking?

Bad alert: “High CPU on web-server-3”

Good alert: “Error rate > 5% on /api/payments since 14:32. Last deploy: 14:15 by @sarah. [View logs] [View trace]”

A few things push alerts toward the second kind: alert on deviation from a baseline rather than an arbitrary threshold, route by ownership (payment errors go to the payments team, not the whole org), and group related errors into one Slack message instead of 50. Keep the loop tight, too — the time between “something broke” and “a developer knows” should be under a minute, which means real-time log streaming rather than five-minute batches, deploy markers on every dashboard, and tracing that survives service boundaries.

Build it for the people who’ll use it

#

The biggest mistake ops teams make is building monitoring for themselves. The fix isn’t sophisticated. Sit with a developer during a debugging session and watch where they get stuck. Ask what questions come up mid-incident — “which service?”, “what changed?”, “what did the request look like?”. Find out which metrics actually matter for the product; it’s rarely CPU, and usually things like checkout completion rate or upload success rate.

Then remove friction. If instrumenting a new service takes a day, it won’t happen. Provide libraries that auto-instrument the frameworks you already use (Django, Express, Spring), copy-paste templates for dashboards and alerts, and self-service metrics so nobody has to file a ticket. And treat the whole thing as code: config in version control, alert rules tested so you know they fire when they should, and multi-region considered early if that’s where you’re headed.

The same habits keep it alive afterwards. Instrumentation goes through code review like any other production code, and a custom metric gets a test. Every postmortem asks what monitoring would have caught this sooner, and the answer goes into the dashboards and alerts that week rather than into the write-up. And a few times a year I go through what’s there and delete what nobody opens, because an alert nobody acts on is training people to ignore the next one.

Small team on AWS? Start with CloudWatch

#

If you’re under 50 engineers and on AWS, CloudWatch is the right starting point, and I’d resist buying anything fancier on day one. EC2, Lambda, RDS and ECS all report to it automatically, so you get visibility without writing instrumentation. It runs $10–50 a month for a small team with no platform fee, and metrics, logs, traces (via X-Ray) and alarms all live in one place. You can have meaningful alerts and dashboards up in an afternoon, not weeks.

CloudWatch Logs Insights is the underrated bit — you can query logs without standing up Elasticsearch:

fields @timestamp, @message
| filter @message like /ERROR/
| stats count() by bin(5m)

Beyond that:

• Composite alarms cut noise — alert when error rate is high and response time is degraded, not on each condition separately.
• Custom metrics are where the real value is. Push business metrics (signups, transactions, feature usage) through the CloudWatch SDK alongside the infrastructure ones.
• CloudWatch Synthetics runs scheduled canaries through real user journeys, so you learn the checkout is broken before your users do.
• X-Ray gives you distributed tracing with minimal setup — good enough for most microservice architectures.

You’ll know when you’ve outgrown it: you go multi-cloud, you want proper anomaly detection, dashboard customisation starts to hurt, or the team passes 50 engineers and needs real collaboration features.

Bigger teams: Datadog

#

Datadog is expensive — $20–100K+ a year for a large org is normal — but at scale it earns its keep. One view across AWS, Azure, GCP, on-prem, containers, serverless, databases and frontend. Watchdog flags anomalies without hand-tuned thresholds, which matters because nobody can manually watch thousands of services. And you get the collaboration layer CloudWatch lacks: team dashboards, RBAC, shared investigation notebooks, PagerDuty/Opsgenie integration, multi-condition alerts, threshold forecasting, maintenance windows, and an APM that goes down to code-level profiling with auto-generated service maps.

Rolling it out, in roughly the order I’d do it: instrument critical services first and tag everything by team and environment from day one. Agree naming conventions, dashboard templates, severity levels and SLO definitions early — retrofitting standards is miserable. Wire it into CI/CD deploy markers, incident management and Slack. Budget time for training, because Datadog is powerful but not self-explanatory. And watch the bill: filter noisy metrics, sample traces on high-volume services, and audit which features you’re actually paying for.

Worth a look before committing: New Relic (similar, sometimes cheaper for high-volume tracing), Dynatrace (strong AIOps, popular in financial services), Splunk (best-in-class log analysis, especially if security already runs it), and Grafana Cloud (the natural home if you’re already on Prometheus/Loki).

In practice a lot of teams land on a hybrid: CloudWatch for AWS-native services because it’s automatic and cheap, Datadog for the application layer, with CloudWatch metrics ingested into Datadog for one unified view. It’s unglamorous, and it works.

CloudWatch vs Datadog at a glance

#

Starting from scratch: the build order

#

The order I’d do things in:

1. Talk to developers about what hurts during incidents.
2. Define SLOs for the flows that matter. What does “working” mean for login, checkout, search?
3. Instrument that critical path first.
4. Add distributed tracing next; it has the best debugging ROI in a microservice setup.
5. Alert on those flows rather than on CPU, and send each alert to the team that owns it.
6. Write a runbook for every alert so 3am-you knows what to check.
7. Put a quarterly review in the calendar to delete stale alerts, fix drifted thresholds, and confirm the dashboards still match the architecture.

And the traps I keep pulling teams out of: too many tools (three that work together beat six that don’t), graphs with no baseline (is 500 rps normal or a 10x spike?), monitoring things users don’t feel (disk I/O on a stateless container), and no redundancy for the monitoring itself — if your alerting dies during an incident, you’re blind exactly when you can least afford it.

None of this needs the fanciest tooling. It needs developers to get the information they need, in the place they’ll actually look, fast.