Search traffic to this blog has been flat for a couple of years and I’ve stopped expecting it to move. So I went looking at where visitors come from instead, and whether the AI assistants are among them. They are, barely. In the twelve months to 16 September 2026, my analytics counted 5,372 sessions here. Twenty-seven of them came from an AI assistant: twelve from chatgpt.com, nine from gemini.google.com, six from Perplexity, and none from claude.ai. Half a percent. The ChatGPT visitors were the good kind, though. They stayed five minutes on average, against 27 seconds for direct traffic and three minutes for Google, because they’d already read the answer and wanted the detail.

So I’ve spent some time on how a small blog gets picked as the source for those answers. This post is what I’ve done on this site and what the logs say about it. Where I have numbers I give them, and where I’m guessing I say so. Three sentences in the first draft turned out to be wrong once I looked, and I’ve left the corrections in.

Two kinds of bot, and only one of them cites you

#

The first thing to get straight is that “AI bots” is two different jobs with different user agents, and blocking one does nothing to the other.

Training crawlers pull pages in bulk to build the next model. OpenAI’s is GPTBot, Anthropic’s is ClaudeBot, Google’s is controlled by the Google-Extended token, Apple’s by Applebot-Extended, and Common Crawl’s CCBot feeds most of the open datasets. Whatever they take ends up as weights, with no link back to you. There’s a case for blocking these and I don’t think it’s a silly one.

Retrieval bots are the other kind. When someone asks ChatGPT or Claude or Perplexity a question, the assistant runs a search, fetches a handful of pages right then, reads them, and writes an answer with citations. Those fetches arrive as OAI-SearchBot and ChatGPT-User, Claude-SearchBot and Claude-User, PerplexityBot and Perplexity-User. Google’s AI Overviews use plain Googlebot, so there’s no separate switch for them at all. These are the bots that send a visitor back. If the aim is to be cited, these are the ones you’re optimising for, and it’s worth knowing that blocking GPTBot does nothing to ChatGPT’s search results in either direction.

This site’s robots.txt is Hugo’s default, which is allow everything:

User-agent: *
Allow: /
Sitemap: https://jared.lynskey.co.nz/sitemap.xml

I’ve thought about splitting it, letting the retrieval bots in and keeping the training ones out, and decided it wasn’t worth doing for a blog whose whole purpose is to be read. If I did want that, it would look like the block below. The reason to show it is the same lesson as in Cloudflare Served Googlebot an Empty Page: the list has to name bots, not vendors. There is no line you can write that says “OpenAI”.

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
Disallow: /

User-agent: *
Allow: /

One thing to check if the site sits behind Cloudflare. Since the middle of 2025, new zones there block AI crawlers by default, and there’s a one-click toggle under the bot settings that does the same to existing ones. It blocks both kinds. This blog moved to Cloudflare Pages last week, so that was on my list. If you turned it on because “AI bots” sounded like something to keep out, you’ve also turned off the citations. Cloudflare has since started writing a Content-Signal line into the robots.txt files it manages, with separate search, ai-input and ai-train flags. Nobody is obliged to honour it, but it’s the right vocabulary: the same split as above, expressed as a preference rather than a wall.

What actually knocks on the door

#

The move to Cloudflare had a side effect: for the first time I can see every request by user agent. Here is what came in over the four days from 13 to 16 September 2026, after throwing out the fakes, which I’ll get to.

Two things stand out. The training crawlers read everything, twice, in the time it took the retrieval bots to read a few dozen pages. And the retrieval bots are not fetching at random: ChatGPT-User and Claude-User both went for the update server post, which is the most specific thing on the site, and the Japanese cold-start post got fetched in Japanese.

Now the fakes. Of 20 requests claiming to be GPTBot, 18 came from addresses outside OpenAI’s published ranges and asked for /.env, /.git/HEAD and /firebase-adminsdk.json. Fifteen of 36 “ChatGPT-User” requests were the same kind of thing. Most of the “PerplexityBot” traffic wanted /id_ed25519. A user agent is a string anyone can send, and vulnerability scanners have noticed that AI bot names get waved through. OpenAI, Anthropic and Perplexity all publish their IP ranges. If you’re going to treat a bot specially, in either direction, check the address, not the name. The 20, 59 and 8 above are the requests that came from the right addresses.

The page has to be readable without a browser

#

Retrieval bots fetch at answer time, with a timeout, and most of them don’t run JavaScript. A page that is an empty div until a bundle loads is a page they don’t read. Hugo writes finished HTML, so this blog gets that for free, and it’s the main reason I’ve never been tempted to move it to anything cleverer. The marketplace I run is the opposite case, a React Native web app, and getting crawlers a real document there took a dynamic-rendering branch and then a caching bug that quietly undid it. The Cloudflare post above is that story.

Speed counts the same way, and here the logs corrected me. I’d assumed a static site behind Cloudflare was served from the nearest edge. It isn’t. This blog is on Cloudflare Pages, the HTML goes out uncached, and from Seoul the response comes back from Los Angeles in about half a second. That’s still fine for a bot with a timeout of several seconds, but it isn’t the tens of milliseconds I’d have claimed, and I only know because I measured it. An assistant assembling an answer from six pages will drop the one that takes three seconds, and it doesn’t tell you it did.

Write so a paragraph can be lifted out

#

The way these systems use a page is not the way a reader does. They split it into chunks, score each chunk against the question, and quote the ones that score. A post that builds up to its point over eight hundred words scores badly on every chunk. A post where each section states its claim in the first sentence and then supports it scores on the section that matches.

That changed how I write here more than any technical thing did. The description in the front matter is now a summary of what the post says, not a hook to get you to read it, because the description is often the only part an assistant shows. Section headings say what the section concludes. Numbers go in the sentence with their units rather than in a chart. Dates are written out, because “last week” means nothing to a model reading the page in a year. And a post is about one thing.

The other half is having something worth quoting, and here the data is mixed. In the logs, the pages the answer-time bots fetch are the specific ones: cold starts, the update server, iOS debugging. Naver’s crawler is called Yeti and Daum’s is Daumoa, a restart on my host is a thirty-second window for 504s, Cloudflare’s on-the-fly Brotli measured worse than gzip on my bundle. Those facts aren’t written down in many places, so a question that needs one of them lands here. But the people who click through from an assistant mostly land on the 2023 and 2024 posts about management, one or two sessions each. The specific posts get read by the machine. The general ones get the click. I don’t have a tidy explanation for that and I’d rather report it than invent one. The lever, if there is one, is still to write down the specific thing you found out this week that you couldn’t find when you searched for it, because that’s what the machines are demonstrably reading.

Say who wrote it, and keep saying the same thing

#

An assistant deciding whether to cite a page is doing something like what a careful reader does: who wrote this, when, and does it agree with everything else I’ve found? The mechanics of that on this site are mostly the theme’s doing. Every post carries Article structured data with an author, a published date and a modified date, and the author is a Person with links to the same public profiles. That’s a few lines in a template, and it’s the difference between “a page” and “a page by a named person with a history”.

The part that isn’t the theme’s doing is consistency. The about page, the llms.txt, my public profiles and the line at the top of every post all describe the same person with the same job history. When something changes, all of them change, or the one I forgot becomes the one that’s wrong. That sounds fussy, but one page saying I’m an engineering manager and another saying I’m freelance is exactly the kind of contradiction that makes a model hedge, or pick someone else.

The modified date has to be honest too. It changes when the content changes and I don’t touch it otherwise. Bumping it to look fresh is the kind of thing that works once and then gets your whole domain discounted.

What llms.txt is, and the fact that nothing reads it

#

llms.txt is a proposal from September 2024 for a Markdown file at the site root that gives a language model a summary of the site and a curated list of links. I added one in February this year. It’s a paragraph about who I am, then sections for management, infrastructure, mobile and community, with one line per post: the title, the URL, and a sentence saying what the post concludes. It reads like a table of contents written for someone in a hurry.

The caveat is that nothing reads it. In the four days of logs I have, the only requests for /llms.txt were my own. Common Crawl has captured 76 pages of this site in its August 2026 crawl and never that file. No major assistant has said it reads it. So it is not why anything here gets cited, and I’d stop short of recommending it to anyone on those grounds. I keep it for two reasons. It cost an hour, and writing one sentence per post that says what the post actually concludes was worth more than the file, because it showed me which posts didn’t conclude anything. It’s also the one page on the site that tells a machine what the site is for, and I’d rather have that and not need it. It has drifted since, because I write it by hand and forget, which is the argument for generating it from the front matter as a Hugo output format instead. That’s on the list.

Feeds, sitemaps and five languages

#

The boring plumbing still counts. The sitemap is how a crawler finds a new post the day it goes up. RSS is how it learns a post changed. Both are Hugo defaults and I’ve never had to think about them. ClaudeBot read the sitemap 36 times in two days, which is how it found 1,380 pages so quickly.

The languages are the part I didn’t expect to matter. Everything here is in English, Japanese, Korean, Simplified Chinese and Vietnamese, with alternate links so a crawler knows they’re the same post. The crawlers treat them as five sites: ClaudeBot’s crawl split almost evenly across the five, and of the 73 post pages Common Crawl took in August, only four were the English ones. Assistants answer in the language they’re asked in and lean towards sources in that language, so a Korean question about cold starts can be answered from the Korean page rather than from a translation of the English one done on the fly, and the Japanese page is the one ChatGPT actually fetched. The condition is that the translation reads as if it were written in that language. The pass I did on that earlier this year, with native speakers checking, was for human readers, but it’s also what makes the Vietnamese page a source rather than a curiosity.

What I can measure

#

More than I thought, once I stopped guessing. Cloudflare’s free plan gives request logs by user agent and path, one day per query, which is where the bot table came from; referrers need the paid plan. The analytics tool has the referrers, and pulling them through its reporting API rather than squinting at a dashboard is what produced the numbers at the top. Together they show the shape of it. In four days the retrieval bots fetched about ninety pages from OpenAI and Anthropic addresses. In the last month, two people arrived from ChatGPT. Read a lot, cited sometimes, clicked rarely. There’s still no way I know of to see the answers that used a page and didn’t get clicked. The most useful check is still the manual one: ask the assistant the question your post answers and see whose page it quotes.

If I had to keep one thing from all of this, it’s the writing rule. The robots.txt, the structured data and the llms.txt are an afternoon’s work and then they’re done. Writing so each section says its claim first, and writing about the specific thing rather than the general one, is a habit, and it’s the habit that decides whether a page is worth quoting at all. The second thing is to measure before you claim. Being trusted by a machine turns out to look a lot like being trusted by a person: say who you are, say what you know, and say it the same way every time.

My marketplace app is a single-page app built with React Native Web. Every listing page is one script tag and an empty div until JavaScript runs. That is fine for people and fatal for search: Google can sometimes execute the JS, Bing will not, and Naver’s crawler — the one that matters most when nearly all your listings are in Korean — certainly will not.

This is the story of making those pages visible, and then of the caching rule that undid it in a way I did not see coming.

Dynamic rendering, which is not cloaking

#

The fix is old and Google-sanctioned: detect a crawler, and serve it server-rendered HTML containing the same content the SPA would have painted. Not different content. The same content, earlier.

So a listing URL like /listings/view?id=... branches. A browser gets the SPA shell. A crawler gets a real document: a heading, the listing’s key facts, an image per photo with alt text, a body excerpt, and links to a handful of related listings so the page is not an orphan.

That last part mattered more than I expected. Before those internal links existed, roughly a thousand listing pages sat with nothing pointing at them, and Search Console showed the crawl settling to seven to eleven requests a day. A crawler that cannot find your pages does not care how good they are.

The trap: a vendor’s name is not their bot’s name

#

Crawler detection is a user-agent match, and here is where it bit.

Korean platforms crawl under names that look nothing like the company. Naver’s crawler is Yeti. Daum’s is Daumoa. Kakao’s link scraper is kakaotalk-scrap. None of them is the vendor name.

Meanwhile those same vendors ship apps with embedded browsers, and those send a real browser user-agent with the app’s name appended. KakaoTalk’s in-app browser says KAKAOTALK/10.5.1. Kakao’s scraper says kakaotalk-scrap/1.0.

Match on kakao and you match both. Which is exactly what I did, and so every KakaoTalk, Naver-app and Daum-app user who opened a link got the crawler page: no JavaScript, no navigation, no login. A perfectly good document, and completely useless to a human.

The fix is an ordering rule, and it is worth stating as one:

import re

# In-app browsers append the app's name to an ordinary browser UA.
INAPP_BROWSER_RE = re.compile(r"kakaotalk/|naver\(inapp|daumapps", re.I)

# Name the bot, never the vendor: Naver crawls as Yeti, Daum as Daumoa,
# Kakao's link-preview scraper as kakaotalk-scrap.
CRAWLER_RE = re.compile(r"googlebot|bingbot|yeti/|daumoa|kakaotalk-scrap", re.I)


def is_crawler(user_agent: str) -> bool:
    # An in-app browser is never a crawler, so that check comes first.
    if not user_agent or INAPP_BROWSER_RE.search(user_agent):
        return False
    return bool(CRAWLER_RE.search(user_agent))

In-app browsers are checked first and win. And every crawler token has to name a bot, never a vendor.

Then I put Cloudflare in front of it

#

The origin is a single shared-cpu machine in Sydney that sustains twenty to thirty requests a second. Googlebot has about 995 URLs to walk. Every one of them was reaching Sydney, because cf-cache-status read DYNAMIC on all of them: Cloudflare does not cache HTML unless a rule tells it to. About 16% of the zone was cached, and none of the part that mattered.

So: a cache rule on the crawler pages. Straightforward. Except that a listing URL does not have one body. It has three.

• A crawler asking for HTML gets the server-rendered page.
• A human gets the SPA shell.
• An agent sending Accept: text/markdown gets markdown, built from the database, because converting an empty HTML shell would produce an empty document.

The origin says Vary: Accept and Vary on the user-agent would be hopeless anyway. And here is the thing I did not know:

Cloudflare ignores Vary on HTML. All three bodies share one cache key.

Whichever body arrives first wins that URL for every later visitor in that edge location, for the hour the origin’s s-maxage allows, plus a day of stale-while-revalidate.

I got to observe this in production in both directions:

• A Naver-app user warmed a listing URL with the crawler page, and plain iPhone Safari then got that page on a HIT. A human staring at a JavaScript-less document.
• A human warmed a URL with the SPA shell, and Googlebot got the empty shell. The exact failure the whole SEO effort existed to prevent, now served faster and from the edge.

The rule that fixes it by narrowing

#

You cannot make Cloudflare respect Vary on HTML. What you can do is make sure only one of those three bodies is ever eligible to enter the cache:

(starts_with(http.request.uri.path, "/listings/")
 and not any(http.request.headers["accept"][*] contains "text/markdown")
 and (lower(http.user_agent) contains "googlebot" or ...)
 and not (lower(http.user_agent) contains "kakaotalk/" or ...))

Match crawler user-agents only, and exclude markdown requests. Now a human request matches no rule at all, so it reads DYNAMIC and goes to the origin as before. Only crawler requests read or write the cache, and the only body that can ever be in there is the crawler one.

Edge TTL is respect_origin, so the lifetime stays owned by the code that renders the page rather than being pinned in a dashboard.

One operational detail that is easy to miss: narrowing a rule does not evict what is already in the cache. You have to purge, or you keep serving the poisoned entries you just stopped creating.

The two lists, and why their errors are not symmetric

#

The rule needs a list of crawler user-agents, and Django already has one. Two copies of the same knowledge in two systems is usually a smell, but here the interesting part is what happens when they drift, because it is not symmetric:

• The bot list in Cloudflare must stay a subset of the one in Django. A token Cloudflare thinks is a bot but Django does not would cache an SPA shell under the crawler-only key, and then serve that to a real crawler.
• The in-app browser exclusion list must stay a superset of Django’s. Excluding too much only costs you a cache hit.

Get either one wrong in the safe direction and you lose some caching. Get the first one wrong in the other direction and you are back to serving Googlebot an empty page. That asymmetry is why the lists use contains rather than a regex, incidentally: matches needs a Business plan, and contains is expressive enough for a list that is allowed to be approximate in a known-safe direction.

An invariant that fails the build instead of the page

#

The crawler pages embed presigned image URLs, valid for seven days. The pages themselves are cached at the edge for an hour, plus a day of stale-while-revalidate.

Those two numbers are related, and if the cache lifetime ever exceeds the image lifetime, the edge starts serving pages whose every photo 403s. Nothing would alert on it; the HTML is fine.

So the code that defines those numbers refuses to import if the relationship breaks:

CRAWLER_IMAGE_TTL = 7 * 24 * 3600  # presigned image URLs
EDGE_S_MAXAGE = 3600
EDGE_STALE_WHILE_REVALIDATE = 24 * 3600

# Fail at import time if the edge cache could outlive the image URLs.
if EDGE_S_MAXAGE + EDGE_STALE_WHILE_REVALIDATE >= CRAWLER_IMAGE_TTL:
    raise RuntimeError("edge cache lifetime must stay below the image URL lifetime")

A constraint between two constants in different files is invisible in code review and obvious at process start. Put it where the process starts.

Two things Cloudflare did that I had to undo

#

Vary: Origin kept the bundle out of the cache. django-cors-headers stamps Vary: Origin on every response it touches, and Cloudflare declines to cache anything that varies on more than Accept-Encoding. That quietly excluded the 1.6 MB JavaScript bundle — the single biggest thing on the critical path — from the edge entirely. The fix is a middleware placed first in the list, so it runs last on the way out, rewriting Vary on the static asset paths.

Compression at the edge was worse than compression at the origin. Letting Cloudflare do Brotli on the fly produced 2,041,742 bytes where the origin’s own gzip produced 1,658,115. Twenty-three percent larger, for the privilege of not thinking about it. On-the-fly compression optimises for CPU, not for ratio; precompressed assets from the origin win.

Telling search engines without waiting to be crawled

#

Two protocols, and they cover disappointingly little between them.

IndexNow takes a ping and covers Bing, Yandex and Seznam. Google and Naver do not participate. I ping it in the background whenever a listing is saved, with a short timeout, and skip listings that are sold or closed. A network failure logs a warning and never fails the save that triggered it, which is the only acceptable behaviour for a best-effort side channel.

Google’s Indexing API accepts exactly two schema types, and one of them is JobPosting, which I have. Live jobs send URL_UPDATED, closed ones send URL_DELETED. That deletion is only honest because closed listings also emit noindex, and the code says so, so that whoever removes the noindex finds out what else they broke.

Two things cost me time there. The quota is 200 URLs a day, and a bulk import of listings would fire it hundreds of times — so imported listings are skipped, or one import spends the entire day’s budget. And the endpoint is urlNotifications:publish, with a colon. Use a slash and Google returns a plain HTML 404, which looks exactly like a disabled API.

What the structured data taught me

#

Three rules, each learned by having Search Console complain:

1. Omit what you cannot parse. Salaries arrive as free text. I used to emit the raw string as baseSalary.value, which is invalid, and flagged every job for a missing unit. Now an unparseable salary is simply absent.
2. Never fabricate a field to satisfy a schema. Job ads rarely carry a street address. The temptation is to invent something plausible for PostalAddress. A fabricated address on a job ad is worse than an incomplete one, so only locality and country go in.
3. Expiry is not optional. Google drops a JobPosting with no validThrough, so one is derived, with a floor so a stale listing does not claim to have expired yesterday.

And a fourth, about sitemaps: do not claim changefreq: daily for everything. Asking Googlebot to re-crawl a thousand listings every day reads as an overloaded host, and it responds by cutting the crawl rate for the whole site. Derive it from how recently the thing actually changed.

The sitemap also has a quality floor: forty characters of title and body combined. Search Console had reported around a thousand listings as “Crawled – currently not indexed,” and one used-goods page’s entire unique content was a one-word title and a phone number. Forty characters is deliberately low, because 471 of 484 listings are in Korean and Korean says more per character; an English-tuned threshold would have dropped half the marketplace. Thin listings stay crawlable through the index pages, and re-enter the sitemap automatically when someone edits them.

What I would take from this

#

• Serving different bodies at one URL is a cache-key problem before it is anything else. Work out what your CDN keys on, and assume it ignores Vary until you have proven otherwise on that exact content type.
• When two systems must agree on a list, work out which direction of error is safe and write that down next to both copies. “These can drift, and here is the only way they are allowed to” beats pretending they cannot.
• Match bots by bot name, never by vendor name. Especially outside the Anglosphere, where the vendor also ships the browser your users are holding.
• Put invariants between distant constants somewhere that fails loudly. Import time is free and nobody can skip it.