I Thought I Was Optimising for Speed

I built a caching layer for my blog. The goal was performance. What I got, alongside that, was a clearer picture of my audience than I had ever had before.

That was not the plan.

What the numbers used to show me

Before, I had Plausible. Clean dashboard, daily visitors, top pages. Reassuring in the way that summaries are reassuring: they tell you something is happening without telling you much about what.

A blog attracts a certain kind of traffic. Some of it is people reading. Some of it is search indexers doing their rounds. Some of it is AI systems pulling content for retrieval pipelines. Some of it is automated probes checking whether your server has vulnerabilities. All of this shows up as traffic. Almost none of it is distinguishable in a standard analytics view.

The crowd that arrived

When the caching article hit Hacker News, thousands of requests came in within the hour. I had not planned for that. What I had planned for was having instruments in place when it happened.

Every request that reaches the site now gets a label: human reader, AI crawler, SEO crawler, security scanner, unknown. Those labels feed a dashboard. So when the spike came, I could see what it was made of.

Human readers, identifiable by dwell time and page depth: people who had seen the link, clicked, and stayed. RSS subscribers, a category I had underestimated: readers who had deliberately chosen to follow the feed, receiving the post through their reader of choice, a habit many had written off as extinct but which showed up clearly in the logs. Federation too: the post propagating through ActivityPub, reaching people on Mastodon and other decentralised platforms who had never visited the site directly but were reading through their own network. Indirect human readers, choosing deliberately, just through different pipes.

Then the other kind of traffic. A vulnerability scanner from Singapore, arriving six minutes into the spike, probing paths that do not exist on Ghost, looking for webshells planted on other servers at other times. A residential proxy swarm with a 96 percent bounce rate and nine seconds average visit duration, arriving and leaving without reading anything. Five categories of arrival, all at once, all labelled.

What seeing changes

The RSS readers and the federated followers were the most useful surprise. They represent deliberate readership: people who opted in, who receive content through infrastructure they control, who are in some ways a more committed audience than someone who clicked a social link and stayed for thirty seconds. They were always there. I just had not looked.

The AI crawlers are also readers, of a kind. They do not render pages. They pull content and pass it on to systems that answer questions on behalf of people. Optimising for them means thinking about caching and headers and response consistency. It is a different kind of publishing, running in parallel with the one I was already doing.

The scanner was useful in a different way. Once it had a name and a fingerprint, it was manageable. A firewall rule catches it before it reaches the Worker. Named things can be handled. Anonymous noise cannot.

What I would encourage

You do not need a custom Cloudflare setup to start seeing more clearly. But it is worth asking what your current analytics are hiding.

RSS still works. Federation works. If you publish without offering a feed or an ActivityPub endpoint, you are not reachable by a meaningful slice of deliberate readers who prefer to control their own reading environment.

Your content is already being read by AI systems, whether you have prepared for that or not. How it is cached, how consistently it is served, what headers it carries: these are not performance details. They are decisions about how your writing moves through the world.

And the gap between “I have analytics” and “I understand my audience” is wider than it looks. The analytics show you what your platform was designed to measure. The actual audience is larger, stranger, and more varied than that.

I thought I was optimising for speed. I ended up learning who I was writing for.

My Visitors Are Not All Human. That Is Fine.

I built a traffic dashboard for my own site. What I found wasn’t alarming, it was interesting. A publisher’s notes on bots, borrowed identities, and editorial agency.

Rob HoeijmakersRob Hoeijmakers

Thirty Years of Caching, Sorted in an Afternoon

HTTP caching never quite made sense, until AI tools made it legible enough to actually implement. And the reason it finally mattered: the audience had quietly changed.

Rob HoeijmakersRob Hoeijmakers

When Bots Become Readers: Publishing in the Age of AI Crawlers

Listening to Matthew Prince on Azim Azhar’s podcast made me reflect on who actually reads my blog. People (like you), machines, or both.

Rob HoeijmakersRob Hoeijmakers