Who Actually Crawls a Developer Documentation Site

/research/who-crawls-a-docs-site/

Everyone quotes a bot-traffic percentage. Almost nobody publishes their logs.

Here are ours: fifteen days, 45,085 parsed requests, one small technical documentation site with no backlinks and no advertising.

The headline

90.9% of requests were not people.

AI assistants fetched more than every search engine combined — 11,317 against 10,156, a ratio of 1.11×.

Why the “unverified” figure is the honest one

It would be easy to write “9.1% human traffic” and it would be wrong.

Classification here is by user-agent string, which is self-reported. A crawler that declares itself is easy to count; one that copies Chrome’s string is indistinguishable from Chrome at the log level. The forged-browser bucket (1.7%) contains only the obvious forgeries — Internet Explorer 6, Opera 7, Windows CE, Android 2.x, each arriving from a single address several hundred times. Software nobody has run in a decade is a spoof. A current Chrome string is not detectable this way at all.

This is why the site’s human numbers come from JavaScript analytics rather than from logs. A crawler that executes JavaScript and reports a plausible user-agent is a much smaller population than one that does neither.

The scanner floor

11.5% of requests were vulnerability scanners: roughly 3,600 probes for /wp-login.php, /.env, /.git/config and similar, from 359 addresses.

This site is a static Astro build with no WordPress, no PHP and no .env anywhere in its output. The probes are indiscriminate and constant, and they arrived within days of the domain first resolving.

A detail worth knowing if you serve a static site with a trailing-slash rule: each probe cost two requests. A request for /.env was 301-redirected to /.env/ before returning 404. Answering with a single 444 instead halves the traffic and removes the redirect from the logs.

What we did and did not block

We block scanners, SEO backlink crawlers and forged ancient user-agents.

We deliberately do not block AI assistants, despite them being the single largest class. Two reasons, and only one of them is philosophical:

The practical one is that Perplexity and Google Gemini already appear as referrers in our analytics, sending real readers. Blocking the crawler removes the citation that produces the visit.

The other is that a documentation site exists to be read. An answer engine quoting a page correctly, with attribution, is the site working — not the site being stolen from.

That is a judgement, not a finding, and a site whose revenue depends on pageviews would reasonably reach the opposite conclusion from the same numbers.

Top AI crawlers by request count

Claude-User (1,803), ChatGPT-User (1,794) and OAI-SearchBot (1,550) led, with GPTBot, PerplexityBot, Perplexity-User, ClaudeBot and Bytespider behind them.

The split between the -User agents and the indexing agents is the interesting part: a -User fetch is generally made because a person asked a question right then, which is closer to a visit than to a crawl.

Raw data

The complete classified dataset is published as JSON — every class, request count, byte total and distinct-address count, with the window and the methodology in the envelope.

@misc{copilotstack_research-who-crawls-a-docs-site_2026,
  title        = {Who Actually Crawls a Developer Documentation Site},
  author       = {{The Copilot Stack}},
  year         = {2026},
  note         = {Version 2026-09-04. Maintained research, not peer reviewed},
  howpublished = {\url{https://thecopilotstack.com/research/who-crawls-a-docs-site/}},
  urldate      = {2026-09-04}
}