Across 20,293 pages audited on 524 websites, 8.9% returned nothing a crawler could read: either the fetch failed outright or the page came back with zero words.
Those pages cannot be cited by an AI answer, because there is nothing in them to cite. This article sets out how large that share is, how it is distributed across sites, why our figure is far lower than the ones you may have seen, and what it costs when a crawler leaves empty handed.
What an AI crawler actually receives
A search crawler and an AI crawler do not do the same work. Googlebot renders a page in a browser before indexing it. Most AI crawlers do not.
Vercel and MERJ measured this across their network in December 2024, on a sample including 569 million GPTBot requests and 370 million ClaudeBot requests in a single month, and stated it without hedging: "none of the major AI crawlers currently render JavaScript." The list covers OpenAI's OAI-SearchBot, ChatGPT-User and GPTBot, Anthropic's ClaudeBot, Meta-ExternalAgent, ByteDance's Bytespider and PerplexityBot. Google's Gemini is the exception because it runs on Googlebot's infrastructure, and AppleBot renders through a browser-based crawler.
The same study found the crawlers do fetch JavaScript files, 11.50% of ChatGPT's requests and 23.84% of Claude's, without executing them. They download the code and cannot run it.
So the question "is my content readable" has a specific meaning here. Not whether a person can see it, and not whether Google can index it, but whether the text exists in the HTML that arrives before any script runs.
8.9% of pages return nothing
Finseo audits pages the way a text-first crawler would: fetch the URL, extract the readable text, record what came back. Across 20,293 URLs on 524 distinct domains between 5 December 2025 and 3 September 2026:
| Outcome | Share of audited pages |
|---|---|
| Fetch failed on every method attempted | 4.9% |
| Fetched successfully, zero readable words | 4.0% |
| Nothing readable, combined | 8.9% |
| Readable but thin, 1 to 299 words | 14.7% |
The two failure modes are different problems. A failed fetch is an access problem: a block, a challenge, a timeout, a server error. In our audits 981 of the 990 failures were recorded after more than one fetch method had been tried and all of them failed, so these are not single-method quirks.
A page that fetches cleanly and yields zero words is the second problem, and it is the one the Vercel finding predicts. The server returned a document. The document contained no text worth extracting, because the text arrives later, from a script the crawler will not run.
Median readable length on the pages that did return text was 868 words, with a mean of 1,152.
Another 14.7% are too thin to be worth citing
Readable is a floor, not a goal. A page with 120 words of text is technically retrievable and practically useless as a source, because an answer engine assembling a reply has better options.
Roughly one page in five, 19.7% of those that fetched, came back with fewer than 300 readable words. Half of the audited pages sat under 868. Whether that is a problem depends entirely on what the page is for: a pricing table does not need 1,500 words. It is a problem when the page is meant to be the source that answers a question, which is the job AI content optimization exists to check. Length alone is not the lever either: when we tested the published GEO techniques against measured outcomes in nine GEO techniques tested, adding words ranked well below answering the right question.
The failures cluster on a minority of sites
The 8.9% is not spread thinly across every site. Of the 524 domains audited:
| Measure | Share of domains |
|---|---|
| At least one unreadable page | 15.3% |
| A quarter or more of audited pages unreadable | 3.4% |
So roughly six sites in seven had no unreadable page at all in their audit, and a small group had a structural problem rather than an isolated one. That shape matters for prioritisation. If your site is in the 84.7%, this is not your bottleneck and effort belongs elsewhere, most likely in AI visibility tracking to find out which prompts you are absent from and why. If a quarter of your pages return nothing, no amount of writing will fix a retrieval problem that starts before the writing is read.
Audits averaged 38.7 pages per site, so a domain here is sampled, not exhaustively crawled.
Why our number is far lower than the ones you have seen
Published figures for the same question run much higher. A May 2026 scan of the top 1,000 websites found that of the 711 sites it could read at all, 403, or 57%, showed a non-rendering agent a nearly empty page, and graded 43% of the top 1,000 at F.
That study and ours do not disagree. They measure different things.
| Finseo audits, 2026 | Top 1,000 scan, May 2026 | |
|---|---|---|
| Unit | 20,293 pages, mostly deep content pages | 1,000 homepages |
| Sites | 524 mid-market and enterprise domains | The 1,000 largest sites on the web |
| Finding | 8.9% returned nothing readable | 57% of readable sites needed JavaScript |
Homepages of the largest platforms are single-page applications by design, and Instagram, YouTube, X and Amazon arriving as an empty shell tells you about how consumer platforms are built, not about how a mid-market company's article pages behave. Our sample is weighted toward the pages a brand actually wants cited. Both numbers are true of their own sample, and the difference between them is roughly the difference between a homepage and an article.
The practical reading: if you run a content site on server-rendered pages, your exposure is closer to 9% than to 57%. If your product pages are client rendered, you are in the other population and should test them specifically.
What a failed crawl costs
Crawling is not free attention, and the exchange rate is bad. Cloudflare published crawl-to-refer ratios from January to July 2025, counting how many pages a platform crawls for every visitor it sends back:
| Platform | Crawls per referred visit, July 2025 |
|---|---|
| Anthropic | 38,066 |
| OpenAI | 1,091 |
| Perplexity | 195 |
| Microsoft | 41 |
| 5 |
Anthropic's ratio had fallen 86.7% over those seven months, from 286,930:1 in January, after Claude added web search with clickable citations. It is still the most crawl-heavy platform by a wide margin.
Read those numbers next to the 8.9%. A crawler that already visits tens of thousands of your pages per referral and finds nothing readable on some of them is not going to compensate later. The fetch is the whole of the interaction. This is the case for watching AI crawler and bot traffic analysis as its own signal rather than inferring it from sessions, and for reading it separately from AI search traffic analytics, which describe a different half of the exchange. It is also worth checking whether the pages being fetched are the ones you would choose, because a crawler follows the sub-queries an engine generated rather than your site structure, a mechanism we set out in query fan-out analysis.
Most of the web has not decided anything
A February 2026 crawl of 999,316 domains from the Tranco top 1 million found that 90.1% have no machine-readable AI policy at all: no AI-specific robots.txt directives, no llms.txt, no ai.txt, nothing. Adoption of the individual standards was 11.1% for AI-specific robots.txt rules and 3.2% for llms.txt across the full million.
Where sites do take a position, GPTBot is the most blocked crawler on the web at 6.9% of domains, roughly 68,700 sites, followed by ClaudeBot at 6.1%.
Two things follow. First, an unreadable page is almost never a deliberate policy, it is an accident of how the page was built. Second, blocking decisions and readability are separate problems, and a site can get both wrong independently.
What to check on your own pages
Fetch a page without JavaScript and read what comes back. Curl the URL, strip the tags, look at the text. If the article is not there, no AI crawler except Gemini and AppleBot will find it either.
Check the pages you want cited, not the homepage. The homepage is the least likely page to be a source and the most likely to be a shell.
Treat a failed fetch as an access problem, not a content problem. Bot challenges, aggressive rate limiting and login walls all present as an empty page to the crawler, and none of them are solved by rewriting the copy.
Separate thin from unreadable. A 200 word page is a different fix from a 0 word page. Only one of them is an engineering task.
Confirm it with what the engines actually cite. Readability is the precondition, not the outcome. Whether the fixed page then enters answers is a question for AI citation tracking, and the two Google surfaces will not agree with each other about it, as we found when we compared AI Mode with AI Overviews.
Method note
Our figures come from Finseo crawlability audits, aggregated across accounts, with no customer, project or private domain identifiable in any number. The sample is 20,293 URLs across 524 distinct domains and 501 projects, from 1,054 audits run between 5 December 2025 and 3 September 2026, averaging 38.7 pages per audit.
What we measure is what a text-first fetcher receives, which is a proxy for what a non-rendering AI crawler receives, not a measurement of GPTBot itself. We do not operate GPTBot and cannot observe its fetches. Where a page returns nothing to us, the safe conclusion is that it would return nothing to a crawler with the same constraints, not that OpenAI specifically saw an empty page.
The sites in this sample chose to run an audit, which means they were already investigating their AI visibility. That selection could push the failure rate in either direction: a team that suspects a problem is more likely to look, and a team that runs audits is more likely to maintain its site. We cannot separate those effects, so read 8.9% as a figure for sites of this kind rather than for the web.
Word counts are of extracted readable text, not of raw HTML, and pages whose fetch failed are excluded from the word count distribution.
External figures are attributed to their study with sample size and date. The Vercel, Cloudflare, Maango and top 1,000 analyses are vendor and independent research, large but not peer reviewed.
FAQ
Do AI crawlers execute JavaScript? Most do not. Vercel and MERJ found in December 2024 that GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, PerplexityBot, Meta-ExternalAgent and Bytespider all fetch raw HTML without rendering it. Gemini uses Googlebot's infrastructure and does render, and AppleBot renders through a browser-based crawler.
How many pages are unreadable to an AI crawler? In our audits, 8.9% of 20,293 pages across 524 sites returned nothing readable, split between 4.9% where the fetch failed and 4.0% that fetched but contained no text. A separate May 2026 scan of the top 1,000 homepages put the figure at 57%, on a very different sample.
Why does my page look fine in a browser but empty to a crawler? Because the browser runs your JavaScript and the crawler does not. The text you see was inserted into the page after it loaded, and the crawler read the version that arrived before that happened.
Is a 200 word page too thin to be cited? Not automatically, but it is competing against pages with more to offer on the same question. In our sample 19.7% of fetched pages held fewer than 300 readable words and the median was 868.
Should I block AI crawlers? That is a business decision, not a technical one, and most sites have not made it: 90.1% of the Tranco top 1 million had no machine-readable AI policy in February 2026. GPTBot is the most blocked crawler at 6.9% of domains.
Does being crawled mean I will get traffic? No. Cloudflare measured Anthropic crawling 38,066 pages per referred visit in July 2025, and OpenAI 1,091. Crawling and referral are only loosely connected.
How do I test this myself? Fetch the page without JavaScript, for example with curl, and read the resulting text. If the content you want cited is missing from that output, a non-rendering crawler will not see it either.
Sources
Ours
- Finseo crawlability audits: 20,293 URLs across 524 domains and 501 projects, 5 December 2025 to 3 September 2026, fetch outcome and readable word count
- Finseo crawlability audits: distribution of unreadable pages across audited domains, same sample
External
- Vercel and MERJ (December 2024), AI crawler behaviour across their network including 569 million GPTBot and 370 million ClaudeBot requests: https://vercel.com/blog/the-rise-of-the-ai-crawler
- Cloudflare (January to July 2025), crawl-to-refer ratios by platform: https://blog.cloudflare.com/crawlers-click-ai-bots-training/
- Maango (February 2026), 999,316 domains from the Tranco top 1 million, AI policy and crawler blocking adoption: https://maango.io/report
- modpagespeed (28 May 2026), AI readability scan of the top 1,000 websites: https://modpagespeed.com/blog/can-ai-read-your-website/
Google Search
Save Finseo as a preferred source to see our articles more often in Top Stories, AI Overviews and AI Mode.
