AI bot traffic analysis
See which AI bots are crawling your website.
AI crawlers visit your site daily, but Google Analytics can't see them. Finseo reads your logs and shows what GPTBot and ClaudeBot fetch.
Latest request
500+ teams measure their AI visibility with Finseo
See it
Every AI crawler visit to your site, in one log.
Finseo reads your logs and shows each request with the bot, the page, the status code, your response time and the city it came from.
Major crawlers are checked against their published IP ranges, so fake bots don't count.
Open any URL to see its visits, bots, problem rate and full user agents.
ChatGPT-User200182 ms
Meta-ExternalAgent40486 ms
Googlebot200214 ms
Bingbot5031,240 msChatGPT-User200182 msMeta-ExternalAgent40486 msGooglebot200214 msBingbot5031,240 msUnderstand it
Which bot reads which page, and why.
Each visit is sorted by what the bot does with your page. A block on the wrong bot keeps you out of answers.
01 Crawlers
The AI crawlers arrive
GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot and PerplexityBot, the major ones checked against their IP ranges.
Verified crawlers · halvard.com
GPTBotOpenAI IP verifiedChatGPT-UserOpenAI IP verified
ClaudeBotAnthropic IP verified
PerplexityBotPerplexity IP verifiedVerified crawlers · halvard.com
GPTBotOpenAI IP verifiedChatGPT-UserOpenAI IP verifiedClaudeBotAnthropic IP verifiedPerplexityBotPerplexity IP verified02 Pages
They fetch your pages
Product pages, guides and filter URLs, with every visit and the status code per bot.
Visits per page
/jackets/storm-shell
142bot visits, last 30 days
ChatGPT-User64200PerplexityBot41200ClaudeBot37403Visits per page
/jackets/storm-shell
142bot visits, last 30 days
ChatGPT-User64200PerplexityBot41200ClaudeBot3740303 Purpose
Sorted by what they do
Live answer while someone asks, search index for AI search, or training for future models.
Visits by purpose · last 30 days
Visits by purpose · last 30 days
04 Blocked
A block, found and fixed
ClaudeBot gets a 403 from the firewall. Once the bot rule allows it, the page answers 200 again.
ClaudeBot
/jackets/storm-shell
Blocked by the firewall403
Readable again200
Allow ClaudeBot in the bot rule
ClaudeBot
/jackets/storm-shell
Blocked by the firewall403
Readable again200
Allow ClaudeBot in the bot rule
Verified crawlers · halvard.com
GPTBotOpenAI IP verifiedChatGPT-UserOpenAI IP verifiedClaudeBotAnthropic IP verifiedPerplexityBotPerplexity IP verifiedVerified crawlers · halvard.com
GPTBotOpenAI IP verifiedChatGPT-UserOpenAI IP verifiedClaudeBotAnthropic IP verifiedPerplexityBotPerplexity IP verifiedWhere they come from
See where crawler requests come from.
The map colours each country by the bot visits from its data centres, or by your average response time. Slow regions show where crawlers wait on your server.
Countries of the bots' servers, mostly cloud data centres, not your visitors.
Countries · 30 days
The blind spot
Google Analytics shows no AI crawlers at all.
AI crawlers don't run JavaScript, so they never reach your analytics. Only your server logs show which pages ChatGPT, Claude and Perplexity read.
Cloudflare integrationLayering for a wet week in Scotland
Google Analytics0 bot sessionsPerplexityBotlast visit today, 14:32Only in server logs
Why it matters
Bots read far more than they send back.
Cloudflare measured how many pages AI crawlers read for every visitor they send back. All the more reason to make sure they read the right pages.
Anthropic
38,000 : 1
pages crawled by Anthropic per visitor referred
OpenAI
1,091 : 1
crawls by OpenAI per visitor referred
Perplexity
195 : 1
crawls by Perplexity per visitor referred
Fix it
Fix what keeps crawlers away from your pages.
Blocked bots, dead URLs and wasted crawl budget, each with what it means and what to change.
Blocked crawlers
Blocked · 401 · 403 · 429
429PerplexityBot
58
visits in 30 days
Your server answered “429 Too Many Requests” and turned the bot away. Perplexity can't read this page right now.
Check the rate limit for PerplexityBot
Crawl instructions
Content pages61.2 %
Crawl budget · all AI bots
Pages that no longer exist
Not found
404
/jackets/storm-shell-2023
38 bot visits in 30 days
Redirect it to the current page
Get started
Connect your logs in three steps.
Log sources
Cloudflare
Worker setup
Akamai
DataStream 2
Server-log API
REST endpoint
Log upload
nginx · Apache · .log
Fastly
Coming soon
AWS CloudFront
Coming soon
Step 01
Connect the source
A guided Cloudflare Worker setup, Akamai DataStream 2 or the server-log API keeps bot visits flowing in.
Step 02
Or upload logs
nginx, Apache or Cloudflare logs as .log or .txt, up to 1 GB per file. Finseo detects the format.
Step 03
Read the bots
Finseo recognises the AI crawlers, checks the big ones against their IP ranges and sorts each visit by purpose.
Bot traffic FAQ
Questions about AI bot traffic.
What customers say
Rated 5.0 — and ready for your security review.
Single sign-on, roles per project and data hosted in Germany, with a data processing agreement for every organisation.
“We’ve worked with Finseo for months and we’re delighted. You can tell it comes out of agency life: smart, clear and genuinely practical.”
Harald Fischl
Founder, Max Online
OMR Reviews
G2SSO via SAML
Login via your identity provider
Data in Germany
EU data residency, GDPR
Roles & audit trail
Every change logged
More features
Other parts of Finseo.
Bot traffic demo
Which AI bots read your website?
Enter your work email and we will show you in 30 minutes which crawlers visit your pages and what they come for.