Cloudflare

Finseo deploys a small Worker named finseo-bot-traffic on your Cloudflare zone. The Worker passes every request through unchanged and, only when the User-Agent matches a known crawler such as GPTBot, ClaudeBot, PerplexityBot, Google-Extended, OAI-SearchBot, or Bingbot, reports the visit to Finseo in the background. Enterprise customers can use a Logpush job on the http_requests dataset instead. Both methods send the same fields per bot request: IP address and User-Agent, host, method, path and query string, referrer, HTTP status and response size, country, city and region, timestamp, and time to first byte. Visits are deduplicated and appear in Bot Traffic alongside manually uploaded logs.

1. 01

   See which pages AI crawlers actually fetch

   Bot Traffic groups Cloudflare requests by crawler and URL, so you know whether GPTBot, ClaudeBot, and PerplexityBot reach your pricing, product, and documentation pages or stop at the homepage.

2. 02

   Catch crawlers that never come

   If Google-Extended or Claude-SearchBot has not requested a single page in 30 days, your robots.txt, a firewall rule, or a WAF setting is probably blocking it. Fix the block before working on AI visibility.

3. 03

   Correlate crawls with citations

   Compare which URLs OAI-SearchBot and Perplexity-User fetch with the sources ChatGPT and Perplexity cite in your tracked prompts. Pages that get crawled but never cited need better structure or clearer answers.

4. 04

   Verify indexing after content changes

   After publishing a new FAQ section or schema markup from a Finseo visibility task, watch the Cloudflare feed for the next GPTBot or Bingbot visit to that URL instead of guessing when AI engines picked it up.

5. 05

   Spot errors AI bots hit

   Cloudflare sends the HTTP status per request. Bot Traffic shows 3xx redirects, 4xx client errors, and 5xx server errors by crawler, so broken paths that keep Gemini and Microsoft Copilot from reading a page surface quickly.

Cloudflare sits in front of roughly 20 percent of all websites, which makes it the most common edge for the sites AI engines crawl. AI crawlers like GPTBot, ClaudeBot, and PerplexityBot do not execute JavaScript, so a client-side analytics snippet never sees them. The only reliable record of what ChatGPT, Claude, Perplexity, Gemini, and Google AI Overviews read on your site is the request log at the edge. Manual log uploads work, but they are always days old. A Cloudflare Worker or Logpush job turns that into a live feed, so Answer Engine Optimization (AEO) and Generative Engine Optimization (GEO) decisions rest on what crawlers did today, not last month.

1. 01

   Collect your Cloudflare IDs

   Open dash.cloudflare.com, select your domain, and copy the Zone ID and Account ID from the API section on the Overview page. Both are needed for the Worker method; Logpush needs only the Zone ID.

2. 02

   Create a custom API token

   Under Profile, API Tokens, create a custom token with Account: Workers Scripts: Edit, Zone: Workers Routes: Edit, and Zone: Zone: Read. For Logpush, the token needs Zone: Logs: Edit instead. Finseo uses the token once and never stores it.

3. 03

   Open the Cloudflare connect dialog in Finseo

   In Finseo, go to Bot Traffic, Sync, and click Connect on the Cloudflare card. Choose Cloudflare Worker (all plans, recommended) or Logpush (Enterprise plan only).

4. 04

   Deploy with one click or paste manually

   Paste the token, Account ID, and Zone ID and click Deploy Worker. Finseo uploads the Worker and adds a route for *yourdomain.com/* covering apex and subdomains. Prefer not to share a token? Copy the Worker script and deploy it yourself under Workers & Pages.

5. 05

   Wait for the first crawler

   The card shows Awaiting first push until the first bot visit arrives, then switches to Connected. Depending on crawl volume this can take a few hours. Logpush sends a test upload immediately.

How does Finseo track AI bots on Cloudflare?

Finseo deploys a lightweight Cloudflare Worker on your zone that passes every request through unchanged and, only when the User-Agent matches a known crawler, reports the visit to Finseo in the background. Enterprise customers can alternatively push the http_requests dataset to Finseo through a Logpush job.

Finseo deploys a lightweight Cloudflare Worker on your zone that passes every request through unchanged and, only when the User-Agent matches a known crawler, reports the visit to Finseo in the background. Enterprise customers can alternatively push the http_requests dataset to Finseo through a Logpush job.

Which Cloudflare plan do I need?

The Worker method works on every Cloudflare plan, including Free. The Logpush method requires the Cloudflare Enterprise plan because the http_requests dataset is only available there. If you are not on Enterprise, choose the Worker and you will get the same fields with lower latency.

The Worker method works on every Cloudflare plan, including Free. The Logpush method requires the Cloudflare Enterprise plan because the http_requests dataset is only available there. If you are not on Enterprise, choose the Worker and you will get the same fields with lower latency.

Which crawlers does the Cloudflare integration capture?

Requests whose User-Agent matches GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-Web, Claude-SearchBot, anthropic-ai, PerplexityBot, Perplexity-User, Google-Extended, GoogleOther, Googlebot, Bingbot, CCBot, Bytespider, Amazonbot, Applebot, meta-externalagent, FacebookBot, DuckAssistBot, cohere, MistralAI, YandexBot, and DuckDuckBot. Matching happens on the User-Agent inside the Worker or Logpush filter, so regular visitor traffic is never transmitted to Finseo.

Requests whose User-Agent matches GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-Web, Claude-SearchBot, anthropic-ai, PerplexityBot, Perplexity-User, Google-Extended, GoogleOther, Googlebot, Bingbot, CCBot, Bytespider, Amazonbot, Applebot, meta-externalagent, FacebookBot, DuckAssistBot, cohere, MistralAI, YandexBot, and DuckDuckBot. Matching happens on the User-Agent inside the Worker or Logpush filter, so regular visitor traffic is never transmitted to Finseo.

Does the Worker slow down my website?

No. The Worker forwards every request unchanged and only reports crawler visits in the background after the response is sent. Visitor responses are never delayed or altered. If the Workers Free plan limit of 100,000 requests per day is exceeded, Cloudflare serves traffic normally and simply skips the Worker.

No. The Worker forwards every request unchanged and only reports crawler visits in the background after the response is sent. Visitor responses are never delayed or altered. If the Workers Free plan limit of 100,000 requests per day is exceeded, Cloudflare serves traffic normally and simply skips the Worker.

Is my Cloudflare API token stored by Finseo?

No. Finseo verifies the token against the Cloudflare API, uses it once to upload the Worker and add the route (or to create the Logpush job), and then discards it. If you prefer not to enter a token at all, copy the generated Worker script and deploy it manually.

No. Finseo verifies the token against the Cloudflare API, uses it once to upload the Worker and add the route (or to create the Logpush job), and then discards it. If you prefer not to enter a token at all, copy the generated Worker script and deploy it manually.

What data does Finseo receive per bot request?

Both methods transmit the same fields: IP address and User-Agent to verify the bot identity, host, method, path and query string, referrer, HTTP status code and response size, country, city and region, timestamp, and time to first byte. Visits are deduplicated before they appear in the Bot Traffic dashboard.

Both methods transmit the same fields: IP address and User-Agent to verify the bot identity, host, method, path and query string, referrer, HTTP status code and response size, country, city and region, timestamp, and time to first byte. Visits are deduplicated before they appear in the Bot Traffic dashboard.

Can I combine Cloudflare sync with manual log uploads?

Yes. Uploaded .log and .txt files and the Cloudflare feed land in the same Bot Traffic project. Finseo deduplicates visits by bot, timestamp, IP, and path, so overlapping data from a historical upload and the live feed is never counted twice.

Yes. Uploaded .log and .txt files and the Cloudflare feed land in the same Bot Traffic project. Finseo deduplicates visits by bot, timestamp, IP, and path, so overlapping data from a historical upload and the live feed is never counted twice.

How do I disconnect Cloudflare from Finseo?

Click Manage, then Disconnect on the Cloudflare card in Finseo. This invalidates the ingest token so further deliveries are rejected immediately. Then remove the finseo-bot-traffic Worker route, the Worker itself, or the Logpush job in your Cloudflare dashboard to stop the pushes at the source.

Click Manage, then Disconnect on the Cloudflare card in Finseo. This invalidates the ingest token so further deliveries are rejected immediately. Then remove the finseo-bot-traffic Worker route, the Worker itself, or the Logpush job in your Cloudflare dashboard to stop the pushes at the source.