Known AI bots
GPTBot, ClaudeBot, PerplexityBot and other AI crawlers, identified by user agent.
Platform
See which AI bots read your site
Server logs show what AI crawlers fetch. Send a copy of each request to Citeflare and see which bots visit, which pages they read and what status they get back.
How it works
How crawler analytics works
01
Create a project token
Each project gets a private crawler token for the ingest endpoint.
02
Add a snippet
Paste a Cloudflare Worker, Next.js middleware or call the API from your own server. It forwards request details without changing the response.
03
Bots are classified
User agents are matched against known AI crawlers such as GPTBot, ClaudeBot and PerplexityBot.
04
Browse the log
Filter by bot, path and status, and see which pages are read and which return errors.
Capabilities
Everything in one place
Cloudflare Worker
Copy-paste snippet that reports requests and passes them through untouched.
Next.js middleware
A small middleware for sites hosted on Vercel or anywhere Next.js runs.
Ingest API
POST request details from any server, CDN or log pipeline with your token.
Status codes
See which pages crawlers read successfully and which return 404 or 5xx.
Visit timeline
Spot when a bot started visiting, or stopped.
In depth
What you can do with it
Visit log
Every bot request, in one log
The log lists each AI crawler visit with its path and response status. Filter to a bot to see what it reads, or to a path to see who read it.
- Time, bot, path and status per request
- Filter by bot, path or status code
- Export-ready table
Coverage
Which pages get read, and which do not
If a bot never reaches the pages you want cited, there is nothing to cite. Compare crawled paths against your important pages and find the ones that are blocked or skipped.
- Most-crawled paths per bot
- Errors served to crawlers
- Compare bots against each other
Setup
Connected in minutes
The Worker snippet runs at the edge, forwards a small record of each request, and returns the original response. Nothing is added to the page.
- Works with Cloudflare, Next.js or the API
- One token per project
- No change to page content or speed for visitors
Metrics
What we measure, defined Every number has one definition, so you can check it against the data.
- Bot hits
- The number of requests a given AI crawler made in the period.
- Pages crawled
- Distinct paths requested by AI crawlers.
- Error rate
- Share of crawler requests that returned a 4xx or 5xx status.
- Last seen
- The most recent time a bot visited, per path.
Related
Works with the rest of the platform
Prompt tracking
Track the prompts your buyers ask AI. Citeflare scans ChatGPT, Claude, Gemini, Perplexity and Google AI Overviews every day and records mentions, position and sentiment.
Citations
See which domains and pages AI engines cite, split into owned, competitor, editorial and user-generated sources, including Reddit and YouTube.
Content agents
Turn a gap into a brief, an article and a published page. Citeflare’s content agents write, schedule and publish to WordPress or a webhook, and refresh existing pages with a diff.
Faq
Questions, answered
Is crawler analytics on every plan?
It is included on Growth and Agency. Starter does not include it.
Does the snippet slow my site?
The Worker reports in the background and returns the original response as normal. Test it on a staging route first if you want to be sure.
Which bots are recognised?
Known AI crawlers such as GPTBot, ClaudeBot and PerplexityBot, matched by user agent. Anything unrecognised is not shown as an AI bot.
Can the user agent be faked?
Yes, user agents can be spoofed. Treat the log as a strong signal, not proof.
What do you store?
Bot name, user agent, path, status code, IP and time for each request. No page content and no visitor data beyond that.
See what AI says about you. Your first scan runs the day you sign up.
Monthly or annual plans. Cancel any time.