See which AI crawlers read your site, and what they get back.
Last reviewed 2026-10-19
AI crawlers never run your page's JavaScript, so Virent captures them where the request arrives - your server, edge, CDN or hosting logs - and classifies each request by provider and purpose.
Choose a collection method#
Open Connect analytics → Hosting integrations in the dashboard for host and CDN methods, with your site's values filled in. For your own server, use a framework guide.
| Where your site runs | Method | Coverage |
|---|---|---|
| Vercel Pro or Enterprise | Log Drains (recommended) | Includes CDN-cached pages, no application code |
| Vercel, any plan | Middleware (proxy.ts) | Requests that reach your app |
| Cloudflare | Workers | Proxied zones, without blocking responses |
| Cloudflare Enterprise | Logpush (recommended) | Complete edge logs, including cached responses |
| AWS | CloudFront real-time logs | Edge logs |
| Akamai | DataStream 2 | Edge logs |
| Netlify Enterprise | Log Drain | Edge logs |
| Google Cloud | Load Balancer log export | Load balancer logs |
| Fastly | Log streaming | Edge logs |
| Framer or Webflow | A Cloudflare Worker in front of the site (Framer, Webflow) | Proxied requests |
| Your own server | Framework guide or the server SDK | Requests that reach your server |
Log-based methods see cached pages that middleware never sees. Middleware cannot see the final response status. See Runtimes and coverage for the details.
Keys#
Crawler capture uses the website's site ID and server secret (vha_sk_…), from Settings → Connection. The secret authenticates writes from your server: keep it in server environment variables, never in browser code. See Keys and secrets.
Server SDK#
For Node.js servers outside the framework guides, use the tracker from @virent.app/sdk/server:
import { createVirentBotTracker } from "@virent.app/sdk/server"const tracker = createVirentBotTracker({ siteId: process.env.VIRENT_SITE_ID!, writeKey: process.env.VIRENT_INGEST_SECRET!,})// In your request handler, with a Fetch API Request:void tracker.trackRequest(request, { statusCode: 200 })Call it for page requests only: skip API routes, static assets, images, favicons, robots.txt and sitemaps. Pass the response status when you know it. Don't await it in the response path; it gives up after 5 seconds. The Express guide shows a middleware that sends the same fields over plain HTTP.
What is stored#
Only requests from AI crawlers are stored and count toward your plan. Search indexers such as Googlebot, SEO tools, scrapers, monitoring and link previews are dropped when they arrive. Before sending, the SDK strips query strings, fragments, credentials, cookies, authorization headers and request bodies.
Crawler purposes#
| Purpose | Shown as | What it means |
|---|---|---|
ai_training | Model training | Collects content that may be used to train models. A crawl does not establish that the page was used. |
ai_search | AI search | Builds the index an AI search or answer engine retrieves from. |
answer_engine | User-initiated fetch | Fetches a page because a person asked an assistant something. |
unknown | Unclassified | An AI crawler whose purpose Virent has not classified yet. |
AI crawlers Virent recognizes#
| Provider | Crawlers |
|---|---|
| OpenAI | GPTBot (training), OAI-SearchBot (search), ChatGPT-User (user fetch) |
| Anthropic | ClaudeBot (training), Claude-SearchBot (search), Claude-User and Claude-Web (user fetch) |
| Google-CloudVertexBot (training), Google-Agent and Google-GeminiNotebook (user fetch). Google-Extended is a robots.txt token, not a crawler, so it never appears in requests. | |
| Perplexity | PerplexityBot and Perplexity-User (user fetch) |
| Meta | meta-externalagent (training), meta-webindexer (search), meta-externalfetcher (user fetch) |
| Amazon | Amazonbot (training) |
| ByteDance | Bytespider (training) |
| Common Crawl | CCBot (training) |
| DeepSeek | DeepSeekBot (user fetch) |
| DuckDuckGo | DuckAssistBot (user fetch) |
| You.com | YouBot (user fetch) |
Virent also classifies AI crawlers from a reviewed open registry, including Mistral, Cohere, Kimi, Exa, Tavily, Firecrawl, Phind and others. Googlebot, Applebot and PetalBot are search indexers, not AI crawlers.
Classification uses the user agent. Anyone can send a crawler's user agent, so a request is evidence of a claimed crawler, not proof of identity.
In the dashboard#
- Crawler requests by provider, and request activity over time.
- Crawled pages, with the providers and request count for each. Select one for its request log.
- Response status, robots and crawl rules, and pages that return errors or blocks.
- A log of recent crawler requests.
Actions turns these into tasks, such as fixing error responses on a path or letting a blocked crawler reach your pages.
Verify#
curl -A "GPTBot" https://your-domain.com/Then refresh the status in Settings → Connection. AI crawler tracking moves from "Waiting for first event" to "Receiving events". The test request checks your integration; it is not a real crawler visit.
Troubleshooting#
No crawler requests arrive#
Check that capture runs before the page renders, covers the path you tested, and uses the full server secret rather than a masked preview.
Static files appear as pages#
Tighten your matcher or route exclusions for framework internals, API routes, images, favicons, robots.txt, sitemaps and static file extensions.
Fewer requests than your logs show#
Middleware does not see CDN-cached responses. Use a log-based method on hosts that offer one.
The secret is masked#
The secret is shown only once. Use Rotate + reveal to issue a new one, then update every install that used the old one.
Human Analytics
Install and configure Virent Human Analytics - the browser script, consent and privacy controls, single-page apps, custom events, identify, goals, funnels and AI referrals.
Visibility overview
How Virent Visibility works - the 9 AI engines, the daily schedule, mention rate, share of voice, citations and the evidence behind every score.