Cleotic
Free AuditSign In

Crawler Analytics

Once you've set up tracking (via pixel or Vercel log drain), the Crawlers tab shows you which AI crawlers are visiting your website, how often, and which pages they're most interested in.

Summary cards

At the top of the Crawlers tab, you'll see two headline numbers:

  • Total visits -- The total number of AI crawler visits detected in the last 30 days
  • Unique pages -- How many distinct pages on your site were crawled

These give you a quick sense of how actively AI companies are indexing your content.

Visits over time

The area chart shows your daily crawler visit count over the last 30 days. Use this to:

  • Spot trends -- Are crawler visits increasing, steady, or declining?
  • Identify spikes -- A sudden increase might follow a new content publish or a crawler updating its index
  • Correlate with visibility -- Rising crawler visits often precede improvements in AI visibility scores

Visits by crawler

The bar chart breaks down total visits by AI crawler identity:

  • GPTBot -- OpenAI's crawler for ChatGPT
  • ClaudeBot / CCBot -- Anthropic's crawlers for Claude
  • PerplexityBot -- Perplexity's crawler
  • Google-Extended -- Google's AI training crawler
  • anthropic-ai -- Anthropic's AI research crawler
  • And dozens of other identified AI crawlers

Each crawler is colour-coded for easy identification. This chart tells you which AI companies are actively indexing your content.

Top crawled pages

The table shows which pages on your site get the most AI crawler attention:

  • Path -- The page URL path (e.g., /blog/comparison-guide, /pricing)
  • Visit count -- How many times AI crawlers visited this page
  • Crawlers -- Which specific AI crawlers visited the page, shown as badges

Use the dropdown filter above the table to narrow the list to a specific provider (e.g., OpenAI) or individual crawler (e.g., gptbot). When filtered, the visit counts reflect only visits from the selected crawler or provider, and the ranking may change to surface pages that are particularly important to that crawler.

This is valuable for understanding which content AI crawlers prioritise. Pages that get crawled more frequently are more likely to influence AI model responses.

How to use crawler data

Confirm indexing of important pages

Check whether your most important pages (product pages, key blog posts, comparison guides) appear in the top crawled pages. If they're missing, AI crawlers may not be reaching them -- check your site's navigation, internal linking, and robots.txt.

Identify content opportunities

If AI crawlers frequently visit your competitor comparison pages but rarely visit your product feature pages, it tells you something about what AI systems consider useful. Create more of the content that gets crawled.

Monitor crawler diversity

If only one or two AI crawlers visit your site, you may have limited visibility across the broader AI landscape. Check your robots.txt to make sure you're not accidentally blocking specific crawlers.

Correlate with visibility changes

When your visibility scores change, check crawler data for the same period. An increase in crawler visits followed by improved visibility suggests your content is making an impact. A decrease in crawler visits might explain a visibility dip.

Tips

  • Don't expect instant results. AI crawler visits and visibility score improvements operate on different timescales. Crawling may happen daily, but model training and updates can take weeks or months to reflect new content.
  • Focus on consistent crawling, not spikes. A page that gets crawled regularly is more likely to be well-represented in AI models than one that gets one big spike.
  • Check your robots.txt. Some sites accidentally block AI crawlers. Make sure your robots.txt allows the crawlers you want to index your content.

Related