Cleotic

Crawler Analytics

Once you have set up tracking (via tracking pixel or server-side tracking), Sources -> AI traffic shows which AI crawlers are visiting your website, how often, and which pages they are most interested in.

The AI traffic explorer combines crawler and referral evidence in one page. Use the Traffic filter to switch between all AI traffic, crawlers only, and referrals only. This page is specifically about reading crawler evidence.

Summary cards

Inside AI traffic, you'll see headline numbers for:

  • AI visits -- total detected AI traffic in the selected date range
  • AI share -- the share of eligible pageviews that came from AI traffic, when total pageviews are available
  • Crawlers -- AI crawler visits in the selected date range
  • Referrals -- AI referral visits in the selected date range
  • Pages -- distinct pages touched by AI traffic

These give you a quick sense of how actively AI systems are indexing and sending traffic to your content.

Visits over time

Use the date range control to review crawler activity over different windows. Use this to:

  • Spot trends -- Are crawler visits increasing, steady, or declining?
  • Identify spikes -- A sudden increase might follow a new content publish or a crawler updating its index
  • Correlate with visibility -- Rising crawler visits often precede improvements in AI visibility scores

Filtering by crawler

The filter row lets you narrow AI traffic by domain, provider, crawler, source, traffic type, and page search. Crawler names include:

  • GPTBot -- OpenAI's crawler for ChatGPT
  • ClaudeBot / CCBot -- Anthropic's crawlers for Claude
  • PerplexityBot -- Perplexity's crawler
  • Google-Extended -- Google's AI training crawler
  • anthropic-ai -- Anthropic's AI research crawler
  • And dozens of other identified AI crawlers

When filtered, the page table and evidence rows reflect only the selected crawler or provider. This tells you which AI companies are actively indexing your content.

AI traffic by page

The table shows which pages on your site get the most AI attention:

  • Path -- The page URL path (e.g., /blog/comparison-guide, /pricing)
  • AI visits/share -- total AI visits and AI share for the page, when available
  • Crawler visits -- how many crawler visits touched this page
  • AI referrals -- how many referral visits landed on this page
  • AI systems -- providers, crawlers, or sources involved
  • Latest seen -- the most recent event time
  • Trend -- page-level movement in the selected period

Click a page row to inspect recent matching evidence for that URL. Use the dropdown filters above the table to narrow the list to a specific provider (e.g., OpenAI) or individual crawler (e.g., gptbot). When filtered, the visit counts reflect only visits from the selected crawler or provider, and the ranking may change to surface pages that are particularly important to that crawler.

This is valuable for understanding which content AI crawlers prioritise. Pages that get crawled more frequently are more likely to influence AI model responses.

How to use crawler data

Confirm indexing of important pages

Check whether your most important pages (product pages, key blog posts, comparison guides) appear in the top crawled pages. If they're missing, AI crawlers may not be reaching them -- check your site's navigation, internal linking, and robots.txt.

Identify content opportunities

If AI crawlers frequently visit your competitor comparison pages but rarely visit your product feature pages, it tells you something about what AI systems consider useful. Create more of the content that gets crawled.

Monitor crawler diversity

If only one or two AI crawlers visit your site, you may have limited visibility across the broader AI landscape. Check your robots.txt to make sure you're not accidentally blocking specific crawlers.

Correlate with visibility changes

When your visibility scores change, check crawler data for the same period. An increase in crawler visits followed by improved visibility suggests your content is making an impact. A decrease in crawler visits might explain a visibility dip.

Tips

  • Don't expect instant results. AI crawler visits and visibility score improvements operate on different timescales. Crawling may happen daily, but model training and updates can take weeks or months to reflect new content.
  • Focus on consistent crawling, not spikes. A page that gets crawled regularly is more likely to be well-represented in AI models than one that gets one big spike.
  • Check your robots.txt. Some sites accidentally block AI crawlers. Make sure your robots.txt allows the crawlers you want to index your content.