Cleotic

Crawler analytics

Open Sources -> AI traffic and set Traffic to Crawlers only to inspect detected crawler requests. Install tracking first.

A crawler event records a request identified from its user agent. It does not confirm indexing, training use, or a recommendation in an AI answer. User-agent identification is also not proof of the visitor's identity.

Read the traffic view

The summary includes AI visits, crawler visits, referrals, pages, and AI share when eligible total pageviews are available. Use the date range, domain, provider, crawler, source, and page-search filters to narrow the evidence.

The page table shows visit counts, systems involved, latest activity, and trends. Open a row to inspect recent events for that page.

Recognise crawler names

Examples you may encounter include GPTBot, ClaudeBot, PerplexityBot, and CCBot. CCBot belongs to Common Crawl, rather than Anthropic. See Common Crawl's crawler documentation.

Google-Extended is a robots.txt control token, not a separate HTTP user-agent string. Do not expect a distinct Google-Extended visit as proof that Google's AI systems accessed your site. See Google's crawler documentation.

Investigate missing visits

  1. Confirm the tracking site is active and its connection is configured.
  2. Widen the date range and clear crawler or page filters.
  3. Check that the domain and subdomain match the installation.
  4. For browser tracking, remember that an HTML-only crawler will not run the snippet.
  5. For server tracking, check delivery and whether caching or request filtering excludes the traffic.

Robots.txt rules may stop a cooperative crawler from requesting a page. A firewall may block a request before it reaches your origin. Neither situation guarantees a recorded visit in Cleotic.

Use the evidence

Check whether important product and comparison pages receive requests. If a page is missing, inspect its public accessibility, internal links, and crawl rules with your site team or Site Readiness.

Compare crawler activity with citations and AI referrals, but keep the signals separate. More crawling alone does not establish that visibility improved.