CleoticBot

CleoticBot is the web crawler of Cleotic, a service that shows businesses how AI assistants describe them and what on their website stops assistants and search engines reading it. It reads public pages only. It does not sign in, submit forms or run scripts, and Cleotic does not use what it reads to train AI models.

How to recognise it

Every request CleoticBot sends carries this user agent:

Mozilla/5.0 (compatible; CleoticBot/1.0; +https://cleotic.ai/bot)

Its robots.txt token is CleoticBot. Requests come from our servers in western Europe. Some checks read a page through a proxy partner in the country a customer sells to, and those requests carry the same user agent.

What it reads

• Site readiness scans. The pages of a site its owner, or an agency working for them, asked Cleotic to check, and sites entered in the free site check on cleotic.ai. A scan follows the site’s links and sitemap up to a page limit.
• Brand details. The homepage and a few pages of a brand’s own site when it is set up in Cleotic, to read its name, description and icon.
• Competitors. The homepage of a competitor a customer tracks.
• Cited sources. Pages that AI assistants cite in the answers Cleotic tracks, to read their title and summary.
• Publication checks. A page a customer has just published from Cleotic, to confirm it is live.

robots.txt

CleoticBot follows the rules in your robots.txt for the CleoticBot token, or for * when there is no CleoticBot group, whenever it reads your site for someone other than you: site readiness scans, the free site check, competitor pages and cited sources. Site readiness scans also honour Crawl-delay, up to 30 seconds between requests.

Brand details and publication checks read a site on behalf of its owner, and the owner sees the result in Cleotic.

To keep CleoticBot off your whole site, add:

User-agent: CleoticBot
Disallow: /

Contact

Questions about CleoticBot, or a request you think it should not have made: email support@cleotic.ai with the address it requested and the time.