Crawlability, robots.txt and AI crawlers

Andy Bright

Director of Product and Technology

Before Google can show one of your pages, or ChatGPT can cite it, a crawler has to be able to fetch it. That ability is called crawlability. Most websites are open to Google without anyone having to do anything. AI crawlers are more often blocked, sometimes by a file on your site called robots.txt, and sometimes by security settings that nobody on your team has looked at.

This chapter explains:

• what crawlers are and how they identify themselves
• how to read your robots.txt file and change it safely
• which AI crawlers exist and what each one does
• how to decide whether AI companies can use your content to train their models
• how to check that your hosting or security settings aren't blocking crawlers you want

If words like crawler and index are new to you, read How search engines and AI assistants find your pages first. It explains the whole process from the beginning.

How crawlers identify themselves

A crawler is a program that fetches web pages automatically. Search engines and AI companies each run their own crawlers, and most run several.

Every time a crawler requests a page, it sends a line of text with the request that says what it is. That line is called the user agent. This is the user agent that OpenAI publishes for its search crawler, OAI-SearchBot:

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot

Most of that line is there for historical reasons. The part that identifies the crawler is its name, OAI-SearchBot. Your robots.txt file refers to crawlers by these short names, so you'll see them throughout this chapter.

Anyone can put any name in a user agent, so a request that says it's from OAI-SearchBot might come from someone else. That only matters when you're reading your logs or setting up firewall rules, and a later section explains how to check.

What robots.txt is

robots.txt is a plain text file that tells crawlers which parts of your website they may visit. It always lives at the same place, the root of your domain. For a site at www.example.com, that's https://www.example.com/robots.txt.

Here is the robots.txt file for Fernhill Coffee, the example online shop used throughout this guide:

User-agent: *
Disallow: /checkout/
Disallow: /account/

Sitemap: https://www.example.com/sitemap.xml

Reading it line by line:

• User-agent: * means the rules underneath apply to every crawler. The * stands for "any".
• Disallow: /checkout/ asks crawlers not to visit any address that starts with /checkout/. Fernhill doesn't want crawlers wandering through its checkout pages.
• Disallow: /account/ does the same for customers' account pages.
• Sitemap: tells crawlers where to find Fernhill's sitemap, the list of pages it wants found.

Anything the file doesn't disallow is allowed. So every crawler may visit /subscriptions, /shop and every other page on Fernhill's site.

Each subdomain needs its own robots.txt. A subdomain is a separate part of your site with its own prefix, such as blog.example.com or shop.example.com. The file at www.example.com/robots.txt doesn't apply to blog.example.com.

What robots.txt can't do

A common belief is that blocking a page in robots.txt keeps it out of Google. It doesn't. Google's introduction to robots.txt⁠(opens in a new tab) says the file is "used mainly to avoid overloading your site with requests; it is not a mechanism for keeping a web page out of Google".

robots.txt stops Googlebot from reading a page, but it doesn't stop Google from knowing the page exists. If other pages link to it, Google can still list its URL in search results. Because Googlebot never read the page, the listing may have no description.

Blocking a page also hides any instructions on it. Pages can carry a noindex tag, a line of HTML that tells search engines not to show the page in results. If robots.txt blocks the page, Googlebot never sees that tag. Google's robots meta tag documentation⁠(opens in a new tab) says the rules on a blocked page "will not be found and will therefore be ignored".

So each tool has its own job:

• To stop crawlers visiting part of your site, for example to reduce the load on your server, use robots.txt.
• To keep a page out of search results, let crawlers reach it and add a noindex tag. The indexing chapter explains how.
• To keep something private, put it behind a login. Anyone can read your robots.txt, including people looking for pages you'd rather they didn't find.

Following robots.txt is voluntary. Google, Bing, OpenAI, Anthropic, Apple and the other large companies all say their main crawlers obey it. But nothing forces a crawler to, and some of the crawlers that fetch pages on behalf of a person say openly that they may not.

Details to know before you edit robots.txt

Google publishes exactly how it reads the file, in its robots.txt specification⁠(opens in a new tab). These are the details most likely to affect you.

Google only understands four kinds of line: user-agent, allow, disallow and sitemap. Some files also include crawl-delay, which asks a crawler to wait a number of seconds between requests. Google ignores it, but some other crawlers, including Anthropic's and Common Crawl's, follow it.

Changes aren't instant. Google generally keeps a copy of your robots.txt for up to 24 hours, so a change can take a day to have an effect.

Two symbols let one rule cover many addresses. A * matches any run of characters, and a $ marks the end of an address. For example, Disallow: /*?sessionid= blocks every address that contains ?sessionid=, wherever it appears.

When two rules disagree, the longer one wins. If Fernhill's file says Disallow: /shop/ and Allow: /shop/subscriptions, crawlers may still visit /shop/subscriptions, because the Allow rule is longer and so more specific. If two rules are exactly as specific, Google follows the less restrictive one.

Google reads only the first 500 KiB of the file, about half a megabyte. A typical robots.txt is a few hundred bytes, so this only matters for very large, automatically generated files.

The response your server gives for the file itself matters most:

• If /robots.txt returns a 404 (not found), Google assumes there are no restrictions. That's fine if you don't need any rules.
• If it returns a server error (a 5xx code) or a 429 (too many requests), Google treats your whole site as blocked. It stops crawling the site for 12 hours. After that, for up to 30 days, it uses the last version of the file it managed to read, while it keeps trying.

Each crawler follows only one group of rules

A group in robots.txt is one or more User-agent lines followed by the rules that apply to them. Fernhill's file has one group, for *. You can add more groups for specific crawlers by name.

A crawler looks for the group with its own name. If it finds one, it follows that group and ignores the * group completely. Google's specification puts it this way: "Only one group is valid for a particular crawler".

Suppose Fernhill wants to make it clear that OpenAI's search crawler is welcome, and adds a group for it:

User-agent: *
Disallow: /checkout/
Disallow: /account/

User-agent: OAI-SearchBot
Allow: /

OAI-SearchBot now ignores the * group, so it's allowed into /checkout/ and /account/ as well. To keep it out of those sections, the group for OAI-SearchBot has to repeat the same rules:

User-agent: *
Disallow: /checkout/
Disallow: /account/

User-agent: OAI-SearchBot
Disallow: /checkout/
Disallow: /account/

A crawler with no group of its own, such as Googlebot here, still follows the * group. Several User-agent lines can also share one group, which the examples later in this chapter use to keep the file short.

The three kinds of AI crawler

Most AI companies run more than one crawler, and each crawler has a different job. The job decides what happens if you block it.

Training crawlers collect text that may be used to train future AI models. A model learns from that text, and what it learns stays fixed until the next version of the model. Answers based on training don't link to any source. Blocking a training crawler stops your new content being collected from then on. It doesn't remove anything already collected.

Search crawlers build the index that an AI assistant searches when it answers a question. When ChatGPT, Claude or Perplexity cite a page with a link, the page usually came from their search index. Blocking a search crawler stops that assistant citing your pages in its search answers.

User-triggered fetchers visit a page because a person asked an assistant to read it, for example by pasting a link into ChatGPT. They fetch one page at a time, at that moment. Because a person made the request, several companies say these fetchers may not follow robots.txt.

The companies themselves say that blocking a training crawler doesn't remove you from their search products. Blocking a search crawler does.

The tables below list each company's crawlers, using the companies' own documentation as of 16 October 2026. The first column is the name to use in robots.txt.

Google

Sources: Google's pages on its common crawlers⁠(opens in a new tab) and user-triggered fetchers⁠(opens in a new tab).

Blocking Google-Extended doesn't keep you out of AI Overviews or AI Mode. Google says Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal" (common crawlers⁠(opens in a new tab)). AI Overviews and AI Mode are part of Google Search, so Googlebot crawls for them, and you control them with page-level tags and a setting in Google Search Console. The AI search chapter explains those controls.

Google-Agent has its own list of IP addresses (user-triggered-agents.json). Google is also testing a way to sign some of its requests, called Web Bot Auth⁠(opens in a new tab), so that websites can confirm a request really came from Google.

OpenAI (ChatGPT)

Source: OpenAI's overview of its crawlers⁠(opens in a new tab).

You can control each OpenAI crawler separately, for example by allowing OAI-SearchBot and blocking GPTBot. If you allow both, OpenAI says "we may use the results from just one crawl for both use cases to avoid duplicative crawling". OpenAI says its systems take "~24 hours" to notice a change to your robots.txt.

OpenAI publishes the IP addresses its crawlers use, in files called searchbot.json, gptbot.json, chatgpt-user.json and adsbot.json. It recommends that you allow requests from those addresses if you want to appear in ChatGPT search. When its crawlers fetch your robots.txt, they may add robots.txt to their user agent, which makes those requests easy to spot in your logs.

Anthropic (Claude)

Source: Anthropic's help centre article⁠(opens in a new tab), updated 7 April 2026.

Anthropic says all three of its crawlers "respect 'do not crawl' signals by honoring industry standard directives in robots.txt". That includes Claude-User, which makes it unusual among user-triggered fetchers. Anthropic also follows Crawl-delay, asks you to add rules to each subdomain you want covered, and says its crawlers won't try to get past CAPTCHAs.

Anthropic publishes its crawlers' IP addresses at https://claude.com/crawling/bots.json. It warns that blocking those addresses isn't a reliable way to opt out, because doing so "impedes our ability to read your robots.txt file". To keep Anthropic's crawlers out, use robots.txt.

Perplexity

Source: Perplexity's crawler documentation⁠(opens in a new tab).

Perplexity publishes IP address lists for both crawlers (perplexitybot.json and perplexity-user.json) and recommends allowing them in your firewall. It says changes to robots.txt "may take up to 24 hours" to take effect.

Microsoft (Bing and Copilot)

Microsoft doesn't run a separate AI crawler. Its crawler, Bingbot, builds Bing's index, and Microsoft Copilot answers questions using that index. Bing's Webmaster Guidelines⁠(opens in a new tab) describe how Bing finds and uses content "across Bing search experiences, Copilot, and grounding API results". If Bingbot can't fetch a page, Copilot can't cite it.

To keep a page out of Copilot's answers, Bing uses a tag on the page instead of a robots.txt rule. Its guidelines say "NOARCHIVE prevents content from being used in Copilot responses and grounding results". Bing also lists its other crawlers⁠(opens in a new tab), such as AdIdxBot and BingPreview, and publishes Bingbot's IP addresses.

Apple (Siri, Spotlight and Safari)

Source: Apple's About Applebot⁠(opens in a new tab) page, published 4 September 2026.

Apple's page also says Applebot ignores crawl-delay and may run JavaScript, and Apple publishes Applebot's IP addresses. Separately from training, a nosnippet tag on a page stops Apple using it for general-knowledge answers in Siri and Search.

Other crawlers you may see

Sources: each company's own pages, from Meta⁠(opens in a new tab), Amazon⁠(opens in a new tab), Common Crawl⁠(opens in a new tab), DuckDuckGo⁠(opens in a new tab) and Mistral⁠(opens in a new tab).

New crawlers appear regularly. When you review your robots.txt, check the companies' documentation again, and look in your logs for crawlers you haven't made a decision about.

AI training: reasons to allow it, reasons to block it

Whether to let AI companies train on your content is a business decision, and there's no answer that suits every site. Someone in your business should make the decision, write it down and review it once in a while. Otherwise your site ends up with whatever a template or a hosting provider chose.

Blocking a training crawler stops it collecting your pages from that point on. Anything it collected before stays in the company's data. It doesn't affect the company's search crawler, so your pages can still be cited in that company's AI search answers.

Reasons to block training crawlers

Your content may be what you sell. If your business is journalism, research, data, courses or similar, that content has value that a licensing deal could pay for. Training use usually comes with no payment, no credit and no visitors.

You may not be allowed to share it. Client work, licensed photos and content written by your users can come with terms that rule out handing it on for AI training.

Crawling has a cost. Vercel measured⁠(opens in a new tab) OpenAI's GPTBot making 569 million requests a month across its network in late 2024, and Anthropic's crawler 370 million. For a small site with pages that are expensive to generate, that load can show up on the hosting bill.

Some owners don't want their writing reused by companies they have no relationship with.

Reasons to allow training crawlers

AI assistants answer many questions from what they learned in training, without searching the web. If your products and your facts were never in that training data, an assistant may know less about you, or rely on how other people describe you. No published study measures how much this affects what assistants say, so treat it as a reasonable expectation rather than a proven effect.

Marketing content is written to be read and repeated. Product pages, help articles, guides and announcements exist to spread information about your business. Blocking training doesn't stop competitors, reviewers or forums describing you instead.

Some crawlers combine jobs. Applebot's data may be used for training unless you also block Applebot-Extended. Meta-ExternalAgent collects content for training and for indexing at the same time. And if you allow both of OpenAI's crawlers, one crawl may serve both purposes. Read each company's description before blocking, because blocking training can sometimes mean blocking something else too.

What doesn't change either way

Training and search are separate decisions. By the companies' own descriptions, blocking GPTBot, ClaudeBot, Google-Extended or Applebot-Extended doesn't remove you from ChatGPT search, Claude's search, Google Search or Apple's search features.

Blocking Google-Extended has no effect on AI Overviews or AI Mode.

Neither choice controls how an AI assistant describes your business.

Many businesses settle on a middle option. They allow everything on their public marketing pages and block training crawlers from specific sections, such as paid content, research or community forums. You can do this with Disallow rules for those paths in the training crawlers' group.

Example robots.txt files

Here are two complete files for Fernhill Coffee. The first allows everything. The second allows search and AI search but asks training crawlers to stay out. Adapt the paths to your own site, and test the result before you publish it.

The first file lets every crawler in, apart from the checkout, account and internal search pages:

User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=yes
Disallow: /checkout/
Disallow: /account/
Disallow: /search?

Sitemap: https://www.example.com/sitemap.xml

The second file does the same for search crawlers, but blocks training crawlers from the whole site:

# Private areas are repeated in every group, because
# a crawler follows only the most specific group that names it.

User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Disallow: /checkout/
Disallow: /account/
Disallow: /search?

# Search and answer crawlers: allowed (named for clarity)
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Disallow: /checkout/
Disallow: /account/
Disallow: /search?

# Training crawlers and tokens: opted out
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
User-agent: CCBot
Disallow: /

Sitemap: https://www.example.com/sitemap.xml

Reading the second file:

• Lines starting with # are comments. Crawlers ignore them, so they're a good place to explain your choices to whoever edits the file next.
• The first group covers every crawler that isn't named elsewhere, including Googlebot and Bingbot.
• The second group names the three AI search crawlers. Naming them isn't strictly necessary, because they'd follow the first group anyway. It makes the decision visible, and it repeats the private paths because of the one-group rule explained earlier.
• The third group names the training crawlers and blocks them from everything with Disallow: /.

Blocking Meta-ExternalAgent also stops it indexing your pages for Meta's products, because Meta uses it for both jobs.

If you use Cloudflare's managed robots.txt setting, Cloudflare adds its own lines to your file. According to its documentation⁠(opens in a new tab), these include search=yes, ai-train=no and an experimental use=reference value, and its example blocks several training crawlers. Check whether that matches the decision your business made.

If your site is built with Next.js

Developers using the Next.js framework can generate robots.txt from a file called app/robots.ts. The MetadataRoute.Robots type accepts a list of crawler names for each rule, as the Next.js documentation⁠(opens in a new tab) shows. This is the second example file above:

// app/robots.ts
import type { MetadataRoute } from 'next'

const privatePaths = ['/checkout/', '/account/', '/search?']

export default function robots(): MetadataRoute.Robots {
  return {
    rules: [
      { userAgent: '*', disallow: privatePaths },
      {
        userAgent: ['OAI-SearchBot', 'Claude-SearchBot', 'PerplexityBot'],
        disallow: privatePaths,
      },
      {
        userAgent: [
          'GPTBot', 'ClaudeBot', 'Google-Extended',
          'Applebot-Extended', 'Meta-ExternalAgent', 'CCBot',
        ],
        disallow: '/',
      },
    ],
    sitemap: 'https://www.example.com/sitemap.xml',
  }
}

This method has no setting for the Content-Signal line. If you want it, your developer can publish robots.txt as a plain file or generate it with a route handler instead.

Crawlers that fetch pages for a person

User-triggered fetchers behave differently from other crawlers. ChatGPT-User, Perplexity-User, Google-Agent, Meta-ExternalFetcher and Amzn-User all visit a page because someone asked an assistant to, and each company says robots.txt may not apply to them. Claude-User is the exception. Anthropic says it follows robots.txt.

If you need to stop these visits, robots.txt won't do it reliably. You'd need a rule in your firewall instead, which is covered in the next section.

Think about who those visitors are before you block them. A user-triggered fetch happens because a person asked an assistant about your page, often while comparing products or deciding what to buy. Blocking the fetch means the assistant can't read your page for that person.

Check your CDN and firewall, not just robots.txt

robots.txt isn't the only thing that can stop a crawler. Many websites sit behind services that filter traffic before it reaches the site, and those services can turn away a crawler that robots.txt allows.

A CDN (content delivery network) is a service, such as Cloudflare or Fastly, that stores copies of your pages on servers around the world so they load quickly. Most CDNs also filter out suspicious traffic.

A firewall, often called a WAF (web application firewall) on websites, is a set of rules that blocks or questions requests that look harmful. Many hosting providers and CDNs include one.

Bot protection is a feature of many CDNs and firewalls that tries to tell people apart from automated programs. When it's unsure, it can show a challenge page, a "check you're human" step. A person can pass the check. A crawler that reads only HTML can't, so to the crawler the challenge page is the same as being blocked. Anthropic also says its crawlers won't try to get past CAPTCHAs (Anthropic⁠(opens in a new tab)).

Google lists this as a basic requirement for its AI features. A site should make sure "crawling is allowed in robots.txt, and by any CDN or hosting infrastructure" (Google⁠(opens in a new tab)).

AI crawler blocking in CDNs became much more common in July 2025, when Cloudflare started asking the owner of every new domain whether to allow AI crawlers. Cloudflare said more than a million of its customers had already switched on its one-click AI crawler block since September 2024 (Cloudflare press release⁠(opens in a new tab)).

To check what's really happening:

1. Find out which CDN, hosting provider or security service your site uses. If you don't know, your developer or hosting provider will.
2. Look through its settings for anything about bots, AI crawlers, "bot fight mode", rate limits or challenges. Note what each one is set to.
3. Ask for your server or CDN logs, and check what response each crawler you want to allow is actually getting. The section on logs below shows what to look for. A 200 means the crawler got the page. A 403 (forbidden), a 429 (too many requests) or a challenge page means it was turned away.
4. If your firewall only lets in crawlers it recognises by IP address, make sure it uses each company's published lists, and that those lists update automatically. OpenAI and Perplexity both recommend allowing their published ranges.

Checking that a crawler is genuine

Because anyone can fake a user agent, you shouldn't trust the name in a request on its own. This matters when you're reading logs, because fake crawlers can make your numbers look wrong, and when you allow crawlers through a firewall. There are two ways to check a request.

The first is a reverse DNS lookup. Every request comes from an IP address, a numeric address such as 66.249.66.1. A reverse DNS lookup asks which domain name that address belongs to. Google's verification guide⁠(opens in a new tab) says a genuine Google crawler's address will belong to googlebot.com, google.com or googleusercontent.com. Then you do a normal lookup on that name and check it leads back to the same IP address. Apple and Common Crawl support the same check.

The second is a published IP list. Google, OpenAI, Anthropic, Perplexity, Apple, Bing, Amazon, Common Crawl, DuckDuckGo and Mistral all publish files listing the IP addresses their crawlers use. If a request's address is on the right list, it's genuine.

These lists move from time to time. In 2026 Google moved its files to developers.google.com/crawling/ipranges/ and renamed googlebot.json to common-crawlers.json. The old addresses now redirect to the new ones (Google Search Central Blog⁠(opens in a new tab)). If a script on your side downloads the old file, check that it follows redirects.

A newer method, Web Bot Auth, lets a crawler sign its requests so a website can verify them cryptographically. Google signs some of Google-Agent's requests⁠(opens in a new tab) this way. It's at an early stage, and you don't need it yet.

Reading your server logs

Your server logs are files that record every request your website receives: who asked, for which page, when, and what response they got. They're the only complete record of what crawlers actually did on your site. Your hosting provider or CDN keeps them, and you may need to ask your developer for access.

A single line from a log looks like this:

66.249.66.1 - - [16/Oct/2026:09:14:02 +0100] "GET /subscriptions HTTP/1.1" 200 18432 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

From left to right, it says that a computer at IP address 66.249.66.1 asked for /subscriptions at 9:14 on 16 October. The server answered with a 200 (success) and sent 18,432 bytes. The last part is the user agent, which here says Googlebot.

Once a month, check your logs for these things:

• which crawlers visited, how often, and which parts of the site they spent time on
• which status codes each crawler received, especially 403s, 429s and server errors
• whether important new pages were fetched within a few days of being published
• whether the AI search crawlers you allow, such as OAI-SearchBot, Claude-SearchBot and PerplexityBot, are reaching the pages you most want cited
• whether new crawlers have appeared that you haven't made a decision about

For developers, two commands give a quick first look at a standard log file:

# Count requests by AI and search crawler user agent
grep -oiE 'GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User|Google-Agent|Applebot|Meta-ExternalAgent|Amazonbot|CCBot|Googlebot|bingbot' access.log \
  | sort | uniq -c | sort -rn

# Which status codes are AI crawlers getting?
grep -iE 'GPTBot|OAI-SearchBot|ClaudeBot|Claude-SearchBot|PerplexityBot' access.log \
  | awk '{print $9}' | sort | uniq -c | sort -rn

The second command tends to be the more useful. When Vercel studied AI crawlers⁠(opens in a new tab) in late 2024, about 34.8% of ChatGPT's crawler requests and 34.2% of Claude's ended in a 404, compared with 8.2% for Googlebot. Many were requests for old pages and files that no longer existed. If your logs show crawlers repeatedly asking for old addresses, redirect them to the closest current page, as the redirects chapter explains.

Testing what a crawler receives

A developer can request a page while pretending to be a crawler, using a command-line tool called curl, to see what the server sends back. This command asks for Fernhill's pricing page using OpenAI's published user agent, and prints only the status code:

curl -s -o /dev/null -w "%{http_code}\n" \
  -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot" \
  https://www.example.com/pricing

OpenAI notes that the version number in the user agent may change (OpenAI⁠(opens in a new tab)). Treat the result as a quick check, not proof. Some CDNs check the IP address as well as the name, so a request from your own computer can be treated differently from one sent by the real crawler. Your logs are better evidence.

Crawl speed, page size and crawl budget

Crawlers slow down when your server struggles. Google adjusts how fast it crawls each site using what it calls a "crawl capacity limit". The limit goes up when your server responds quickly and goes down when it slows or returns errors (Google's crawl budget guide⁠(opens in a new tab)). Google treats 429 and 5xx responses as a signal to slow down, and if the errors continue, it eventually drops the affected pages from its index (Google⁠(opens in a new tab)). A fast, reliable server helps every crawler, not only Google's.

There's also a limit on how much of each page Google reads. Googlebot reads the first 2 MB of an HTML file, measured before compression, and ignores anything after that (Googlebot⁠(opens in a new tab)). Each file the page loads, such as a script or stylesheet, has its own 2 MB limit. Google's other crawlers default to 15 MB, and Google says the limit "is not set in stone and may change over time" (Inside Googlebot⁠(opens in a new tab)). Most pages are far smaller than 2 MB. Pages that carry large amounts of code or data in the HTML can get close, so keep the page's title, description and main content near the top of the HTML.

Crawl budget is the number of pages a search engine is willing to crawl on your site in a given period. Most sites don't need to think about it. Google says its crawl budget guide⁠(opens in a new tab) is for sites with roughly a million or more unique pages that change weekly, sites with 10,000 or more pages that change daily, or sites with a large share of pages stuck as "Discovered – currently not indexed" in Search Console.

If your site is that large, Google recommends these steps:

• Combine duplicate pages, so crawlers spend their time on unique ones.
• Use robots.txt to stop crawlers visiting low-value addresses, such as every combination of filters and sort orders on a shop.
• Don't rely on a noindex tag to save crawling, because Google still has to fetch the page to read the tag.
• Return a 404 or 410 for pages you've removed for good, and fix pages that show an error message but return a 200.
• Keep your sitemaps up to date, and avoid chains of several redirects in a row.
• Make pages fast, and support HTTP caching so the server can reply "not modified" (a 304) when a page hasn't changed.

Common mistakes

These are the problems that most often stop crawlers reaching a site:

• A Disallow: / line left over from when the site was being built. It blocks every crawler from the whole site. Check robots.txt after every launch.
• Blocking the files that make pages work, such as folders called /assets/, /static/ or /_next/. Google and Apple need them to see the finished page.
• Using robots.txt to remove a page from search. Use a noindex tag on a page crawlers can reach instead.
• A robots.txt that returns a server error. Google stops crawling the site for up to 12 hours. If you have no rules, let the address return a 404.
• Forgetting subdomains. blog.example.com needs its own robots.txt.
• Adding a group for a named crawler without repeating the shared rules in it.
• Blocking AI crawlers by accident, through a CDN default, a copied template or a security plugin.

What to do next

1. Open your robots.txt and check it doesn't block your whole site.
2. Decide, as a business, whether to allow AI training crawlers, and write the decision down.
3. Make sure the AI search crawlers you want, such as OAI-SearchBot, Claude-SearchBot and PerplexityBot, aren't blocked.
4. Repeat your shared rules in every named group.
5. Check your CDN and firewall settings for bot blocking and challenges.
6. Ask for 30 days of logs and check the status codes each crawler received.
7. Check robots.txt again after any change to your hosting, CDN or website platform.

Further reading

• Introduction to robots.txt⁠(opens in a new tab), Google's beginner-level explanation of what the file does and doesn't do.
• How Google interprets the robots.txt specification⁠(opens in a new tab), the full rules, including status codes, caching and how conflicting rules are resolved.
• Overview of OpenAI crawlers⁠(opens in a new tab), OpenAI's crawlers and the IP addresses they use.
• Does Anthropic crawl data from the web?⁠(opens in a new tab), Anthropic's crawlers and how to control them.
• Perplexity crawlers⁠(opens in a new tab), Perplexity's two crawlers.
• Google's common crawlers⁠(opens in a new tab), including what Google-Extended does and doesn't control.
• Giving users choice with Cloudflare's new Content Signals Policy⁠(opens in a new tab), the background to the Content-Signal line.