Indexing, canonicals and duplicate content

Andy Bright

Director of Product and Technology

A search engine can fetch one of your pages and still not show it. Sometimes it leaves the page out of its results entirely. Sometimes it shows a different address for the page from the one you wanted, such as a version with a tracking code on the end. Both problems are usually caused by small settings in the page's code, and both can be fixed once you know where to look.

This chapter explains:

• what it means for a page to be indexed
• how to keep a page out of search results, and out of AI answers, when you want to
• what the codes your server sends back tell search engines
• what duplicate content and canonical URLs are, and how to make search engines pick the right version of a page
• how to check which of your pages are indexed, and why others aren't

It builds on How search engines and AI assistants find your pages, which introduces the index and the other steps a page goes through.

What indexing means

A search engine's index is its database of the pages it has stored and can show in results. When a page is indexed, it's in that database. Being crawled is a separate, earlier step. A crawler can fetch a page and the search engine can still decide not to store it.

Indexing now decides whether a page can appear in AI answers too. Google says a page must be "indexed and eligible to be shown in Google Search with a snippet" before it can appear as a link in AI Overviews or AI Mode (AI features and your website⁠(opens in a new tab)). Microsoft's Bing says "many LLMs rely on data grounded in the Bing index or other search indexes" (Bing Webmaster Blog⁠(opens in a new tab)). An LLM, or large language model, is the technology behind AI assistants such as ChatGPT. So a page that isn't in a search index is usually missing from AI answers as well.

Keeping a page out of search results

Some pages shouldn't appear in search results. Fernhill Coffee, the example shop in this guide, has an order confirmation page, a "thanks for subscribing" page and its own internal search results. None of these are useful to someone arriving from Google.

To keep a page out of results, you add a noindex instruction. The usual way is a line in the page's <head>, the part of the HTML that holds information about the page rather than the visible content:

<meta name="robots" content="noindex, follow">

This is called a robots meta tag. noindex tells search engines not to store the page. follow tells them they can still follow the links on it. You can also aim the tag at one crawler by name. This version only applies to Googlebot:

<meta name="googlebot" content="nosnippet">

Files that aren't web pages, such as PDFs, have no <head> to put a tag in. For those, the server sends the instruction in an HTTP header, a line of information the server attaches to a file when it sends it. That header is called X-Robots-Tag. Google's robots meta tag documentation⁠(opens in a new tab) gives examples for the two most common web servers, which a developer can adapt:

# NGINX: keep all PDFs out of the index
location ~* \.pdf$ {
  add_header X-Robots-Tag "noindex, nofollow";
}

# Apache (.htaccess)
<Files ~ "\.pdf$">
  Header set X-Robots-Tag "noindex, nofollow"
</Files>

Other ways noindex goes wrong

A noindex added or removed by JavaScript is unreliable. Google renders every page that returns a 200 status "unless a robots meta tag or header tells Google not to index the page" (JavaScript SEO basics⁠(opens in a new tab)). So if the HTML says noindex and a script is supposed to remove it, Google may never run the script. In December 2025 Google added to its documentation⁠(opens in a new tab) that a noindex changed by JavaScript "is not well defined and might change". The noindex, or its absence, should be in the HTML the server sends.

The tag belongs in the <head>. Google said in March 2026 that it "will respect robots meta tags in the body section" of a page too (robots meta tag documentation⁠(opens in a new tab)). Other crawlers may not, and a tag in the wrong place usually means something is wrong with the page template.

The most damaging mistake is a noindex copied from a test site. Suppose Fernhill rebuilds its website on a private test address first, with noindex on every page so the unfinished site stays out of Google. If the new site goes live with those tags still in place, Google starts dropping Fernhill's pages from its results. Check for noindex on your important pages after every launch or redesign.

Controlling snippets and AI answers

A snippet is the short description under a page's link in search results. Google uses the same text when it quotes a page in AI Overviews and AI Mode. Three instructions control it, and they're described in Google's robots meta tag documentation⁠(opens in a new tab).

nosnippet tells search engines not to show a text snippet for the page. Google says it also stops the content being used "as a direct input for AI Overviews and AI Mode".

max-snippet:[number] limits a snippet to that many characters. Google says the same limit applies to how much of the page its AI features can use.

data-nosnippet works on part of a page instead of the whole page. You add it to a section of the HTML, and search engines leave that section out of snippets. Google supports it on span, div and section elements. Fernhill sells to cafés as well as to the public, and doesn't want its trade terms quoted in search results or AI answers:

<p>We supply cafés across the UK.
  <span data-nosnippet>Trade discounts start at 15% for orders over 20 kg.</span>
</p>

These controls have a cost. nosnippet also removes the ordinary description from search results, so fewer people are likely to click. And because Google's AI features only use pages that are "eligible to be shown in Google Search with a snippet", a page with nosnippet can't appear as a link in them at all (AI features and your website⁠(opens in a new tab)).

The usual approach is to use data-nosnippet for specific passages, such as pricing terms, legal text or personal details, and keep nosnippet for pages where you've accepted losing the snippet. If you want to keep your whole site out of Google's AI features, Search Console has a separate setting for that, covered in the AI search chapter.

All the controls in one place

This table lists every control in this chapter, plus two from elsewhere in the guide that people often confuse with them.

The last two rows are often mixed up. AI Overviews and AI Mode are part of Google Search, so they follow Search's rules: Googlebot, robots meta tags and the Search Console setting. Google-Extended covers Gemini and model training, so blocking it doesn't take you out of AI Overviews.

What status codes tell search engines

Every time a crawler requests a page, your server replies with an HTTP status code, a three-digit number that says what happened. Search engines read it before they look at anything on the page. Google's status code documentation⁠(opens in a new tab) explains how it treats each group of codes:

When one page redirects to another, which redirects to a third, that's called a redirect chain. Googlebot follows up to 10 steps in a chain and gives up after that. Aim for a single step. The redirects chapter explains redirects in detail.

Soft 404s

A soft 404 is a page that tells visitors "not found" but tells crawlers everything is fine, by returning a 200 status.

Fernhill stops selling its Kenyan single-origin coffee. The product page now says "Sorry, this coffee is no longer available", but the server still returns a 200. To a crawler, it looks like a normal page with very little on it. Search Console reports pages like this as soft 404s in its Page indexing report. They waste crawlers' time and can stay in the index looking thin.

There are two better options for a page like this. If there's a close replacement, such as another Kenyan coffee, redirect the old address to it. If there isn't, return a 404 (not found) or a 410 (gone for good).

Soft 404s are common on sites that send every unknown address to the home page or to a sign-in page. They also appear on single-page apps, which are websites built mostly in JavaScript. The JavaScript chapter explains how to fix them there.

Duplicate content and canonical URLs

Duplicate content is the same page, or almost the same page, available at more than one URL. It's very common, and it's usually caused by how a website works rather than by anyone copying text.

Fernhill's subscription page can be reached at all of these addresses:

• https://www.example.com/subscriptions
• https://example.com/subscriptions, without www
• http://www.example.com/subscriptions, without the secure https
• https://www.example.com/subscriptions/, with a slash on the end
• https://www.example.com/subscriptions?utm_source=newsletter, with a tracking code from an email campaign

Other common causes are capital letters in addresses, sort and filter options in a shop, print versions of pages, and the same product appearing under several categories.

To a crawler, each address is a separate page with the same content. The search engine groups them together and picks one to store and show. The chosen address is called the canonical URL, and the process of choosing it is called canonicalisation. It happens whether or not you do anything. Your job is to make it obvious which address you want, so the search engine doesn't pick the version with the newsletter tracking code.

How to tell search engines which version you want

Google's canonical URL guide⁠(opens in a new tab) lists three main ways to say which version you prefer, and how much weight it gives each one:

The same guide says Google prefers addresses that start with https over http, and addresses that are part of a set of language versions, when choosing a canonical. The international SEO chapter explains language versions.

The canonical tag goes in the page's <head>. On Fernhill's subscription page, it looks like this, and it's the same whichever address the page was reached at:

<link rel="canonical" href="https://www.example.com/subscriptions">

For PDFs and other files, the server can send the canonical as an HTTP header instead, as shown in Google's guide⁠(opens in a new tab):

Link: <https://www.example.com/brewing-guide.pdf>; rel="canonical"

Rules for canonical tags

These rules come from Google's guide⁠(opens in a new tab), with a few practical additions.

Write the full address in the tag, including https:// and the domain. https://www.example.com/subscriptions works reliably. /subscriptions on its own can be misread.

Give every page you want indexed a canonical tag that points to itself. This is called a self-referencing canonical. It means that when someone adds a tracking code to the address, the tag still names the clean version.

Use one canonical tag per page. If a website platform adds one and a plugin adds another, and they disagree, search engines usually ignore both.

Don't use robots.txt or noindex to deal with duplicates. If robots.txt blocks a duplicate, the search engine can't see that it's a duplicate. A noindex removes the page from search entirely, instead of passing its value to the preferred version.

Make every signal point to the same address. Suppose Fernhill's canonical tag names /subscriptions, its sitemap lists /subscriptions/ with a slash, and its menu links to /subscriptions?ref=menu. The signals disagree, so the search engine has to decide for itself.

Link to the preferred address from your own pages. Links within your site are a small signal, but they add up. The site architecture chapter covers internal links.

Choose one version of your domain, https with or without www, and redirect the others to it. Do the same with trailing slashes and capital letters. Pick one form and redirect the other.

A canonical tag is a suggestion, not an order. Google can choose a different canonical if your other signals point elsewhere. The URL Inspection tool in Google Search Console shows both the canonical you named and the one Google chose. After you fix conflicting signals, give Google time to notice. Its canonicalisation troubleshooting guide⁠(opens in a new tab) says it "might hold pages in a duplicate cluster for up to two weeks" before it looks at them again.

Canonical tags set by JavaScript

This section is mainly for developers. Some websites add the canonical tag with JavaScript instead of including it in the HTML the server sends. Google clarified its rules⁠(opens in a new tab) for this in December 2025, and its JavaScript SEO basics⁠(opens in a new tab) describe two acceptable approaches:

1. The preferred approach is to put the canonical in the HTML the server sends, and make sure no script changes it. Google says "you shouldn't use JavaScript to change the canonical URL to something else" than the one in the original HTML.
2. If that really isn't possible, leave the canonical out of the original HTML and add it only with JavaScript, so there's only ever one value.

The thing to avoid is two different values, one in the original HTML and another after the JavaScript runs. Use the first approach if you can. Most AI crawlers don't run JavaScript, as the JavaScript chapter explains, so a canonical added by a script is invisible to them.

On sites built with the Next.js framework, set the canonical through its Metadata API, so it's included in the HTML the server sends. With metadataBase set, a short path becomes a full address⁠(opens in a new tab) automatically:

// app/layout.tsx
export const metadata = {
  metadataBase: new URL('https://www.example.com'),
}

// app/coffees/[slug]/page.tsx
export async function generateMetadata({ params }) {
  const { slug } = await params
  return {
    alternates: { canonical: `/coffees/${slug}` },
  }
}

Filtered category pages

Online shops usually let visitors filter a category page. Fernhill's coffee page can be filtered by roast, by grind and by origin, and sorted by price. Each combination creates a new address, such as /coffees?roast=dark&grind=espresso&sort=price. A shop with a few thousand products can end up with millions of these addresses. Google calls this faceted navigation.

Most of those addresses aren't worth having in search results, and crawling them all wastes time that crawlers could spend on your real pages. Google's guide to faceted navigation⁠(opens in a new tab) recommends different approaches depending on whether you want any filtered pages in search.

If you don't need filtered pages in search, block the filter parameters in robots.txt, or put the filters after a # in the address, which Google generally doesn't crawl. Keep the product pages and the unfiltered category page crawlable.

If some filtered pages should appear in search, keep the addresses tidy. Separate the parameters with &, always put filters in the same order, and never repeat a filter. When a combination has no products, return a 404 rather than redirecting to a general "not found" page.

Canonical tags and nofollow links can help a little, but Google says both are "generally less effective in the long term" than robots.txt or using #. A nofollow link only helps if every link to that filtered address carries it.

This robots.txt is adapted from Google's example. It blocks Fernhill's roast, grind and sort filters, but lets crawlers see the page that shows every roast:

User-agent: *
Disallow: /*?*roast=
Disallow: /*?*grind=
Disallow: /*?*sort=
Allow: /*?roast=all$

Test patterns like these carefully, because a rule that's too broad can block pages you want indexed. Don't use noindex to stop crawlers visiting filtered pages either. Google still has to fetch a page to read its noindex, so it doesn't reduce crawling⁠(opens in a new tab).

Some filtered pages match what people actually search for, such as "decaf espresso beans". This guide recommends turning those into proper pages, with their own text, a clean address, a self-referencing canonical and a link from your menu, rather than relying on a filtered version being indexed. That's a recommendation, not Google guidance.

Duplicate content and AI answers

Duplicate content doesn't get your site penalised. Bing's post on the subject⁠(opens in a new tab) says it "doesn't trigger search penalties on its own". The cost is that duplicates divide your signals between several addresses, and the search engine has to guess which version matters.

The same post explains the effect on AI answers. Bing says LLMs "group near-duplicate URLs into a single cluster and then choose one page" to represent the group. If the versions are very similar, the AI system may pick an out-of-date one, or one you never meant to promote.

Bing describes four common sources of duplicates. This is how to handle each one:

1. Syndicated content, which is your articles republished on other sites. Ask those sites to add a canonical tag pointing to your original, or to rewrite enough that the pages are clearly different.
2. Old campaign and landing pages. When a campaign ends, redirect its page to the permanent page with a 301, rather than leaving near-copies online.
3. Regional versions. Pages for different countries that only change the currency symbol look like duplicates. Make real local differences, and connect the versions with hreflang tags, explained in the international SEO chapter. Google says translated pages are only treated as duplicates if the main content is left untranslated⁠(opens in a new tab).
4. Technical variants, such as www and non-www. Fix these with redirects on the server, then back them up with canonical tags and consistent internal links.

How to check what's indexed

Google Search Console is Google's free service for website owners, and its Page indexing report shows which of your pages are indexed and why others aren't. These are the statuses to watch:

• "Crawled – currently not indexed" means Google fetched the page but hasn't stored it. It may still be indexed later.
• "Discovered – currently not indexed" means Google knows the address but hasn't fetched it yet.
• "Duplicate, Google chose different canonical than user" means Google thinks the page is a duplicate, and picked a different address from the one your canonical tag names.
• "Soft 404" means the page looks like an error page but returns a 200.

Compare the number of indexed pages with the number you expect. A sudden drop can be the first sign of a stray noindex or a robots.txt change.

To check a single page, paste its address into the URL Inspection tool at the top of Search Console. It shows whether the page is indexed, which canonical Google chose, and the HTML Google saw after running the page's JavaScript.

Bing Webmaster Tools offers the same kind of inspection for Bing, plus reports on your sitemaps, your IndexNow submissions and how often Copilot cites you. Searching Google for site:yourdomain.com gives a rough idea of what's indexed, but the count isn't reliable, so use Search Console for real numbers.

When a page isn't indexed

Work through these checks in order. Each one rules out a cause before you move on to the next.

1. Check the status code. The page should return a 200, with no redirect in the way.
2. Check robots.txt, to make sure it doesn't block the page.
3. Check for noindex. Look in the HTML the server sends and in the HTML after JavaScript runs, and ask a developer to check the X-Robots-Tag header.
4. Check the canonical tag. It should be a full address, it should point to the page itself, and it should be the same before and after JavaScript runs.
5. Check that the signals agree. The sitemap, internal links, hreflang tags and canonical tag should all name the same address.
6. Inspect the page in Search Console and Bing Webmaster Tools, and compare the canonical you named with the one they chose.
7. Look at the content. If the page is very short, or almost the same as another page, the fix may be to improve or combine the pages rather than change any code.

A developer can check the status code, headers and tags from the command line:

# Status code, redirects, X-Robots-Tag and Link headers
curl -sI https://www.example.com/subscriptions | grep -iE '^HTTP|^location|^x-robots-tag|^link'

# Canonical and robots meta tags in the HTML the server sends
curl -s https://www.example.com/subscriptions \
  | grep -ioE '<link[^>]+rel="canonical"[^>]*>|<meta[^>]+name="robots"[^>]*>'

What to do next

1. Open Search Console's Page indexing report and look for important pages that aren't indexed.
2. Check your most important pages for a noindex that shouldn't be there, especially after a redesign.
3. Make sure no page is blocked in robots.txt and marked noindex at the same time.
4. Add a self-referencing canonical tag to every page you want indexed.
5. Choose one version of your domain and redirect the others to it.
6. If you run a shop with filters, decide which filtered pages, if any, belong in search, and block the rest.
7. Use URL Inspection to check that Google chose the canonical you named on your key pages.

Further reading

• AI features and your website⁠(opens in a new tab), Google's short page on what AI Overviews and AI Mode need from a page.
• Robots meta tag, data-nosnippet, and X-Robots-Tag specifications⁠(opens in a new tab), every indexing and snippet instruction Google supports.
• How to specify a canonical URL⁠(opens in a new tab), Google's guide to canonical URLs and how much weight each signal carries.
• Fix canonicalization issues⁠(opens in a new tab), for when Google picks a different canonical from yours.
• How HTTP status codes affect Google Search⁠(opens in a new tab), how Google treats every response code.
• Managing crawling of faceted navigation URLs⁠(opens in a new tab), if your shop has filtered category pages.
• Does duplicate content hurt SEO and AI search visibility?⁠(opens in a new tab), Bing on how duplicates affect search results and AI answers.