AI search and answer engines

Andy Bright

Director of Product and Technology

AI assistants such as ChatGPT, Claude, Perplexity, Microsoft Copilot and Gemini answer questions directly, and often link to the pages they used. So do Google's AI Overviews, the AI-written summaries at the top of some Google results, and AI Mode, Google's chat-style search. Getting your pages used and linked in those answers is called answer engine optimisation (AEO) or generative engine optimisation (GEO). The two names mean the same thing.

Most of the technical work behind AEO is the SEO covered in the earlier chapters of this guide. AI assistants find pages in much the same way search engines do. Google's guide to optimising for generative AI⁠(opens in a new tab), published in May 2026, says that "optimizing for generative AI search is optimizing for the search experience, and thus still SEO".

Some things are different, though. There are more crawlers doing different jobs, each AI company has its own settings, there are new reports to read, and a lot of advice about AI search does nothing. This chapter explains:

• how AI assistants find and choose pages
• what Google's AI features need from your site
• the settings that control each AI service, and how to avoid blocking one by accident
• what to fix first, and what you can skip
• what llms.txt is, and whether you need one
• how to make your site usable by AI agents
• how to measure whether AI assistants use your pages

How AI assistants find pages

The first chapter introduced two ideas from Google's guide⁠(opens in a new tab).

The first is grounding, also called retrieval-augmented generation (RAG). When someone asks a question, the AI system searches an index for relevant pages, reads them, and writes its answer from what it found, with links to the pages it used.

The second is query fan-out. The AI system turns one question into several related searches and runs them all. When someone asks "what's a good coffee subscription in the UK?", it might also search for "coffee subscription prices" and "ethical coffee roasters". A page can be used in the answer because it matches one of those extra searches.

The SEO consultant Lily Ray, writing in March 2026⁠(opens in a new tab), argues that every major AI search product works this way, which is why ordinary SEO carries over to AI search. Bing says⁠(opens in a new tab) "many LLMs rely on data grounded in the Bing index or other search indexes". An LLM, or large language model, is the kind of AI model behind these assistants.

AI companies don't always say which search index they use. Google's AI features use Google's index, and Microsoft Copilot uses Bing's. Lily Ray points to independent reports⁠(opens in a new tab) that ChatGPT uses Google's results, but notes that OpenAI hasn't confirmed this.

Assistants also answer some questions from what their model learned during training, without searching at all. Those answers don't link to any source. Whether to let AI companies train on your content is a separate decision, covered in the crawlability chapter.

Seven stages, each with its own failure

For Fernhill Coffee's subscription page to be cited in an answer, seven things have to happen in order. If the page isn't appearing in AI answers, check each stage in turn, because each one depends on the one before.

Technical SEO mostly decides the first three stages, and affects the fourth. The last three depend much more on the content itself, the business's reputation, and how many people are asking.

Report each stage separately. A page can be crawled but never cited, and cited without the answer recommending the business. A business can also be mentioned in an answer without any of its own pages being cited.

What Google's AI features need

Google sets out what AI Overviews and AI Mode need in its AI optimisation guide⁠(opens in a new tab) and its page on AI features and your website⁠(opens in a new tab):

1. The page must be "indexed and eligible to be shown in Google Search with a snippet". A snippet is the short piece of text shown under a search result.
2. The site must be included in Search generative AI features, a setting in Google Search Console. Sites are included unless someone changes the setting, which the next section explains.

That's all. Google says there are "no additional technical requirements", that "You don't need to create new machine readable files", and that there's "no special schema.org markup you need to add".

Google's list of good practices also includes a good page experience, allowing crawling "in robots.txt, and by any CDN or hosting infrastructure", and making important content "easily findable through internal links" (AI features and your website⁠(opens in a new tab)). A CDN, or content delivery network, is a service many sites use to deliver their pages quickly from servers around the world. CDNs can block crawlers too.

Google has said that clicks from search results with AI Overviews are "higher quality, where users are more likely to spend more time on the site" (Google Search Central Blog, May 2025⁠(opens in a new tab)). Google hasn't published the data behind that claim.

The settings for each AI service

Each AI company gives site owners different settings, and some do two different jobs. Mixing them up can take you out of ChatGPT search when all you meant to do was stop OpenAI training its models on your pages.

The table has three columns. "To be eligible" is what you need for your pages to appear in that service's answers. "To opt out of AI answers" is how to keep your pages out of those answers. "To opt out of training" is how to stop the company using your pages to train its AI models. Most settings are rules in your robots.txt file, which the crawlability chapter explains, along with the crawler names. The table reflects each company's documentation as of 16 October 2026.

For example, if Fernhill wanted to appear in ChatGPT search but keep its pages out of OpenAI's training, it would allow OAI-SearchBot and disallow GPTBot in its robots.txt.

Other crawlers, including those from Meta, Amazon, DuckDuckGo and Mistral, are covered in the crawlability chapter.

Google's Search generative AI setting

Google announced this setting in June 2026⁠(opens in a new tab) and made it available to all sites on 31 August 2026. Every site in Google Search Console now has it, under Settings › Search generative AI (Search Console Help⁠(opens in a new tab)). There are three options:

• "Include" is the default. Your pages can appear and be linked in AI Overviews, AI Mode and the AI features in Google Discover.
• "Exclude" means your pages won't be linked or used to support answers in those features. Google says "You won't receive any traffic or impressions" from them.
• "Inherit from parent" applies to sections of a site that are set up as their own property in Search Console, and is the default for them. It copies the setting of the main site.

Google's help page⁠(opens in a new tab) lists some limits:

• The setting "isn't used as a ranking or inclusion signal" for the rest of Google Search.
• It "doesn't affect AI training", which is what Google-Extended controls.
• Excluding your site doesn't take it out of Google entirely: "Your site's content may still be used to power Google Search as a whole."
• It "doesn't override publishers' other choices to participate in particular services, such as Merchant Center and Google Ads."
• Changes "generally take a few days", and some pages take longer to drop out because Google keeps stored copies for a while.

Search Engine Journal linked the worldwide rollout⁠(opens in a new tab) to rules set for Google by the UK's Competition and Markets Authority (CMA). It reported that the CMA has given Google until March 2027 to add settings for individual pages, and that "the regulator is also expecting click-through data". This area is likely to keep changing.

Whether to exclude your site is a business decision. Excluding it removes your links from Google's AI features completely. For most businesses that want customers to find them, that's a high price. Publishers who want to license their content to AI companies may decide differently. If you're unsure, look at how often your pages appear in Google's AI features first, using the report described in Measuring AI visibility. Then check the setting on every site you manage, in case someone changed it and forgot.

Training and answers are separate

Every AI company uses separate crawlers for training and for answering questions, and blocking one doesn't block the other. OpenAI says⁠(opens in a new tab) that blocking GPTBot doesn't remove a site from ChatGPT search, but blocking OAI-SearchBot does. The arguments for and against allowing training are in the crawlability chapter.

Google's John Mueller, asked in June 2026 how to prepare a site for AI agents, answered⁠(opens in a new tab): "don't block agents. I think that hurdle will be the biggest, for most sites".

What to fix first

This list puts the work in order, based on the documentation above and the crawler research covered in this guide. You can give it to your developer as it is.

1. Let AI search crawlers in, through robots.txt and through your CDN or firewall. If a crawler gets an error or a security check instead of the page, nothing else on this list helps. See the crawlability chapter.
2. Put important content in the raw HTML. The best available research, Vercel's crawler study⁠(opens in a new tab), found that the major AI crawlers it studied didn't run JavaScript. Aleyda Solis's AI search checklist⁠(opens in a new tab) gives the same advice. Prices, product details, stock levels, policies and contact details are the content most often hidden this way. See the JavaScript chapter.
3. Get indexed in both Google and Bing. Google's AI features use Google's index, Copilot uses Bing's, and other assistants also use search indexes.
4. Check that the Search Console generative AI setting is what you intend.
5. Keep one clear main page for each topic. Bing says⁠(opens in a new tab) AI systems group near-identical pages and pick one to represent them. See the indexing chapter.
6. Keep your dates accurate, and tell search engines when pages change. That means an accurate lastmod date in your sitemap, a visible "last updated" date, dateModified in your structured data, and IndexNow notifications to Bing when you publish. Don't change dates unless the page has changed. See the sitemaps chapter.
7. Make it clear who your business is. Use Organization structured data with sameAs links, a clear About page, and the same facts on your site and your profiles elsewhere. See the structured data chapter.
8. Write pages that answer questions clearly. Use headings that say what each section covers, put the answer near the top, write short paragraphs that make sense on their own, use tables for comparisons, and give sources for claims. Bing's recommendations for content it cites⁠(opens in a new tab) include clear headings, tables and FAQ sections, "examples, data, and cited sources", and making text, images and video describe the same things. Google says⁠(opens in a new tab) writing well for people is enough for AI too.

What Google says you can skip

Google's May 2026 guide⁠(opens in a new tab) lists "AEO/GEO hacks" that it says don't help in Google Search:

• llms.txt and other special files, which the llms.txt section below explains.
• Splitting content into small pieces for AI, sometimes called chunking. Google says "There's no requirement to break your content into tiny pieces for AI", that its systems understand pages that cover several topics, and that there's no ideal page length.
• Rewriting content specially for AI. "You don't need to write in a specific way just for generative AI search."
• Trying to get fake mentions of your business on other websites.
• Focusing too much on structured data, which isn't required for generative AI features.

The same guide⁠(opens in a new tab) warns against creating a separate page for every possible fan-out search, mainly to manipulate rankings or AI answers. It says this "violates Google's scaled content abuse spam policy. This is also an ineffective long-term strategy."

Not everyone agrees with all of Google's list. iPullRank's November 2025 guide⁠(opens in a new tab) argues that "LLMs can and do use structured data as part of the RAG pipeline", and recommends detailed structured data. Google's guidance covers Google Search, and other AI companies haven't published anything as detailed. None of the tactics on Google's list replaces content that's useful to the people reading it, whichever AI service you have in mind.

Tactics that can backfire

Lily Ray's March 2026 article⁠(opens in a new tab) lists five GEO tactics that she believes can harm a site's search traffic:

• publishing large amounts of AI-written content
• changing page dates without changing the content, to make pages look new
• listicles that rank the publisher's own business as the best option, such as "The 10 best coffee subscriptions" with Fernhill at number one
• "Summarize with AI" buttons that open an AI assistant with hidden instructions to promote the business
• large numbers of "alternatives to" and comparison pages

She reports that traffic fell on sites with many self-promoting listicles from around 20 to 21 January 2026. She also notes that Microsoft has described the hidden instructions in "Summarize with AI" buttons as a security threat. That technique is called prompt injection: hiding instructions in content so that an AI system follows them. Aleyda Solis's checklist⁠(opens in a new tab) also warns against prompt injection and fake signals.

Google clarified in May 2026⁠(opens in a new tab) that its spam policies apply to AI answers in Search. Anything designed to manipulate an AI answer rather than inform it is a risk.

Common myths

llms.txt

llms.txt is a proposed file, placed at /llms.txt on a website, that describes the site and links to its most useful pages, for AI models and agents to read. It's written in Markdown, a simple way of formatting text with symbols, such as # for a heading and - for a list item.

Different sources say very different things about llms.txt, including different parts of Google.

The case for it

The idea comes from Jeremy Howard, who published version 2 of the proposal⁠(opens in a new tab) in August 2026. The proposal says "thousands of sites publish an llms.txt file", that some documentation platforms create one automatically, and that OpenAI, Anthropic and Gemini publish llms.txt files for their developer documentation. Version 2 also recommends offering Markdown versions of pages, with tags (rel="alternate" type="text/markdown" and rel="describedby") that tell software where to find them.

Google Search ignores it

Google's AI optimisation guide⁠(opens in a new tab) says Google Search doesn't use llms.txt, and that creating one "will neither harm nor help your site's visibility or rankings in Google Search, as Google Search ignores them". It adds that it's fine to keep one for other services that use it.

Chrome's Lighthouse checks for it

Lighthouse, the page-testing tool built into Chrome, added an experimental "Agentic Browsing" category in version 13.2, and turned it on by default in version 13.3 in May 2026 (Lighthouse scoring documentation⁠(opens in a new tab)). It includes an llms.txt check⁠(opens in a new tab). The check fails if requesting the file causes a server error, and is skipped if the file doesn't exist, "as providing the file is optional at the moment". Version 13.5, released in September 2026, added an Agent Resource Discovery check and grouped it with the llms.txt check under "agent discovery" (release notes⁠(opens in a new tab)).

As Search Engine Land pointed out⁠(opens in a new tab), these checks are about AI agents and browser tools, not Google Search rankings, so they don't contradict Google Search's advice.

Google's John Mueller is doubtful

John Mueller, who answers site owners' questions for Google Search, compared llms.txt to the keywords meta tag in 2025. That tag is an old HTML tag that search engines stopped using because site owners filled it with keywords. In February 2026 he said⁠(opens in a new tab) that converting pages to Markdown for bots was "such a stupid idea". In May 2026 he told Lily Ray⁠(opens in a new tab), "For non-developer sites, I don't think this makes much sense". In June 2026 he called it⁠(opens in a new tab) "purely speculative for now", while saying he likes WebMCP, which the agents section explains.

Should you have one?

No major AI search company has published evidence that llms.txt affects whether a site is cited, and none says it uses llms.txt to find or rank pages. The file is cheap to create and does no harm if it's accurate. It's most useful for developer documentation, programming interfaces and software tools, because AI coding assistants do look for it. For most business websites it's optional, a long way below the list in What to fix first, and it never replaces pages that crawlers can read.

If Fernhill published one, it would look like this:

# Fernhill Coffee

> Fernhill Coffee is a small-batch coffee roaster in the UK.
> It sells coffee beans and coffee subscriptions online, with UK, Irish and German delivery.

## Shop
- [Coffee subscriptions](https://www.example.com/subscriptions): Plans, prices in GBP including VAT, and delivery
- [All coffees](https://www.example.com/coffees): Every coffee we roast, with tasting notes

## Help
- [Delivery and returns](https://www.example.com/delivery): Delivery times, charges and returns

## Guides
- [Brewing guides](https://www.example.com/brewing-guides): How to brew with a cafetière, filter and espresso machine

If you publish one, make sure /llms.txt returns either the file or a normal "not found" (404) response, never a server error, because a server error is what fails the Lighthouse check. And make sure its facts match the facts on your pages.

Making your site usable by AI agents

AI agents are AI tools that use websites on someone's behalf, to compare products, fill in forms or place orders. Someone could ask an agent to order a bag of Fernhill's Ethiopian coffee and have it delivered to their office. The agent would then visit Fernhill's site, find the product, and go through the checkout.

Google documents Google-Agent, the name its agents use when they visit sites. OpenAI, Anthropic, Perplexity and others have similar visitors that act for a user, described in the crawlability chapter. Google's AI optimisation guide⁠(opens in a new tab) includes some early guidance on agents, noting that "protocols like Universal Commerce Protocol (UCP) are emerging that will allow Search agents to do more".

web.dev's guide to building agent-friendly websites⁠(opens in a new tab) says agents see a site in three ways: "screenshots, raw HTML, and the accessibility tree". The accessibility tree is the description of a page that browsers create for screen readers, listing each button, link, heading and form field. Most of the guide's advice is the same as the advice for making sites accessible to disabled people:

• Use the proper HTML elements, such as <button>, <a>, <label>, <form> and headings, not general-purpose elements made clickable with JavaScript. Where that isn't possible, add the right role and tabindex settings.
• Connect each form field to its label with <label for="...">, and write clear error messages.
• Avoid invisible layers that cover buttons and links.
• Make buttons and other interactive elements visible, and "larger than 8 square pixels", so they aren't ignored when the agent analyses a screenshot.
• Keep layouts stable, so buttons don't move between pages or while a page loads. Agents that click by screen position fail when content moves. The page experience chapter explains how to measure this.

<!-- A form field that agents and screen readers can understand -->
<label for="postcode">Delivery postcode</label>
<input id="postcode" name="postcode" autocomplete="postal-code">
<button type="submit">Check delivery</button>

Three more steps help agents complete tasks:

• Put important information in text, not only in images or video.
• Remove obstacles you don't need. Each CAPTCHA, forced account sign-up or pop-up is a point where an agent may give up.
• Make prices and product details clear and readable in the HTML. Agents comparing options for someone need numbers they can read.

WebMCP is a proposed standard that lets a website list the actions agents can take on it, such as "check delivery" or "add to basket", using JavaScript or HTML. web.dev warns⁠(opens in a new tab) that "WebMCP is in active development and may change". For now it's worth following rather than building for.

Measuring AI visibility

Measuring how AI assistants use your site is still difficult. The reports are new, and they show less than you'd want. There are four sources.

Google Search Console

Visits from AI Overviews and AI Mode are included in Search Console's standard Performance report, as part of "Web" search (AI features and your website⁠(opens in a new tab)). That report can't separate them from other search traffic.

The generative AI performance report⁠(opens in a new tab), introduced in June 2026 and available to all sites since 31 August 2026, shows impressions from AI Overviews and AI Mode. An impression means a link to your site was shown. The report breaks impressions down by page, country, device and date (Search Console Help⁠(opens in a new tab)). Google Discover has its own report.

The report doesn't show clicks or the questions people asked. As Search Engine Roundtable⁠(opens in a new tab) put it, there's "no click data and no query data". An impression tells you a link to your site was shown in an AI answer. It doesn't tell you whether anyone clicked, or what the answer said about you.

Bing Webmaster Tools

Bing's AI Performance report⁠(opens in a new tab) launched as a preview on 10 February 2026. It covers Microsoft Copilot, AI-written summaries in Bing, and some partner services. It shows:

• total citations, meaning how often your pages were shown as a source
• the average number of your pages cited each day
• grounding queries, a sample of the searches the AI used to find your pages
• citations for each page, and how they change over time

Bing called it "an early step toward Generative Engine Optimization (GEO) tooling". It notes that citation counts show how often a page was cited, not how important the page was or where it appeared in the answer.

In June 2026 Bing added Intents, Topics, Compare and Citation Share⁠(opens in a new tab). Citation Share is your share of all the citations for a grounding query. Bing says it's designed as "an observational metric", not a ranking or a way to compare yourself with competitors, and it doesn't show which other sites were cited.

In October 2026 Bing added a Topics Portfolio⁠(opens in a new tab), still in testing. It shows "opportunity queries", meaning queries "where other sites appear and yours does not", with search numbers given as ranges. Bing's advice is to "treat the numbers as directional signals for planning, not as a precise ledger".

As of October 2026, OpenAI, Anthropic and Perplexity don't offer site owners reports like these.

Your server logs and analytics

Your website's server logs record every request to your site, including which crawler asked for which page and what status code it got. The crawlability chapter explains how to read them.

Count requests from AI search crawlers and user-triggered fetchers separately from training crawlers. The ones to track are OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Agent and bingbot. Requests from user-triggered fetchers are a rough sign that your pages are being read to answer questions. A request doesn't prove a page was indexed or cited. If there are no requests at all, check robots.txt and your CDN or firewall first.

Your analytics can show visits that come from AI assistants. When someone clicks a link, their browser usually tells the site which page they came from. This is called the referrer. Visits from AI assistants show referrers such as chatgpt.com, perplexity.ai, claude.ai, gemini.google.com and copilot.microsoft.com. Group these visits together, and track how many of them lead to a sale or sign-up. Some assistants and apps don't pass on a referrer, so the real number of visits from AI assistants is higher than your analytics show.

Asking the assistants

None of these reports tells you what AI assistants say about your business when they don't cite you, how you compare with competitors, or how ChatGPT, Claude and Gemini answer the questions your customers ask. The only way to find out is to ask them, many times.

Answers change from one try to the next, so a single check tells you very little. Aleyda Solis's checklist⁠(opens in a new tab) suggests asking each question several times (for example, 3 to 5 times) within a set period (for example, 24 to 72 hours), and reporting the results without overstating them.

Fernhill might ask "what's a good coffee subscription in the UK?", "which UK coffee roasters deliver to Ireland?" and "is Fernhill Coffee any good?" in each assistant, five times each. For each answer, it would record the date, the assistant, the country it asked from, whether Fernhill was mentioned, and which pages were cited. Repeated over months, the records show trends that a single check can't.

What to do next

1. Check that AI search crawlers can reach your site, both in robots.txt and in your CDN or firewall settings.
2. Check that the Search generative AI setting in Search Console is set the way you intend.
3. Work through the What to fix first list with your developer.
4. Look at the generative AI report in Search Console and the AI Performance report in Bing Webmaster Tools, and note the numbers as a starting point.
5. Set up a group in your analytics for visits from AI assistants.
6. Write down five to ten questions your customers ask, and start asking them in the main AI assistants each month.

Further reading

• Optimizing your website for generative AI features on Google Search⁠(opens in a new tab), Google's guide, including the tactics it says you can skip and its position on llms.txt.
• AI features and your website⁠(opens in a new tab), Google's short page on what its AI features need and how their traffic appears in Search Console.
• Search generative AI control⁠(opens in a new tab), the Search Console help page for the setting and its limits.
• Overview of OpenAI crawlers⁠(opens in a new tab), which explains what each OpenAI crawler does, and which one to block to opt out of training.
• Introducing AI Performance in Bing Webmaster Tools⁠(opens in a new tab), what Bing's citation reports show and what they don't.
• Your GEO strategy might be destroying your SEO⁠(opens in a new tab), Lily Ray on the GEO tactics that can backfire.
• Build agent-friendly websites⁠(opens in a new tab), web.dev's practical advice on making pages usable by AI agents.
• The /llms.txt file⁠(opens in a new tab), the proposal itself, if you're deciding whether to publish one.