Glossary

Andy Bright

Director of Product and Technology

This glossary explains the terms used in this guide. Each entry links to the chapter that explains the term in more detail, usually with an example. The table of status codes at the end lists the numbers servers send back when a page is requested.

A

AEO (answer engine optimisation) and GEO (generative engine optimisation) are two names for the same work: getting your pages used and linked in answers from AI assistants. See AI search.

Accessibility tree is the description of a page that browsers create for screen readers, listing each button, link, heading and form field. AI agents use it too. See AI search.

AI agents are AI tools that use websites on someone's behalf, for example to compare products, fill in forms or place orders. See AI search.

AI crawlers are crawlers run by AI companies. There are three kinds: training crawlers, search crawlers and user-triggered fetchers. See crawlability.

AI Mode is Google's chat-style search, where you ask questions and get AI-written answers with links. See AI search.

AI Overviews are the AI-written summaries that appear at the top of some Google search results. See AI search.

Anchor text is the visible, clickable text of a link. See site architecture.

B

Backlink is a link from another website to one of your pages. See redirects.

Bing Webmaster Tools is Microsoft's free service for website owners. It shows how Bing crawls and indexes your site, and how often Microsoft Copilot cites your pages. See sitemaps and AI search.

Bot protection is a feature of many CDNs and firewalls that tries to tell people apart from automated programs. When it's unsure, it can show a challenge page, a "check you're human" step that crawlers can't get past. See crawlability.

Breadcrumbs are the row of links near the top of a page that shows where the page sits on the site, such as Home › Coffee › Single origin. See site architecture.

C

Canonical URL is the address that search engines treat as the main version of a page, when the same content is available at more than one address. A canonical tag is a line of HTML that tells search engines which address you prefer. See indexing.

CDN (content delivery network) is a service, such as Cloudflare or Fastly, that stores copies of your pages on servers around the world so they load quickly. CDNs can also block crawlers. See crawlability.

Citation is a link to your page in an AI answer, shown as one of the sources the answer used. See how search works.

Click depth is the number of clicks it takes to reach a page from the home page. See site architecture.

Client-side rendering is a way of building a website where the server sends a mostly empty page and JavaScript in the visitor's browser builds the content. See JavaScript.

Cloaking means showing search engines different content from what visitors see. It breaks Google's spam policies. See JavaScript.

Core Web Vitals are three measurements of page experience that Google uses in its rankings: Largest Contentful Paint (LCP), Interaction to Next Paint (INP) and Cumulative Layout Shift (CLS). See page experience.

Crawl budget is the number of pages a search engine is willing to crawl on your site in a given period. It mostly matters for very large sites. See crawlability.

Crawler is a program that visits web pages automatically and downloads them. Crawlers are also called bots or spiders. Googlebot is Google's main crawler, and GPTBot is one of OpenAI's. See how search works.

CrUX (Chrome UX Report) is Google's collection of speed measurements from real visitors using Chrome. Google uses it to judge Core Web Vitals. See page experience.

Cumulative Layout Shift (CLS) is a score for how much the content on a page moves unexpectedly while someone is using it. A good score is 0.1 or less. See page experience.

D

DNS (domain name system) is the system that turns your domain name into the numeric address of the server that hosts your site. See redirects.

Duplicate content is the same page, or almost the same page, available at more than one address. See indexing.

Dynamic rendering means detecting crawlers and sending them a pre-built copy of each page, while visitors get a version built with JavaScript. Google calls it a workaround. See JavaScript.

E

Entity is a distinct thing that search engines keep track of, such as a business, a person, a product or a place. See structured data.

F

Faceted navigation is the set of filters on a category page, such as roast, grind or price, where each combination of filters can create a new address. See indexing.

Field data is speed measurements taken from real visitors. Lab data comes from a test that loads the page once in a simulated browser. Google uses field data to judge Core Web Vitals. See page experience.

Firewall is a set of rules that blocks or questions requests that look harmful. On websites it's often called a WAF (web application firewall). Firewalls can block crawlers you want to allow. See crawlability.

G

Google Search Console is Google's free service for website owners. It shows which of your pages are indexed, how they perform in search, and any problems Google has found. See indexing.

Googlebot is Google's main crawler. See how search works.

Grounding, also called retrieval-augmented generation (RAG), is when an AI system searches an index, reads the pages it finds, and writes its answer from them, with links to the pages it used. See AI search.

H

hreflang is a tag that tells search engines which version of a page is for which language and country. A group of pages that list each other with hreflang is called an hreflang cluster. See international SEO.

HTML is the code that holds a page's text, headings, links and images. It's what a server sends when a page is requested. See JavaScript.

HTTP header is a line of information a server attaches to a file when it sends it. Headers can carry instructions such as noindex for files that aren't web pages, like PDFs. See indexing.

HTTP status code is a three-digit number a server sends back with every response, saying what happened. The table at the end lists the common ones. See indexing.

HTTPS is the secure version of the web's communication method, shown by the padlock in a browser's address bar. See page experience.

Hub and cluster is a way of organising pages about a topic. The hub page introduces the topic and links to every page about it. Each cluster page covers one part of the topic and links back to the hub. See site architecture.

Hydration is the step where JavaScript in the browser takes over a page that was built on the server, so the page becomes interactive. See JavaScript.

I

Index is a search engine's database of the pages it has stored and can show in results. A page that's in the index is indexed. See indexing.

IndexNow is a free service for telling search engines that a page has been added, changed or deleted. Bing uses it, but Google doesn't. See sitemaps.

Infinite scroll is when more items appear as you scroll down a page, instead of on a new page. See site architecture.

Interaction to Next Paint (INP) is the delay between someone clicking, tapping or pressing a key and the page visibly reacting. A good score is 200 milliseconds or less. See page experience.

Internal links are links from one page on your site to another page on the same site. See site architecture.

Intrusive interstitial is Google's term for a pop-up that covers a page as soon as someone arrives from a search result. See page experience.

IP address is the number that identifies a computer on the internet. Some AI companies publish the IP addresses their crawlers use, so you can check a crawler is genuine. See crawlability.

J

JavaScript is a programming language that runs in the visitor's browser and can add or change content after the page arrives. See JavaScript.

JSON-LD is the format Google recommends for structured data. It sits in its own block in a page's HTML, separate from the visible content. See structured data.

L

Largest Contentful Paint (LCP) is the time from when someone opens a page until the biggest image or block of text on their screen appears. A good score is 2.5 seconds or less. See page experience.

lastmod is the date in a sitemap that says when a page last changed. Search engines only use it if it's accurate. See sitemaps.

Lazy loading means waiting to load images or sections of a page until the visitor scrolls near them. See JavaScript.

Lighthouse is the page-testing tool built into Chrome. It measures speed in a simulated browser, and includes experimental checks for AI agents. See page experience.

LLM (large language model) is the kind of AI model behind assistants such as ChatGPT, Claude and Gemini. See AI search.

llms.txt is a proposed file that describes a website and links to its most useful pages, for AI models and agents to read. Google Search ignores it. See AI search.

M

Manual action is a penalty applied to a site by a person on Google's spam team. See structured data.

Markdown is a simple way of formatting text with symbols, such as # for a heading and - for a list item. See AI search.

Metadata is the information in a page's <head> section that visitors don't see on the page itself, such as its title, description and canonical tag. See JavaScript.

Mobile-first indexing means Google crawls and judges your site using its smartphone crawler, so it sees what the mobile version of each page contains. See page experience.

N

noindex is an instruction that tells search engines not to show a page in their results. It can be a robots meta tag in the page's HTML or an HTTP header. See indexing.

O

Orphan page is a page that no other page on the site links to. Crawlers can easily miss it. See site architecture.

P

Page experience is Google's name for how easy a page is to use, apart from what it says: its speed, whether it works on a phone, whether it's secure, and whether pop-ups get in the way. See page experience.

Pagination is splitting a long list, such as a blog's articles or a shop's products, across numbered pages. See site architecture.

Parameter is the part of an address after a ?, such as ?sort=price. See site architecture.

Permanent redirect (status 301 or 308) tells search engines that a page has moved for good, so they show the new address in results. A temporary redirect (status 302 or 307) tells them to keep showing the old address. See redirects.

Prompt injection is hiding instructions in content so that an AI system that reads it follows them. See AI search.

Q

Query fan-out is when an AI system turns one question into several related searches and uses the results of all of them. See how search works.

R

Ranking is the order in which a search engine shows the pages that match a search. See how search works.

Raw HTML is the HTML exactly as the server sends it, before any JavaScript runs. It's what most AI crawlers read. Rendered HTML is what exists after JavaScript has run. See JavaScript.

Real user monitoring (RUM) means collecting speed measurements from your own visitors with a small script on your site. See page experience.

Redirect is an instruction on your website that sends anyone who asks for one address to a different address instead. A redirect chain is a redirect that leads to another redirect. A redirect map is the list of old addresses and the new address each one should redirect to. See redirects.

Referrer is the information a browser passes to a website about which page a visitor came from. Analytics use it to show visits from AI assistants. See AI search.

Rendering means running a page's JavaScript to build the finished page, the way a browser does. Google renders pages, and most AI crawlers don't. See JavaScript.

Rendering strategy is the way a website produces each page's HTML, such as static site generation, server-side rendering or client-side rendering. See JavaScript.

Retrieval is the step where a search engine or AI system finds the pages in its index that match a search or question. See how search works.

Reverse DNS lookup is a check that a request claiming to come from a crawler, such as Googlebot, really came from that company's network. See crawlability.

Rich results are search listings that show extra details, such as star ratings, prices or event dates. They come from structured data. See structured data.

Robots meta tag is a line of HTML in a page's <head> that gives crawlers instructions about that page, such as noindex. See indexing.

robots.txt is a plain text file at the root of a website that tells crawlers which parts of the site they may visit. See crawlability.

S

schema.org is the shared vocabulary for structured data that Google, Bing and other search engines agreed on. In schema.org, each kind of thing is a type, such as Product, and each detail of it is a property, such as price. See structured data.

Search crawlers build the index that an AI assistant searches when it answers a question. OAI-SearchBot, Claude-SearchBot and PerplexityBot are search crawlers. See crawlability.

Search generative AI setting is a setting in Google Search Console that controls whether your pages can appear in AI Overviews and AI Mode. See AI search.

Server is the computer that stores a website and sends out its pages. See how search works.

Server logs are files that record every request your website receives: who asked, for which page, when, and what response they got. See crawlability.

Server-side rendering is a way of building a website where the server builds each page's complete HTML at the moment someone asks for it. See JavaScript.

Single-page app is a website that loads once and then uses JavaScript to swap in new content as you move around, instead of loading a new page each time. See JavaScript.

Site architecture is the way the pages on a website are organised and connected. See site architecture.

Site migration is any change that alters a website's addresses or where it's hosted, such as a new domain or a new platform. See redirects.

Sitemap, or XML sitemap, is a file on your site that lists the pages you want search engines to know about, usually with the date each one last changed. A sitemap index is a file that lists several sitemaps. See sitemaps.

Snippet is the short description under a page's link in search results. Snippet controls such as nosnippet, max-snippet and data-nosnippet limit what Google can show, including in its AI features. See indexing.

Soft 404 is a page that tells visitors "not found" but tells crawlers everything is fine, by returning a 200 status. See indexing.

Soft navigation is a move between pages in a single-page app, where the browser doesn't load a whole new page. See page experience.

Staging site is a private copy of a website, used to test changes before they go live. See redirects.

Static site generation is a way of building a website where each page's HTML is built once, when the site is published, and stored ready to send. See JavaScript.

Streaming means sending a page in pieces, so the browser can start showing it before the whole page is ready. See JavaScript.

Structured data is code you add to a page to describe its content in a standard format that machines can read, such as a product's price and stock level. See structured data.

Subdomain is a separate part of your site with its own prefix, such as blog.example.com. See crawlability.

T

Technical SEO is the work of making sure search engines and AI assistants can find, read and store the pages on your website. See the guide's introduction.

Third-party scripts are code from other companies that runs on your pages, such as chat widgets, tag managers and analytics. They often slow pages down. See page experience.

Time to first byte (TTFB) is the time a server takes to start sending a page after it's requested. See page experience.

Training crawlers collect text that may be used to train future AI models. GPTBot, ClaudeBot and Google-Extended are training crawlers or controls. See crawlability.

TTL (time to live) is how long other computers keep a copy of your DNS settings before checking for changes. See redirects.

U

URL (uniform resource locator) is a page's full address, such as https://www.example.com/subscriptions. See how search works.

User agent is the name a crawler or browser gives when it requests a page, such as Googlebot or GPTBot. robots.txt rules are written for user agents. See crawlability.

User-triggered fetchers visit a page because a person asked an AI assistant to read it, for example by pasting a link into ChatGPT. ChatGPT-User, Claude-User and Perplexity-User are user-triggered fetchers. See crawlability.

W

WebMCP is a proposed standard that lets a website list the actions AI agents can take on it, such as "add to basket". It's still changing. See AI search.

X

x-default is the hreflang value that names the page for visitors who don't match any language or country version you list. See international SEO.

XML is a text format for storing information in a structured way, using labels in angle brackets. Sitemaps are written in XML. See sitemaps.

X-Robots-Tag is an HTTP header that gives crawlers instructions such as noindex, for files that can't carry a robots meta tag, such as PDFs. See indexing.

Common status codes

Every time a browser or crawler requests a page, the server replies with a three-digit status code. These are the ones you'll meet most often in this guide.

The indexing chapter explains how search engines treat each of these.