Technical SEO audit checklist
Updated 16 October 2026 · 18 min read
Andy Bright
Director of Product and Technology
On this page
- Ten checks you can do yourself
- Before a full audit
- 1. Crawling and access
- 2. Indexing and canonicals
- 3. Site architecture and internal links
- 4. Redirects and migrations
- 5. XML sitemaps and IndexNow
- 6. JavaScript rendering
- 7. Page experience and Core Web Vitals
- 8. Structured data
- 9. International and hreflang
- 10. AI search
- How often to check
- After the audit
- Further reading
A technical SEO audit is a check of everything on a website that affects whether search engines and AI assistants can find, read and use its pages. This chapter lists every check from the technical SEO guide in one place, with a priority for each and a way to test it.
Use it for a first audit of your site, before and after a redesign or a move to a new platform, or as a regular health check every few months.
Many of the checks need a developer, or an SEO crawler, a program that visits every page of your site the way a search engine does and reports problems. Screaming Frog and Sitebulb are two widely used examples. If you don't have a developer to hand, start with the ten checks you can do yourself.
Each section links to the chapter that explains its checks. Every check has a priority. These priorities are this guide's own judgement, not an industry standard:
- P1 problems can stop pages being crawled, indexed or read. Fix these first.
- P2 problems weaken the signals you send to search engines, or waste crawlers' time. Fix them soon.
- P3 checks cover improvements and tidying up. Plan them in.
Passing every check won't guarantee rankings, AI citations or traffic. It removes the technical reasons your pages might be missed. The rest depends on your content, your reputation and how many people are looking for what you offer.
Ten checks you can do yourself
These need only a browser, Google Search Console and Bing Webmaster Tools. The chapter for each check explains what to do if it fails.
- Open
/robots.txton your site, and check it doesn't containDisallow: /underUser-agent: *. That line blocks the whole site. (Crawlability) - In Search Console, open the Page indexing report and check that your most important pages are indexed. (Indexing)
- Type a made-up address on your site, such as
/this-page-does-not-exist, and check you get a "page not found" page rather than your home page or a sign-in page. (Indexing) - Open
/sitemap.xml, check that your important pages are listed, and check that thelastmoddates aren't all the same. (Sitemaps) - On each important type of page, choose "View page source" and search for a sentence from the main content, such as a price. If it isn't there, crawlers that don't run JavaScript can't see it. (JavaScript)
- Open your Core Web Vitals report in Search Console, and note which groups of pages fail on mobile. (Page experience)
- Run your home page through Google's Rich Results Test, and check for
Organizationmarkup. (Structured data) - If you have country versions, view the source of a UK page and search for
en-UK, which should been-GB. (International) - In Search Console, check that Settings › Search generative AI is set the way you intend. (AI search)
- Ask whoever manages your CDN or firewall whether it blocks or challenges AI crawlers. (Crawlability)
Before a full audit
A full audit goes much faster with these ready:
- access to Google Search Console and Bing Webmaster Tools, verified for every version of your domain and every subdomain that matters
- two crawls of your site from an SEO crawler, one with JavaScript turned on and one with it turned off
- at least 30 days of server logs, the records your server or CDN keeps of every request, including which crawler made it and what status code it got
- your XML sitemaps, and a list of every page address from your website platform or database
- a list of your most important page templates, such as the home page, category pages, product pages, articles, pricing and contact
- the bot and security settings in your CDN and firewall
- your current robots.txt, and any written decisions about AI crawlers
Comparing the two crawls shows content, links and tags that only appear after JavaScript runs. Google runs JavaScript, and on the available evidence most AI crawlers don't. Server logs and CDN settings usually mean asking a developer or your hosting company, so ask early. If nobody has written down a decision about AI crawlers yet, record that as the audit's first finding.
1. Crawling and access
If a crawler can't fetch a page, nothing further down this list matters. The crawlability chapter explains these checks.
| Check | Priority | How to check |
|---|---|---|
| robots.txt returns a 200 status on every domain and subdomain (or a 404 if you don't want one), never a 5xx server error | P1 | Run curl -I https://www.example.com/robots.txt. A 5xx error makes Google stop crawling the site for 12 hours(opens in a new tab) |
No Disallow: / left over from a test site | P1 | Read the robots.txt on every live domain and subdomain |
| CSS, JavaScript and image files aren't blocked | P1 | Look for blocked folders such as /assets/, /static/ and /_next/ |
| Rules meant for every crawler are repeated in each group for a named crawler | P2 | Crawlers follow only the most specific group(opens in a new tab) that names them, so a named group doesn't include the rules under User-agent: * |
| robots.txt is under 500 KiB | P3 | Google ignores anything beyond that(opens in a new tab) |
| There's a written decision about each kind of AI crawler: search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot), training crawlers (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent, CCBot) and user-triggered fetchers | P1 | Compare robots.txt with the written decision. Check the current crawler names in OpenAI's(opens in a new tab), Anthropic's(opens in a new tab) and Perplexity's(opens in a new tab) documentation |
If robots.txt has a Content-Signal line, it matches your decision, including any line your CDN adds | P3 | Read the robots.txt your live site sends, not the copy in your code |
| Your CDN or firewall doesn't block or challenge crawlers you've decided to allow | P1 | Check the CDN's bot settings, then search your logs for 403 and 429 errors and security challenge pages sent to AI crawlers |
| Someone has chosen, on purpose, how the firewall treats user-triggered fetchers (ChatGPT-User, Claude-User, Perplexity-User, Google-Agent) | P2 | Firewall rules and logs |
| Logs show search engine and AI crawlers getting 200 responses on important pages | P2 | Filter the logs by crawler name, then group by status code |
| Crawlers are verified before you act on log data or add them to an allowlist | P3 | Google recommends a reverse DNS lookup followed by a forward one(opens in a new tab), a check that the request really came from Google's network |
Lists of allowed crawler IP addresses update automatically from each company's current files. Google moved its files to /crawling/ipranges/ in 2026 | P2 | Check which addresses the update process fetches |
| The server responds quickly and reliably, with few 429 or 5xx errors | P2 | Search Console's Crawl stats report, and your logs |
| Every subdomain that should be crawled has its own robots.txt | P2 | List your subdomains and fetch each file |
HTML pages are well under Googlebot's 2 MB limit, with the <head> tags and main content near the top | P3 | Check the uncompressed size of the HTML against Googlebot's limit(opens in a new tab) |
| On large sites, low-value addresses created by filters and tracking codes are kept out of the crawl | P2 | Crawl stats, and logs grouped by address pattern. Google's crawl budget guide(opens in a new tab) explains this |
2. Indexing and canonicals
These checks make sure the right address for each page gets indexed, and that every signal you send (canonical tags, sitemaps, internal links and hreflang) names the same address. The indexing chapter explains them.
| Check | Priority | How to check |
|---|---|---|
| Important pages return 200 and are indexed | P1 | Search Console's Page indexing report and URL Inspection tool |
No accidental noindex, in the HTML or in X-Robots-Tag headers | P1 | Crawl both the raw and the rendered HTML, and check the headers |
No page is blocked in robots.txt while you rely on noindex to remove it from search | P2 | If crawlers can't fetch the page, they never see its noindex(opens in a new tab) |
Robots meta tags are in the <head> of the raw HTML, not added or removed by JavaScript | P2 | Compare the raw and rendered HTML |
One version of the domain (with or without www, always https), with a single permanent redirect from each of the others | P1 | Run curl -I on each version |
| Each page that should be indexed has one canonical tag, giving its own complete address | P2 | Crawl report |
| The canonical tag is in the raw HTML, and JavaScript doesn't change it(opens in a new tab) | P2 | Compare the raw and rendered HTML |
| Canonical tags, sitemaps, internal links and hreflang all name the same address | P2 | Compare them in the crawl report |
| Google's chosen canonical matches yours on important templates | P2 | URL Inspection |
| Missing pages return 404 or 410, not 200. No soft 404s, and unknown addresses don't redirect to the home page or a sign-in page | P2 | Request a made-up address on each template |
Addresses created by filters and sorting(opens in a new tab) are controlled on purpose, through robots.txt, # fragments or indexing rules | P2 | Crawl the site and look for large numbers of addresses with ? in them |
| Near-identical pages (old campaigns, almost identical country pages, copies published on other sites) are combined | P3 | Bing says(opens in a new tab) AI systems group near-identical pages and pick one |
nosnippet, max-snippet and data-nosnippet appear only where you mean to limit snippets and AI use | P2 | These also limit Google's AI features(opens in a new tab) |
3. Site architecture and internal links
Crawlers find pages by following links, so links need to be real ones they can follow. The site architecture chapter explains these checks.
| Check | Priority | How to check |
|---|---|---|
Menus and important links are <a href> links in the raw HTML | P1 | Crawl without JavaScript and compare the number of links with the JavaScript crawl. Google's link guidelines(opens in a new tab) explain what it can follow |
| Every important page is linked from at least one other page, so there are no orphan pages | P2 | Compare the addresses in your sitemap or website platform with the ones the crawl found |
| Link text describes where the link goes, with no "click here" on important links | P3 | Your crawler's link text report |
| Important pages are a few clicks from the home page, and each main topic has a hub page | P3 | Your crawler's click depth report |
| Each page of a paginated list has its own address and a canonical tag pointing to itself. Lists that load more as you scroll are backed by numbered pages(opens in a new tab) | P2 | Turn off JavaScript and click through the pages |
| Breadcrumbs are real links | P3 | View the page source |
Addresses are readable, lower case and use hyphens, with as few ? parameters as possible | P3 | Crawl report |
| Internal links point to final addresses, not to redirects | P3 | Your crawler's internal redirects report |
| No broken internal links | P2 | Your crawler's report of links to 4xx pages |
4. Redirects and migrations
Redirects decide whether old links and bookmarks still reach the right page after an address changes. The redirects chapter explains these checks.
| Check | Priority | How to check |
|---|---|---|
| Permanent moves use server redirects with a 301 or 308 status(opens in a new tab), not JavaScript | P2 | Run curl -I on a sample of old addresses |
| Each redirect goes straight to its final address, and to the closest matching page rather than the home page | P2 | Google follows up to 10 redirects in a row(opens in a new tab), but aim for one |
| Old addresses that crawlers or AI assistants still request are redirected, where a sensible new page exists | P2 | Look for 404 errors by crawler in your logs, and check where AI referral visits land |
| Redirects from past moves are still working. Keep them for at least a year, and permanently for addresses that other sites link to | P2 | Test the old list of redirects |
| For a site move: a complete list of old and new addresses, redirects working from launch day, Google's Change of Address tool used for every verified version of the old domain, and a plan for checking afterwards | P1 | Review the plan against Google's site moves guide(opens in a new tab) |
After a change of domain or brand name, sameAs links, profiles and directory listings point to the new domain | P3 | Check your Organization structured data and main profiles |
5. XML sitemaps and IndexNow
These checks are about whether your sitemap is accurate, including its lastmod dates. The sitemaps chapter explains them.
| Check | Priority | How to check |
|---|---|---|
| A sitemap exists, is listed in robots.txt, and is submitted to Google and Bing | P2 | The sitemap reports in Search Console and Bing Webmaster Tools |
| It lists only complete, canonical addresses that can be indexed and return 200 | P2 | Fetch every address in the sitemap |
| It includes every public page you want found, including tools and landing pages | P2 | Compare it with the list of pages from your website platform |
| Each file stays within the limits of 50,000 addresses or 50 MB uncompressed | P2 | Check against Google's sitemap guide(opens in a new tab) |
lastmod shows the last real change to each page, not when the site was last rebuilt, and matches dateModified and the date shown on the page | P2 | Compare with the edit dates in your website platform. If every page shows today's date, it's the rebuild time |
Nothing relies on changefreq or priority | P3 | Google(opens in a new tab) and Bing(opens in a new tab) both ignore them |
| IndexNow(opens in a new tab) notifications are sent when pages are published, updated or deleted | P3 | The IndexNow report in Bing Webmaster Tools |
| The IndexNow key file can be fetched, and notifications get a 200 or 202 response | P3 | Fetch the key file with curl, and check the records of notifications sent |
6. JavaScript rendering
Run these checks against the raw HTML your server sends, because on the best available evidence that's all most AI crawlers see. The JavaScript chapter explains how to fix what you find.
| Check | Priority | How to check |
|---|---|---|
| The main content of each important template is in the raw HTML | P1 | View the source, or use curl, and search for a sentence from the content. The major AI crawlers in Vercel's study(opens in a new tab) didn't run JavaScript |
The title, description, canonical tag, robots tags and structured data are in the <head> of the raw HTML | P1 | View the source and read the <head> section |
| Prices, product details, stock levels and contact details aren't added only by JavaScript | P1 | Turn off JavaScript and look at the page |
| The main content appears without a click, a scroll, a login or a cookie choice | P1 | Turn off JavaScript, and check what's inside tabs and accordions |
No noindex in the raw HTML that JavaScript is meant to remove | P1 | Google may not render pages with noindex(opens in a new tab) |
| The rendered HTML in Search Console's URL Inspection tool matches what visitors see | P2 | Search Console |
Pages in a single-page app have normal addresses, not addresses with # | P2 | Click through the site and watch the address bar |
| Addresses that don't exist return a real 404 status from the server | P2 | Request a made-up address |
| Next.js sites: content that should be indexed is loaded in Server Components, and metadata is set with the Metadata API | P2 | Ask your developer |
Next.js sites: someone has decided whether pages should stream metadata(opens in a new tab) to AI crawlers (the htmlLimitedBots setting) | P3 | Use curl with an AI crawler's user agent, and check whether the metadata is in the <head> |
| The main image at the top of the page isn't lazy loaded | P2 | View the source |
| New sites don't rely on dynamic rendering | P3 | Google calls it a workaround(opens in a new tab) |
7. Page experience and Core Web Vitals
Most of these checks are P2 or P3. The P1 is making sure your mobile pages contain everything your desktop pages do. The page experience chapter explains them.
| Check | Priority | How to check |
|---|---|---|
| Core Web Vitals(opens in a new tab) pass for 75% of visits on mobile and desktop: LCP 2.5 seconds or less, INP 200 ms or less, CLS 0.1 or less | P2 | Search Console's Core Web Vitals report, and PageSpeed Insights |
The main image(opens in a new tab) is in the HTML, isn't lazy loaded, and has fetchpriority="high" | P2 | The Performance panel in Chrome's developer tools |
No scripts in the <head> that block the page without a good reason | P2 | View the source |
The server responds quickly (time to first byte), HTML is stored at the CDN where possible, and the server supports 304 Not Modified responses | P2 | PageSpeed Insights and your CDN's reports |
| Images and embedded videos have a set size, and banners that load late don't push content down | P2 | The layout shift highlights in Chrome's developer tools |
| Long-running JavaScript is split up, and third-party scripts have been checked for their effect on INP | P3 | Chrome's developer tools, and real user monitoring |
| Real user monitoring covers all three measurements | P3 | Google's web-vitals library, or a monitoring service |
| Single-page apps: someone has looked at the experimental soft navigation measurements(opens in a new tab) | P3 | CrUX data |
| Mobile pages have the same content, links, metadata and structured data as desktop pages | P1 | Compare the mobile and desktop versions. Google's mobile-first indexing guidance(opens in a new tab) lists what to match |
Every page uses HTTPS, http:// addresses redirect to https://, and HSTS is turned on | P2 | Run curl -I http://… and check the response |
| No pop-ups cover the page when someone arrives from search | P2 | Open important pages from a search result on a phone |
8. Structured data
Start with Organization markup. The P1 here is markup that doesn't match the page. The structured data chapter explains which types are still worth adding.
| Check | Priority | How to check |
|---|---|---|
Organization(opens in a new tab) markup, or a more specific type, on the home page or About page, with logo, sameAs links and identifiers | P2 | Rich Results Test |
WebSite markup on the home page | P3 | View the source |
The right types on important templates, such as Article with a real author, datePublished and dateModified, Product, BreadcrumbList or LocalBusiness | P3 | Search Console's rich result reports |
| Markup describes only content that's visible and accurate, with no star ratings from reviews of your own business that you control | P1 | Compare the markup on each template with the page |
| Structured data is in the raw HTML, near the top | P3 | View the source and search for application/ld+json |
| No work spent on FAQ or HowTo markup to get Google rich results | P3 | Google has stopped showing both(opens in a new tab) |
| Product prices and details in the markup match the page and your Merchant Center product information | P2 | Compare a few products |
| Facts about your business, products and prices are the same on your site, in your markup and on your profiles elsewhere | P2 | Compare your About page, Organization markup and main profiles |
9. International and hreflang
If your site is in one language for one country, you can skip this section, apart from the lang check. The international SEO chapter explains these checks.
| Check | Priority | How to check |
|---|---|---|
| Each language or country version has its own address(opens in a new tab). Content doesn't change by cookie or location | P1 | Open each version directly |
| Every version lists itself and the other versions(opens in a new tab), and every listed version lists it back | P1 | Your crawler's hreflang report |
Valid codes: en-GB (not en-UK), no country codes on their own, no es-419 | P1 | Crawl report, and search your templates for en-UK |
| Every hreflang tag uses a complete address | P2 | Crawl report |
x-default names a page for visitors who don't match any version | P3 | View the source, or check the sitemap |
| Each language version's canonical tag points to itself | P1 | Crawl report |
| hreflang is added in one way (HTML, HTTP headers or sitemap), from one shared list of versions | P3 | Ask your developer |
| No automatic redirects by location that stop crawlers reaching some versions | P2 | Open the site from different countries, or use a VPN |
The <html> tag has a lang attribute, plus dir="rtl" for right-to-left languages | P3 | View the source |
| Country versions have real local differences in the HTML, such as currency, VAT, delivery and contact details | P2 | Compare the versions side by side |
10. AI search
Most of what helps in AI search is covered in the sections above. These checks cover the settings and reports that AI search adds, and the tactics to avoid. The AI search chapter explains them.
| Check | Priority | How to check |
|---|---|---|
| Search Console's "Search generative AI" setting(opens in a new tab) is what you intend, on every property | P1 | Settings › Search generative AI |
| Your decision about Google-Extended is written down. It covers training and Gemini grounding, not Google Search | P3 | robots.txt |
No unintended NOARCHIVE tags, which keep content out of Copilot(opens in a new tab) | P2 | Crawl the site for robots meta tags |
| Someone has looked at the generative AI performance report(opens in a new tab) for impressions by page | P2 | Search Console. It shows impressions only, without clicks or questions |
| Someone has looked at Bing's AI Performance report(opens in a new tab): citations, cited pages, grounding queries and Citation Share | P2 | Bing Webmaster Tools |
| Visits from AI assistants are tracked separately from crawler requests | P2 | Groups in your analytics, and filters in your logs |
| Important facts (what you do, who for, where, and how you charge) are stated clearly in the HTML of the pages you want cited | P2 | Read the raw HTML as a stranger would |
| Pages that change show a visible "last updated" date | P3 | Check a sample of templates |
| No manipulative tactics(opens in a new tab): hidden instructions to AI, "Summarize with AI" buttons with hidden prompts, date changes without content changes, or large numbers of near-identical pages | P1 | Review your content and templates |
| Forms and important actions work for AI agents(opens in a new tab): proper HTML with labels, stable layouts, and no unnecessary CAPTCHAs on public pages | P3 | The accessibility tree in Chrome's developer tools |
If you publish llms.txt, it's accurate and returns 200. If you don't, /llms.txt returns a normal 404 | P3 | Run curl -I on /llms.txt. Lighthouse fails a 5xx error(opens in a new tab). The file is optional |
| Your most important questions are asked repeatedly in each assistant, in each country you sell to | P3 | Ask each question several times within a set period, and record the results |
How often to check
An audit shows how the site looked on the day you ran it. Sites change with every update, so some checks need repeating. If an update goes live with a test site's noindex still in place, checks run on every update catch it that day, rather than weeks later when traffic falls.
| How often | What to check |
|---|---|
| Every site update | Automatic checks that important pages return 200, robots.txt hasn't changed unexpectedly, canonical and robots tags are present, no noindex has been copied from a test site, structured data is valid, and the main content is in the raw HTML |
| Weekly | Errors in Search Console and Bing Webmaster Tools, Core Web Vitals getting worse, and visits from AI assistants |
| Monthly | Server logs by crawler, including AI crawlers and any new ones. The Page indexing report, the generative AI performance report and Bing's AI Performance report. Your list of questions in each AI assistant |
| Every three months | A full audit against this checklist. Read the Google Search changelog(opens in a new tab), the Google crawling changelog(opens in a new tab) and AI companies' crawler documentation for changes, and check your CDN's bot settings again |
| Before any site move | A complete list of addresses and their redirects, a crawl of the new site before launch, and daily checks for the first few weeks after launch |
After the audit
An audit only helps if the fixes go live.
- Fix the P1 problems first, then crawl the site again to check they're fixed.
- Turn the P2 problems into tasks, each with a person responsible and a date.
- Record your starting numbers: indexed pages, the share of pages passing Core Web Vitals, AI impressions in Search Console, citations in Bing, requests from AI crawlers, and visits from AI assistants. Measure each stage separately (crawled, indexed, cited, visited), and don't treat any one of them as proof of the next.
- Run the audit again after every major update, and at least every three months. AI crawler names, Search Console settings and website framework defaults all changed during 2026, so check each company's documentation every time.
Further reading
- Latest Google Search documentation updates(opens in a new tab), the quickest way to spot changes to how Google crawls, indexes and shows pages.
- Google crawling documentation changelog(opens in a new tab), for changes to Google's crawlers, IP addresses and limits.
- How Google interprets the robots.txt specification(opens in a new tab), the rules behind several of the P1 checks.
- Understand JavaScript SEO basics(opens in a new tab), Google's explanation of how it renders pages, including what happens when JavaScript changes
noindexor canonical tags. - Site moves and migrations(opens in a new tab), Google's guide to use alongside the site move checks.
- The rise of the AI crawler(opens in a new tab), Vercel and MERJ's study of how AI crawlers behave, from December 2024.
- Introducing AI Performance in Bing Webmaster Tools(opens in a new tab), what Bing's AI citation report shows.