Cleotic

Site architecture and internal linking

Updated 16 October 2026 · 11 min read

Andy Bright

Director of Product and Technology

On this page

Crawlers find most pages by following links from pages they already know. The way your pages link to each other therefore decides which pages crawlers find, how often they come back to them, and what search engines think each page is about. A page that nothing links to can go unnoticed for months.

This chapter explains:

  • what a link looks like to a crawler, and why some links can't be followed
  • how the words in a link help search engines
  • how to find pages that nothing links to
  • how to group pages by topic so their relationship is clear
  • breadcrumbs, lists that run over several pages, and choosing web addresses

If you're new to crawlers, read How search engines and AI assistants find your pages first.

What site architecture means

Site architecture is the way the pages on a website are organised and connected. The connections are internal links, which are links from one page on your site to another page on the same site. Your menu, footer, product listings and the links inside your articles are all internal links.

Fernhill Coffee, the example shop in this guide, has a simple structure. The home page links to the shop, the subscription page and a section of brewing guides. The shop links to categories such as single-origin coffee and blends, and each category links to its products. Every page can be reached by following links from the home page.

On a web page, a link is written in HTML as an <a> element with an href attribute. The href holds the address the link goes to:

<a href="/subscriptions">Coffee subscriptions</a>

Google's link best practices⁠(opens in a new tab) state the rule like this: "Generally, Google can only crawl your link if it's an <a> HTML element (also known as anchor element) with an href attribute".

Some websites build things that look like links but aren't written this way. They might be buttons that run JavaScript when clicked, or text with a click handler attached. A person can click them, but a crawler doesn't click anything, so it never finds the page they lead to:

<!-- Crawlers can follow this -->
<a href="/brewing-guides/pour-over">Pour-over brewing guide</a>

<!-- Crawlers can't reliably follow these -->
<a onclick="goTo('/brewing-guides/pour-over')">Pour-over brewing guide</a>
<span data-href="/brewing-guides/pour-over">Pour-over brewing guide</span>
<a href="javascript:goTo('pour-over')">Pour-over brewing guide</a>

Links that JavaScript adds to the page are fine for Google, as long as they end up as proper <a href> links once the page has been rendered. For websites built as single-page apps, Google also recommends real addresses over ones that use # to switch content (JavaScript SEO basics⁠(opens in a new tab)).

AI crawlers are stricter. In late 2024, Vercel and MERJ found⁠(opens in a new tab) that the major AI crawlers they studied, including those from OpenAI, Anthropic and Perplexity, didn't run JavaScript at all. If Fernhill's menu or its "related guides" box only appeared after JavaScript ran, those crawlers couldn't use it to find anything. The menu, breadcrumbs and the most important links on each page should be in the HTML your server sends. The JavaScript chapter shows how to check.

Anchor text is the visible, clickable text of a link. Google uses it to understand what the page being linked to is about. Its link guidance⁠(opens in a new tab) asks for anchor text that's descriptive, reasonably short and relevant to that page.

VagueDescriptive
Click hereRead our pour-over brewing guide
Learn moreCompare our coffee subscription plans
This articleHow to store coffee beans

The same guidance warns against cramming keywords into links, which Google treats as spam, and against putting several links one after another with no words between them, because readers lose the context. Writing link text for a person scanning the page works well for crawlers too.

Link to every page you care about

Google's advice⁠(opens in a new tab) is direct: "every page you care about should have a link from at least one other page on your site".

A page that no other page links to is called an orphan page. Crawlers can still find it through your sitemap, but nothing on your site tells them it matters or what it's about. A page can become an orphan when a redesign or a menu change removes the only link to it, or when a landing page is built for one campaign and never linked from the rest of the site.

Google also says there's no ideal number⁠(opens in a new tab) of links on a page. Add a link where it helps a reader, not to reach a target.

Finding orphan pages

There isn't one report that lists every orphan page. The reliable way is to compare three lists of addresses:

  1. The pages you expect to exist. Your sitemap or your website platform can give you this list.
  2. The pages a crawler finds by following links. A site-crawling tool starts at your home page and follows every link, the way a search engine's crawler does, and lists every page it reaches.
  3. The pages that people and crawlers actually visit, from your analytics or your server logs.

Any page in list 1 but not list 2 is an orphan. Any address in list 3 but not list 1 is often an old page that needs a redirect to its replacement.

If your lists are in text files, a developer can compare them with two commands:

# Pages in the sitemap that the crawl didn't reach
comm -23 sitemap-urls.txt crawled-urls.txt > orphans.txt

# Addresses people or crawlers request that aren't in the sitemap
comm -13 sitemap-urls.txt logged-urls.txt  > unknown-requested-urls.txt

Grouping pages by topic

When several pages cover parts of the same subject, linking them together shows search engines how they relate. A common way to do this is a hub and cluster structure. The hub is a page that introduces a topic and links to every page about it. Each cluster page covers one part of the topic in depth, links back to the hub, and links to related cluster pages where that helps the reader.

Fernhill's brewing guides are set up this way. The hub page introduces home brewing and links to a guide for each method:

/brewing-guides                 ← the hub
├── /brewing-guides/pour-over   ← cluster pages
├── /brewing-guides/aeropress
├── /brewing-guides/french-press
└── /brewing-guides/cold-brew

This guide is built the same way. Its overview page is the hub, and each chapter is a cluster page.

Google's page on AI features⁠(opens in a new tab) asks that important content be "easily findable through internal links". SEO practitioners writing about AI search say the same. The agency iPullRank⁠(opens in a new tab) lists a site's information architecture first among its five technical foundations for AI search, alongside entities and structured data, crawlability, rendering and performance. That's a practitioner's view rather than a search engine's statement, but it's consistent with Google's advice about links.

Click depth is the number of clicks it takes to reach a page from the home page. Google doesn't set a limit, and you'll see different numbers quoted as rules. The practical aim is to keep your most important pages, such as Fernhill's subscription page and its best guides, a few clicks from the home page, and to link to them from related pages rather than only from long archive lists.

Don't make a separate page for every question

Google's guide to AI features⁠(opens in a new tab) explains that its AI systems often run several related searches to answer one question, a technique called query fan-out. It can be tempting to respond by making a separate page for every variation of a question, such as one page each for "best coffee subscription", "top coffee subscription" and "coffee subscription for beginners".

Google warns against this. Creating separate content for every variation, mainly to influence rankings or AI answers, "violates Google's scaled content abuse spam policy. This is also an ineffective long-term strategy." The SEO consultant Lily Ray, writing in March 2026⁠(opens in a new tab), lists mass-produced AI content and large numbers of "alternatives" and comparison pages among the tactics that can harm a site's search visibility. A smaller number of useful pages, well linked to each other, is the safer approach.

Breadcrumbs are the row of links near the top of a page that shows where the page sits on the site. On one of Fernhill's product pages, they read:

Home › Coffee › Single origin › Ethiopia Yirgacheffe

Each step is a link back up to a broader page, so breadcrumbs give every page a crawlable path back to the home page. They should be real links in the HTML:

<nav aria-label="Breadcrumb">
  <ol>
    <li><a href="/">Home</a></li>
    <li><a href="/coffee">Coffee</a></li>
    <li><a href="/coffee/single-origin">Single origin</a></li>
    <li aria-current="page">Ethiopia Yirgacheffe</li>
  </ol>
</nav>

The aria-label and aria-current attributes tell screen readers what the links are and which page is the current one. AI agents that read pages the way screen readers do use them too, as the AI search chapter explains.

You can also describe breadcrumbs to search engines with structured data, explained in the structured data chapter. Since January 2025, Google has only shown breadcrumbs in its desktop results, not on phones (Google's documentation updates⁠(opens in a new tab)). The links themselves still help crawlers on every device.

Lists that run over several pages

Long lists, such as Fernhill's blog archive or a category with hundreds of products, are often split across several pages. This is called pagination. Page 2 of Fernhill's blog lives at /blog?page=2, page 3 at /blog?page=3, and so on. Google's pagination guide⁠(opens in a new tab) gives five rules.

Give each page its own address. Google treats each page in the series as a separate page, and either ?page=2 or /page/2/ works.

Don't point page 2's canonical tag at page 1. Google says not to use the first page as the canonical for the whole series, and to give each page its own canonical instead. Canonical tags are explained in the indexing chapter.

Link the pages to each other with ordinary <a href> links, from each page to the next. Google adds that linking every page back to page 1 can hint that the first page is the best place to start.

Don't rely on rel="next" and rel="prev" tags. Google no longer uses them, though other search engines may.

Make "load more" buttons and infinite scroll work without clicking. Infinite scroll is when more items appear as you scroll down, instead of on a new page. Google says its crawlers "don't 'click' buttons and generally don't trigger JavaScript functions that require user actions". So each batch of items needs its own address, and the pages need ordinary links to those addresses. As the visitor scrolls, the address in the browser's address bar should update, using what developers call the History API, so each batch can also be opened directly.

Shop category pages with filters, such as roast and grind, raise separate issues. Google's guide to faceted navigation⁠(opens in a new tab) recommends keeping filtered addresses out of the crawl unless people actually search for them. The indexing chapter explains how.

Choosing web addresses

Good web addresses are easy to read, don't change, and follow the same pattern across the site. Google's URL structure guide⁠(opens in a new tab) gives these rules:

  • Use words people can read rather than ID numbers where you can. /coffee/ethiopia-yirgacheffe is better than /p?id=4821.
  • Separate words with hyphens, not underscores.
  • Don't use # to switch between different content. Give each piece of content a real address.
  • Use as few parameters as you can. A parameter is the part of an address after a ?, such as ?sort=price.
  • Avoid creating endless addresses, such as a calendar that keeps generating next-month pages, or filters that combine without limit.

Some other habits save trouble later. Use lower case throughout and redirect any capitalised versions. Decide whether addresses end with a slash or not, and redirect the other form. Keep one address for each piece of content. Leave dates and ID numbers out of addresses unless they mean something to readers, because they make it harder to reorganise the site later.

The most important habit is not to change addresses for cosmetic reasons. Every changed address needs a redirect from the old one, and search engines take time to transfer what they know about the old address to the new one. The redirects chapter explains how to change addresses safely when you have to.

When a page's address changes, every link to the old address now leads to a redirect, or to a 404 error page if nobody set up a redirect. Each one costs crawlers a wasted request.

AI crawlers seem to waste more requests this way. When Vercel studied crawler traffic⁠(opens in a new tab) in late 2024, 34.82% of the requests from ChatGPT's crawler and 34.16% of those from Claude's ended in a 404 error, compared with 8.22% for Googlebot.

To keep your links current, do three things regularly:

  • When a page moves, update your own links to point straight at the new address, rather than relying on the redirect.
  • Fix or remove links to pages that no longer exist. A site-crawling tool lists them.
  • Set up a redirect for any old address that crawlers keep asking for, if there's a sensible replacement. The redirects chapter explains how.

A developer with a list of the addresses your site links to can check them all at once:

# Print any linked address that doesn't return a 200
while read -r url; do
  code=$(curl -s -o /dev/null -w "%{http_code}" "$url")
  [ "$code" != "200" ] && echo "$code $url"
done < internal-link-targets.txt

What to do next

  1. View the source of a few pages and check that your menu links are real <a href> links.
  2. Replace vague link text such as "click here" with words that describe the page being linked to.
  3. Compare your sitemap with a crawl of your site to find orphan pages, and link to the ones that matter.
  4. Group related pages under a hub page, and link them to each other.
  5. Check that your most important pages are a few clicks from the home page.
  6. If you have lists that run over several pages, make sure each page has its own address and its own canonical tag.
  7. Fix links to pages that have moved or no longer exist.

Further reading