JavaScript rendering

Andy Bright

Director of Product and Technology

Many websites use JavaScript to build their pages. Google can usually see content that JavaScript adds. Most AI crawlers can't. This chapter explains why, shows you how to check what crawlers see on your own pages, and lists what to ask your developer to change.

The first chapter introduced the problem with Fernhill Coffee's subscription page. Its prices are added by JavaScript, so Google sees them and most AI crawlers see an empty table. This chapter picks up that example.

This chapter explains:

• what HTML, JavaScript and rendering are
• how Google handles JavaScript
• which crawlers run JavaScript, and which don't
• the ways a website can produce its pages, and which suit public pages
• common problems on sites built with Next.js and other JavaScript frameworks
• how to test what crawlers see

HTML, JavaScript and rendering

When you visit a web page, your browser asks the website's server for it. The server sends back HTML, the code that holds the page's text, headings, links and images. HTML wraps each piece of content in labels called tags. <h1>Coffee subscriptions</h1> is a main heading, and <a href="/subscriptions"> is a link.

JavaScript is a programming language that runs in your browser after the HTML arrives. It can add content to the page, change it, or respond when you click something.

Rendering means running a page's JavaScript to build the finished page. Your browser does this automatically, so you never see a page before its JavaScript has run. Crawlers don't always render. Some read the HTML exactly as the server sent it and never run the JavaScript.

So every page has two versions. The raw HTML, also called the initial HTML, is what the server sends. The rendered HTML is what exists after JavaScript has run. On many pages the two are almost identical. On pages built mainly with JavaScript, the raw HTML can be nearly empty.

What Fernhill's page looks like to a crawler

This is the raw HTML of Fernhill's subscription page, shortened:

<html>
  <head>
    <title>Coffee subscriptions | Fernhill Coffee</title>
  </head>
  <body>
    <h1>Coffee subscriptions</h1>
    <p>Freshly roasted coffee, delivered on your schedule.</p>
    <table id="plans"></table>
    <script src="/assets/plans.js"></script>
  </body>
</html>

The table where the plans should be is empty. The last line loads a script, plans.js, which asks Fernhill's server for the plans and prices and fills in the table. After rendering, that part of the page looks like this:

<table id="plans">
  <tr><td>Weekly</td><td>£9.50 a bag</td></tr>
  <tr><td>Fortnightly</td><td>£10.50 a bag</td></tr>
  <tr><td>Monthly</td><td>£11.50 a bag</td></tr>
</table>

A visitor sees the full table. Google sees it too, because Google renders pages. An AI crawler that reads only the raw HTML sees a heading, one sentence and an empty table. If someone asks ChatGPT how much Fernhill's subscription costs, that crawler has collected nothing that answers the question.

Anything you want search engines or AI assistants to find and quote should be in the raw HTML.

How Google handles JavaScript

Google renders JavaScript well. In March 2026 it removed an old warning from its documentation, noting that "using JavaScript to load content is not 'making it harder for Google Search'" (Google Search documentation updates⁠(opens in a new tab)).

Google's JavaScript SEO basics⁠(opens in a new tab) describes three stages:

1. Googlebot fetches the page and reads the raw HTML.
2. The page waits in a queue. Later, Google's Web Rendering Service runs the page in a recent version of the Chrome browser and produces the rendered HTML.
3. Google uses the rendered HTML to index the page, and follows any new links it finds there.

Google reads the raw HTML before it renders anything, and three details follow from that.

A noindex tag in the raw HTML can stop rendering. Google's guide says pages that return a 200 status go to the rendering queue "unless a robots meta tag or header tells Google not to index the page". If a developer puts noindex in the raw HTML and plans to remove it with JavaScript, Google may never run that JavaScript, and the page stays out of search. Pages with other status codes might not be rendered at all. The indexing chapter explains noindex and status codes.

Rendering starts fresh every time. Google's March 2026 post on how Googlebot works⁠(opens in a new tab) says the Web Rendering Service "clears local storage and session data between requests". Googlebot doesn't log in, accept a cookie banner, scroll or click. Content that only appears after one of those actions won't be seen.

Googlebot reads only the first 2 MB of each file. The limit applies separately to the HTML and to each JavaScript and CSS file, and it's measured before compression. Anything past the limit isn't used for indexing (Googlebot⁠(opens in a new tab)). Very few pages come close. The hydration section below explains how some frameworks can push a page towards it.

Even though Google renders well, its guide still recommends building pages on the server, because "not all bots can run JavaScript".

Which crawlers run JavaScript

The best public data comes from a study by Vercel and MERJ⁠(opens in a new tab), published in December 2024. It found that none of the major AI crawlers it examined ran JavaScript. ChatGPT's and Claude's crawlers did download JavaScript files, which made up 11.50% and 23.84% of their requests, but they didn't run them.

Read the table with two cautions. The study is nearly two years old, and crawlers change. And no AI company says whether its crawlers run JavaScript. In September 2026 one site owner reported on Indie Hackers⁠(opens in a new tab) that OpenAI's crawlers, checked against OpenAI's published IP addresses, had started running JavaScript on their Next.js site. OpenAI hasn't confirmed this, and it's a single report.

Until a crawler's operator says it runs JavaScript, assume it reads only the raw HTML. Building for that costs little, and every crawler can use the result. Aleyda Solis's AI search checklist⁠(opens in a new tab) gives the same advice: test both the raw and the rendered HTML, and don't make important text or product details depend on JavaScript running in the browser.

How a website produces its pages

Developers choose how and when each page's HTML is produced. This choice is called the rendering strategy, and a site can use different strategies for different pages. You don't need to know how each one works. What matters for crawlers is whether the content is in the raw HTML.

For public pages, use one of the first four. Client-side rendering is fine for interactive parts of a page, such as filters, calculators and product configurators, as long as the content you want indexed is already in the raw HTML. Vercel's advice is to build "main content (articles, product information, documentation), meta information (titles, descriptions, categories), and navigation structures" on the server (The rise of the AI crawler⁠(opens in a new tab)).

Building pages on the server also makes them load faster for visitors. web.dev's guide to Largest Contentful Paint⁠(opens in a new tab) recommends it because images and text can be found in the raw HTML, without waiting for JavaScript. The page experience chapter explains Largest Contentful Paint.

For Fernhill, the fix is for the server to look up the plans and put them in the HTML before sending the page. The raw HTML then contains the full price table, and the page looks the same to visitors.

Dynamic rendering

Dynamic rendering means detecting crawlers and sending them a pre-built copy of each page, while visitors get the JavaScript version. It used to be a common fix for sites built entirely with JavaScript.

Google now describes it in the past tense. Its documentation⁠(opens in a new tab) says "Dynamic rendering was a workaround and not a long-term solution", and recommends server-side rendering, static rendering or hydration instead. iPullRank's guide to technical foundations for AI search⁠(opens in a new tab) still lists dynamic rendering for crawlers as an option for older sites.

For an older site that can't be rebuilt soon, dynamic rendering can fill the gap, as long as crawlers and visitors get the same content. Showing search engines different content from visitors is called cloaking, and it breaks Google's spam policies⁠(opens in a new tab). Google's dynamic rendering page⁠(opens in a new tab) says that "as long as your dynamic rendering produces similar content, Googlebot won't view dynamic rendering as cloaking".

Dynamic rendering relies on a list of crawlers to detect, and that list needs updating every time a new AI crawler appears. New sites should build pages on the server, and older sites that use dynamic rendering should plan to move away from it.

Problems on Next.js sites

This section is for developers. Next.js is a popular framework for building websites with React, a JavaScript library for building interfaces. Next.js builds pages on the server by default, using React Server Components. Four patterns can still hide content or page details from crawlers.

1. Content loaded in the browser

This is how Fernhill's price table problem looks in Next.js:

'use client'
// ❌ The server sends an empty table. Crawlers that don't run JavaScript see no plans.
export default function Subscriptions() {
  const [plans, setPlans] = useState([])
  useEffect(() => { fetch('/api/plans').then(r => r.json()).then(setPlans) }, [])
  return <PlanTable plans={plans} />
}

Loading the data in a Server Component puts the plans in the raw HTML:

// ✅ The server looks up the plans, so the HTML includes them.
export default async function Subscriptions() {
  const plans = await getPlans()
  return <PlanTable plans={plans} />
}

Use client components for interactive parts, such as toggles, forms and carousels. Don't use them to load the content you want indexed.

2. Metadata in client components

A page's metadata is the information in its <head> section, such as its title, description and canonical tag. In Next.js, the metadata object and the generateMetadata function only work in Server Components (generateMetadata⁠(opens in a new tab)).

If a client component sets the title or adds a canonical tag with JavaScript, crawlers that don't run JavaScript never see it. Changing a canonical tag with JavaScript also goes against Google's JavaScript guidance⁠(opens in a new tab). Set metadata on the server instead:

// app/coffees/[slug]/page.tsx
import type { Metadata } from 'next'

export async function generateMetadata(
  { params }: { params: Promise<{ slug: string }> }
): Promise<Metadata> {
  const { slug } = await params
  const coffee = await getCoffee(slug)
  return {
    title: coffee.name,
    description: coffee.summary,
    alternates: { canonical: `/coffees/${slug}` },
  }
}

3. Streaming metadata and AI crawlers

Streaming means sending a page in pieces, so the browser can start showing it before the whole page is ready. Since version 15.2, Next.js can stream metadata. On pages whose metadata depends on information looked up when the page is requested, Next.js sends the page first. It then adds the title, description and canonical tags inside the <body> once they're ready, instead of in the <head>.

The Next.js documentation⁠(opens in a new tab) says this has been tested with crawlers that run JavaScript, such as Googlebot. For crawlers it calls "HTML-limited bots", which it recognises by their user agent, Next.js waits and puts the metadata in the <head> as usual.

As of October 2026, the default list of HTML-limited bots⁠(opens in a new tab) is:

[\w-]+-Google|Google-[\w-]+|Chrome-Lighthouse|Slurp|DuckDuckBot|baiduspider|yandex|sogou|bitlybot|tumblr|vkShare|quora link preview|redditbot|ia_archiver|Bingbot|BingPreview|applebot|facebookexternalhit|facebookcatalog|Twitterbot|LinkedInBot|Slackbot|Discordbot|WhatsApp|SkypeUriPreview|Yeti|googleweblight

The list includes Bingbot, Applebot and the crawlers that build link previews for social networks. It doesn't include GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot or any other AI crawler. On pages that stream metadata, those crawlers find the title, description and canonical tag in the <body>.

Google follows robots meta tags outside the head⁠(opens in a new tab). The AI companies' crawler documentation doesn't say how their crawlers treat metadata in the body.

There are two ways to avoid the question. The first is the htmlLimitedBots⁠(opens in a new tab) setting, which takes your own list of crawlers. Your list replaces the default list rather than adding to it, so copy the defaults and add the AI crawlers to the end:

// next.config.ts
import type { NextConfig } from 'next'

const config: NextConfig = {
  // Next.js's default list (October 2026) plus AI crawlers.
  // Re-check the default list when you upgrade Next.js.
  htmlLimitedBots:
    /[\w-]+-Google|Google-[\w-]+|Chrome-Lighthouse|Slurp|DuckDuckBot|baiduspider|yandex|sogou|bitlybot|tumblr|vkShare|quora link preview|redditbot|ia_archiver|Bingbot|BingPreview|applebot|facebookexternalhit|facebookcatalog|Twitterbot|LinkedInBot|Slackbot|Discordbot|WhatsApp|SkypeUriPreview|Yeti|googleweblight|GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User/i,
}

export default config

The second is to set htmlLimitedBots: /.*/, which turns off streaming metadata for every visitor. Pages then take slightly longer to start loading (generateMetadata⁠(opens in a new tab)). For a site that wants to appear in AI answers, the first option keeps the speed benefit for visitors and gives AI crawlers metadata in the <head>.

Pages that Next.js builds ahead of time aren't affected. Their metadata is always in the raw HTML.

Streaming also affects testing. Google's URL Inspection tool and Rich Results Test identify themselves as Google-InspectionTool, which matches Google-[\w-]+ in the default list, so they get metadata in the <head>. Googlebot itself isn't on the list, so it gets streamed metadata. On pages that stream metadata, Google's testing tools don't show exactly what Googlebot sees.

4. Loading screens and Suspense

Next.js can show a loading screen, set in a loading.tsx file or a React <Suspense> boundary, while part of the page is still being prepared. The loading screen is sent first, and the real content follows later in the same response. Crawlers that read the whole response usually see the content. If the content is loaded by JavaScript in the browser after the page arrives, though, it won't be in the HTML at all. Check the raw HTML, as described in the testing section, rather than what the browser shows you.

Problems on single-page apps

A single-page app is a website that loads once and then uses JavaScript to swap in new content as you move around, without asking the server for a whole new page each time. Sites built with React, Vue or Angular often work this way. Google's JavaScript guidance⁠(opens in a new tab) lists the problems they run into, with fixes.

Give each page its own address

Some single-page apps put the page name after a # in the address, such as /#/subscriptions. Google doesn't reliably treat these as separate pages. Use normal addresses like /subscriptions, which developers can set up with the browser's History API.

Use real links

Every link that leads to another page should be an <a> tag with an href. Google says it "can only discover your links if they are <a> HTML elements with an href attribute". A button or other element that changes the page when clicked is invisible to crawlers as a link. The site architecture chapter covers links in more detail.

Return a real 404 for missing pages

In a single-page app, the server often answers every address with a 200 status, including addresses that don't exist. The app then shows a "not found" message, but the status says the page is fine. This is a soft 404, which the indexing chapter explains.

Google suggests two fixes in JavaScript. One sends visitors to an address that returns a real 404. The other adds a noindex tag to the error page:

// Option 1: send missing pages to an address the server answers with 404
if (!coffee) window.location.href = '/not-found'

// Option 2: mark the error page noindex
if (!coffee) {
  const meta = document.createElement('meta')
  meta.name = 'robots'
  meta.content = 'noindex'
  document.head.appendChild(meta)
}

Both fixes only work for crawlers that run JavaScript. It's better for the server to return a 404 status for unknown addresses, because every crawler understands it. The same applies to sites that send every unknown address to the home page or a sign-in page, which turns every broken link into a soft 404.

Let Google fetch fresh files

When it renders, Google may ignore your cache settings, the instructions that tell browsers how long to keep a saved copy of a file. Google's guidance⁠(opens in a new tab) recommends including a code that changes with every version in each file name, such as main.2bb85551.js. Each new version then has a new name, so Google can't use an out-of-date copy. This is called fingerprinting, and most build tools do it automatically.

Don't block your own files

If robots.txt blocks a JavaScript file, "Google Search won't render JavaScript from blocked files or on blocked pages" (JavaScript SEO basics⁠(opens in a new tab)). Applebot⁠(opens in a new tab) also needs your JavaScript and CSS files to render pages. Keep folders like /_next/, /static/ and /assets/ open to crawlers.

Lazy loading

Lazy loading means waiting to load images or sections of a page until the visitor scrolls near them. It makes pages faster, and Google supports it⁠(opens in a new tab) as long as the content loads when it comes into view. Three points to check:

• Use the built-in loading="lazy" setting on images, but not on the main image at the top of the page. That image is usually the page's Largest Contentful Paint, and lazy loading it makes the page feel slower. The page experience chapter explains this.
• Don't lazy-load the main text of a page. Google's lazy-loading guide⁠(opens in a new tab) says "Google Search does not interact with your page", so it won't scroll, click or swipe to load more. Crawlers that don't run JavaScript won't load it at all.
• If a page keeps loading more items as you scroll, such as a long list of products, also give each batch its own page address, so crawlers can reach every item.

Content in tabs or accordions, the sections that open when you click a heading, is a different case. If the content is in the HTML and only hidden from view, crawlers can read it. If it's fetched only when someone opens the tab, they can't.

Hydration data and the 2 MB limit

Many JavaScript frameworks send each page's HTML along with a copy of the data used to build it, so JavaScript in the browser can take over the page and make it interactive. This step is called hydration. The data usually sits inside the HTML as a large block of code. In Next.js it's the React Server Components payload, or __NEXT_DATA__ on older Next.js sites. In Nuxt it's window.__NUXT__.

On data-heavy pages, this block can be large. Googlebot stops reading HTML after 2 MB, so a large block placed before the main content could push that content past the limit. Three habits keep pages well clear of it:

• Keep the <head> tags (title, robots, canonical, hreflang and structured data) and the main content near the start of the HTML.
• Don't send data the page doesn't show. Give client components only the data they need.
• Move large scripts and styles into separate files, and don't paste images into the HTML as text, a technique called base64 encoding.

Testing what crawlers see

Compare three views of each important type of page. The first view is the quickest, and it shows what AI crawlers get.

1. The raw HTML

This is what crawlers that don't run JavaScript get. "View page source" in your browser shows it, as described above. Developers can also use curl, a command-line tool that fetches a page and prints the raw HTML:

# Is the main content there without JavaScript?
curl -s https://www.example.com/subscriptions | grep -c "Fortnightly"

# Are the title, description, canonical and robots tags in the <head>?
curl -s https://www.example.com/subscriptions | sed -n '/<head>/,/<\/head>/p' \
  | grep -iE '<title>|name="description"|rel="canonical"|name="robots"'

# How big is the HTML, uncompressed?
curl -s https://www.example.com/subscriptions | wc -c

curl can also send an AI crawler's user agent, which shows whether the site treats that crawler differently. It's a quick way to catch streamed metadata, and rules set at your CDN (content delivery network), the service many sites use to deliver their pages from servers around the world:

curl -s -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot" \
  https://www.example.com/subscriptions | grep -iE '<title>|rel="canonical"'

Treat the result as a rough check. Some CDNs check a crawler's IP address as well as its user agent, so a request from your own computer can be treated differently from the real crawler.

2. The rendered HTML

This is what Googlebot indexes. The URL Inspection tool in Google Search Console shows Google's rendered HTML, a screenshot, and any files it couldn't load. On Next.js pages that stream metadata, this tool shows metadata in a different place from Googlebot, as explained above.

3. The page with JavaScript turned off

Turn off JavaScript in your browser's settings and reload the page. If the page is empty or broken, crawlers that don't run JavaScript are in the same position.

Check several types of page, not only the home page. Product pages, articles, category pages and pricing pages are often built differently.

What to do next

1. View the source of each important type of page on your site, and search for a sentence from its main content.
2. Check that the title, description and canonical tag are in the raw HTML.
3. If content is missing from the raw HTML, ask your developer how those pages are rendered, and whether the server can include the content.
4. Check that robots.txt doesn't block your JavaScript and CSS files.
5. Check that an address that doesn't exist on your site returns a 404 status, not a 200 or a redirect.
6. If your site uses Next.js, ask your developer whether any pages stream metadata, and whether to add AI crawlers to htmlLimitedBots.

Further reading

• Understand JavaScript SEO basics⁠(opens in a new tab), Google's guide to how it crawls, renders and indexes JavaScript, with fixes for single-page apps.
• The rise of the AI crawler⁠(opens in a new tab), Vercel and MERJ's study of which crawlers run JavaScript. It's from December 2024, and it's still the best public data.
• Googlebot⁠(opens in a new tab), including the 2 MB limit on HTML and on each file.
• Dynamic rendering as a workaround⁠(opens in a new tab), Google's current position on sending crawlers a pre-built page.
• Fix lazy-loaded website content⁠(opens in a new tab), on lazy loading that doesn't hide content from Google.
• generateMetadata⁠(opens in a new tab) and htmlLimitedBots⁠(opens in a new tab), the Next.js documentation behind the streaming metadata section.