Rendering for AI Crawlers: What They Can Read

Microsoft's own webmaster guidelines are 125 kilobytes of HTML containing fifteen visible words. The rest needs JavaScript.

The document that cannot read itself

Start with a page that proves the chapter.

Bing publishes webmaster guidelines. They exist to tell you how to be crawlable. We fetched them on 20 August 2026. Browser user agent, no JavaScript engine.

The response was 125,562 bytes.

Strip the tags and the script blocks. Count what is left: fifteen words. Here they are, complete.

That is the entire readable content of Microsoft's advice on being readable. Sit with that for a second.

Which is not a gotcha. It is the most common failure on the web, published by people who know exactly how it works. Read that again.

Your page probably has two bodies. One your server ships. One a browser builds after running your code. Most machines reading you only ever get the first. Which one is yours?

So this chapter is about the gap between those two bodies. Three questions: who sees which one, what the evidence for that actually is, and what it costs you to close. Three earlier chapters sent debts here.

We are paying all three.

What ten vendors actually say

Chapter 3 made a flat claim. ChatGPT executes no JavaScript, so content that requires rendering is invisible to it. That claim needs its evidence shown. And the first thing to show you is what is not evidence.

OpenAI has never said it.

Its crawler documentation, fetched 20 August 2026, runs 629 words. It documents four bots: GPTBot, OAI-SearchBot, OAI-AdsBot and ChatGPT-User. The word JavaScript appears zero times. So do render, DOM, headless and timeout.

Anthropic's crawler page, last updated 7 April 2026: zero.

Amazon: zero. Meta: zero.

Perplexity's: zero. Its documentation site does say render and DOM in its page source. Every occurrence is site furniture: stylesheet comments, script handlers, a cookie variable name.

ByteDance does publish something, on its Toutiao search webmaster platform. That page is client-side rendered too. So its position is unknown here.

So five of the largest crawler operators on the web have never addressed it in writing. Not denied it. Never raised it. Would you build a roadmap on that?

Four operators do address it, and only one says no.

Google states it plainly. Every page returning a 200 goes into a queue. Then "a headless Chromium renders the page and executes the JavaScript."

Microsoft said the same in October 2018. Then again in October 2019, moving bingbot onto Edge.

Its current webmaster guidelines carry a JavaScript section too. Nobody here can tell you what it says.

That page is the one this chapter opened with. Fifteen words. A request to enable JavaScript.

Apple hedges. Applebot "may render the content of your website within a browser."

Common Crawl is the only operator that states the negative. Nine words, in its FAQ: "Currently, JavaScript is not executed and Cookies are not used."

No hedge. No qualifier.

That one matters out of proportion to its size. Common Crawl is not answering anybody's question. It is the corpus these models were built on. Not the one answering your question today. The one that formed what they already believe about you.

So the honest form of Chapter 3's claim is narrower than Chapter 3 stated it. OpenAI does not say it fails to render. OpenAI says nothing at all. Somebody else went and measured.

So where does the claim come from

One study. You should know its shape before you build a roadmap on it.

In December 2024 Vercel published a network log study run with MERJ. The sample: its own network, plus two job board sites on different stacks, over a few months.

The method is the part worth knowing. They did not infer execution from server logs, which cannot tell you.

Their earlier Googlebot study used a client-side beacon. Code that only fires if JavaScript actually runs. Fires means executed. Silence means fetched and ignored.

The finding, in their words: "The results consistently show that none of the major AI crawlers currently render JavaScript."

The named list is specific. OAI-SearchBot, ChatGPT-User and GPTBot. ClaudeBot. Meta-ExternalAgent. Bytespider. PerplexityBot.

Then the sentence everybody skips.

Read that twice. They download your JavaScript. They never run it. Why would they?

Which is why your access logs are misleading. You will see these bots pulling bundle.js all day long. A fetch is not an execution. Most rendering audits we have been shown confuse the two.

Now the part that should make you hold this loosely.

That study is dated 17 December 2024. Today is 20 August 2026. Twenty months, and it is still the only measurement of its kind at network scale.

Two smaller tests point the same way. A German consultancy built a page in October 2025 with prices hidden in eight different places. Then it asked five AI systems what the products cost. The JavaScript-rendered price was found by Gemini. And by nobody else.

An independent consultant ran a black box version in August 2025. ChatGPT and Claude both refused on a client-side rendered page. They said they could not read it.

Neither is strong on its own. The first is one page, one vendor, five queries per system. And Claude found none of the eight prices in it, including the plain visible control. Which means Claude's row tells you nothing about JavaScript at all.

What is missing is anything on the other side. We looked hard.

No 2026 replication. No vendor announcement that rendering has started. No academic paper. Cloudflare publishes crawler volumes, robots.txt compliance and crawl-to-click ratios, and nothing at all about rendering.

So here is the whole evidence base. One network-scale study from 2024. Two small tests from 2025. Zero contradictions, and five vendors who have never addressed it.

That is thinner than the confidence with which the field repeats it. We believe it. But we would re-test before spending a quarter of engineering budget on it. So should you.

The exception nobody mentions

Gemini renders, and the same study says so.

Their wording: Gemini runs on Google's infrastructure, which gives it "the same rendering capabilities" they measured for Googlebot, "allowing it to process modern web applications fully."

Which follows from how Google works. Every 200 goes to the render queue, JavaScript on the page or not. And the index is built from the rendered HTML. Not from the raw response.

How long the queue actually is

Rendering is not instant, and the folklore around that is worse than the numbers.

The same team matched 37,000 pages against their render times. The median wait was ten seconds. Three quarters were done inside twenty-six seconds.

So for most pages the queue is not a problem at all.

The tail is where it hurts. At the ninetieth percentile the wait is about three hours. At the ninety-fifth, six. At the ninety-ninth, eighteen hours.

Which matters for one thing only: content with a short shelf life. A price that changes daily. A stock level. A live event. Is that you?

If your content is stable for a week, ignore this. If it is stale in six hours, the raw HTML is the only version anybody can trust.

So one of the largest AI surfaces on the web reads your second body. Does that carry to AI Overviews?

Google has never said.

Four documents were checked for this chapter: the AI features page, the AI optimization guide, How Search Works, and the most detailed crawler post Google has published in years. None of them connects the two.

The closest Google gets is eligibility. To appear in AI Overviews or AI Mode, "a page must be indexed and eligible to be shown in Google Search with a snippet, fulfilling the Search technical requirements." Then it adds: "There are no additional technical requirements."

Put those beside each other and the inference is strong. AI Overviews draw from the index. The index is built from rendered HTML. Therefore rendered content is reachable.

We think that inference is correct. It is still an inference. Google has never written the sentence, and a strong inference is not a citation.

One trap worth naming, because it is easy to fall into. Google's AI optimization guide does use the word DOM. It sits in a passage about "browser agents" that inspect "the DOM structure." Those are third-party agents visiting your site. Not Google describing its own AI surfaces. Anybody quoting it that way has misread the page.

The two megabyte wall

On 31 March 2026 Google published the most detailed account of its crawler in years. It contains a limit most people have never read.

Read the third sentence against the first.

The cap applies per resource. Your JavaScript bundle is a resource. So ship three megabytes of it, and the renderer receives the first two. Then tries to run that.

Which fails in the way truncated code always fails. Silently, and not on your machine.

The same post adds something else. Google's renderer "operates statelessly," and "it clears local storage and session data between requests."

So anything that only appears on a visitor's second view does not appear. No accumulated state, no remembered preference, no progressive reveal.

The debt from Chapter 7

Chapter 7 said markup injected by JavaScript has an address, and this is the address.

Google supports it explicitly. It even documents Google Tag Manager as one way to do it. The sentence in its structured data guidance is short, and the conditional in it is the whole thing.

"Google Search can understand and process structured data that's available in the DOM when it renders the page."

When it renders. Google Search can. That sentence is about Google, and Google is the operator that renders.

How much of the web does this? Two percent.

The 2025 Web Almanac measured it directly across roughly 16 million sites. Their finding: "just 2% of mobile and desktop crawls have structured data added via JavaScript," unchanged from 2024. So the blast radius is small. And if you are in that two percent, you probably know it.

Has anyone measured whether the non-rendering crawlers see it? Once, and the answer looks damning until you read the control.

That October 2025 test included a price published only as JSON-LD injected by JavaScript. No AI system found it. Seven configurations, zero hits.

Now the control, which is the part that keeps this honest. The same page also served plain JSON-LD in the HTML. No JavaScript involved. That was invisible to all seven too.

So the failure is over-determined. You cannot blame JavaScript when the version without JavaScript failed identically. The tester says so himself. He adds that his test covers direct fetching, not the indexed path that Google and Bing surfaces use.

Which leaves a plain recommendation and a warning.

The recommendation: put your JSON-LD in the HTML your server ships. One template change. It costs nothing at runtime and removes a dependency for free.

The warning: nobody should sell you a schema-into-tag-manager migration as an AI visibility project. No measurement supports the upside. And Chapter 7 already priced what structured data does and does not do.

What belongs in the first body

The advice usually stops at server-render everything. Which is right and not very useful, because it does not tell you what to check.

Four things, in this order.

Your main content, obviously. The h1, the body copy, the answer to whatever question the page exists to answer. If a sentence matters for citation, it belongs in the response your server sends.

Your links. A crawler that does not execute code cannot follow a route it never saw. Client-side routing with no anchor tags in the HTML means whole sections of your site are unreachable to anything except a renderer.

Your metadata. Title, description, canonical. The 2025 Web Almanac found the canonical injected by JavaScript on about 2% of pages, which is the same small share as the structured data problem and the same easy fix.

Your structured data, per the section above. That is the whole list.

Now the reassuring part, because the picture across the web is better than the panic suggests.

That Almanac measured raw and rendered word counts across roughly 16 million sites. The median home page shows about 18% more words after rendering on desktop, 16% on mobile.

An eighteen percent gap is normal. It is the usual furniture: menus, widgets, cookie notices and the odd late-loading module.

The Almanac also notes that some of that gap is not client-side rendering at all. It is content that sits in the HTML and gets hidden by a mobile breakpoint, inside an accordion or a tab. Which a non-rendering crawler reads perfectly well.

So most sites are fine. The ones in trouble are not near the median, and they are not subtle. They are the ones where the raw response is a nearly empty shell.

Like the one this chapter opened with.

The failures that have nothing to do with JavaScript

Rendering gets the attention. These five are cheaper to fix, and more likely to be broken.

They are spending a third of their visits on nothing

The same Vercel data measured where AI crawlers actually land. ChatGPT spent 34.82% of its fetches on 404 pages. Claude, 34.16%. Googlebot, 8.22%.

So a third of what these bots spend on you may be going nowhere. To URLs that do not exist. Have you checked?

Which is a sitemap and redirect problem, not a rendering problem. Check your logs by user agent before you touch your framework.

Tabs and accordions are safe, but only for one reader

Google is explicit that hiding content behind tabs is fine. It goes further and recommends it: "consider moving content into accordions or tabs to save space."

That guarantee is Google's alone.

It holds because Google renders and then reads the DOM, hidden or not.

No other operator has made the same promise. But the mechanics are kinder than they sound. If the content is in your HTML, a non-rendering crawler gets it, click or no click. If it loads on click, nobody gets it.

So the test is not visible versus hidden. It is present versus fetched-later.

Lazy loading, and the line it crosses

Google's rule is one line: lazy loading must trigger on the viewport, never on an interaction. Their words: "To ensure that Google sees all content on your page, make sure that your lazy-loading implementation loads all relevant content whenever it is visible in the viewport."

A crawler does not click. It does not scroll in the way you think either.

Infinite scroll has a documented fix, and it is unglamorous. Every chunk needs its own persistent URL. Google suggests "absolute page numbers in the URL, for example by using ?page=12 as a query parameter."

The single page app that returns 200 for everything

Client-side routing means your server answers 200 to URLs that do not exist. Google calls the result a soft 404. It names two remedies: redirect to a URL that really does return 404, or inject a noindex robots tag onto the error view.

Neither remedy helps the crawlers that do not run your JavaScript. For those, the first option is the only one that works.

Blocking your own scripts

The oldest mistake on this list. Disallow your JavaScript or CSS in robots.txt and the renderers that would have read you cannot.

Google is blunt: "Google Search won't render JavaScript from blocked files or on blocked pages." Apple says the same for Applebot. It names XHR, JS and CSS specifically.

Open your robots.txt. Look for a Disallow line covering /static, /assets, /_next or /dist. It is a two-minute check and we still find it.

Traced to source

Five claims you will hear about rendering. Here is where each one actually comes from.

Row two is the one to watch for in a pitch deck.

It is the difference between a vendor saying something and a researcher measuring it. Both can be right. Only one of them can be quoted back at the vendor later.

Perplexity publishes a blog post on the difference between agents and bots. We tried to read it for this chapter.

What came back was nine words, and the last seven were these: "Enable JavaScript and cookies to continue."

Which is a bot challenge, not a rendering failure. A different problem with the same result, and worth separating, because this chapter is about separating them.

What to do about it

Four real options, and one that Google has retired.

Dynamic rendering means detecting the crawler and serving it a pre-built page. Google used to recommend it. Google now writes about it in the past tense.

"Dynamic rendering was a workaround and not a long-term solution for problems with JavaScript-generated content in search engines."

Google still says it does not treat it as cloaking, so long as the content matches. Bing said the same in 2018, and more generously than that.

So who is the risk here? Not the search engines.

No AI vendor publishes a policy on serving different content to its crawler. Not OpenAI. Not Anthropic. Not Perplexity. The crawler documentation and the publisher FAQs were checked for this chapter, and there is nothing there.

Which does not mean there is no rule. It means there is no rule you can read.

Then August 2026. Perplexity de-indexed a major magazine for serving markdown-formatted ads to AI agents. It called the practice deceptive and warned that publishers doing it risk their trust score. And it cited no published policy, because there is not one.

That is the live risk with dynamic rendering in 2026. Not a search penalty. An unwritten rule, enforced by a company that never wrote it down. Against a site with no notice and no appeal. How would you argue that one?

So serve everyone the same HTML. It is simpler. It is cheaper to reason about, and it is the only position that cannot be reinterpreted later.

The ten-minute test

Before you scope any of that, find out whether you have a problem at all. Most teams assume they do. About half of them are wrong.

Nobody will show you what they saw

Step 6 of that artifact only covers Google. There is a reason, and it is not an oversight.

Google renders your page back to you on request. Search Console shows a screenshot, the loaded resources and the JavaScript output. It is the most generous tool any operator offers, by a distance.

Bing has a URL inspection tool with a live check. Its documentation promises "the option to see a HTML format and HTTP response details of the same." It never says rendered.

Then the rest. OpenAI, nothing. Anthropic, nothing. Perplexity, nothing.

No fetch-as-our-bot. No render preview. No log of what their crawler took away from your site.

So of the ten operators in Table 10.1, exactly one will show you what it rendered. You cannot check the rest. You can only control the input, and assume the worst about the processing.

Which is the argument for the boring fix, stated one more time. If everything important is in the HTML your server sends, you do not need a tool to tell you it arrived.

What this costs

Less than the meeting about it usually costs. Three cases.

If you are on a modern framework, server rendering is a configuration change and a redeploy. Half a day, including the arguing.

If you are on an older single page app, this is real engineering. Weeks, not days. And the honest way to scope it is to fix your highest-value templates first, rather than the whole site at once.

If you are on a hosted platform, you already have server-rendered HTML. This chapter is then a ten-minute check that confirms it.

The recurring cost is close to zero. Static generation and hydration add nothing at runtime. Server rendering adds compute per request. A real line item at scale, a rounding error below it.

The cost of the alternative is the part nobody prices. Ship an empty first body, and every non-rendering crawler that visits takes nothing away with it. There is no partial credit here. Nothing is what it gets.

And it will keep visiting.

If you are already doing it

Some teams will run the test and pass. Good. Two things still worth ten minutes.

Check your bundle sizes against the two megabyte cap. Any single resource over it is truncated for Google before rendering begins.

Then check the 404 rate by user agent in your access logs. Are your AI crawlers burning a third of their fetches on missing URLs? That is crawl budget you did not know you had.

Then one more, and it is the one teams skip.

Re-run the diff after your next major release. Rendering regressions do not announce themselves. A component gets moved client-side, a library changes its default, and the raw response quietly empties out.

Nothing in your monitoring will catch that. Your page still loads. Your users still see it. Your uptime is fine.

Two word counts, once a quarter. Put it in the release checklist and forget about it.

None of these is a rendering fix. All of them fit on the same afternoon's ticket.

The objections this chapter has to answer

"You built a whole chapter on one study from 2024."

Correct, and this chapter said so twice before you got here. It is the only network-scale measurement anyone has published. It has two smaller corroborations, and nothing contradicts it. And if that is too thin for your risk appetite, note that the mitigation is cheap anyway.

"Server rendering everything is expensive and you are hand-waving it."

Fair for legacy stacks. Wrong for modern ones. That is why Table 10.5 separates them, and why the artifact tells you to test before you scope.

"If ChatGPT cannot render, how did it read my client-side page last week?"

Two likely answers, and neither one is rendering.

First: content in the initial response that is not HTML still travels. Vercel notes that JSON payloads and server component data "may still be indexed since AI models can interpret non-HTML content." Your framework may already be saving you.

Second: it read a cached copy from a search index, and never fetched you at all.

Neither means your page is safe. Both mean the diagnosis is different.

A fourth, which cuts at us. "You sell the fix."

We do. Rendering audits are a small line for us and framework migrations are a large one, so we have an interest in you finding a problem here. Which is exactly why Artifact 10.6 is six steps you run yourself, in ten minutes, without calling anybody.

Run it before you take a quote. From us or from anyone.

What would change our mind

Three things, and the first one is overdue.

A 2026 replication at network scale. The current evidence is twenty months old and the crawlers have shipped a lot since.

Any vendor documenting rendering. One sentence from OpenAI or Anthropic would settle more than any third-party test.

Google connecting its own two statements. It says AI Overviews come from the index. It says the index is built from rendered HTML. If it ever writes the sentence joining them, we can stop calling that an inference.

What is left

So the ledger for this chapter, in four lines:

Two operators document that they render. One documents that it does not. Six have never said.

The measurement everyone cites is real, careful about the fetch and execute distinction, and old.

The fix is a configuration change on a modern stack and a project on an old one. Either way you should test before you scope, and the test is ten minutes.

Which closes Part III, and closes the last of the debts. You now know what the machines read, where they read it, and whether they can read you at all.

Which leaves the question every chapter so far has quietly deferred. If you do all of this, how would you know it worked?

That is Chapter 11, and it is the hardest chapter in the book, because the honest answer starts with what cannot be measured.