You are being indexed by four separate systems, each with its own rules. Three of them never tell you what they found. You are not indexed once. You are indexed four times.
There is no such thing as the AI index
Your most expensive assumption in GEO: there is one place to get into. Optimize for AI, the thinking goes, and every assistant sees the result.
They do not share an index. They do not share a crawler. They do not agree on what a good source looks like.
Chapter 1 read the 12% overlap figure one way: as a ranking problem. Read it again as an access problem.
Four crawlers. Four sets of rules. Four ways to be locked out. Three of them never tell you.
So what does a single averaged AI visibility score tell you? Nothing useful. It hides the engine that is broken.
Google: the index you already have
Google is the only one that documents this for you. The documentation is short.
Its generative features run retrieval-augmented generation over the core Search index. Google's words: they rely "on our core Search ranking systems to retrieve relevant, up-to-date web pages from our Search index." Their index. Not a new one.
So the eligibility rules are ones you already know, plus one you probably do not:
- The page must be indexed and eligible to appear with a snippet. Blocked from a snippet means blocked from the answer.
- The site must not be excluded from generative AI features in Search Console. The control is under Settings, then Search generative AI. Inclusion is the default for every property, so this only bites if someone on your team switched it off.
One more thing changed while everyone was writing about crawlers. Chapter 1 told you AI Overviews and AI Mode merged in 2026 (Pichai confirmed it on the Q2 call in July). One surface, one report, one set of rules.
Which retires a lot of 2025 advice. Optimizing for them separately is last year's problem.
The Google-Extended trap, and what it actually controls
Plenty of sites blocked the Google-Extended token believing it kept them out of AI answers.
But it does not.
Google-Extended governs two things: whether your content trains future Gemini models, and whether it grounds answers in Gemini Apps and Vertex AI. It has no effect on Google Search.
Your page is already in the Search index. The AI surfaces retrieve from it regardless. Blocking the token changes nothing.
Training → Gemini. Grounding → Gemini Apps and Vertex. Search → untouched.
Which makes Google-Extended the one token in this chapter that is not a free block. It costs you Gemini answers.
So if you genuinely need out, there are two exits. They cost different amounts:
- The Search Console Exclude control (Settings, Search generative AI) removes your site from those features and leaves the rest of Search untouched. Google states it is not used as a ranking or inclusion signal elsewhere. This is the cheap exit, and almost nobody knows it exists.
- nosnippet and data-nosnippet also work, and they are blunter. They cost you the featured snippet and the rich result too.
Most teams reach for the second. Why? Nobody told them about the first.
ChatGPT: the one that reads you twice
OpenAI splits its crawling into four named bots. Confusing them is the most common technical mistake we find on audits. There are four:
- OAI-SearchBot: surfaces sites in ChatGPT search. Block it and you are out of the answers (a bare navigational link can survive, OpenAI says). This is the one that matters.
- GPTBot: collects training data. Blocking it is a licensing call, not a search one.
- ChatGPT-User: fetches a page because a user asked for it. OpenAI states it is "not used to determine whether content may appear in Search," and that because the fetch is user-initiated, "robots.txt rules may not apply."
- OAI-AdsBot: validates pages for the advertising product OpenAI began testing (February 2026).
We see the same pattern on audit after audit. A security team blocks OpenAI wholesale: all four tokens, one ticket. The brand disappears from ChatGPT search while the rankings stay perfect.
Who signed that ticket at your company?
Nobody notices, because the dashboard is green. Marketing was never told.
Where ChatGPT's results actually come from
OpenAI's documentation and the independent measurement do not agree. Know both.
OpenAI's help center says ChatGPT search "sometimes partners with other search providers" and names Bing and Shopify. That is the official position.
An independent study captured ChatGPT's retrieval stream directly in mid 2026. It found something different.
Only 1.5% of the URLs returned by OpenAI's own in-house retrieval hub appeared in Bing's top 20 for the same fan-out queries. Not one snippet matched a Bing snippet either.
That hub feeds nearly every search citation in ChatGPT's free instant mode. One exception: near-live questions, where it splits about 50/50 with scraped Google. Note whose results those are. Not Bing's.
Which gives you one instruction: optimizing for Bing does not get you into ChatGPT. It gets you into Bing.
The cache mechanics that matter more than the index question
The same study documented how ChatGPT stores what it fetches. Four findings. Each one is an instruction:
- JSON-LD is stripped on the read path. Read path only: the index is a separate step, unmeasured.
- JavaScript is never executed. Content that requires rendering is invisible. Chapter 10 takes this apart properly.
- Pages over 4MB are rejected outright. Not truncated. Rejected.
- Cached copies were served more than 90 days after the original fetch.
That last one is the uncomfortable one. A page you fixed last week may still be answering questions in its broken form. The copy on file is old.
If your content requires JavaScript to appear, a large part of the AI web cannot read it. If your page weighs more than 4MB, at least one engine refuses it outright.
This is independent measurement, not vendor documentation. Trust the direction. The digits are one product, one moment.
How to get a stale copy replaced
The 90-day finding raises an obvious question the write-ups never answer. What do you actually do about it?
Four levers, in order of bluntness:
- Change the URL. A new URL is a new fetch. Blunt, expensive, and it works.
- Fix your validators. Correct
Last-ModifiedandETag, honest 304s. A lying validator teaches every cache to keep what it has. - Fix
lastmodin your sitemap. Most are wrong, and a sitemap that claims every page changed today is a sitemap that says nothing. - Check your edge cache headers. You may be making the crawler's copy stickier than you meant.
And one writing consequence, which belongs to Chapter 5 but starts here.
If a claim can be quoted 90 days after it was fetched, date it inside the passage. "As of Q3 2026" costs four words and makes a stale quote indict itself.
An undated claim quoted three months late is just wrong.
Perplexity: the one that built its own
Chapter 2 covered their retrieval pipeline. What matters here is the crawler behind it, and the scale it runs at.
Their index tracks over 200 billion unique URLs, against 200 million queries a day.
Crawl scheduling is a machine learning problem for them, not a fixed frequency. A model predicts two things: whether a URL needs indexing at all, and when to come back. It weighs page importance against how often the page changes.
Not a schedule. A prediction.
A page that never changes gets visited less. Perplexity puts no number on it and nobody has measured one. The direction is not in doubt.
They also run two bots with different rules. PerplexityBot builds the search index and honors robots.txt. Perplexity-User fetches a page because a person asked, and Perplexity's own documentation says it "generally ignores robots.txt rules." One reads your file. One does not.
The stealth crawler dispute, and the lesson under it
In August 2025 Cloudflare published evidence of undeclared crawling. A stealth crawler ran 3 to 6 million requests a day (against 20 to 25 million declared). It spoofed a macOS Chrome user agent and rotated outside the published IP ranges.
Cloudflare pulled Perplexity from its verified bot list. Perplexity disputed the characterization.
You do not need a position on that dispute. You need the operational lesson:
robots.txt is a request, not a wall. If access genuinely must be controlled, it is controlled at the edge with authentication or a firewall rule. Not with a text file.
Being found, not just being allowed
Everything so far is about whether a crawler may fetch you. None of it is about how it finds you.
Perplexity's scheduler predicts whether a URL needs indexing at all. Which makes change signaling a retrieval variable, not a housekeeping task.
Your levers are thin and worth using:
- Sitemaps with honest
lastmod. The one signal every one of these systems can read cheaply. - IndexNow, with a caveat. It pings Bing and its partners. Since ChatGPT's provenance is not Bing, it does less than it is sold as.
And the flat fact underneath: none of these engines offers a submit-a-URL path. There is no Search Console for ChatGPT.
The crawl is the only door.
Claude: the one that publishes nothing
Anthropic documents three bots that touch your site: ClaudeBot for training, Claude-User for user-initiated fetches, Claude-SearchBot for search. Anthropic's words on the last one: it "analyzes online content specifically to enhance the relevance and accuracy of search responses." That is the entire public record.
All three honor robots.txt.
Beyond that you get nothing. No index size. No crawl policy. No re-ranking model. Nothing on what a good source looks like.
So how would anyone know what Claude rewards?
Anthropic has published no equivalent of Google's optimization guide or Perplexity's architecture paper. So when someone tells you, ask where they read it.
They did not.
| robots.txt token | Purpose | Verify by | Block it and |
|---|---|---|---|
| Googlebot | Search index, which feeds the AI surfaces | rDNS googlebot.com | You are gone from everything |
| Google-Extended | Gemini training and Gemini/Vertex grounding | Not a crawler, a token | No effect on Search. Removes you from Gemini answers |
| OAI-SearchBot | Inclusion in ChatGPT search | Published IP ranges | Gone from ChatGPT search |
| GPTBot | Model training | Published IP ranges | No documented search effect |
| ChatGPT-User | Fetches a page a user asked for | Published IP ranges | Little: robots.txt may not apply |
| PerplexityBot | Search index | Published IP ranges | Gone from Perplexity |
| Perplexity-User | User-initiated, ignores robots.txt | Published IP ranges | Nothing: it ignores robots.txt |
| Claude-SearchBot | Search result quality | Published IP ranges | Reduced visibility in Claude search |
| ClaudeBot | Model training | Published IP ranges | No documented search effect |
Read the last column twice. Four of those nine tokens decide whether an engine can see you. Two are pure training. Two are user-initiated fetchers that may ignore your file entirely.
And one, Google-Extended, is neither: it is free to block for Search and expensive to block for Gemini. Which of the nine are named on yours?
The robots.txt that actually works
Every GEO article tells you to audit robots.txt. None prints one. Here is ours. The repetition in it is deliberate:
# Global rules. Every named group below must repeat these, # because a crawler with its own group never reads this one. User-agent: * Disallow: /cart/ Disallow: /account/ Disallow: /search # Search crawlers. These decide whether you exist in the answer. User-agent: Googlebot Disallow: /cart/ Disallow: /account/ Disallow: /search User-agent: OAI-SearchBot Disallow: /cart/ Disallow: /account/ Disallow: /search User-agent: PerplexityBot Disallow: /cart/ Disallow: /account/ Disallow: /search User-agent: Claude-SearchBot Disallow: /cart/ Disallow: /account/ Disallow: /search # Training crawlers. A licensing decision, not a search one. # Blocking these has no documented effect on search citation. User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / # Google-Extended is NOT in the same category. It also governs # grounding in Gemini Apps and Vertex, which is a live answer # surface. Blocking it costs you nothing in Google Search and # removes you from Gemini answers. Decide that one separately. # User-agent: Google-Extended # Disallow: / # Discovery. The only door these engines offer you. Sitemap: https://example.com/sitemap.xml
User-agent: * group only when no named group matches it. Name a crawler in its own group and it stops reading your global rules entirely. Deploy a file with bare Allow: / groups, add a global Disallow: /staging/ six months later, and those four crawlers will walk straight into it. So every global rule is repeated inside every named group.
Anything not named here is allowed. Bingbot, Applebot-Extended, CCBot, Bytespider, meta-externalagent, and every crawler invented after this book was printed. There is no default deny in robots.txt.
Two things worth knowing about the syntax.
Disallow: /search is a prefix match, so it covers /search/ and /search?q= alike. And Crawl-delay is absent because Google ignores it, others honor it inconsistently, and it is the wrong instrument anyway: rate limiting belongs at the edge, where you can scope it to training crawlers only.
Two tokens are deliberately absent. ChatGPT-User and Perplexity-User are user-initiated fetchers, and both vendors state that robots.txt may not apply to them. Listing them buys you the feeling of control rather than the control.
The training block is optional, and Google-Extended is not part of it. GPTBot and ClaudeBot are pure training, so blocking them costs nothing in citations. Google-Extended is commented out on purpose: it also governs Gemini and Vertex grounding, which is a live answer surface. Blocking it is free in Google Search and expensive in Gemini. Three decisions, not one.
That repetition rule is the one people delete in review. It looks redundant. It is the only thing holding the file together.
So you have two options and no third: repeat every global rule inside every named group, or never name a crawler at all. Which is it on yours?
The argument against blocking training, which we think loses
Before you take the default, hear the counter-case.
Model weights carry entity knowledge. Chapter 8 is entirely about the model naming you unprompted, before any retrieval happens, and that recall comes from training rather than from search.
So a site that has trained nothing for three years may be a weaker entity in exactly the prior that decides which three brands get named.
Nobody has measured this. So the recommendation stands on cost, not on proof: the training crawl returns you nothing you can see, the search crawl returns you everything.
If your brand is already well known, that argument is weaker than it looks.
Stop auditing robots.txt. Audit reachability.
A clean robots.txt does not mean a reachable site. Not even close.
Which is why reachability, not the file, is the first thing to check. When did anyone last run that on yours?
Edge blocking is real, and narrower than the headlines suggest.
Cloudflare began asking every new domain at sign-up whether to allow AI crawlers: that was 1 July 2025. From 15 September 2026 it blocks training and agent crawlers by default on ad-monetized pages. Search crawlers stay allowed. That distinction is the whole chapter.
Nobody has published a credible count of who blocks at the edge. The causes are site-specific: a WAF rule, a bot-management default, a rate limit nobody remembers setting. Marketing has never seen any of it.
None of it appears in robots.txt. Which is why you test the response instead of reading the file.
Marketing reads a clean robots.txt and concludes access is fine. Meanwhile the site returns 403s to OAI-SearchBot. Nobody is looking.
So audit the response, not the file.
- Get a host that looks like a crawler. The cheapest VM at any cloud provider. Real crawlers arrive from datacenter address space, and most bot management scores network reputation alongside the user-agent string. Run this from your office and you are testing your own allowlist. From home, a residential IP no crawler uses.
- Fetch each of your ten highest-value URLs six times. Once with a normal browser user-agent as your control, then once as each crawler. Record status, final URL after redirects, and body size in bytes.
- Read the control first. A 403 for OAI-SearchBot means nothing until you know the control returned 200 from the same host. Control also blocked? You found a network-level block and every other row is noise.
- Compare bytes, not just codes. Any crawler row under about 30% of the control byte count is the interstitial case: a challenge page, a consent wall, a soft 404, an edge-cached error. It logs as success at your origin and arrives empty at the engine.
- Know what this cannot tell you. curl does not reproduce a real crawler's TLS and HTTP/2 fingerprint, and bot management reads both. A 403 here proves you are blocked. A 200 here does not prove you are reachable. That asymmetry is why step 6 is not optional.
- Pull logs from the edge, not the origin. If the block lives at your CDN the request never reaches your application, so a zero row in origin logs cannot distinguish never-crawled from turned-away-at-the-door. Ask whoever administers the CDN for 30 days of request logs filtered to these agents. If nobody can produce edge logs, that is the first finding, and its owner is infrastructure.
- Verify before you count. Forward-confirmed reverse DNS for Googlebot: reverse-resolve the IP, forward-resolve the hostname, confirm it returns to the same IP. For everyone else, membership of the vendor's published IP-range file. OpenAI's bots have no reverse-DNS path at all, so IP range is the only check.
| What comes back | What it means | Who owns the fix | Time |
|---|---|---|---|
| 200, full HTML | Reachable. Your problem is content, not access | Content | Nothing to fix |
| 200, JS shell | Reachable and unreadable. ChatGPT executes no JavaScript. Chapter 10 is entirely about this row | Engineering | Weeks |
| 403 | Deliberate block. Usually WAF or bot management, often a default | Infrastructure | Hours |
| 429 | Rate limited. You are throttling the crawler, not blocking it | Infrastructure | Hours |
| 200, tiny payload | Interstitial, soft-404 or edge-cached error. Reads as reachable in your logs, arrives empty. Most Cloudflare challenges return 403 instead | Infrastructure | Hours |
| Over 4MB ChatGPT read path | Rejected outright. ChatGPT only: no other engine documents a cap | Engineering | Days |
| 5xx on /robots.txt | Complete disallow, every compliant engine, until the next successful fetch. The highest-severity row here | Infrastructure | Minutes |
| 200 behind a consent wall | Server-side cookie gate. The most common cause of the tiny-payload row in EU markets | Legal and engineering | Weeks |
| No log entry at all | Either never crawled, or blocked at the edge before reaching origin. Origin logs cannot tell these apart | Infrastructure | Hours |
The file that deletes you when your server hiccups
One more thing about robots.txt, and it inverts every intuition you have about it.
Under RFC 9309, a robots.txt that returns 5xx or times out means the crawler must assume complete disallow. A 404 means allow everything.
Read that twice. A missing robots.txt is safe. A flaky one is fatal.
So the failure mode is not a bad rule in the file. It is your WAF challenging /robots.txt, your rate limiter catching it during a crawl spike, or your maintenance page returning 503 on it for twenty minutes.
Twenty minutes of 503 on that one path. Every compliant engine treats your whole site as disallowed until it fetches again.
Which gives you a five-minute job: exclude /robots.txt from every WAF rule, every rate limit, every bot challenge and every maintenance page. Then monitor its status code separately from your homepage.
Nobody does this. When did you last check yours?
Verify the crawler before you believe the log
One more thing the Perplexity dispute should teach you.
A user-agent string is an unauthenticated string. Anyone can send any of them. Including your competitor.
So every blocking rule and every traffic claim built on user agent alone is built on sand. Verify two ways before you count a visit or write a rule: reverse DNS, or the vendor's published IP ranges. The string is not evidence.
Otherwise your AI crawler dashboard is measuring whoever felt like claiming to be GPTBot that week. When did anyone last check yours?
What the crawl-to-refer ratio should decide
There is a real argument for refusing AI crawlers. Not an ideology argument: an economics one.
In the first week of August 2025, Cloudflare measured one number: pages crawled for every visitor sent back.
Anthropic crawled roughly 50,000 pages for every visitor it returned. OpenAI 887. Perplexity 118. Around 80% of that crawling was for training, and user-action crawling grew more than fifteenfold across 2025, so the split has moved since.
Read the gap, not the digits. And note that this is Cloudflare's whole network, which skews publisher. Your ratio is a different number.
Read it honestly in both directions. Which business are you in?
If you are a publisher whose business is pageviews, those ratios are an extraction problem. Blocking is rational.
If you are a B2B company whose business is being chosen, they are the price of admission. A 50,000 to 1 ratio still means being present in the answer when a buyer decides.
You were never going to monetize the crawl.
The objection this chapter has to answer
"You are telling me to open my site to companies that take 50,000 pages for every visitor they return. My infrastructure bill is real, and my content team is negotiating a license with two of these firms right now. Access granted free is a term I have already conceded. Why is unilateral disarmament the right move?"
If your company is negotiating a content license, the crawl decision is not yours. Ask your licensing lead before you touch robots.txt. Skip the rest of this section.
For everyone else, the leverage argument is weaker than it looks. Here is why.
Blocking does not remove you from the answer. The engine still describes you, from G2, from Reddit, from comparison pages, from your competitors' "alternatives to" posts.
What blocking removes is your version of you.
And your collective does exist. It is your CDN. Cloudflare changed its defaults for a large slice of the web on 15 September 2026, and it is acting on your behalf whether you opted in or not.
So the question is not block or do not block. It is which of your CDN's defaults you accept, and whether anyone told marketing they changed.
The exception worth taking seriously is bandwidth cost. And that is a rate-limiting problem, not a blocking one.
Bound it: never rate-limit the four search tokens. Apply budgets to training tokens only. If the bandwidth number is genuinely material, that is a pay-per-crawl or edge-cache conversation, not a robots.txt one.
Rate-limit. Do not disappear.
The objection nobody makes, and it is the better one
"My content is already crawled, already cached, already in the weights. You are offering me a reversible upside and an irreversible downside. You cannot untrain a model."
That is the strongest version. And the search-training split is the answer to it.
Grant search. Withhold training. The irreversible half never happens, and the reversible half is the one that shows up in an answer this quarter.
Bing publishes the fan-out: nobody else does
One asymmetry is worth exploiting. Chapter 2 showed you why it matters.
In February 2026 Bing Webmaster Tools added an AI Performance report (Copilot, AI summaries in Bing, select partners). Alongside citation counts it exposes grounding queries: the phrases the AI actually searched to find you. Not the phrases you guessed.
That is a fan-out, published. Google's generative AI report gives you impressions by page, device and country. No query dimension at all. None.
So does any tool really show you Google's fan-out? No. They are modeling, exactly like the prompt in Chapter 2.
Verify the property and read the report monthly, even if Bing sends you almost no traffic. The value is not traffic.
The value is seeing which sub-questions your content actually gets pulled into, then diffing them against the fan-out you predicted.
Every phrase Bing shows that you did not predict is a defect in your prompt.
So fix the prompt before you commission a single page. The same defect is corrupting the other 140 sub-questions you generated.
One thing this chapter contradicts
Chapter 7 is about structured data. This chapter just told you ChatGPT strips JSON-LD on its read path.
Those two facts have to be reconciled. Most schema writing never tries.
Schema is a Google-pipeline asset, and a Perplexity-pipeline maybe. On ChatGPT's read path it is not an asset at all.
The rule: anything an assistant needs to know must exist as visible prose.
Structured data is a supplement to that prose. Never a substitute.
Where does your pricing live? If the answer is a JSON-LD block, one of the four engines in this chapter cannot see it at all.
Four systems read your site this quarter. Three will never tell you what they found.
Stop waiting to be told.
- Run the reachability test before anything else. Ten URLs, four crawler user agents, from outside your network. Status code and byte count.
- Then pull thirty days of logs by user agent, verified against published IP ranges rather than the string itself.
- Deploy the robots.txt in Artifact 3.2. Repeat every global rule inside every named group, or a named crawler will ignore your global rules entirely.
- Check Settings, then Search generative AI, in Search Console. Inclusion is the default, so you are looking for someone having switched it off.
- Verify Bing Webmaster Tools this week and read the grounding queries in 30 days. Diff them against your predicted fan-out.
- Test each engine separately. An averaged AI visibility score hides the one that is broken.
- Google Search Central, "Optimizing your website for generative AI features on Google Search," published May 2026, updated 10 July 2026. Retrieval-augmented generation over the core Search index, and the eligibility conditions.
- Google, Search Console help documentation on generative AI features. Inclusion is the default control for all properties.
- Google, common crawlers documentation. Google-Extended governs Gemini training and Gemini Apps and Vertex AI grounding, not Search.
- Alphabet Q2 2026 CEO remarks, 22 July 2026. AI Overviews and AI Mode brought together into one Search experience.
- OpenAI, bot documentation, developers.openai.com, and the "ChatGPT Search" help article. Four named crawlers and their separate purposes.
- RESONEO, ChatGPT retrieval study, July to August 2026, and the Search Engine Land write-up, 17 August 2026. In-house retrieval hub versus Bing overlap, cache conversion, JSON-LD stripping, no JavaScript execution, the 4MB cap, 90-day cached copies. Independent measurement, not vendor-confirmed.
- Perplexity Research, "Architecting and Evaluating an AI-First Search API," 25 September 2025. Index scale and machine-learning crawl scheduling. Perplexity crawler documentation, docs.perplexity.ai.
- Anthropic support documentation on ClaudeBot, Claude-User and Claude-SearchBot, updated 7 April 2026.
- Cloudflare, "Perplexity is using stealth, undeclared crawlers," 4 August 2025. "AI crawler traffic by purpose and industry," 28 August 2025. Crawl-to-refer ratios for the week of 1 August 2025, and the training share of AI crawling.
- RFC 9309, the Robots Exclusion Protocol. Product-token matching, and the rule that the wildcard group applies only when no named group matches.
- Cloudflare, "Content Independence Day," 1 July 2025 (new domains asked at sign-up) and the July 2026 default-blocking announcement effective 15 September 2026, which leaves search crawlers allowed.
- Bing Webmaster Blog, "Introducing AI Performance in Bing Webmaster Tools, Public Preview," 10 February 2026. Grounding queries, described by Bing as a sample of overall citation activity.
- Ahrefs, "Only 12% of AI Cited URLs Rank in Google's Top 10 for the Original Prompt," 11 August 2025. 15,000 long-tail queries, four engines, Perplexity the outlier at 28.6%.