=== AI Crawler Control by J6 Venture - Block GPTBot, ClaudeBot & AI Bots + Crawl Log ===
Contributors: j6venture
Tags: ai crawler, gptbot, claudebot, ai bot blocker, aeo
Requires at least: 6.0
Tested up to: 7.1
Requires PHP: 7.4
Stable tag: 1.1.0
License: GPLv2 or later
License URI: https://www.gnu.org/licenses/gpl-2.0.html

By J6 Venture. See which AI crawlers hit your site, what they cost in bandwidth, how many visitors they send back, and block the rest.

== Description ==

Built and maintained by **J6 Venture**.

AI crawlers now account for a large share of the bot traffic hitting ordinary websites, and the exchange is lopsided. Published measurements put ClaudeBot at roughly 2,200 pages crawled for every one visitor referred, GPTBot at around 217:1, and Mistral's crawler at over 3,000:1. Meanwhile Googlebot sits nearer 5:1.

Those are industry averages. **This plugin measures the ratio for your site**, from your own server, and gives you a switch for every crawler.

= What you get =

**A real crawl log.** Every request from a known AI crawler: which bot, which URL, what status, how many bytes you served. Ordinary human visitors are never logged.

**Bandwidth accounting.** The plugin records the size of the HTML your server generates for each crawler request (before compression; files served outside PHP, such as images and PDFs, are not counted), so "AI bots are hammering my site" becomes a number you can point at rather than a feeling.

**Your own crawl-to-referral ratio.** The plugin also counts visitors arriving from chatgpt.com, claude.ai, perplexity.ai, gemini.google.com, copilot.microsoft.com and others, then divides. No other free WordPress plugin shows you this, and it is the only figure that tells you whether a crawler is worth its bandwidth.

**Identity verification that actually verifies.** A user agent string is a text field - anyone can type "GPTBot" into it, and plenty of scrapers do. This plugin checks the requesting IP address against the official published ranges from OpenAI, Anthropic, Perplexity, Google, Microsoft and Apple, plus forward-confirmed reverse DNS where a vendor uses that instead. A request that claims to be a crawler and fails the check is flagged. Where a vendor publishes nothing, the result is shown as "no published source" - never as a false accusation. Verification uses the real connection IP by default; if your site sits behind a proxy or CDN (such as Cloudflare), point it at the forwarded header on the Settings screen so a spoofed header cannot fake an identity.

**Blocking that works on crawlers that ignore robots.txt.** OpenAI's ChatGPT-User, Perplexity-User and Meta-ExternalFetcher all document that they bypass robots.txt, because a human asked for the page. For those, a robots.txt rule is decoration. Set them to Block and the plugin refuses the request at the server with a 403 before WordPress renders anything.

**robots.txt done properly,** including the Content Signals Policy (search / ai-input / ai-train) with its Article 4, EU Directive 2019/790 reservation of rights.

**The long tail, not just the names.** Bot-like requests that are not in the catalog are captured under "Unrecognised crawlers", so a new or niche AI crawler shows up the day it first visits instead of being silently dropped. They are logged, never blocked.

**Real long-range history.** Daily totals per crawler are kept well beyond the detailed log, so the headline numbers, the crawl-to-referral exchange and the chart go back 90 days even if you keep the per-request detail for only a week.

= Crawlers covered =

OpenAI (GPTBot, OAI-SearchBot, ChatGPT-User, OAI-AdsBot) - Anthropic (ClaudeBot, Claude-SearchBot, Claude-User) - Perplexity (PerplexityBot, Perplexity-User) - Google (Google-Extended, GoogleOther, Googlebot) - Microsoft (Bingbot) - Apple (Applebot, Applebot-Extended) - Meta (Meta-ExternalAgent, Meta-ExternalFetcher) - ByteDance (Bytespider) - Common Crawl (CCBot) - Amazon (Amazonbot) - Mistral, Cohere, Allen Institute, Diffbot, ImageSift, You.com, DuckAssistBot, Firecrawl, Timpi.

= Three traps this plugin will stop you falling into =

Most advice about blocking AI crawlers is wrong in one of three specific ways. The plugin labels each of them in the interface, at the moment you would otherwise make the mistake.

1. **Blocking GPTBot does not remove you from ChatGPT.** GPTBot collects training data. `OAI-SearchBot` builds the search index. Block the second one and you disappear from ChatGPT's answers; block the first and nothing about your visibility changes.
2. **Google-Extended does not control AI Overviews.** It governs Gemini training and Gemini app grounding only. AI Overviews and AI Mode are grounded by Googlebot, and Google does not let you separate them. The only way out of AI Overviews is out of Google Search.
3. **Applebot-Extended does not remove you from Siri.** It is a training opt-out. Apple's search products keep using Applebot.

= What this plugin will not tell you =

It will not claim that blocking AI crawlers improves your rankings. It does not. It will not sell you an "AEO score". And it will not tell you that an llms.txt file gets you cited - Ahrefs crawled 137,210 domains and found 97% of llms.txt files received zero requests, with AI retrieval bots accounting for 1.1% of the traffic the remaining 3% did get.

This plugin does one thing that is measurable and unrefuted: it shows you what crawlers actually cost you and lets you decide.

= No catch =

* No account, no signup, no email required.
* No API key.
* No paid tier, so no feature is being withheld from you.
* No upsell notices, no dashboard takeover, no admin nags.
* No telemetry. The only outbound requests this plugin ever makes are to the vendors' own published IP range files, and only if you leave verification switched on. You can turn that off and it becomes fully offline.

= Built for performance =

A request from a human costs a single lowercase conversion and a short run of substring comparisons against the crawler list - no database query, no output buffer, no options bloat. Only crawler requests are logged, and rows older than the retention window you set are pruned on a schedule, so the table cannot grow without bound. DNS lookups for identity verification run on cron, never inside a page request.

== Installation ==

1. Upload the plugin folder to `/wp-content/plugins/`, or install it from the Plugins screen.
2. Activate it.
3. Open **AI Crawlers** in the admin menu.

Nothing is blocked until you choose to block it. The plugin observes first.

== Frequently Asked Questions ==

= Will blocking AI crawlers hurt my SEO? =

Blocking the AI-specific crawlers listed here has no effect on classic Google or Bing rankings - those are Googlebot and Bingbot, which are separate. But blocking `OAI-SearchBot`, `Claude-SearchBot` or `PerplexityBot` will remove you from those products' answers, which may cost you referral traffic. The plugin flags which is which on the Bots screen so the trade-off is explicit before you make it.

= Does this stop AI companies scraping my content? =

Partly, and it is worth being honest about the limits. It stops crawlers that identify themselves, which is most of the major ones most of the time. It cannot stop a scraper that uses a residential IP address and a plain Chrome user agent, because at that point the request is indistinguishable from a person. Anyone who tells you their plugin stops all AI scraping is selling something. For determined evasion you need network-level defences such as a WAF or a CDN with bot management.

= Why does a crawler show as "not verified"? =

The IP address that made the request is not in the range the vendor publishes for that crawler. That usually means something is impersonating the crawler. Occasionally it means the vendor changed their ranges and the local copy is stale - use **Refresh now** on the Settings screen to re-download them.

= Why does a crawler show as "no published source"? =

That vendor does not publish IP ranges or use verifiable reverse DNS, so there is nothing to check against. The plugin says so plainly rather than guessing.

= Google-Extended never appears in my log. Is it broken? =

No. `Google-Extended` and `Applebot-Extended` are robots.txt tokens, not crawlers. They have no user agent and never make a request, so they cannot appear in any log. They are listed so you can set a rule for them; that is all they do.

= Will this slow down my site? =

No. A human request is checked with one lowercase conversion and a short list of substring comparisons, then left alone. There is no query, no output buffer and no autoloaded option payload on a normal page view.

= Does it work with a caching plugin? =

Yes, with one caveat that applies to every logging plugin: if a page cache serves a crawler from a static file without loading PHP, the plugin never sees the request. Most caching plugins bypass the cache for unknown user agents, but if your crawl counts look low, that is why. Blocking still works if your cache loads WordPress; if it serves from disk before PHP, add the block at the server or CDN level instead.

= Can I keep the log longer than 30 days? =

Yes, up to 365 days. That setting controls the per-request detail (URLs, IPs, the raw log). Daily totals per crawler are kept far longer, so the dashboard totals, the crawl-to-referral exchange and the chart still show a full 90 days even on a short detail window. The crawler inventory, hot paths and the raw log reach back only as far as the detail is kept, and say so when you pick a range beyond it.

= Does it collect any personal data? =

Crawler IP addresses are stored, because verifying a crawler's identity is impossible without them. They are pruned with the rest of the log. Visitors arriving from AI answers are recorded as a URL and a source name only - no IP, no user agent, no identifier of any kind. Nothing is ever transmitted off your server.

== Screenshots ==

1. The dashboard: crawler requests, HTML served, visitors sent back, and requests blocked.
2. Crawl-to-referral ratio per company, measured on your own site.
3. Every crawler, what it does, whether it obeys robots.txt, and your rule for it.
4. The raw crawl log with identity verification.
5. robots.txt preview including the Content Signals Policy.

== Changelog ==

= 1.1.0 =
* New: "Unrecognised crawlers" - bot-like requests that are not in the catalog are now captured and shown on the dashboard (logged, never blocked), so a new or niche AI crawler surfaces the day it first visits. Toggle it off under Settings if you only want named crawlers.
* New: real long-range history. Daily totals per crawler are kept beyond the detail window, so the headline totals, the crawl-to-referral exchange and the chart go back 90 days even on a short retention setting. The 90-day range is available again; row-level cards note when they only reach back to the detail window.
* The retention setting now clearly separates per-request detail (pruned on your schedule) from daily totals (kept far longer).

= 1.0.1 =
* Identity verification now uses the direct connection IP by default. Forwarded headers (X-Forwarded-For, CF-Connecting-IP and similar) are trusted only when you declare a proxy on the Settings screen, so a spoofed header can no longer fake a "verified" crawler.
* Added a confirmation step before turning away a classic search engine (Googlebot, Bingbot, Applebot) or an AI search crawler, with a plain warning about the visibility you would lose.
* Bandwidth is now labelled honestly as the HTML generated before compression, excluding static files served outside PHP.
* The crawl-to-referral ratio now carries a note that it is a floor: referrals are under-counted when a browser strips the referrer.
* Date ranges are capped to your retention window so the dashboard never offers a period whose detail has been pruned.
* CSV export now neutralises spreadsheet formula injection from crawler-supplied fields.
* Dashboard polish: honest tile labels, a clearer empty state with a call to action, right-aligned numeric table headers, a reading-friendly chart with date and peak labels plus a screen-reader table, and accessibility fixes.
* Per-crawler identity now shows "N of M verified" while confirmation is still in progress, rather than a premature "Verified".

= 1.0.0 =
* First release.
* Crawl logging with per-request bandwidth measurement for 28 AI and search crawlers.
* Crawl-to-referral ratio per company, measured from your own traffic.
* Crawler identity verification against vendor-published IP ranges and forward-confirmed reverse DNS.
* Per-crawler policy: allow, robots.txt rule, or hard block at the server.
* Content Signals Policy support in robots.txt.
* CSV export, dashboard widget, automatic log rollup and pruning.

== Upgrade Notice ==

= 1.1.0 =
Adds an "Unrecognised crawlers" view so new AI bots show up by themselves, and real 90-day history that survives a short detail-retention setting.

= 1.0.1 =
Hardens crawler verification against spoofed IP headers, adds a deindex warning before blocking a search engine, and makes the bandwidth and ratio figures honest. Recommended for everyone.

= 1.0.0 =
First release.
