Free WordPress plugin · by J6 Venture

AI Crawler Control + Log.

AI bots are crawling your site right now, training on your content and running up your bandwidth. This free plugin shows you exactly which ones, how often, and what they cost, then lets you allow or block any of them in one click.

↓  Download for WordPress, free See what it does →
Free forever. No account, installs in under a minute, and all data stays on your server. Privacy policy
AI crawler log · last 24hwp-admin · AI Crawler Control
CrawlerHitsBandwidthStatus
GPTBot1,28442.1 MBBlocked
ClaudeBot90331.7 MBAllowed
Google-Extended61218.9 MBAllowed
PerplexityBot48814.2 MBAllowed
Bytespider2,05161.4 MBBlocked
The problem

AI crawlers are the traffic you can’t see.

Standard analytics hide bot traffic, so AI crawlers pull your content and burn your bandwidth completely unseen. You can’t tell GPTBot from a real visitor, you can’t prove what your server is spending on bots, and robots.txt is a one-way guess with no feedback. Block blindly and you risk shutting out the answer engines you actually want to be cited in.

Without it
  • AI crawler traffic is invisible in your analytics.
  • No idea which bots hit you, how often, or what they cost.
  • robots.txt is a guess, with no way to see if it works.
  • Blocking is all-or-nothing, so you risk losing citations.
With AI Crawler Control
  • Every AI crawler logged by name, hits, bandwidth and cost.
  • See exactly what each bot is spending on your server.
  • Allow the answer engines you want, block the rest, per crawler.
  • All inside wp-admin, with nothing leaving your server.
What you can do

See every AI crawler, then decide who stays.

One free plugin to watch AI crawlers, control them per bot, and prove what they do to your traffic. Here is what that looks like inside wp-admin.

Per-crawler control

Allow the ones that cite you.

Every AI crawler in one table, grouped by vendor and labelled by what it actually does. Keep the search bots that cite you, block the pure training and ad bots, and set each one to Allow, Ask to stay out, or Block.

  • Grouped by OpenAI, Anthropic, Perplexity, Google
  • Training, search, user fetch or ads, spelled out
  • An honest column for bots that ignore robots.txt
AI Crawler Control
CrawlerObeys robots.txtYour setting
OpenAI
GPTBot AI
Model training
YesAllowAskBlock
OAI-SearchBot AI
AI search index
YesAllowAskBlock
ChatGPT-User AI
User-triggered fetch
NoAllowAskBlock
Crawl log

See who takes and who sends back.

A raw log of every crawler hit, with status, size, IP and verified identity. It also catches humans arriving from AI answers, so you can see the crawl-to-referral ratio: which bots only take, and which actually send traffic back.

  • When, source, URL, status, size and IP
  • Referrals from AI answers logged alongside crawls
  • Filter by Today, 7, 30 or 90 days
Crawl Log
Today7 days30 days90 days
WhenSourceStatusSizeIdentity
14:02GPTBot20014.2 KBVerified
13:58OAI-SearchBot20022.8 KBVerified
13:51ChatGPT Referral2009.6 KBVisitor
13:44ClaudeBot20031.4 KBPending
Content signals

Put your position on the record.

Publish a machine-readable Content-Signal line that splits search, AI input and AI training into separate permissions. It doubles as a reservation of rights under Article 4 of EU Directive 2019/790. Compliance is voluntary, but your position is explicit and on the record.

  • The Cloudflare Content-Signal standard
  • search=yes, ai-input=yes, ai-train=no
  • Extra robots.txt rules appended verbatim
Content Signals
robots.txt · Content-Signal
searchAllowedNot allowed
ai-inputAllowedNot allowed
ai-trainAllowedNot allowed
Signals a reservation of rights under Article 4 of the EU DSM Directive.
Identity verification

Confirm the bot is who it claims.

User-agents are trivial to fake, so a request saying it is GPTBot often is not. The plugin checks every crawler against the official IP ranges each vendor publishes. Those vendor files are the only outbound requests it ever makes, and nothing about your site is ever sent anywhere.

  • Verified against OpenAI, Anthropic, Google, Apple
  • Spoofed user-agents flagged as failed
  • The only outbound request the plugin makes
Identity Check
Verify crawler identity
Ranges refreshed 12 minutes agoRefresh now
GPTBot
matched OpenAI ranges
Verified
ClaudeBot
matched Anthropic ranges
Verified
GPTBot (claimed)
no matching range
Failed
Fetching official IP ranges is the only outbound request this plugin makes.
Clean controls

Answer blocked crawlers on your terms.

Choose exactly how blocked bots are met, keep the log tidy on its own, and read the summary from your WordPress dashboard. Sensible defaults out of the box, with nothing to configure before it starts working.

  • 403 is honest, 429 says slow down, 410 says gone
  • Retention rolls old rows into daily totals
  • Ignore paths and a dashboard summary widget
Settings
Response for blocked crawlers
403 Forbidden
Keep detailed log for
30 days
Setup

Install, activate, done.

  1. Install and activate
    Install from the WordPress plugin directory, or upload the zip under Plugins → Add New → Upload, and activate.
  2. Let it log
    The crawler log starts populating immediately, with no configuration needed.
  3. Open and control
    Open AI Crawler Control in your admin sidebar to see who is crawling, what they cost, and allow or block any of them.

Take back control of your content.

Free, self-hosted, and private. See which AI bots are crawling your WordPress site in the next five minutes.

↓ Download for WordPress, free Get the Chrome extension
See our work Book Strategy Session →