How to Block AI Bots on Your Server

Why would you block AI bots?

AI crawlers have become a normal part of Internet traffic. Some crawl content for model development, some build indexes for AI-powered search, and some retrieve a page because an individual user asked an AI service to access it.

Those are not the same use cases. A publisher may want to be discoverable in AI search while refusing model-training crawls. An e-commerce site may welcome referral traffic but want to stop unknown scrapers from repeatedly rendering expensive product, search or API pages.

Common reasons to restrict automated AI traffic include:

• preventing unwanted scrapers from consuming original content;
• reducing PHP, application, database, search and bandwidth usage;
• protecting expensive dynamic endpoints and APIs;
• keeping fake search or AI crawlers out;
• controlling which services may crawl for model development;
• reducing noise in logs, analytics and operational monitoring.

Method 1: publish your policy in robots.txt

The standard starting point is /robots.txt. The Robots Exclusion Protocol gives crawler operators a machine-readable way to discover which paths you ask them to allow or disallow.

A simple policy that blocks several model-development crawlers while leaving search-oriented access available could look like this:

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Google-Extended
Disallow: /

User-agent: PerplexityBot
Allow: /

This is an example policy, not a universal recommendation. The right configuration depends on whether you want training, AI-search discovery and user-triggered retrieval to have access to your content.

OpenAI: GPTBot and OAI-SearchBot are separate controls

OpenAI currently documents GPTBot and OAI-SearchBot as independent robots.txt controls. A site can allow OAI-SearchBot for visibility in ChatGPT search while disallowing GPTBot to indicate that crawled content should not be used for training OpenAI's generative AI foundation models.

Anthropic: different bots for different purposes

Anthropic documents ClaudeBot for content that may contribute to model development, Claude-SearchBot for search, and Claude-User for user-directed retrieval. Anthropic states that these bots honor robots.txt directives.

Google-Extended is a robots.txt control token

Google-Extended is important because it does not have its own HTTP User-Agent string. Google says the token controls certain Gemini model-training and grounding uses of content already crawled by Google, and that using Google-Extended does not affect inclusion or ranking in Google Search.

Perplexity distinguishes search crawling and user-triggered retrieval

Perplexity documents PerplexityBot for surfacing websites in search results and Perplexity-User for user-triggered fetches. Its documentation notes that Perplexity-User generally ignores robots.txt because the fetch was requested by a user. This is a good example of why the modern crawler landscape cannot be reduced to one global “AI bot” switch.

Why robots.txt is not enough when access must be blocked

The most important limitation is part of the standard itself. RFC 9309 says that robots.txt rules are not a form of access authorization. Google makes the same practical point in its documentation: robots.txt instructions cannot force an arbitrary crawler to obey them.

The request still reaches your server. A cooperative crawler reads the policy and decides whether to continue. An unknown scraper can ignore the file completely.

This does not make robots.txt useless. It means robots.txt and access control solve different problems.

Method 2: block known User-Agents at nginx or Apache

If you want a simple hard block, your web server can reject selected User-Agent strings before the request reaches the application.

if ($http_user_agent ~* "(GPTBot|ClaudeBot|PerplexityBot)") {
    return 403;
}

For Apache with mod_rewrite, the same basic idea can be expressed as:

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (GPTBot|ClaudeBot|PerplexityBot) [NC]
RewriteRule ^ - [F,L]

These examples are useful when you intentionally want to reject a known crawler identity. They are not crawler verification.

The crawler identity problem: a User-Agent is only a string

Any HTTP client can claim to be Googlebot, GPTBot, ClaudeBot or an ordinary Chrome browser. Google explicitly warns that its crawler User-Agent strings can be spoofed and recommends verification rather than trusting the name alone.

Simple User-Agent blocking therefore fails in two directions:

• a scraper can avoid the rule by changing its User-Agent to look like a browser;
• a malicious client can pretend to be a trusted crawler and hope your allow rules trust the label.

When crawler identity matters, combine identity with network, reputation and request-context signals rather than relying on one self-declared string.

Method 3: use server-side bot intelligence with BotDetect

BotDetect is designed for the case where you want more than a static User-Agent rule but do not want to move your website behind another CDN or reverse proxy.

Your backend sends request metadata to a regional BotDetect API node and receives an explainable decision that your application can use:

Bot intelligence as an API — not another CDN

The key architectural difference is control. You keep your hosting, DNS, CDN if you already use one, reverse proxy and application. BotDetect adds an intelligence layer rather than becoming the mandatory path for all of your traffic.

Explainability instead of a black-box yes/no

BotDetect returns a 0–10 score together with a reason and contributing signals. A suspicious crawler decision can therefore be explained in operational terms, for example:

{
  "score": 9,
  "reason": "fakecrawler",
  "action": "block",
  "signals": [
    "googlebot_asn_mismatch",
    "browser_headers_missing",
    "ip_reputation"
  ]
}

The exact signals available depend on the request and current scoring logic. The important point is that the administrator can see why a request was considered suspicious instead of receiving only blocked=true.

Shared reputation: benefit from abuse observed elsewhere

A single website only sees its own traffic. BotDetect also uses shared reputation built from security observations across protected services and distributed to regional detection nodes. When the same abusive source appears elsewhere, reputation can already exist as an additional signal.

Reputation is not treated as a permanent automatic punishment. It is one input combined with current request context and other signals.

Why this is stronger than robots.txt

BotDetect is not a replacement for robots.txt. It solves the enforcement and verification side of the problem:

• robots.txt tells legitimate crawler operators what policy you want;
• BotDetect helps evaluate the request that actually arrived;
• your server decides whether to allow or reject it.

For WordPress, bot protection is also traffic intelligence

On WordPress, unwanted requests can be disproportionately expensive because a request may trigger PHP, the WordPress bootstrap, plugins, database queries and theme rendering.

The BotDetect WordPress plugin is therefore useful for more than simply returning a block response. A site owner can start in monitor mode, see where automation is hitting the site, review decisions and only then enable stronger enforcement.

Typical WordPress areas worth watching include:

• wp-login.php and authentication traffic;
• XML-RPC;
• the WordPress REST API;
• admin-ajax.php;
• search, forms, comments and parameterized URLs;
• frontend content targeted by AI scrapers or fake crawlers.

Unwanted AI traffic has a real infrastructure cost

A single HTTP request may look almost free. At scale, it can include TLS handling, application execution, cache misses, database queries, search queries, API calls, logging, storage and bandwidth.

As a deliberately simple illustration, if the effective infrastructure cost of one dynamic request were only €0.0001:

These are illustrative numbers, not a universal estimate. Real request cost depends on hosting, caching, application architecture, database work, bandwidth and workload.

The point is economic rather than mathematical: a tiny per-request cost becomes meaningful when automated traffic reaches millions of requests. Rejecting unwanted automation before expensive application work can therefore save real infrastructure capacity.

robots.txt vs User-Agent blocking vs BotDetect

A practical layered policy

1. Decide what AI access you actually want. Separate model-development crawling, AI search and user-triggered retrieval.
2. Publish the policy in robots.txt. Cooperative crawler operators need a clear, standard signal.
3. Monitor real traffic. Do not assume the bots in documentation are the only automation hitting your site.
4. Do not trust crawler names by themselves. A User-Agent is self-declared and can be spoofed.
5. Enforce important restrictions server-side. If access really must be denied, your server must make the final decision.
6. Start with monitoring before aggressive blocking. Review what would be blocked and reduce false-positive risk.

Frequently asked questions