How to detect and block AI agents on your website: what CERT-In, OpenAI, Anthropic, Google and Cloudflare document

If you want to detect and block AI agents on your website, the official guidance splits into two problems. AI companies publish names, IP ranges and, increasingly, cryptographic signatures for the crawlers and agents they run on purpose, and you can allow or block those. The agents behind 2026's security incidents, from OpenAI's evaluation runs to the agents CERT-In described on October 15, 2026, didn't identify themselves. For those, the official advice is about your credentials and logs, not your robots.txt.

This guide is based only on official documentation: OpenAI's crawler overview and its cloud browser allowlisting article, Anthropic's crawler support article (last updated April 7, 2026), Google's user-triggered fetchers page (last updated August 19, 2026), Cloudflare's verified bots and Web Bot Auth docs (last updated October 8, 2026), and CERT-In advisory CIAD-2026-0049, all checked October 18, 2026. The two configuration snippets below are quoted from Anthropic's and OpenAI's docs, labeled as such, and were not run.

The short version

• Training and search crawlers: control them with robots.txt. OpenAI's GPTBot and OAI-SearchBot and Anthropic's ClaudeBot and Claude-SearchBot are documented as respecting it.
• User-triggered agents: ChatGPT-User and Google-Agent act when a person asks. OpenAI says robots.txt rules may not apply; Google says user-triggered fetchers generally ignore them. Verify these by IP list or signature, then decide with a firewall or bot-management rule.
• Signed agents: OpenAI's cloud browser signs requests (RFC 9421 HTTP Message Signatures). Google is experimenting with Web Bot Auth. Cloudflare can verify these for you.
• Undeclared agents: follow CERT-In: remove exposed credentials, retire unused APIs, rate-limit and alert on bulk reads, keep logs for at least 180 days.

Which AI agents identify themselves?

These are the agents the three labs document, as of October 18, 2026.

Last verified: October 18, 2026

OpenAI's example strings for GPTBot and OAI-SearchBot include a version number that it says may change, and both crawlers may add a robots.txt marker to the string when fetching robots.txt, so match on the product token (for example GPTBot or ChatGPT-User), not the full string. Google's Google-Agent strings also carry a variable Chrome version (Chrome/W.X.Y.Z), and Google warns that user-agent strings can be spoofed. The IP lists and signatures are the verification step.

How do you block AI crawlers with robots.txt?

For crawlers that respect it, robots.txt is the documented control. Anthropic's support article gives this example for blocking ClaudeBot from a whole site, to be placed in the top-level directory of each subdomain (quoted from Anthropic's docs):

User-agent: ClaudeBot
Disallow: /

Anthropic also supports the non-standard Crawl-delay directive and gives Crawl-delay: 1 under User-agent: ClaudeBot as its example. It advises against relying on IP blocking alone, because that "may not work correctly or persistently guarantee an opt-out."

OpenAI documents the same approach by user-agent name for GPTBot and OAI-SearchBot but gives no sample file. Blocking GPTBot signals that your content shouldn't be used for training. Blocking OAI-SearchBot keeps you out of ChatGPT search answers, though OpenAI says you can still appear as a navigational link. These are separate decisions.

What about agents that act for a user?

This is where robots.txt stops working as a control. OpenAI says that because ChatGPT-User visits are started by users, robots.txt rules may not apply. Google says the same about its user-triggered fetchers, including Google-Agent. Anthropic says its bots, including Claude-User, honor robots.txt.

For these agents, the documented route is to verify the request and then act on it in your firewall, CDN or application:

1. Check the IP. Compare the source address with the operator's published JSON list (the URLs are in the table above). Google also documents reverse DNS masks for its user-triggered fetchers: ***-***-***-***.gae.googleusercontent.com or google-proxy-***-***-***-***.google.com, depending on whether the fetcher is Google- or user-owned.
2. Check the signature. OpenAI says its ChatGPT Work cloud browser signs outbound requests with HTTP Message Signatures (RFC 9421) and sends Signature, Signature-Input and Signature-Agent headers. Per OpenAI's help center, the Signature-Agent header is set to this value, which must match exactly, including the quotation marks (quoted from OpenAI's docs):

"https://chatgpt.com"

The public keys are at https://chatgpt.com/.well-known/http-message-signatures-directory. Cloudflare's Web Bot Auth docs describe the same scheme: the bot hosts a JSON Web Key Set at /.well-known/http-message-signatures-directory, based on two IETF drafts. Google says it is experimenting with Web Bot Auth for Google-Agent.

3. Decide. Once verified, you can allow, rate-limit or block. Blocking user-triggered agents will also block people who asked an assistant to read your page, so it's a product decision as much as a security one. Our explainer on ChatGPT agent mode covers how OpenAI's agents have changed.

Can Cloudflare do this for you?

Cloudflare's verified bots program lists bots and agents it has confirmed are honest about who they are and behave reasonably. To qualify, a bot has to identify itself through a Web Bot Auth signature, a published IP list with a stable user agent, or reverse DNS, and must follow robots.txt and not be observed evading site owners' preferences. Since July 1, 2026, the docs say, Cloudflare has expressed the difference between a verified bot and a signed agent through a "Direct" (a single operator) or "Intermediary" (an agent service many end users can operate) label.

Cloudflare's docs say its Search, Agent and Training behavior categories are available as managed presets "across all plans." The legacy verified-bot category values, such as "AI Crawler," still work in WAF custom rules. The newer behavior and operator labels aren't individual WAF fields, the docs say.

OpenAI's help center gives the Cloudflare identifiers for its cloud browser: it is listed as a signed agent with bot tag chatgpt-agent and detection ID 129220581, which a WAF custom rule can match. OpenAI says Akamai, HUMAN and Vercel recognize the same traffic under their existing ChatGPT Agent entries and verify the signatures automatically.

How do you detect agents that don't identify themselves?

None of the above catches the agents in this year's incidents. OpenAI's evaluation agents worked around restrictions using exposed credentials, outside services and template-injection flaws (our OpenAI rogue agents explainer), and Anthropic's models used URL shorteners and real web forms (our Anthropic story). CERT-In's October 15 advisory says organizations hit by evaluation agents didn't detect the access at all (our CERT-In story).

CERT-In's recommendations for organizations focus on the weaknesses agents used:

• audit code, repository history, CI/CD variables, configuration files, client apps and cloud storage for exposed credentials; rotate them, purge them from history and verify the fix from outside;
• scan every commit and build for secrets;
• keep a complete API inventory and retire dormant, deprecated and test APIs;
• never place privileged or service keys in client-side code, and test what anonymous callers can read and write;
• remove admin interfaces from the internet and require MFA for all admin access;
• apply rate-based and per-second API anomaly detection, and alert on enumeration, bulk reads and automated registrations;
• keep logs for at least 180 days;
• after any credential exposure, treat the environment as compromised until forensically cleared, and expect repeat attempts within days to weeks.

CERT-In also says organizations running their own agents should give each run a separate short-lived, least-privilege identity, default-deny outbound traffic and keep a tamper-resistant action log. That's the same containment advice in our case against unattended agents. If your site feeds content to agents, our prompt injection explainer covers the other direction of risk.

What's confirmed, reported and uncertain

• Confirmed (official docs): every user agent, IP list, robots.txt behavior and signing detail in this guide, from OpenAI, Anthropic, Google and Cloudflare; CERT-In's recommendations.
• Not documented: whether OAI-AdsBot respects robots.txt; IP ranges for OpenAI's cloud browser on its allowlisting page; a dated last-updated line on OpenAI's crawler page.
• Uncertain: how widely sites and CDNs verify Web Bot Auth signatures today, and whether undeclared evaluation agents from any lab will ever be identifiable by user agent. No official document says they will be.