Should You Block GPTBot? A Decision Framework for SaaS Sites and Publishers

Should you block GPTBot?

Block GPTBot if keeping your pages out of OpenAI's model training matters more to you than anything that training could do for you, and leave it open if it does not. That is a decision about one crawler, and the answer can differ from the one you give OpenAI's search crawler or Google's training token.

• GPTBot is OpenAI's training crawler. OAI-SearchBot is a separate crawler for ChatGPT search features, so you can block one and allow the other.
• Google-Extended controls Gemini training and grounding use. Google says it does not affect inclusion in Search or ranking.
• robots.txt is a request. Google's documentation says a crawler decides whether to obey it, so a firewall rule is the stronger tool.
• In our reading, publishers whose product is the content have the strongest case for blocking. Most SaaS marketing sites have more to gain from being read.
• This page's own contribution is a stated-criteria matrix by business model and a worked robots.txt example. It contains no survey data of its own.

What does GPTBot actually do?

GPTBot is OpenAI's web crawler, and OpenAI's documentation describes it as collecting content to train generative AI foundation models. It announces itself with the robots.txt token GPTBot and a user-agent string that points to openai.com/gptbot.

Search Logistics prints the string with version 1.0, while Passionfruit prints version 1.1. The two guides disagree on the number, which is a good reason to match on the token and not on a full string.

Both guides say GPTBot reads public pages only. Search Logistics reports that OpenAI designed it to filter out paywalled content and personal information, and Passionfruit says it does not try to get past logins. Those are descriptions of intent, not a guarantee you can build a paywall on.

Scale is the other open question. Vercel says OpenAI generates hundreds of millions of monthly requests and calls it the most active AI crawler it sees, but gives no date or method in the text we retrieved. Your own logs are the only reliable number for your site.

Which AI crawlers feed search and which feed training?

Each major AI company runs separate crawlers for training, for search and for visits a user requests, so one blanket rule is the wrong tool. The table lists the ones this decision touches, with the purpose each company documents and what disallowing it does.

How the table was put together: we took each company's own documentation as the only source for what its crawler does, and left out any bot whose purpose we could not attribute to its operator. That excludes lesser-known crawlers, so it is a map of the major ones, not an inventory of the web.

One of the ranking guides lists only GPTBot, ChatGPT-User and ChatGPT-Plugins as OpenAI's bots and does not mention OAI-SearchBot. OpenAI's own crawler overview is the authority, so read it before you copy a bot list from anyone else, including this page.

What are Google-Extended and ClaudeBot, and do they change the answer?

Google-Extended and ClaudeBot are the training-side controls for Google and Anthropic, and the decision logic is the same as for GPTBot. Google-Extended is the less obvious of the two, because it is not a crawler you will find in a log.

Google's common crawlers page says Google-Extended has no separate user-agent string: crawling is done with existing Google user agents, and the token is used only as a control. It governs whether content Google crawls may be used to train future Gemini models and to ground answers in Gemini Apps and Grounding with Google Search on Vertex AI. The same page says it does not impact a site's inclusion in Google Search and is not a ranking signal.

That answers the most common worry. Disallowing Google-Extended does not remove you from Search, and Googlebot keeps its own group of rules. Google's page also lists GoogleOther, a generic crawler that affects no specific product.

ClaudeBot is Anthropic's counterpart to GPTBot. Anthropic's documentation describes it as collecting web content that may contribute to model training, with Claude-SearchBot and Claude-User as separate crawlers for search and user requests. A ClaudeBot rule therefore does not decide whether Claude can find or cite you.

Passionfruit claims that GPTBot and ClaudeBot can strain servers, with surges of up to 30 TB, but the extracted text gives no method. Treat that as an unverified claim from one secondary source and look at your own bandwidth before deciding.

What does blocking GPTBot change, and what does it not?

Blocking GPTBot changes what a compliant OpenAI training crawler may fetch from your site in the future, and almost nothing else. The research supports four limits on what a block does.

1. It is a request. Google's robots.txt introduction says the instructions cannot enforce crawler behaviour and that respectable crawlers obey while others might not. Search Logistics makes the same point about relying on GPTBot to follow the rules.
2. It applies to the future. Anthropic describes a ClaudeBot block as excluding future content from training. No source in the research says a block removes content a crawler has already collected.
3. It stays inside its lane. A GPTBot rule does not touch OAI-SearchBot, Googlebot or Google-Extended, because each token has its own group.
4. It does not hide pages. Google says a page disallowed in robots.txt can still be indexed if other sites link to it, and that noindex or a password is the method for keeping a page out.

The cost side is mostly deferred. One ranking guide argues that blocking AI crawlers cuts you out of featured snippets and AI summaries on Google, but for the tokens in this decision that does not hold: GPTBot belongs to OpenAI, and Google says Google-Extended does not affect Search.

Why do some site owners block GPTBot?

Owners block GPTBot mainly to keep their content out of model training and to protect the revenue the content earns. The guides in this research list four reasons.

• Unpaid use of the work. Passionfruit describes the arrangement as unfair when a model trained on your pages answers questions without linking back.
• Server and security load. The same guide says another automated visitor adds complexity to monitoring and firewall rules, and Vercel notes that even legitimate AI crawlers can become unwanted if they exceed fair use.
• Legal uncertainty. Passionfruit says the fair-use argument for AI training remains unsettled and that no clear precedent exists, which is why it suggests caution for finance, healthcare and law.
• Principle. Some owners block it as a statement about AI use, whatever the traffic effect.

Passionfruit reports that more than 3.5% of websites block GPTBot, and that The New York Times, CNN and more than 30 of the Top 100 websites have done so. The extracted text gives no date or method, so read the figure as a rough marker of how common the choice is, not a benchmark to match.

Why do others allow GPTBot?

Owners allow GPTBot when they would rather be read and described accurately than withheld. The case rests on representation more than on traffic.

Passionfruit argues that if GPTBot cannot read your site, a model must describe your brand from second-hand sources that may be outdated or wrong. It cites about 800 million weekly ChatGPT users and says allowing the crawler gives you more say in how the brand is described. That is the guide's argument, and it is unproven here.

The same guide is candid about the traffic: AI search drives under 1% of traffic, though it says earlier data showed those visitors converting 23 times better than organic search visitors. It does not give the source for that comparison, so we do not rely on it.

There is also a logical gap in the pro-allow case. Under OpenAI's own descriptions, being cited in ChatGPT search answers depends on OAI-SearchBot, not GPTBot. If visibility is your goal, the search crawler is the one to keep open, and GPTBot becomes a separate question about training.

Which choice fits your business model?

The choice follows what your content is worth to you, so a publisher and a SaaS company should usually decide differently. The matrix below applies four criteria to eight common site types.

1. Is the content itself the product, or does it support a product you sell?
2. Does a buyer find you through AI answers, which depends on search crawlers and not training crawlers?
3. Can you accept your pages being used for training without a licence?
4. Can you enforce a block at the server, or only ask in robots.txt?

Two notes on reading it. First, most real sites are several of these at once, which is why the marketing site, the docs host and the app should each get their own decision. Second, a lean is not a verdict: if you cannot say what blocking would protect, the default that costs least is to leave a training crawler open and revisit it.

The research has no measured result showing that either choice raised or lowered revenue for any kind of site. The matrix is a way to make your own reasoning explicit, and the sources behind each row are guides and documentation, not outcome data.

How do robots.txt groups decide which rule applies to a crawler?

A crawler obeys only the single group whose user-agent line most specifically matches it, and ignores every other group. Google's robots.txt specification page states this, and it is the most common way a partial block fails.

Take this file, which we wrote for illustration. The owner wants everything under /private/ off limits to all crawlers, and wants GPTBot kept out of /drafts/ and Google-Extended out of the whole site.

• Group 1: User-agent: * then Disallow: /private/
• Group 2: User-agent: GPTBot then Disallow: /drafts/
• Group 3: User-agent: Google-Extended then Disallow: /

GPTBot has its own group, so group 1 is ignored for it and /private/ is open to GPTBot, which is the opposite of what the owner intended. The fix is to repeat Disallow: /private/ inside group 2.

Three more rules from the same page matter here. The order of groups in the file is irrelevant, the file applies only to the host, protocol and port it is served from, and Google caches a robots.txt for up to 24 hours. A rule on example.com does not cover docs.example.com, and an edit will not take effect for Google instantly.

The caveat is scope. This is Google's documented reading of the Robots Exclusion Protocol, and Google itself notes that each crawler might interpret rules differently, so confirm OpenAI's and Anthropic's behaviour on their own pages.

When is robots.txt not enough, and how do you check who is crawling you?

robots.txt is not enough when the crawler may ignore it, when the page is private, or when a platform setting overrides what you wrote. In those cases you enforce the block at the server and then check the logs.

Two enforcement routes appear in the research. Search Logistics shows an .htaccess rule that returns 403 Forbidden to any request with GPTBot in the user agent, and calls it a surer block than a request. Vercel describes a firewall rule on the user agent that can deny, challenge or rate limit GPTBot, applies globally in about 300 milliseconds, and stops blocked requests reaching your functions or pages, so you are not charged for them.

Both depend on the crawler identifying itself honestly. A rule keyed to a user agent stops a bot that says who it is, which is an inference from how the rule works, not a claim any source makes about particular bots.

Also check what your site serves today. Cloudflare documents a managed robots.txt setting, Netlify has a guide to blocking AI bots, and Vercel offers a firewall template for AI bots. Threads in the research ask whether a CDN setting changed their rules without notice, so open yoursite.com/robots.txt, not the file in your repository.

To see who is visiting, search your logs or firewall traffic for the GPTBot and ClaudeBot tokens, and record the request count for a week before deciding. Kanso's GEO Agent checks how readable your site is to AI systems and estimates how citable it looks; that is a model estimate, and Kanso does not query ChatGPT, Gemini, Claude or Perplexity.

For the wider picture of what Google does and does not require for AI features, read GEO Without the Hype. The weekly growth checklist carries a shorter crawler table, the llms.txt article covers the file Google ignores, and the Generative AI performance report guide shows where Google reports AI feature impressions.

What is documented and what is our analysis?

The documented points come from Google, OpenAI, Anthropic and the guides named in each sentence, and the matrix and its leans are Kanso's own reading. The lists keep them apart.

• Documented: Google-Extended has no separate user agent, controls Gemini training and grounding use, and does not affect Search inclusion or ranking (Google, retrieved 27 October 2026).
• Documented: a robots.txt crawler follows one most specific group, groups can be in any order, the file is valid per host, and Google may cache it for up to 24 hours (Google specification page).
• Documented: robots.txt cannot enforce crawler behaviour and does not keep a page out of Google (Google robots.txt introduction).
• Documented by vendors: GPTBot trains, OAI-SearchBot supports ChatGPT search, ClaudeBot trains and Claude-SearchBot supports Claude search (OpenAI and Anthropic documentation, recorded 13 October 2026).
• Reported by guides, not verified here: the 3.5% block rate, the 30-of-Top-100 figure, the 30 TB surge claim, the 23-times conversion claim, and Vercel's request volume.

• Analysis: the four criteria and every lean in the decision matrix.
• Analysis: the point that being cited in ChatGPT search depends on the search crawler and not GPTBot, which follows from the vendor descriptions.
• Analysis: the worked robots.txt example, written for illustration and applied under Google's documented rules.
• Analysis: the inference that a user-agent rule only stops crawlers that identify themselves.

What this page does not have: a measurement of how many sites block each crawler, any test of whether blocking changed traffic or citations, and a source for what happens to content collected before a block. If you need those, ask the vendors directly and log your own results.

GPTBot and robots.txt: frequently asked questions

Sources

• Google's common crawlers, Google Crawling Infrastructure
• Introduction to robots.txt, Google Search Central
• How Google interprets the robots.txt specification, Google Crawling Infrastructure
• Overview of OpenAI crawlers, OpenAI
• Does Anthropic crawl data from the web, and how can site owners block the crawler?, Anthropic
• How to block bots from OpenAI GPTBot, Vercel Knowledge Base
• What Is The GPTBot And How Do You Block It?, Search Logistics
• What Is GPTBot? Should You Block OpenAI's Web Crawler? | 2025 Guide, Passionfruit