GPTBot, ClaudeBot and AI Crawlers: Block or Allow?
Vishal Arora
Founder, The Marketing Mavericks™ | 6 min read
For most businesses that want to be recommended by AI tools, allow the search and user-request crawlers and decide separately on training crawlers. Blocking OAI-SearchBot, Claude-SearchBot or PerplexityBot can remove you from those AI search answers. Blocking training bots like GPTBot or ClaudeBot keeps your content out of future model training but does not affect Google rankings.
On this page
Not All AI Bots Do the Same Job
AI companies now run separate crawlers for different purposes. Some collect content to train models. Others fetch pages in real time to answer a search. Others visit only when a user asks the assistant to open a link. Blocking them all treats very different jobs the same way.
| Crawler | Company | Purpose | Block it? |
|---|---|---|---|
| GPTBot | OpenAI | Model training | Your choice: no effect on search answers |
| OAI-SearchBot | OpenAI | ChatGPT search results | Allow if you want ChatGPT citations |
| ChatGPT-User | OpenAI | Fetches pages a user asks for | Allow |
| ClaudeBot | Anthropic | Model training | Your choice |
| Claude-SearchBot | Anthropic | Claude search results | Allow if you want Claude citations |
| Claude-User | Anthropic | Fetches pages a user asks for | Allow |
| PerplexityBot | Perplexity | Perplexity search index | Allow if you want Perplexity citations |
| Google-Extended | Gemini training and grounding | Your choice: does not affect Google Search | |
| CCBot | Common Crawl | Open dataset used for training | Your choice |
The Trade-Off
Most brands lose more than they protect by blocking search crawlers.
Publishers selling content have strong reasons to block training. Service businesses, agencies, ecommerce brands and clinics usually want the opposite: to be named when buyers ask AI tools for recommendations.
Robots.txt Rules to Copy
This setup allows AI search and user fetches while opting out of training. Adjust it to your own policy.
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
If you want maximum AI visibility, remove the training blocks too. Some engines lean on training data as well as live search when they describe brands.
Check your current AI crawler setup.
How to Confirm AI Bots Are Reaching You
Open your server access logs or your host analytics and filter by user agent for OAI-SearchBot, Claude-SearchBot, PerplexityBot and ChatGPT-User. Regular visits mean the door is open. No visits for weeks usually means a block somewhere between robots.txt, your firewall and your CDN.
Then test from the user side. Ask ChatGPT or Claude to open and summarise one of your pages. If the assistant reports it cannot access the page, check the response your server sends to that bot.
- Logs show AI search bots visiting: access is working
- Robots.txt allows bots but logs are empty: check the firewall and CDN
- Assistants cannot open your pages: check for bot challenges and error codes
Check Your Firewall Too
Many CDNs and security plugins now block AI bots by default. A clean robots.txt means nothing if the firewall returns errors to the crawler. Check bot settings in your CDN and security plugin, then confirm visits in server logs. Crawler access is step one of AI search visibility, followed by llms.txt and content work. We set this up as part of our AI SEO services.
Frequently Asked Questions
Does blocking GPTBot remove me from ChatGPT?
Not from ChatGPT search. GPTBot is for training. ChatGPT search uses OAI-SearchBot, so allow that one if you want to appear in its answers.
Will blocking Google-Extended hurt my Google rankings?
No. Google-Extended only controls use of your content for Gemini models and grounding. Googlebot and Search are unaffected.
Do AI crawlers respect robots.txt?
The major ones from OpenAI, Anthropic, Google and Perplexity say they do. Some smaller or unidentified bots do not, which is where firewall rules help.
Get Your AI Access Set Up Right
We configure crawler access, schema and content so AI tools can find and recommend you, for businesses across the USA, UK, Australia, Canada, the UAE and India.
Talk To Our Team →
