AI Crawler Robots.txt Checker
Check if your website blocks AI crawlers like GPTBot, ClaudeBot, and PerplexityBot. Generate robots.txt rules to control which AI systems can access your content.
Check Your Robots.txt
Copy content from yoursite.com/robots.txt and paste here
Generate Robots.txt Rules
How It Works
RFC 9309 compliance: This tool implements the official robots.txt standard (RFC 9309). It parses User-agent groups, merges all groups matching the target agent, applies longest-match rules for Allow/Disallow directives, supports * and $ wildcards, and correctly handles precedence where Allow overrides Disallow on ties. Empty Disallow directives are ignored.
Checking methodology: Paste your robots.txt content. The parser normalizes line endings (CRLF), strips inline # comments, identifies user-agent groups, and merges all groups matching each AI crawler. Status: "blocked site-wide" (Disallow: / or /* with no Allow overrides), "partially blocked" (some paths blocked), or "allowed" (no blocking or Allow overrides Disallow).
Generator methodology: The generator creates properly formatted robots.txt rules. Each selected crawler gets its own User-agent block with a Disallow directive. The "block all" option uses "User-agent: *" which applies to every crawler unless overridden by a more specific rule.
Compliance caveat: Robots.txt is a voluntary standard. Major AI companies (OpenAI, Anthropic, Google, Perplexity) state they respect robots.txt, but it's not legally binding. Some crawlers may not honor these rules. This is a first line of defense, not a guarantee.
AI Crawler Reference
GPTBot
OpenAI training crawler
User-agent: GPTBotOAI-SearchBot
OpenAI search
User-agent: OAI-SearchBotChatGPT-User
ChatGPT web browsing
User-agent: ChatGPT-UserClaudeBot
Anthropic training crawler
User-agent: ClaudeBotClaude-User
Claude user requests
User-agent: Claude-UserClaude-SearchBot
Claude search indexing
User-agent: Claude-SearchBotPerplexityBot
Perplexity AI crawler
User-agent: PerplexityBotPerplexity-User
Perplexity user requests
User-agent: Perplexity-UserFrequently Asked Questions
How do I block AI crawlers from my website?
Add specific User-agent rules to your robots.txt file. For example, add "User-agent: GPTBot" followed by "Disallow: /" to block OpenAI crawlers. Each AI company respects different user agents, so you need separate rules for each crawler you want to block.
Do all AI companies respect robots.txt?
Major AI companies (OpenAI, Anthropic, Google, Perplexity) state they respect robots.txt, but compliance is voluntary and not legally required. Some smaller or research crawlers may not honor robots.txt. The effectiveness depends on each crawler operator following RFC 9309 standard.
What is the difference between GPTBot and ChatGPT-User?
GPTBot is OpenAI's crawler for training data collection. ChatGPT-User is the user agent when ChatGPT's browsing feature accesses websites on behalf of users. You may want to block GPTBot (training) but allow ChatGPT-User (user requests).
Can I block AI crawlers but still allow Google Search?
Yes. Block Google-Extended (AI training) while allowing Googlebot (search indexing). Add "User-agent: Google-Extended" followed by "Disallow: /" to block AI training without affecting search rankings.
Related Tools
Control your content, intelligently
GBot helps you create content while giving you control over how AI systems use it. Try it free.
Try GBot Free