AI crawler checker: robots.txt and llms.txt
The short answer: type a website's domain and see which AI crawlers its robots.txt lets in (OpenAI's GPTBot, Anthropic's ClaudeBot, Google-Extended, PerplexityBot and the rest) and whether the site has an llms.txt file.
To read the two files, dejo.run fetches /robots.txt and /llms.txt from the domain you type, and nothing else. It only fetches public websites. The rules are read in your browser.
The AI crawlers, and what each one does
| Crawler | Company | What it is for |
|---|---|---|
| GPTBot | OpenAI | Content that may be used to train OpenAI's models |
| OAI-SearchBot | OpenAI | Showing sites in ChatGPT search |
| ChatGPT-User | OpenAI | Visits a ChatGPT user asks for; robots.txt may not apply |
| ClaudeBot | Anthropic | Content that could help train Claude |
| Claude-SearchBot | Anthropic | Improving Claude's search results |
| Claude-User | Anthropic | Pages a Claude user asks for |
| Google-Extended | Whether the site may be used to train Gemini models (a product token, not a separate crawler) | |
| PerplexityBot | Perplexity | Showing sites in Perplexity results, not model training |
| Perplexity-User | Perplexity | Visits a Perplexity user asks for; generally ignores robots.txt |
Each company's own documentation, checked on 10 November 2026, is linked below.
How the rules are read
A crawler follows the group in robots.txt that names it, and only falls back to the User-agent: * group when none does. Inside a group the longest matching rule wins, and Allow wins a tie. This is the standard in RFC 9309. The checker shows whether each crawler can open the home page, and says "some paths blocked" when other parts of the site are off limits.
No llms.txt yet? Make one with the llms.txt generator. To see which crawler a request in your logs came from, paste its user agent into the user agent parser.
Sources
- OpenAI crawlers: developers.openai.com/api/docs/bots
- Anthropic crawlers: support.claude.com
- Google: Google's common crawlers
- Perplexity: docs.perplexity.ai/guides/bots
- llms.txt: llmstxt.org
FAQ
How do I block AI crawlers in robots.txt?
Add a group for each crawler you want to block, such as `User-agent: GPTBot` followed by `Disallow: /`. Each company publishes its crawler names: OpenAI uses GPTBot, OAI-SearchBot and ChatGPT-User; Anthropic uses ClaudeBot, Claude-SearchBot and Claude-User; Google uses Google-Extended for Gemini training; Perplexity uses PerplexityBot and Perplexity-User.
Does blocking GPTBot remove my site from ChatGPT?
Not from search. OpenAI says GPTBot collects content that may be used for training, while OAI-SearchBot is what shows sites in ChatGPT search. Blocking one does not block the other.
Do all AI bots obey robots.txt?
Visits a person asks for are different. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores robots.txt, because a user started the request.
What is llms.txt?
A markdown file at a site's root that gives AI tools a short summary and a list of the pages worth reading. It starts with an H1 naming the site. Make one with the llms.txt generator.