# AI crawler checker: GPTBot, ClaudeBot and robots.txt

> Type a domain and see which AI crawlers its robots.txt allows (GPTBot, OAI-SearchBot, ClaudeBot, Google-Extended, PerplexityBot and more) and whether it has an llms.txt file. Free robots.txt checker for AI.

Canonical: https://dejo.run/tools/ai-crawler-checker

> For AI agents: the index of this site is https://dejo.run/llms.txt, and every page is also served as markdown (add .md to its address, or send Accept: text/markdown).

**The short answer:** type a website's domain and see which AI crawlers its robots.txt lets in (OpenAI's GPTBot, Anthropic's ClaudeBot, Google-Extended, PerplexityBot and the rest) and whether the site has an llms.txt file.


To read the two files, dejo.run fetches `/robots.txt` and `/llms.txt` from the domain you type, and nothing else. It only fetches public websites. The rules are read in your browser.

## The AI crawlers, and what each one does

| Crawler | Company | What it is for |
|---|---|---|
| GPTBot | OpenAI | Content that may be used to train OpenAI's models |
| OAI-SearchBot | OpenAI | Showing sites in ChatGPT search |
| ChatGPT-User | OpenAI | Visits a ChatGPT user asks for; robots.txt may not apply |
| ClaudeBot | Anthropic | Content that could help train Claude |
| Claude-SearchBot | Anthropic | Improving Claude's search results |
| Claude-User | Anthropic | Pages a Claude user asks for |
| Google-Extended | Google | Whether the site may be used to train Gemini models (a product token, not a separate crawler) |
| PerplexityBot | Perplexity | Showing sites in Perplexity results, not model training |
| Perplexity-User | Perplexity | Visits a Perplexity user asks for; generally ignores robots.txt |

Each company's own documentation, checked on 10 November 2026, is linked below.

## How the rules are read

A crawler follows the group in robots.txt that names it, and only falls back to the `User-agent: *` group when none does. Inside a group the longest matching rule wins, and `Allow` wins a tie. This is the standard in RFC 9309. The checker shows whether each crawler can open the home page, and says "some paths blocked" when other parts of the site are off limits.

No llms.txt yet? Make one with the [llms.txt generator](https://dejo.run/tools/llms-txt-generator). To see which crawler a request in your logs came from, paste its user agent into the [user agent parser](https://dejo.run/tools/user-agent-parser).

## Sources

- OpenAI crawlers: [developers.openai.com/api/docs/bots](https://developers.openai.com/api/docs/bots)
- Anthropic crawlers: [support.claude.com](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)
- Google: [Google's common crawlers](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers)
- Perplexity: [docs.perplexity.ai/guides/bots](https://docs.perplexity.ai/guides/bots)
- llms.txt: [llmstxt.org](https://llmstxt.org/)

## FAQ

## FAQ

### How do I block AI crawlers in robots.txt?

Add a group for each crawler you want to block, such as `User-agent: GPTBot` followed by `Disallow: /`. Each company publishes its crawler names: OpenAI uses GPTBot, OAI-SearchBot and ChatGPT-User; Anthropic uses ClaudeBot, Claude-SearchBot and Claude-User; Google uses Google-Extended for Gemini training; Perplexity uses PerplexityBot and Perplexity-User.

### Does blocking GPTBot remove my site from ChatGPT?

Not from search. OpenAI says GPTBot collects content that may be used for training, while OAI-SearchBot is what shows sites in ChatGPT search. Blocking one does not block the other.

### Do all AI bots obey robots.txt?

Visits a person asks for are different. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores robots.txt, because a user started the request.

### What is llms.txt?

A markdown file at a site's root that gives AI tools a short summary and a list of the pages worth reading. It starts with an H1 naming the site. Make one with the llms.txt generator.

## Related

- [Dejo: browser automation MCP that remembers websites](https://dejo.run/)
- [Dejo MCP server setup: install in a minute](https://dejo.run/docs)
