For AI agents: the index of this site is https://dejo.run/llms.txt, and every page is also served as markdown (add .md to its address, or send Accept: text/markdown).

dejo

AI crawler checker: robots.txt and llms.txt

The short answer: type a website's domain and see which AI crawlers its robots.txt lets in (OpenAI's GPTBot, Anthropic's ClaudeBot, Google-Extended, PerplexityBot and the rest) and whether the site has an llms.txt file.

To read the two files, dejo.run fetches /robots.txt and /llms.txt from the domain you type, and nothing else. It only fetches public websites. The rules are read in your browser.

The AI crawlers, and what each one does

CrawlerCompanyWhat it is for
GPTBotOpenAIContent that may be used to train OpenAI's models
OAI-SearchBotOpenAIShowing sites in ChatGPT search
ChatGPT-UserOpenAIVisits a ChatGPT user asks for; robots.txt may not apply
ClaudeBotAnthropicContent that could help train Claude
Claude-SearchBotAnthropicImproving Claude's search results
Claude-UserAnthropicPages a Claude user asks for
Google-ExtendedGoogleWhether the site may be used to train Gemini models (a product token, not a separate crawler)
PerplexityBotPerplexityShowing sites in Perplexity results, not model training
Perplexity-UserPerplexityVisits a Perplexity user asks for; generally ignores robots.txt

Each company's own documentation, checked on 10 November 2026, is linked below.

How the rules are read

A crawler follows the group in robots.txt that names it, and only falls back to the User-agent: * group when none does. Inside a group the longest matching rule wins, and Allow wins a tie. This is the standard in RFC 9309. The checker shows whether each crawler can open the home page, and says "some paths blocked" when other parts of the site are off limits.

No llms.txt yet? Make one with the llms.txt generator. To see which crawler a request in your logs came from, paste its user agent into the user agent parser.

Sources

FAQ

How do I block AI crawlers in robots.txt?

Add a group for each crawler you want to block, such as `User-agent: GPTBot` followed by `Disallow: /`. Each company publishes its crawler names: OpenAI uses GPTBot, OAI-SearchBot and ChatGPT-User; Anthropic uses ClaudeBot, Claude-SearchBot and Claude-User; Google uses Google-Extended for Gemini training; Perplexity uses PerplexityBot and Perplexity-User.

Does blocking GPTBot remove my site from ChatGPT?

Not from search. OpenAI says GPTBot collects content that may be used for training, while OAI-SearchBot is what shows sites in ChatGPT search. Blocking one does not block the other.

Do all AI bots obey robots.txt?

Visits a person asks for are different. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores robots.txt, because a user started the request.

What is llms.txt?

A markdown file at a site's root that gives AI tools a short summary and a list of the pages worth reading. It starts with an H1 naming the site. Make one with the llms.txt generator.