AI-search readiness checker

Check whether ChatGPT, Claude, Perplexity and Google's AI crawlers are allowed to read your site, and whether you're accidentally blocking them.

Domain or URL

Check

We fetch /robots.txt, /llms.txt, /llms-full.txt, /sitemap.xml and the homepage. Nothing is stored.

What this checks

• robots.txt rules for AI training crawlers (GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended, Bytespider, meta-externalagent, Amazonbot), AI search and answer bots (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User), and classic search (Googlebot, Bingbot). We apply RFC 9309 group matching and longest-match for the homepage path.
• llms.txt and llms-full.txt. These are optional. An Ahrefs study found that 97% of llms.txt files never got a single request (source). Having one is fine, but crawl access matters far more.
• Sitemap declared in robots.txt or at /sitemap.xml, and noindex in the homepage meta robots tag or the X-Robots-Tag header.

FAQ

Should I block GPTBot?

It depends on what you want. Blocking GPTBot stops OpenAI from training on your content. ChatGPT search uses OAI-SearchBot, so if you want to appear in ChatGPT answers, keep OAI-SearchBot allowed. The same split applies to ClaudeBot (training) versus Claude-SearchBot (search).

Which bots does the checker test?

16 user agents: AI training crawlers (GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended, Bytespider, meta-externalagent, Amazonbot), AI search and answer bots (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User) and classic search crawlers (Googlebot, Bingbot).

How is the 0-100 score calculated?

Up to 60 points for the six AI search and answer bots being allowed, 10 for Googlebot, 10 for Bingbot, 10 for no noindex on the homepage, 5 for an XML sitemap and 5 for a valid llms.txt. If the homepage cannot be loaded at all, the score is capped at 30.

Does llms.txt matter?

Only a little. llms.txt is optional and most AI crawlers do not request it yet; an Ahrefs study found that 97% of llms.txt files never received a single request. Crawl access in robots.txt, no noindex and a sitemap matter far more.

Does robots.txt actually stop AI bots?

The major vendors (OpenAI, Anthropic, Google, Perplexity, Apple) say they respect it. Some crawlers do not. robots.txt is a request, not a firewall; use your CDN or firewall bot rules if you need enforcement.