01
AI crawler rules
robots.txt allow/deny for major training, search, and live-answer user agents.
Free tool for marketing and SEO teams
Check your AI crawlability
See whether ChatGPT, Claude, Perplexity, and other AI crawlers can access your site, and get clear fixes when they cannot.
How it works
How to check AI crawlability
See which AI crawlers can access a site in a few seconds. Built for marketing and SEO teams.
Step 1
Enter your domain
Paste a domain like example.com. A full URL works too.
Step 2
Review crawler access
We check robots.txt, sitemap.xml, llms.txt, and major AI bot rules.
Step 3
Apply the fixes
Copy the recommended robots.txt updates for any blocked crawlers.
Coverage
What we scan
Four signals from the site root, plus allow/block status for 14 major AI crawler user agents.
02
Homepage fetch
Whether the site responds over HTTPS so crawlers can load a page at all.
03
sitemap.xml
A published sitemap so bots can discover URLs beyond the homepage.
04
llms.txt
Optional map for coding agents. Useful, but never treated as a hard failure.
AI crawlers we check
- GPTBot · OpenAI · Training
- OAI-SearchBot · OpenAI · ChatGPT search
- ChatGPT-User · OpenAI · Live answers
- ClaudeBot · Anthropic · Training
- Claude-SearchBot · Anthropic · Claude search
- Claude-User · Anthropic · Live answers
- PerplexityBot · Perplexity · Answer engine
- Google-Extended · Google · Gemini training
- Applebot-Extended · Apple · Apple Intelligence
- Amazonbot · Amazon · Alexa / search
- meta-externalagent · Meta · AI training
- Bytespider · ByteDance · Training
- CCBot · Common Crawl · Open datasets
- cohere-ai · Cohere · Training
Score guide
What the score means
AI crawlability is 0 to 100. Higher means more AI crawlers can fetch the site and discovery files are in place. It is not a ChatGPT mention score.
0-29
Restricted
Major AI crawlers are blocked, or the site could not be reached.
30-49
Limited access
Some bots can crawl, but key training or search crawlers are still denied.
50-69
Mostly accessible
Most AI bots can fetch the site. Tighten robots.txt and discovery files next.
70-84
Crawl ready
AI crawlers can access the site. A sitemap or llms.txt would improve discovery.
85-100
Fully open
robots.txt, major bots, and discovery signals look open to AI systems.
Why this tool
Crawl access first, then readiness
If GPTBot or ClaudeBot cannot fetch the site, citation tools will not help. Fix crawl access here, then check brand identity with the AI readiness checker.
What it measures
Whether major AI crawlers can fetch the site root via robots.txt, and whether homepage, sitemap.xml, and llms.txt respond.
What you leave with
A 0–100 crawlability score, allow/block status per bot, and copy-paste robots.txt or llms.txt fixes when something is blocked.
What it doesn’t
It is not a ChatGPT mention score, not Domain Rating, and not a promise that allowing a bot will make models cite your brand.
Related: AI readiness checker · Domain Rating checker
FAQ
Common questions about AI crawlability
Short answers for the questions people actually type.
AI crawlability means whether AI systems can fetch your pages. That starts with robots.txt rules for bots like GPTBot, ClaudeBot, and PerplexityBot, plus discovery files like sitemap.xml and llms.txt.
We check major crawlers from OpenAI, Anthropic, Perplexity, Google, Apple, Amazon, Meta, Common Crawl, Cohere, and ByteDance, including training, search, and live-answer user agents such as GPTBot, ClaudeBot, PerplexityBot, and Google-Extended.
No. This tool only answers whether AI bots can crawl the site. The AI readiness checker also looks at brand identity, schema, and citation signals.
Not automatically. Crawl access is the first requirement. Citation still depends on whether models trust and retrieve your content. Anny tracks whether ChatGPT and other models actually mention you.
Most bots treat a missing robots.txt as allow-all. We still flag that as a gap, because crawlers lack an explicit map and you cannot signal AI-specific Allow rules.
Yes. No account, no credit card. Paste a domain and see which AI crawlers are allowed or blocked.
Next step: Run the AI readiness checker
Crawl access is only the first step
Get a managed plan to show up in ChatGPT, Perplexity, and AI Overviews
Anny's GEO team builds the strategy, runs the work, and keeps auditing performance, so your brand doesn't just get crawled, it gets cited.