Robots rules
Detects broad blocks and crawler-specific rules for leading AI user agents.
AI crawlability audit
AI Agent Website Crawlability Checker
Run a practical, no-sign-up audit of robots.txt, sitemap.xml, llms.txt, page accessibility and rules for GPTBot, ClaudeBot, Google-Extended, PerplexityBot and other AI crawlers.
What the audit checks
Discovery files
Checks sitemap.xml and llms.txt availability and content.
Readable page
Tests status, content type, title, description and extractable text.
FAQ
Does a pass guarantee AI inclusion?
No. It only verifies technical accessibility. Each provider decides whether and how to crawl or use content.
Is the target page stored?
No. The Worker fetches public URLs for the report and does not persist the response.