Robots.txt Validator
Check your crawl and indexing setup against our Website Standards: robots.txt rules, AI-bot policy, staging protection, with evidence for every finding
Enter your site address; we read robots.txt and your crawl rules from there
What This Tool Checks
Favicon at least 96x96 crawlable, square, multiple of 48
Robots.txt: never block CSS/JS; don't block AI crawlers by default
Noindex on thin/utility pages
Staging/preview never indexable (Coolify needs explicit header)
Server-render all indexable content (View Source)
Allow search-tier AI bots; decide training-tier per brand
Clean accessibility tree = agent-usable
Forms submittable by an agent (labels, native elements, no captcha-before-content)
Llms.txt
Every result quotes the evidence we found. Anything we cannot measure from outside is listed plainly instead of guessed.
Frequently Asked Questions
Common questions about this tool and how to use it effectively
Robots.txt is a plain-text file at your domain root that tells crawlers which parts of your site they may fetch. A correct one keeps utility pages out of crawl budget; a wrong one can block CSS and JavaScript files Google needs to render your pages, or accidentally hide your whole site from search.
Yes. A single 'Disallow: /' under 'User-agent: *' blocks everything, and it ships disabled-by-default on many staging setups, then sneaks into production at launch. That is exactly why our standards include both a robots.txt check and a separate staging-must-never-be-indexable check.
No, never. Google renders pages like a browser; blocking CSS or JS means it sees a broken page, which can hurt rankings. Our check verifies your robots rules leave rendering resources crawlable.
Treat search-tier and training-tier AI bots as separate decisions. Letting answer engines read your site can win you citations and traffic; allowing model-training crawlers is a business choice. Our standards say allow the search tier and make a deliberate call on the training tier, plus consider an llms.txt file to guide AI agents.
A convention for a plain-text file that gives AI assistants a clean, curated summary of your site and its key pages. It is young but cheap to add, and it is one of the items this tool checks for.
Disallow blocks crawlers from the paths you list; Allow carves exceptions back out of a blocked area (for example disallowing /admin/ but allowing /admin/help/). Order and specificity matter, and different crawlers resolve conflicts slightly differently, so keep rules as simple as possible.
No. Robots.txt is public and only advisory: polite crawlers obey it, bad actors read it as a map of what you tried to hide. Protect sensitive areas with authentication, and keep thin utility pages out of search with a noindex tag instead, another check this tool runs.
Still have questions?
Contact our team for helpRelated Tools You Might Like
Continue optimizing your website with these complementary tools