Robots.txt Tester
Fetch and parse any site's robots.txt, see its rules and sitemaps, and test whether a specific URL is allowed for search crawlers. Free, instant.
Frequently asked
Which user-agent does it test?
The global `User-agent: *` group, which is what most crawlers fall back to. Googlebot-specific groups can differ.
Does a Disallow guarantee deindexing?
No. robots.txt blocks crawling, not indexing — a blocked URL can still be indexed if linked. Use a noindex meta tag to keep a page out of the index.