Tools / SEO & Web
Robots.txt Validator
Fetch or paste robots.txt, check groups, rules and sitemaps
The lookup is made by our server against public DNS; only the domain/IP you enter is used.
About this tool
Fetch a site's robots.txt or paste your own and check its syntax, user-agent groups, allow and disallow rules and sitemap lines. It warns when the whole site is blocked from crawling.