Robots.txt Tester
This free robots.txt tester downloads a site's live robots.txt, applies Google's matching rules and tells you whether the crawler you choose (Googlebot, Bingbot, GPTBot, ClaudeBot, PerplexityBot and more) may fetch a given URL, which rule decides it, which user-agent group applies, and which sitemaps the file lists. The file itself is shown for review.
Robots.txt Tester
Technical SEO
About this tool
One wrong line in robots.txt can hide a whole site from Google, and it happens more often than anyone admits: a Disallow: / left over from staging, a rule meant for one folder that matches a hundred, or a block on CSS and JavaScript that stops Google rendering pages. The same file now decides which AI crawlers may read your content, so it deserves a proper test rather than a quick look.
The tester fetches the robots.txt at the root of the domain, parses it the way Google does (groups by user-agent, the crawler's own group wins over the * group, the longest matching rule wins, Allow wins a tie, * and $ are wildcards) and evaluates the path you give for the crawler you choose. It names the deciding rule, shows which group was used, lists every Sitemap line and prints the file so you can read the rules in context.
Robots.txt controls crawling, not indexing: a blocked page can still appear in results if other sites link to it, and a page you want removed needs a noindex tag or removal request instead. Crawlers other than Google may interpret wildcards differently, and rules in the file only apply to the host it was fetched from, so www and non-www each need their own.
Frequently asked questions
Does blocking a page in robots.txt remove it from Google?
No. Robots.txt stops Google fetching the page, but if other pages link to it Google may still list the URL without a description. To keep a page out of results, let Google crawl it and add a noindex meta tag or header, or use the removal tool in Search Console.
How do I block AI crawlers such as GPTBot or ClaudeBot?
Add a group for each: User-agent: GPTBot followed by Disallow: /, and the same for ClaudeBot, CCBot or Google-Extended. Each crawler has its own token and its own group. Remember that blocking AI crawlers also keeps your content out of AI search answers, so decide what you want before blocking everything.
Which rule wins when Allow and Disallow both match?
Google uses the most specific rule, meaning the longest matching path. If both are the same length, Allow wins. So Disallow: /blog/ with Allow: /blog/guide allows the guide page. Wildcards count by their literal length, so a rule with * can lose to a longer plain path.
Why does Google ignore my Crawl-delay line?
Google does not support Crawl-delay and never has; it sets its own crawl rate, which you can lower in Search Console. Bing and Yandex do honour it. The tester reports only Allow, Disallow and Sitemap lines because those are what Google reads.
My robots.txt returns 404. Is that a problem?
Not on its own: a missing robots.txt means everything may be crawled. A robots.txt that returns a server error (5xx) is worse, because Google may stop crawling the site until it recovers. If you have nothing to block, serve a small file with User-agent: * and an empty Disallow, plus your sitemap line.