Try the live tool: Open robots.txt Tester — Check Crawler Access Rules →
What this tool does
Paste robots.txt and check whether a selected user-agent can crawl a specific path using a documented local matching approximation.
1. Paste the robots.txt content
Copy the real robots.txt file into the input or use the built-in sample. The tool never requests your website.
2. Select a crawler and URL path
Choose Googlebot, Bingbot, GPTBot, generic or a custom user-agent. Enter a path such as /admin/ or an absolute URL.
3. Test and review matching rules
Click Test URL path. Inspect ALLOWED or DISALLOWED, the winning rule, line number and advisory syntax findings. Clear all fields and results before another file.
Practical example
User-agent: * with Disallow: /admin/ blocks /admin/ but an Allow: /admin/public/ exception permits /admin/public/page.
Limits and responsible use
This tool approximates common Google/Bing matching behavior; it does not fully emulate every crawler, fetch remote files, verify actual crawling or guarantee deindexing.
Keep the original markup for comparison. Review facts against the real page, and use the dated Google feature catalogue or official Google Search Central documentation where relevant. A local check is not a search-ranking or rich-result guarantee.
Authoritative references
Frequently asked questions
Answers are educational. FAQ markup does not guarantee Google FAQ rich results.
Does Disallow prevent a page from appearing in Google Search?
Not necessarily. robots.txt controls crawling; it does not guarantee a page will be removed from search results.
Are wildcard rules supported?
Yes. The tester supports common * wildcard and $ end-anchor rules and longest-match precedence.
Does the tool download robots.txt from my website?
No. Paste the text locally; no remote URL is fetched.
