🔒 Browser-local: pasted data and uploaded CSV files stay on this device. These tools do not submit your input to a server, create an account, or save your source code in a database. “Check this result” transfers output within the current browser tab and removes it after loading.
Test crawler rules for a URL path
Paste your robots.txt. No website is fetched. This tester approximates common Google/Bing matching rules, not every crawler's implementation.
Disallow controls crawling, not guaranteed indexing.
Ready. Nothing has been uploaded.
Output
Matching rules and limitations
The tester reads User-agent groups, Allow and Disallow paths, wildcard (*) and end-of-path ($) anchors, plus Sitemap URLs. Longest applicable matching rule wins and Allow wins ties. It is not a live crawler, and robots.txt cannot guarantee removal from search results. Check official crawler documentation for edge cases.
Step-by-step tutorial: Read the robots.txt Tester guide and FAQs →
