Local processing

DEVELOPER TOOLS

robots.txt tester online

Paste your robots.txt, add the URLs you want to check and pick a crawler. You get a verdict per URL — blocked or allowed — plus the exact rule and line number that decided it. One stray Disallow can pull a whole section out of the index for weeks before anyone notices.

Use this tool

LOCAL PROCESSING

Everything happens in your browser

The file and text never leave your device.

Paste your robots.txt above. Without URLs, only the syntax is checked.

What the parser reports

Directives are grouped by User-agent, with the Allow and Disallow rules that belong to each one plus any declared Sitemap lines. That makes it obvious when a rule written for one specific bot is actually applying to every crawler, which is the costliest and most common mistake.

The mistake that costs the most traffic

A Disallow: / under User-agent: * blocks crawling of the entire site. It usually arrives by being promoted from a staging environment, where that exact rule is correct. Nothing alerts you: indexing simply decays until pages start dropping out of results.

Blocking a crawl is not deindexing

Robots.txt controls whether a crawler may fetch a URL, not whether it may list it. A page blocked by Disallow can still appear in Google if external links point to it, because the engine knows the URL exists even though it cannot read the content. Removing a page from the index requires a noindex tag, and that means the crawler must be allowed to read the page in the first place.

FREQUENTLY ASKED QUESTIONS

Is my file uploaded anywhere?

No. The text you paste is parsed by JavaScript inside your own browser and never leaves your machine.

Does it test whether a specific URL is blocked?

Yes, that is what it is for. Add one URL per line, choose a user-agent, and each one comes back as blocked or allowed with the rule and line that decided it. Matching follows RFC 9309: the longest rule wins, Allow wins ties, and a bot with its own group ignores the * group instead of adding to it.

Where does robots.txt need to live?

At the domain root, as /robots.txt. A file placed in a subdirectory is ignored, and every subdomain needs its own.

What if my site has no robots.txt at all?

It is treated as permission to crawl everything. Having no file is not an error by itself, though you lose the standard place to declare your sitemap.