Robots.txt Tester & Analyzer

Paste your robots.txt, enter a URL and instantly see whether it is allowed or blocked for a given crawler. The tester follows the RFC 9309 matching rules: the most specific user-agent group wins, the longest matching path wins, and Allow beats Disallow on a tie, with support for * wildcards and $ end anchors. A table shows the verdict for fifteen popular bots at once, including AI crawlers such as GPTBot and ClaudeBot.

PrivateFreeWorks offline

Reviewed by Vincenzo Manto, Founder & Lead Developer · Last updated September 30, 2026

How it works

  1. 1

    Paste the content of your robots.txt file into the left box.

  2. 2

    Enter a URL or path and choose a user-agent such as Googlebot.

  3. 3

    Read the verdict and the exact rule that matched, then review the table of common bots.

Example

robots.txt
User-agent: *
Disallow: /admin/
Allow: /admin/help.html

User-agent: GPTBot
Disallow: /
Verdicts
Googlebot  /admin/settings   -> Blocked
Googlebot  /admin/help.html  -> Allowed
GPTBot     /blog/post        -> Blocked

How robots.txt matching works

A crawler picks the group whose User-agent line most specifically matches its name; only if none matches does it fall back to the * group. Within that group, every Allow and Disallow path is compared with the URL path, where * matches any sequence of characters and $ anchors the end of the URL. The longest matching pattern decides the outcome, and if no pattern matches the URL is allowed.

Common mistakes to avoid

A stray Disallow: / under User-agent: * blocks your whole site, which often happens when a staging file is deployed to production. Blocking CSS or JavaScript folders can stop Google from rendering your pages correctly. Remember that each crawler reads only its own group, so rules under * do not apply to a bot that has its own more specific group.

Frequently asked questions

Why does the tool ask me to paste the file instead of fetching it?

Browsers block cross-origin requests to most sites, so a client-side tool cannot download another domain's robots.txt. Open yoursite.com/robots.txt, copy the content and paste it here. In exchange nothing is sent to any server.

How are conflicting rules resolved?

As in RFC 9309 and Google's implementation, the rule with the longest matching path wins. If an Allow and a Disallow have the same length, Allow wins.

Does robots.txt hide a page from Google?

Not reliably. Robots.txt controls crawling, not indexing: a blocked URL can still be indexed if other sites link to it. Use a noindex directive on a crawlable page to keep it out of search results.

Can I block AI crawlers?

Yes. Add a group for the crawler, for example User-agent: GPTBot followed by Disallow: /. The table shows how your file treats GPTBot, ClaudeBot, PerplexityBot and Google-Extended.

Related tools

Hreflang Tag Generator & Validator

Build hreflang tags, HTTP headers or sitemap XML for multilingual sites, with validation.

Schema Markup Generator (JSON-LD)

Generate JSON-LD structured data for FAQ, Local Business, How-to, Product and Article pages.

Open Graph Tag Analyzer & Social Preview

Paste page HTML to preview how it looks on Facebook, X and LinkedIn and spot Open Graph errors.