Free Robots.txt Tester, Validator & Checker

Fetch any site's robots.txt, catch the lines crawlers skip or misread, and test whether Googlebot, Bingbot and every major AI crawler can reach a page. Free, no account.

Enter your robots.txt URL or domain. Paste a page URL instead to test whether crawlers can reach that page.

Need to build a new robots.txt? Use the free robots.txt generator, with presets for WordPress, Next.js, Shopify and blocking AI training crawlers.

How do crawlers decide between Allow and Disallow?

Google and Bing apply the most specific rule: the matching Allow or Disallow with the longest path wins, and Allow wins a tie. So with Disallow: / and Allow: /blog/, /blog/post can be crawled because the Allow path is longer, while every other page stays blocked.

How to check a robots.txt file

  1. 1

    Enter a domain or a page

    Type a domain to check its robots.txt, or a full page URL to also test whether crawlers may fetch that exact path.

  2. 2

    Read the crawler table

    Googlebot, Bingbot and every major AI crawler is tested against the file with Google's longest-match rules, showing the rule that decided each one.

  3. 3

    Fix the highlighted lines

    Lines crawlers skip or misread are marked in the file view: unknown directives, rules before any User-agent, relative Sitemap URLs and the unsupported Noindex.

  4. 4

    Confirm your sitemap is declared

    Every Sitemap line is fetched. Add one if it is missing so all crawlers find your sitemap, not only the ones you submit it to.

Test Crawler Access for Any Page Path

Paste a full page URL instead of a domain and the tester checks that exact path, query string included, against every crawler in the table. Rules are matched the way Google matches them: a crawler follows its own User-agent group if it has one, otherwise User-agent: *, and the longest matching Allow or Disallow path wins, with Allow winning a tie. Wildcards (*) and end anchors ($) are supported.

Each row shows the rule that decided the result, so an unexpected block points straight at the line to change.

AI Bot Crawler Support (GPTBot, ClaudeBot, Perplexity)

Googlebot and these AI crawlers are tested on every check. Search crawlers fetch pages to answer questions now, so blocking them removes you from that engine's answers. Training crawlers collect data for future models, and blocking them does not.

AI search crawlers

  • OAI-SearchBot · OpenAI SearchBot (ChatGPT search)
  • ChatGPT-User · ChatGPT browsing
  • PerplexityBot · Perplexity search
  • Perplexity-User · Perplexity browsing
  • Claude-SearchBot · Claude search
  • Claude-User · Claude browsing
  • Bingbot · Bingbot (Bing, Copilot)

AI training crawlers

  • GPTBot · OpenAI GPTBot (training)
  • ClaudeBot · Anthropic ClaudeBot (training)
  • Google-Extended · Google-Extended (Gemini training)
  • Applebot-Extended · Applebot-Extended (Apple training)
  • CCBot · Common Crawl (training datasets)
  • Bytespider · ByteDance Bytespider (training)

robots.txt only says who may crawl. To see what an AI crawler actually gets from a page, with no JavaScript run, use the AI crawler checker. Crawler access is the first group of the GEO audit checklist.

Common Robots.txt Syntax Errors Caught by This Validator

Every line is linted, and these are marked in red in the file view:

  • Lines with no colon, which crawlers skip as not being a directive.
  • Directives crawlers don't recognise, which Google ignores.
  • Allow or Disallow rules placed before any User-agent line, so they apply to no one.
  • Paths that don't start with / or *.
  • Relative Sitemap lines. A Sitemap must be a full URL, including https://.
  • Noindex, which Google stopped supporting in 2019.

6 Common Robots.txt Mistakes and How to Fix Them

MistakeWhat happensFix
Noindex: in robots.txtIgnored by Google since September 2019, so the page can still be indexed.Use a noindex robots meta tag or an X-Robots-Tag header, and leave the page crawlable.
No trailing slash on a folderRules match by prefix, so Disallow: /private also blocks /private-policy.Write Disallow: /private/ to block only the folder.
Blocking CSS and JavaScriptGooglebot can't render the page, so it may judge layout and content wrong.Allow the folders that hold your styles and scripts, such as /assets/ or /_next/.
Relative Sitemap URLSitemap: /sitemap.xml is skipped; crawlers need the full address.Write Sitemap: https://yourdomain.com/sitemap.xml.
Invisible charactersA zero-width space or other hidden character pasted from a document can break a directive. Google skips a leading byte order mark, but other crawlers may not.Save the file as plain UTF-8 and strip hidden characters before uploading.
robots.txt returns a 5xx errorGoogle treats a server error as a full block and stops crawling the site.Serve the file with a 200, or a 404 if you have no rules. Never a 5xx or 429.

Pasted rules from a doc or chat? Run them through the hidden character remover first. Declaring a sitemap you haven't built yet? Make one with the sitemap generator.

Check the rest of the site too

The free audit checks robots.txt alongside sitemaps, canonicals, indexability and AI-search readiness.

Frequently Asked Questions About Robots.txt Testing

What does a robots.txt checker test?

Whether the file is served correctly as plain text, whether every line is a directive crawlers understand, which crawlers are allowed or blocked on a given page, and whether the sitemaps it declares actually load.

How does Google decide between conflicting Allow and Disallow rules?

The most specific rule wins: the matching rule with the longest path. If an Allow and a Disallow match with the same length, Allow wins. A crawler follows only the most specific User-agent group that names it, and falls back to User-agent: * otherwise.

Does blocking GPTBot remove my site from ChatGPT?

No. GPTBot collects training data. ChatGPT's search answers use OAI-SearchBot and ChatGPT-User. Blocking the training crawlers is a legitimate choice; blocking the search crawlers keeps your pages out of their answers.

What happens if robots.txt returns an error?

A 404 means no rules, so everything may be crawled. A 5xx server error is worse: Google treats it as a full block until the file answers again, and can drop pages if it lasts.

Why isn't Googlebot respecting my Noindex directive in robots.txt?

Google stopped supporting Noindex in robots.txt on 1 September 2019, so the line is ignored. Put a noindex robots meta tag on the page, or send an X-Robots-Tag: noindex header, and leave the page crawlable so Google can see it.

Does robots.txt prevent pages from appearing in Google search?

No. It stops crawling, not indexing. A blocked URL can still be indexed from links, without a description. To keep a page out, allow crawling and add a noindex meta tag or X-Robots-Tag header.

Need to check more than one issue? Run our free SEO audit tool for 210 automated SEO and AI search checks, with evidence and priority fixes in a shareable report. No email or signup required.