robots.txt checker
Fetch any site's robots.txt, catch the lines crawlers skip or misread, and test whether Googlebot, Bingbot and every major AI crawler can reach a page. Free, no account.
Enter your robots.txt URL or domain. Paste a page URL instead to test whether crawlers can reach that page.
No robots.txt yet? Build one with the robots.txt generator.
How to check a robots.txt file
- 1
Enter a domain or a page
Type a domain to check its robots.txt, or a full page URL to also test whether crawlers may fetch that exact path.
- 2
Read the crawler table
Googlebot, Bingbot and every major AI crawler is tested against the file with Google's longest-match rules, showing the rule that decided each one.
- 3
Fix the highlighted lines
Lines crawlers skip or misread are marked in the file view: unknown directives, rules before any User-agent, relative Sitemap URLs and the unsupported Noindex.
- 4
Confirm your sitemap is declared
Every Sitemap line is fetched. Add one if it is missing so all crawlers find your sitemap, not only the ones you submit it to.
Check the rest of the site too
The free audit checks robots.txt alongside sitemaps, canonicals, indexability and AI-search readiness.
Frequently asked questions
What does a robots.txt checker test?
Whether the file is served correctly as plain text, whether every line is a directive crawlers understand, which crawlers are allowed or blocked on a given page, and whether the sitemaps it declares actually load.
How does Google decide between Allow and Disallow?
The most specific rule wins: the matching rule with the longest path. If an Allow and a Disallow match with the same length, Allow wins. A crawler follows only the most specific User-agent group that names it, and falls back to User-agent: * otherwise.
Does blocking GPTBot remove my site from ChatGPT?
No. GPTBot collects training data. ChatGPT's search answers use OAI-SearchBot and ChatGPT-User. Blocking the training crawlers is a legitimate choice; blocking the search crawlers keeps your pages out of their answers.
What happens if robots.txt returns an error?
A 404 means no rules, so everything may be crawled. A 5xx server error is worse: Google treats it as a full block until the file answers again, and can drop pages if it lasts.
Can robots.txt keep a page out of Google?
No. It stops crawling, not indexing. A blocked URL can still be indexed from links, without a description. To keep a page out, allow crawling and add a noindex meta tag or X-Robots-Tag header.