Free robots.txt generator
Build a valid robots.txt, choose which search engines and AI crawlers may read your content, add a sitemap line, then copy or download the file. Runs entirely in your browser, with no signup.
Unchecked crawlers get an explicit block. Blocking removes you from those AI answers.
Your robots.txt
User-agent: * Allow: / Disallow: /admin Disallow: /cart Disallow: /*?s=
Upload this file so it is reachable at /robots.txt on your domain root, then re-run an audit to verify.
Common robots.txt examples
Standard production site
User-agent: * Allow: / Sitemap: https://example.com/sitemap.xml
Everything crawlable, sitemap advertised. The right default for most sites.
Staging or private environment
User-agent: * Disallow: /
Blocks all crawling. Pair it with HTTP auth, since robots.txt is a request, not a lock.
Allow search, block AI training crawlers
User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: CCBot Disallow: / User-agent: * Allow: /
Keeps Google and Bing while opting out of AI training crawlers. It also reduces AI answer citations.
How to create a robots.txt file
- 1
Enter your website URL
Used to build the absolute Sitemap line, which must point at a full URL such as https://example.com/sitemap.xml.
- 2
Choose whether crawlers may read the site
Leave 'Allow crawling of the whole site' on for production. Turn it off only for staging, which emits Disallow: / for every crawler.
- 3
List the paths to keep out of search
Add one path per line, such as /admin, /cart or /*?s=. Use it for low-value or duplicate URLs, not for private data.
- 4
Decide which AI crawlers may use your content
Uncheck any AI agent you want to block. Blocking removes your pages from that engine's answers, so weigh traffic against control.
- 5
Copy the file to your domain root
Save the output as robots.txt in your public folder so it is served at https://yourdomain.com/robots.txt, then reload that URL to confirm.
Mistakes to check before you publish
- Disallow is not noindex. Blocked URLs can still be indexed without a snippet. Use a noindex meta tag on pages you want out of results.
- Do not block CSS or JS. Search engines render your pages, and blocked assets make them render broken.
- One file per host. Subdomains and the www variant each need their own robots.txt.
- The Sitemap line is absolute. Relative paths are ignored.
- Rules are case sensitive.
/Admindoes not match/admin.
See whether your current robots.txt is blocking anything
Run a free SEO and GEO audit to check robots.txt rules, AI crawler access, sitemap discovery, indexability and schema in one report.
Frequently asked questions
- What is a robots.txt file?
- robots.txt is a plain text file at the root of a domain that tells crawlers which paths they may request. It is a crawling instruction, honoured voluntarily by well-behaved bots, and it is the first file most search engines fetch.
- Where does robots.txt go?
- It must be served at the root of each host, for example https://example.com/robots.txt. A file at /pages/robots.txt or on a different subdomain is ignored.
- Does robots.txt stop a page from being indexed?
- No. Disallow stops crawling, not indexing. A blocked URL can still appear in results from external links, without a snippet. To keep a page out of the index, allow crawling and use a noindex meta robots tag or an X-Robots-Tag header.
- How do I block ChatGPT, Perplexity or Claude?
- Add an explicit block for their user agents: GPTBot and OAI-SearchBot for OpenAI, PerplexityBot for Perplexity, ClaudeBot for Anthropic, and Google-Extended for Gemini and AI Overviews. This generator writes those blocks for any agent you uncheck.
- Should I add a Sitemap line?
- Yes, if you have a sitemap. The Sitemap directive gives crawlers an absolute URL to your sitemap.xml and works independently of the User-agent blocks around it.
- What is the most common robots.txt mistake?
- Shipping a staging file with Disallow: / to production, which quietly removes the whole site from crawling. The second most common is blocking CSS or JS folders, which stops search engines rendering the page as users see it.