Free Robots.txt Generator & AI Crawler Rule Builder
Build a valid robots.txt, choose which search engines and AI crawlers may read your content, add a sitemap line, then copy or download the file. Runs entirely in your browser, with no signup.
Start a line with Allow: for an exception, such as Allow: /wp-admin/admin-ajax.php.
Unchecked crawlers get an explicit block. Blocking removes you from those AI answers.
Your robots.txt
User-agent: * Allow: / Disallow: /admin Disallow: /cart Disallow: /*?s=
Upload this file so it is reachable at /robots.txt on your domain root, then test your robots.txt file to confirm every crawler is treated the way you meant.
Build Production-Ready Crawling Directives
The generator writes the layout every major crawler parses the same way: a group for each crawler you block, then a User-agent: * group with Allow: / and your Disallow paths, then an absolute Sitemap line. The preview updates as you type, so what you copy is exactly what crawlers will read.
Keep the file short. Disallow only low-value or duplicate URLs such as carts, internal search and admin pages, and never block the CSS and JavaScript search engines need to render your pages.
Control AI Search & LLM Scraping Bots
AI crawlers come in two kinds. Search crawlers such as OAI-SearchBot, PerplexityBot and Claude-SearchBot fetch pages to answer a question now, so blocking them takes your site out of those answers. Training crawlers such as GPTBot, ClaudeBot, Google-Extended and CCBot collect data for future models, and blocking them is an opt-out that does not affect search.
Uncheck a crawler above to give it its own Disallow: / block. Google Search and AI Overviews use Googlebot, so blocking Google-Extended only stops Gemini training.
Recommended Robots.txt Templates for WordPress, Next.js, and Shopify
Pick a starting point, then adjust it in the generator. Your website URL and crawl delay carry over.
WordPress
User-agent: * Allow: / Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php Disallow: /?s=
Keeps crawlers out of the admin and internal search, but leaves admin-ajax.php open because themes and plugins call it from public pages. A real robots.txt in the site root replaces the virtual one WordPress serves.
Next.js
User-agent: * Allow: / Disallow: /api/
Keeps crawlers out of API routes. Never block /_next/, which holds the JavaScript and CSS Google needs to render your pages. Save it as public/robots.txt, or return the same rules from app/robots.ts.
Shopify
User-agent: * Allow: / Disallow: /cart Disallow: /checkout Disallow: /account Disallow: /search
Shopify serves its own robots.txt and you can't upload one. To add rules such as AI crawler blocks, add a robots.txt.liquid template to your theme and append them to Shopify's defaults.
Block AI training crawlers
User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: CCBot Disallow: / User-agent: Bytespider Disallow: / User-agent: * Allow: /
Keeps Google, Bing and AI search crawlers, and opts out of model training. Your pages can still appear in ChatGPT, Perplexity and Claude answers.
Staging or private site
User-agent: * Disallow: /
Blocks all crawling. Pair it with a password, since robots.txt is a request, not a lock.
How to create a robots.txt file
- 1
Enter your website URL
Used to build the absolute Sitemap line, which must point at a full URL such as https://example.com/sitemap.xml.
- 2
Choose whether crawlers may read the site
Leave 'Allow crawling of the whole site' on for production. Turn it off only for staging, which emits Disallow: / for every crawler.
- 3
List the paths to keep out of search
Add one path per line, such as /admin, /cart or /*?s=. Use it for low-value or duplicate URLs, not for private data.
- 4
Decide which AI crawlers may use your content
Uncheck any AI agent you want to block. Blocking removes your pages from that engine's answers, so weigh traffic against control.
- 5
Copy the file to your domain root
Save the output as robots.txt in your public folder so it is served at https://yourdomain.com/robots.txt, then reload that URL to confirm.
Mistakes to check before you publish
- Disallow is not noindex. Blocked URLs can still be indexed without a snippet. Use a noindex meta tag on pages you want out of results.
- Do not block CSS or JS. Search engines render your pages, and blocked assets make them render broken.
- One file per host. Subdomains and the www variant each need their own robots.txt.
- The Sitemap line is absolute. Relative paths are ignored.
- Rules are case sensitive.
/Admindoes not match/admin.
See whether your current robots.txt is blocking anything
Run a free SEO and GEO audit to check robots.txt rules, AI crawler access, sitemap discovery, indexability and schema in one report.
Robots.txt Setup & Deployment FAQ
What is a robots.txt file?
robots.txt is a plain text file at the root of a domain that tells crawlers which paths they may request. It is a crawling instruction, honoured voluntarily by well-behaved bots, and it is the first file most search engines fetch.
Where should the robots.txt file be uploaded on a web server?
It must be served at the root of each host, for example https://example.com/robots.txt. A file at /pages/robots.txt or on a different subdomain is ignored.
Does robots.txt stop a page from being indexed?
No. Disallow stops crawling, not indexing. A blocked URL can still appear in results from external links, without a snippet. To keep a page out of the index, allow crawling and use a noindex meta robots tag or an X-Robots-Tag header.
How do I block ChatGPT and Claude while allowing Google search?
Give their crawlers their own Disallow: / blocks and leave User-agent: * allowed, so Googlebot keeps crawling. For OpenAI that is GPTBot, OAI-SearchBot and ChatGPT-User; for Anthropic, ClaudeBot, Claude-SearchBot and Claude-User. Google-Extended blocks only Gemini training and does not affect Google Search or AI Overviews, which use Googlebot. This generator writes those blocks for any agent you uncheck.
Should I add a Sitemap line?
Yes, if you have a sitemap. The Sitemap directive gives crawlers an absolute URL to your sitemap.xml and works independently of the User-agent blocks around it.
What is the most common robots.txt mistake?
Shipping a staging file with Disallow: / to production, which quietly removes the whole site from crawling. The second most common is blocking CSS or JS folders, which stops search engines rendering the page as users see it.