Free tool / Robots.txt Generator
Choose which search and AI crawlers may visit your site, list any folders to keep private and add your sitemap, and the generator writes a robots.txt file you can copy or download. It checks the result the way crawlers read it (RFC 9309), so you can see each crawler's verdict before you publish.
Free, with no sign-up and no email.
Reviewed by M. NoumanCo-founder, RankNexus · Updated 19 October 2026
User-agent: GPTBot User-agent: ClaudeBot User-agent: Google-Extended User-agent: Applebot-Extended User-agent: meta-externalagent User-agent: CCBot Disallow: / User-agent: * Disallow:
Upload the file to the root of your site, then check the live file with the Robots.txt Tester. Crawlers may take up to a day to notice changes.
The guide
| Line | What it does | Example |
|---|---|---|
| User-agent | Starts a group and names the crawler it applies to; * means every crawler | User-agent: GPTBot |
| Disallow | A path the crawler shouldn't fetch; left empty, nothing is blocked | Disallow: /checkout/ |
| Allow | An exception inside a blocked path | Allow: /search/help |
| Sitemap | The full address of a sitemap, anywhere in the file | Sitemap: https://www.example.com/sitemap.xml |
| * and $ | * matches any characters; $ marks the end of a path | Disallow: /*.pdf$ |
Paths start with a slash and are case-sensitive. Google ignores the crawl-delay line, so the generator leaves it out.
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://www.example.com/wp-sitemap.xmlUser-agent: *
Disallow: /cart
Disallow: /checkout
Disallow: /account
Disallow: /search
Sitemap: https://www.example.com/sitemap.xmlUser-agent: *
Disallow: /Swap in your own paths and sitemap address. On a staging site a password is safer than robots.txt, because a blocked page can still be indexed if something links to it.
Each crawler has its own job, so you can opt out of AI training without leaving AI search. These are the tokens the generator covers, from each company's own documentation:
| Token | Company | Type | What it controls |
|---|---|---|---|
| Googlebot | Search and AI answers | Google Search, which AI Overviews and AI Mode draw on. | |
| Bingbot | Microsoft | Search and AI answers | Bing search, which Microsoft Copilot draws on. |
| OAI-SearchBot | OpenAI | Search and AI answers | Showing your site in ChatGPT search answers. |
| Claude-SearchBot | Anthropic | Search and AI answers | Indexing pages for search inside Claude. |
| PerplexityBot | Perplexity | Search and AI answers | Showing and linking your site in Perplexity answers. |
| Applebot | Apple | Search and AI answers | Search in Spotlight, Siri and Safari. |
| meta-webindexer | Meta | Search and AI answers | Meta AI search results and the sources it cites. |
| ChatGPT-User | OpenAI | User-requested visits | Visiting a page when a ChatGPT user's request needs it. |
| Claude-User | Anthropic | User-requested visits | Visiting a page when a Claude user's question needs it. |
| Perplexity-User | Perplexity | User-requested visits | Fetching a page a Perplexity user's question needs, so it can be cited. |
| GPTBot | OpenAI | AI model training | Content that may be used to train OpenAI's models. |
| ClaudeBot | Anthropic | AI model training | Content that may be used to train Anthropic's models. |
| Google-Extended | AI model training | Gemini training and grounding. It does not affect Google Search. | |
| Applebot-Extended | Apple | AI model training | Apple's model training. It does not affect Apple search results. |
| meta-externalagent | Meta | AI model training | Meta's AI training and product indexing. |
| CCBot | Common Crawl | AI model training | Common Crawl's open web archive, widely used to train AI models. |
Blocking a training token such as GPTBot or Google-Extended doesn't remove you from search. Blocking a search crawler such as OAI-SearchBot or PerplexityBot keeps you out of the answers it feeds.
After any change, test the live file with the Robots.txt Tester and check the robots.txt report in Google Search Console. Crawlers may keep the old file for up to a day.
The AEO Checker shows whether a page is ready to be quoted, and technical SEO covers crawling and indexing beyond one file.
The short answers
Pick the crawlers to allow or block, add any folders crawlers should skip and your sitemap address, then copy or download the file. Save it as robots.txt at the root of your site, so it opens at https://yourdomain.com/robots.txt, and test the live file with our Robots.txt Tester.
Groups of rules, each starting with one or more User-agent lines followed by Disallow or Allow lines, plus optional Sitemap lines. A simple file that allows everything is two lines: User-agent: * and an empty Disallow:.
Block the training crawlers, such as GPTBot, ClaudeBot, CCBot and the Google-Extended and Applebot-Extended control tokens, and leave the search crawlers, such as OAI-SearchBot, Claude-SearchBot and PerplexityBot, allowed. The generator's “Block AI training” preset does exactly that.
Questions
At the root of your domain, the same folder as your homepage. Each subdomain needs its own file. On WordPress an SEO plugin can edit it; on Shopify you edit the robots.txt.liquid theme template.
WordPress serves a virtual robots.txt. Most SEO plugins have a file editor for it, or you can upload your own robots.txt to the site root, which replaces the virtual one.
Blocking training crawlers such as GPTBot or Google-Extended doesn't affect Google Search. Blocking Googlebot, Bingbot or the AI search crawlers keeps you out of the results and AI answers they feed.
Partly. Blocking OAI-SearchBot and GPTBot stops OpenAI's automatic crawling, but OpenAI says ChatGPT-User, which fetches a page when a person asks, may not follow robots.txt.
No. Google says a blocked page can still be indexed if other sites link to it. Use a noindex tag, and leave the page crawlable so Google can see it.
Fill in this generator and download the file, or write the rules in a plain text file named robots.txt. Upload it to the root of your domain so it opens at yourdomain.com/robots.txt, then check it with our Robots.txt Tester.
Shopify creates robots.txt for you. To change it, go to Online Store, open Edit code on your theme, add a new template called robots and edit the robots.txt.liquid file. Shopify warns this is an unsupported customization and that mistakes can cost all your traffic, so change as little as you need.
No. The file is built in your browser, and nothing you type is sent to us.
A useful next conversation
The tools show what is on your site. Our free audit looks at how ChatGPT, Google and Perplexity describe you, and what we would fix first. A specialist sends a recorded walkthrough within 2 business days.