Free tool / Robots.txt Tester
Enter your website to see whether Googlebot, Bingbot and the crawlers behind ChatGPT, Claude, Perplexity, Gemini, Apple and Meta can reach it under your robots.txt. We read the file the way the robots.txt standard (RFC 9309) describes, show the exact rule behind each verdict and check for an llms.txt file.
Free, with no sign-up and no email.
Reviewed by M. NoumanCo-founder, RankNexus · Updated 19 October 2026
The guide
Not always. With no robots.txt file (a 404), crawlers may fetch every page, which suits many small sites. You need one to keep crawlers out of areas such as internal search results, carts or staging folders, or to give AI crawlers different rules. Google notes that on platforms such as Wix or Blogger you may not need to, or be able to, edit the file yourself.
Know what it can't do. Robots.txt manages crawling, not indexing: Google says a page blocked in robots.txt can still be indexed if other sites link to it. To keep a page out of search, use a noindex tag or password-protect it.
Each group starts with one or more User-agent lines, followed by its rules. A crawler follows only the group that names it, or the group for all crawlers if none does. Copy the example that matches your choice, publish it, then test it above.
User-agent: *
Disallow:
Sitemap: https://www.example.com/sitemap.xmlUser-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: CCBot
Disallow: /
User-agent: *
Disallow:In that second example, search crawlers such as Googlebot, OAI-SearchBot, Claude-SearchBot and PerplexityBot fall under the last group, so they can still read your pages for search results and AI answers.
User-agent: *
Disallow: /checkout/
Disallow: /search
Allow: /search/helpWhen two rules match, the longer one wins, so /search/help stays open while the rest of /search is blocked.
GPTBot collects content that may be used to train OpenAI's models. ChatGPT search uses a different crawler, OAI-SearchBot, and OpenAI says sites that opt out of OAI-SearchBot are not shown in ChatGPT search answers. So block GPTBot by name and leave OAI-SearchBot alone:
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /OpenAI says robots.txt changes take about 24 hours to reach its search systems. ChatGPT-User, which fetches a page when someone asks ChatGPT to, may not follow robots.txt, because a person asked for the page.
Write the corrected file with the Robots.txt Generator, which shows each crawler's verdict before you publish and explains how to edit robots.txt on WordPress, Shopify and Wix. Then test the live file here again and check the robots.txt report in Google Search Console.
If you also want a plain summary for AI tools, the llms.txt Generator writes one, and technical SEO covers crawling and indexing problems in full.
The short answers
Allow the search crawlers if you want to appear in AI answers: Googlebot and Bingbot, plus OAI-SearchBot for ChatGPT search, Claude-SearchBot, PerplexityBot and Applebot. Blocking them keeps you out of the answers they feed.
Training crawlers such as GPTBot, ClaudeBot and CCBot, and control tokens such as Google-Extended, only decide whether your content can train AI models. You can block those without leaving search.
No. GPTBot collects training data. ChatGPT search uses a different crawler, OAI-SearchBot, so you can block GPTBot and still appear in ChatGPT search answers, as long as OAI-SearchBot is allowed.
A crawler follows the group that names it, or the group for all crawlers (User-agent: *) if none does. Within that group, the most specific matching rule wins, and Allow wins a tie. An asterisk matches anything and a dollar sign marks the end of a path.
A missing file (a 404) means crawlers may fetch everything. A server error (a 5xx) is treated by the standard, and by Google, as a block on the whole site for as long as it lasts, which can quietly hide a whole site, so it is worth fixing quickly.
Questions
Reputable search and AI crawlers say they do. Some user-requested fetchers, such as ChatGPT-User and Perplexity-User, may not apply robots.txt because a person asked for the page. Blocking by robots.txt is a request, not a lock.
No. Google says Google-Extended only controls whether your content is used for Gemini training and grounding. Search, including AI Overviews, is governed by Googlebot.
Crawlers may cache robots.txt for up to a day, so allow 24 hours before checking again.
No. A blocked page can still be indexed if other sites link to it. To keep a page out of search, use a noindex tag and leave it crawlable.
Not always. Without one, crawlers may fetch every page, which suits many small sites. You need one to keep crawlers out of areas such as carts or internal search results, or to set separate rules for AI crawlers. Our Robots.txt Generator writes one.
Yes, if it blocks nothing you need: crawlers then treat the whole site as open. What you should never do is leave a staging rule such as Disallow: / on a live site, because it blocks everything.
A plain text file at the root of your site that tells search engine and AI crawlers which pages they may fetch. Google says it's used mainly to avoid overloading your site, and that it isn't a way to keep a page out of Google: use a noindex tag for that.
Add /robots.txt to the domain, for example example.com/robots.txt. If the site has one, it opens as plain text, and this tester fetches it and shows what each crawler may access.
A few short groups of rules: a User-agent line naming the crawler, then Allow or Disallow lines for paths, plus a Sitemap line with your sitemap's full address. A small site often needs only User-agent: *, an empty Disallow line and its sitemap. Our Robots.txt Generator writes one for you.
Only your robots.txt and llms.txt files, from public websites, with a short timeout. We keep results for up to an hour to avoid fetching the same files repeatedly.
A useful next conversation
The tools show what is on your site. Our free audit looks at how ChatGPT, Google and Perplexity describe you, and what we would fix first. A specialist sends a recorded walkthrough within 2 business days.