Test what your robots.txt allows, then build a better one.
Enter any URL. The tester fetches the site's robots.txt, reads it the way Google does and shows for Googlebot, Bingbot and 13 AI bots whether that URL may be crawled, and which rule decides. The generator below writes a new file for you.
robots.txt is a request, not a lock. Firewalls and CDN bot protection can block crawlers before the file is ever read.
- 15
- crawlers and AI bots evaluated
- 500 KiB
- Google's size limit, checked
- 4
- generator presets
What the check tests
Google's robots.txt rules, applied to the live file. Each item says what was found and what Google does with it.
- 01
HTTP status
What Google does with the answer: a 2xx file is parsed, a 4xx (except 429) counts as no robots.txt and no restrictions, a 429, 5xx or network error makes Google pause crawling for 12 hours and then fall back to its last good copy.
- 02
Size and format
File size against Google's 500 KiB limit, the content type, UTF-8, and the common case of an HTML page served under /robots.txt.
- 03
Syntax
Lines that are not valid directives, rules before the first user-agent line, unknown fields, paths that don't start with /, and blank lines that look like they separate groups but don't.
- 04
Directives Google ignores
crawl-delay, noindex, nofollow, host and other fields some crawlers know and Google doesn't support, with what to use instead.
- 05
Sitemap lines
Whether each Sitemap line is a full URL, and whether the first three answer with 2xx.
- 06
Your URL, per crawler
For Googlebot, Bingbot and 13 AI bots: may this URL be crawled, and which rule on which line decides. Matching follows Google's rules: the most specific group, the longest matching rule, allow wins a tie.
- 07
Control tokens
Google-Extended and Applebot-Extended are evaluated too and labelled for what they are: tokens that control AI use of content, not crawlers.
Generate a robots.txt
Pick a preset, add the paths you want to keep crawlers out of and your sitemap, then copy the file. It runs in your browser; nothing you type is sent anywhere.
Your robots.txt
What each group does
Questions
Didn't Google have its own robots.txt tester?
It did. Google retired the robots.txt tester in Search Console in late 2023 and replaced it with the robots.txt report, which shows which robots.txt files Google found for the top 20 hosts of your site, when it last fetched them and which lines it could not parse. To check whether a specific URL is blocked, Google points to the URL Inspection tool and its open-source robots.txt parser. This tester applies the same matching rules to any site, including ones you don't have Search Console access for.
Does Disallow keep a page out of Google?
No. Disallow stops crawling, not indexing. If other pages link to a blocked URL, Google can still index it, just without its content. To keep a page out of search results, let Google crawl it and add a noindex robots meta tag or X-Robots-Tag header. Google stopped supporting noindex inside robots.txt in 2019.
Should I block AI bots?
Separate training from search. Training bots such as GPTBot, ClaudeBot and CCBot collect content for future models; blocking them is a business decision. Search bots such as OAI-SearchBot, Claude-SearchBot and PerplexityBot decide whether assistants can find and cite you; OpenAI says sites that block OAI-SearchBot are not shown in ChatGPT search answers. Google-Extended only controls Gemini training and grounding and has no effect on Google Search. The AI Crawler Check also tests whether your server or CDN blocks these bots before robots.txt is read.
Why does a bot ignore my rules for User-agent: *?
A crawler follows only the most specific group that names it. Once a group for GPTBot exists, GPTBot ignores the * group completely, so paths you want blocked for every crawler have to be repeated in that group. Several groups naming the same crawler are merged, and blank lines don't end a group.
How fast do robots.txt changes take effect?
Google generally caches robots.txt for up to 24 hours, and you can ask for a faster recrawl in the Search Console robots.txt report. OpenAI says its search systems need about 24 hours to adjust. Other crawlers don't publish a timeframe.
Crawlers can read you. Do AI assistants cite you?
The AI Search Visibility Audit tests the questions your buyers ask in ChatGPT, Perplexity, Gemini and Google's AI Overviews, measures how often you are mentioned and cited against your competitors, and shows what to change.