Robots.txt Generator and Tester

Free robots.txt generator and tester: build the rules for Googlebot, Bingbot and AI crawlers, then check if a URL is blocked using Google’s matching rules.

Also: Meta Tag Generator · Open Graph Generator · UTM Builder

What it does

robots.txt sits at the root of a site and tells crawlers which paths they may fetch. Rules are grouped by user agent; a crawler follows the most specific group that names it, or the * group. Inside a group, the rule with the longest matching path wins and allow wins a tie; * matches any sequence and $ marks the end of the URL. Blocking a page in robots.txt does not remove it from search results: use a noindex tag for that.

How to use it

  1. In Generate, choose what to allow or block, add AI crawlers and your sitemap.
  2. Copy the result into a file named robots.txt at the root of your site.
  3. In Test, paste any robots.txt and check a URL for a given crawler.

Frequently asked questions

Does robots.txt stop a page from being indexed?
No. It stops crawling, but a blocked URL can still appear in results if other sites link to it. To keep a page out of the index, allow crawling and add a noindex meta tag.

Which crawlers can I block for AI training?
The best known are GPTBot (OpenAI), Google-Extended (Google’s AI models, without affecting Search), CCBot (Common Crawl), ClaudeBot (Anthropic), PerplexityBot and Applebot-Extended. Their owners say they respect robots.txt.