Free tool

robots.txt Generator

Generate your robots.txt with per-bot rules, sitemap and AI crawler blocking. 100% in your browser.

Built by

Miguel Ángel Colorado Marin (MACM)

Full-Stack Developer · Guadalajara, España

I develop web apps, digital tools and full projects — from design to deployment.

GitHubLinkedInBuy me a coffee
Contact me

The robots.txt generator creates the file that tells web crawlers (search engines, AI bots) which parts of your site they can or can't visit. Choose the quick mode (allow everything or block everything) for simple cases, or define custom rules per User-agent with specific paths to allow or block. You can block the best-known AI training bots with one click (OpenAI's GPTBot, Anthropic's ClaudeBot, Common Crawl's CCBot and more) if you don't want your content used to train models, add your sitemap URL to help indexing, and set a Crawl-delay if you need to limit crawl frequency. The result is generated instantly, ready to copy or download directly as robots.txt.

Features

  • ✓Quick mode: allow or block everything
  • ✓Custom rules per User-agent
  • ✓Direct blocking of known AI bots (GPTBot, ClaudeBot, CCBot...)
  • ✓Optional Sitemap and Crawl-delay
  • ✓100% in browser, no uploads

How to generate a robots.txt?

  1. 1

    Choose the mode

    Allow all, block all, or custom rules.

  2. 2

    Add rules if needed

    Paths to allow or block for each User-agent.

  3. 3

    Block AI bots if you want

    Check the ones you don't want crawling your content.

  4. 4

    Download the file

    Upload it to your domain's root as /robots.txt.

Frequently asked questions

Where do I need to upload the file?

At your domain's root, accessible at yourdomain.com/robots.txt — it doesn't work at any other path.

Does blocking an AI bot guarantee it won't use my content?

robots.txt is a request that respectful crawlers voluntarily follow, not a real technical barrier. Most major bots (OpenAI, Anthropic, Common Crawl) do respect it, but it's not an absolute guarantee against actors who ignore it.

Is my site's information uploaded to a server?

No, the file is generated by combining your options directly in your browser.

Related tools

.gitignore Generator

Combine technologies to generate your .gitignore.

LICENSE Generator

Generate your project's LICENSE file.

JSON Formatter

Format, validate and minify JSON.

URL Encoder

Encode and decode URLs instantly.

Embed robots.txt Generator on your site

Add robots.txt Generator to any web page with a simple iframe. Free, with attribution to miguelacm.es.

<iframe
  src="https://miguelacm.es/embed/robots-txt-generator"
  width="100%"
  height="700"
  frameborder="0"
  title="robots.txt Generator — miguelacm.es"
></iframe>
View embed in new tab →