robots.txt is one of the oldest, most genuinely impactful files in technical SEO: a simple text file at your domain's root that tells crawlers what they can and can't index. And since generative AI models train on web content at scale, it has also become the first line of defense in deciding whether your content gets used to train those models.
What it is and where it lives
The robots.txt file always sits at the domain root (yourdomain.com/robots.txt) and contains rules that crawlers (search engines, AI, or any other kind) should voluntarily respect. It's not a security mechanism: it's a convention well-behaved bots follow, but it doesn't technically stop a malicious one from ignoring the rules.
The basic structure
Each rule targets a User-agent (the bot's name) and defines allowed (Allow) or blocked (Disallow) paths:
User-agent: *
Disallow: /admin/
Disallow: /api/
Sitemap: https://yourdomain.com/sitemap.xml
User-agent: * applies the rule to every bot; you can target a specific bot using its exact name (User-agent: Googlebot).
Why including the sitemap matters
Adding the Sitemap: line at the end of robots.txt is, for many crawlers, the first way they discover where your site's complete URL map is. It's a line that costs nothing to add and actively makes it easier to crawl the pages you actually want indexed.
Blocking AI training crawlers
Besides traditional search engine crawlers, there are specific bots AI companies use to collect content to train their models: GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended, among others. These bots respect robots.txt like any other serious crawler, so blocking them specifically is as simple as adding a dedicated rule:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
This is independent of blocking traditional search engines: you can keep letting Google and Bing index your content to appear in search results, while specifically blocking AI training bots.
How to generate your robots.txt
- Choose which bots to allow or block, including specific AI crawlers if you want to exclude them.
- Define the paths that shouldn't be crawled (admin panels, internal APIs, duplicate content).
- Add your sitemap's URL.
- Download or copy the generated file and upload it to your domain's root.
You can do it free with the robots.txt generator on this site.
Why it isn't a security measure
It's important not to confuse robots.txt with real access protection. Blocking a path in robots.txt doesn't stop someone from directly accessing that URL by typing it into the browser, nor does it stop a malicious bot from crawling it while ignoring the rules. For content that truly needs to be protected, you need real authentication, not an entry in this file.
Frequently asked questions
Does blocking GPTBot affect my Google rankings? No, they're completely independent bots; blocking AI crawlers doesn't affect Googlebot's crawling or traditional SEO.
Is having a robots.txt mandatory? No, but its absence means, by default, all content is crawlable with no explicit restriction.
Can I block a folder but allow a specific file inside it? Yes, using more specific Allow rules alongside the folder's general Disallow.
Does blocking a bot in robots.txt technically block it? No, it only asks it not to crawl; a bot that decides to ignore the convention can still access the content, so it doesn't replace real server-level blocking measures if that's what you need.
Generate your robots.txt for free with the robots.txt generator, with per-bot rules, sitemap and AI crawler blocking in seconds.