AI crawler control guide
Block AI Crawlers in Robots.txt
Some publishers want AI crawlers to avoid training or citation use. Robots.txt is the first public control layer to understand.
Short answer
You can request AI crawler blocking with robots.txt user-agent groups, but support varies by crawler. Blocking AI crawlers may reduce unwanted access, but it can also reduce AEO and GEO visibility because answer engines may not read or cite your pages.
Example robots.txt pattern
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xmlKeep the public crawler group separate from AI-specific groups so Googlebot and Bingbot can still crawl pages you want indexed.
Related crawl-control pages
Frequently asked questions
Can robots.txt block all AI crawlers?+
Robots.txt can request that compliant crawlers stay away, but it is not an access-control system and support varies. Sensitive content should be protected with authentication, not robots.txt.
Will blocking AI crawlers hurt AEO or GEO?+
It can. If answer engines cannot crawl a page, they are less likely to summarize or cite it. Blocking is a control tradeoff, not a ranking tactic.