SEO infrastructure · crawl control · indexing
Robots.txt Generator
Create a clean robots.txt file for Google, Bing, and other crawlers. Choose an indexing preset, add private paths, include your XML sitemap, then copy the finished file.
Short answer
A robots.txt file tells crawlers which URLs they may crawl; it does not force Google to index a page or remove a URL from search. For SEO, keep important pages crawlable, block only low-value/private paths, and always include an absolute Sitemap line so crawlers can discover your canonical URLs faster.
Generate your robots.txt
Start with a safe crawl-control preset, then add one path per line. The output updates instantly.
Upload the file to /robots.txt on your canonical host.
Related SEO infrastructure
Robots.txt directives reference
| Directive | SEO use | Example |
|---|---|---|
| User-agent | Selects which crawler group the rules apply to | User-agent: * |
| Allow | Explicitly permits a path, useful when a broader folder is blocked | Allow: /blog/ |
| Disallow | Requests that crawlers do not crawl a path | Disallow: /admin/ |
| Crawl-delay | Optional throttle for crawlers that support it; Google ignores it | Crawl-delay: 2 |
| Sitemap | Points crawlers to your XML sitemap or sitemap index | Sitemap: https://example.com/sitemap.xml |
Robots.txt is a crawl directive, not an access-control system. Do not use it to hide private data.
Robots.txt, SEO, AEO and GEO
Robots.txt is one of the first files crawlers request. For classic SEO, it protects crawl budget by keeping low-value URLs out of the crawl queue. For AEO and GEO, it matters because answer engines need clean access to the pages, structured data, FAQs, and reference tables you want them to cite. Blocking the wrong folder can make a technically perfect page invisible to the systems that summarize the web.
The safest default is simple: keep public canonical pages crawlable, include your XML sitemap, and avoid using robots.txt as a deindexing tool. If a URL should not appear in search, use a crawlable noindex tag or remove the URL with the correct HTTP status instead of hiding it from crawlers.
How this generator builds the file
The generator outputs conservative robots.txt syntax that is easy for search crawlers and humans to audit.
- Choose a crawler group. The default User-agent: * applies rules to all compliant crawlers. Advanced users can target a named bot such as Googlebot.
- Normalize paths. Folder-like paths are normalized with a leading and trailing slash, while wildcard patterns such as /*.pdf$ are preserved.
- Keep discovery open. The SEO-safe preset blocks low-value private paths but keeps public pages, assets, and the XML sitemap crawlable.
Assumptions
- The generated file is intended for public websites, not password-protected staging environments.
- Robots.txt cannot guarantee deindexing; use noindex or HTTP status codes for removal workflows.
- Crawl-delay is included only when requested because major crawlers differ in support.
Sources
Last reviewed: August 21, 2026
Frequently asked questions
What should I put in robots.txt for SEO?+
For most sites, keep the file simple: allow important public pages, disallow low-value or private areas such as /admin/ and /api/, and include a Sitemap directive with the absolute URL of your XML sitemap. Do not block CSS, JavaScript, images, or pages you want Google to understand and rank.
Does robots.txt control indexing?+
Not directly. Robots.txt controls crawling, not indexing. A blocked URL can still appear in Google if other pages link to it, but Google may show it without a useful title or snippet because it could not crawl the content. To remove a page from the index, use noindex on a crawlable page or return the correct HTTP status.
Should I add my sitemap to robots.txt?+
Yes. A Sitemap line is safe and useful because it gives crawlers a stable discovery path to your canonical XML sitemap. Use the full absolute URL, such as Sitemap: https://example.com/sitemap.xml. Large sites can list multiple sitemap URLs.
Can I block AI crawlers in robots.txt?+
Some AI and answer-engine crawlers document user-agent names and may respect robots.txt, but support varies. If you block them, understand the tradeoff: less training/citation exposure, but more control. For AEO/GEO growth, many publishers prefer to keep useful pages crawlable and provide clean structured data and llms.txt files.
Where do I upload the generated robots.txt file?+
Place it at the root of your site: https://example.com/robots.txt. A file at /blog/robots.txt or another folder does not control the whole host. If you use a framework like Next.js, create an app/robots.ts route or place a static robots.txt in the public folder, depending on your setup.
This tool creates syntax guidance for public crawler directives. It is not a security control and does not protect private files from direct access.