robots.txt Generator & Tester
Generate and test robots.txt rules for search engines, AI crawlers, and private routes.
Generate
Test a live robots.txt
How to set robots.txt for SEO, AEO, and GEO
Allow important public pages
Search and answer engines need access to indexable public content. Block app shells, dashboards, private APIs, and staging pages, but keep marketing pages, docs, guides, and comparison pages crawlable.
Name AI crawler policy explicitly
Use explicit user-agent sections when you want a clear policy for OpenAI, Anthropic, Perplexity, Google, Apple, Common Crawl, and other AI-related crawlers. Treat search access and training access as separate policy choices.
Always include sitemap
Add a Sitemap line so crawlers can discover canonical URLs quickly. Keep sitemap.xml updated with accurate lastmod dates.
FAQ
Can robots.txt force a page out of search?
Use noindex for that. robots.txt blocks crawling, but blocked URLs can still be discovered from links.
Should API routes be blocked?
Usually yes for public marketing SEO. APIs should also return noindex headers when appropriate.
Letting AI crawlers in only helps if what they read is right. usedocs checks every help article against your code and proposes the fix.
Start a 7 day free trial