Free Robots.txt Generator
General Settings
Configure basic robots.txt settings
Include Comments
Robot Rules
Define rules for different user agents
Sitemaps
Add sitemap URLs for search engines
Generated Robots.txt
Preview and download your robots.txt file
About Robots.txt
The robots.txt file tells search engine crawlers which URLs they can access on your site. It's a simple text file placed in the root of your website that helps control crawler traffic and prevents indexing of sensitive or duplicate content. Support for all major directives including user agents, crawl delays, sitemaps, and custom rules.