Free Robots.txt Generator By
Mansoor Bhanpurawala
Global Site Settings
Site MetadataUser-Agent Directives
Define search engine specific access permissions
Quick Add Agent:
No user-agent blocks configured
Add a user-agent block to specify crawl permissions.
Directives & Syntax Cheat Sheet
Quick Reference
User-agent:
Defines which web robot the following rules apply to. Use * for all bots.
Disallow:
Specifies directory or URL paths crawlers must not request. Must start with /.
Allow:
Overrides Disallow rules for subfolders (e.g. Allow `/wp-admin/admin-ajax.php`).
Sitemap:
Absolute URL to XML sitemap file. Place anywhere in file (usually at bottom).
robots.txt
0 lines
0 chars
0.0 KB / 500 KB
Syntax & Health Inspector
Passed
Keyboard Shortcuts
Copy Output
Ctrl + Shift + C
Download File
Ctrl + S
Undo / Redo
Ctrl + Z / Y