Robots.txt Generator
Pick what to block, add your sitemap, copy the file. Includes an option to block common AI training bots.
How to use the generator
Choose a default rule, list paths to block, add any exceptions and your sitemap URL. Copy the output into a plain text file named robots.txt and upload it to your site root.
Blocking AI training bots does not affect Google Search rankings, because Google-Extended is a separate control from Googlebot. It only reflects what each company says it respects. Not every crawler obeys robots.txt.
Test the file before and after upload with the robots.txt checker, and in Google Search Console.
Want this done for your site?
Send your URL on WhatsApp and I will tell you what to fix first.
Common questions
What is a robots.txt file?
A text file at the root of your site that tells crawlers which paths they may fetch.
Does robots.txt remove pages from Google?
No. It blocks crawling. To keep a page out of results, use a noindex tag and let Google crawl it so it sees the tag.
Should I block AI bots?
That is a business choice. Blocking cuts the chance of your content being used for training. It may also reduce your visibility in some AI answers.
Where does the file go?
At the root: yourdomain.com/robots.txt. A file in a subfolder is ignored.
Leave a Reply