Robots.txt Checker
Paste your robots.txt, pick a bot and a URL. See whether it is allowed, which rule decided it, and any syntax problems.
How the checker decides
It finds the group that best matches the bot name, falling back to User-agent: *. Then it tests every Allow and Disallow rule against the path. The longest matching rule wins. If an Allow and a Disallow tie, Allow wins.
Paste the file contents. This tool cannot fetch your live file from another domain because browsers block that. Open yourdomain.com/robots.txt, copy, and paste.
Want this done for your site?
Send your URL on WhatsApp and I will tell you what to fix first.
Common questions
Why paste instead of entering my domain?
Browsers block pages from reading other sites files. Pasting keeps your data in your browser and avoids a server.
What does Disallow: with nothing after it mean?
It blocks nothing. The bot may crawl everything.
Do wildcards work?
Google supports * for any characters and $ for end of URL. Not every crawler does.
Can I trust this over Search Console?
Use both. Search Console shows what Google actually fetched.
Leave a Reply