SEO Tool
Robots.txt Analyzer
Analyze the robots.txt file at any domain root. Check for sitemap declarations, crawl-delay directives, and specific bot rules to ensure search engines can find your content.
How to use the Robots.txt Analyzer
Your robots.txt file controls which pages search engines are allowed to crawl — one wrong “Disallow” line can quietly hide your entire site from Google. This analyzer breaks down the rules at your domain root so you can spot mistakes and confirm your sitemap is referenced.
- 1Enter any domain — the tool checks /robots.txt automatically.
- 2Review the allowed and disallowed paths per user-agent.
- 3Check for a sitemap reference and any crawl-delay rules.
- 4Loosen any rule that blocks pages you want indexed.
Frequently asked questions
What is a robots.txt file?
robots.txt is a plain-text file at your domain root that tells search-engine crawlers which parts of your site they may or may not crawl. It can also point crawlers to your sitemap.
What does this robots.txt analyzer check?
It fetches the robots.txt at the domain root and breaks down its user-agent rules, allowed and disallowed paths, crawl-delay directives, and any sitemap references, so you can spot rules that accidentally block important pages.
Can robots.txt stop my pages from ranking?
Yes — if you disallow a path, crawlers will not fetch those pages, which can keep them out of search results. A misconfigured robots.txt is a common reason pages silently disappear from Google.
Is this tool free?
Yes. Paste any domain and analyze its robots.txt for free, no account needed.