Robots.txt Checker

What is robots.txt?

robots.txt is a file that tells search engine crawlers which pages or sections of a website they can or cannot access. It's located at the root of a website (e.g., example.com/robots.txt) and uses simple directives to control crawler behavior. robots.txt helps manage crawler traffic and protect sensitive areas of your website.

Our free Robots.txt Checker fetches and analyzes robots.txt files. It extracts user agents, disallow rules, allow rules, and sitemap locations. This helps verify that your robots.txt is configured correctly and accessible to search engines.

Robots.txt Directives

• User-agent: Specifies which crawler the rules apply to (* for all)
• Disallow: Blocks crawlers from specific paths
• Allow: Allows crawlers to access specific paths
• Sitemap: Specifies sitemap location

Example robots.txt

User-agent: *
Disallow: /admin/
Disallow: /private/
Allow: /public/

Sitemap: https://example.com/sitemap.xml

Best Practices

• Place at Root: robots.txt must be at domain root
• Use Correct Format: Follow robots.txt syntax rules
• Test Thoroughly: Verify rules work as expected
• Include Sitemap: Add sitemap location
• Don't Block Important Pages: Avoid blocking indexable content

FAQs

Where should robots.txt be located?

robots.txt must be at the root of your domain: https://example.com/robots.txt

Does robots.txt block search engines?

robots.txt is a request, not a command. Well-behaved crawlers respect it, but it's not a security measure.

Can I use wildcards in robots.txt?

Yes, you can use * for user-agent (all crawlers) and some patterns in paths, though support varies.

Should I include sitemap in robots.txt?

Yes, including sitemap location in robots.txt helps search engines discover your sitemap.