Robots.txt Tester
Check what your robots file allows and blocks, because one wrong line is enough to hide an entire site.
- ✓ Test any URL
- ✓ Any crawler, including AI ones
- ✓ Wildcards handled
- ✓ Free, no account
The robots file tells crawlers where they may go. It is a few lines of plain text and it is entirely capable of removing a whole site from search with one of them, which is why it is worth reading rather than assuming.
Catch a blanket block
The staging rule that shipped with the site is the classic one.
Test the pages that matter
Home, services, location pages, individually.
Check AI crawlers too
They follow their own rules and their own names.
Test a URL against your robots file
Enter a URL and a crawler to see whether it is allowed through.
Matching follows Google’s rules (wildcards *, end-anchor $, most-specific rule wins). Free tool.
How It Works
Load your robots file
Read what is actually there, not what you remember.
Test the important URLs
One at a time, starting with the pages that earn.
Look for a leftover disallow
That single line explains most cases.
What blocking really does
It stops a page being crawled. It does not reliably keep it out of results. A blocked page can still be listed, with no description under it, which is the worst of both outcomes. Use noindex when you want something genuinely excluded.
The mistake that keeps happening
A development site carries a rule blocking everything, so it stays out of search while it is being built. Then it goes live and the rule goes with it. The site looks perfect, nothing gets indexed, and the cause sits in a file nobody has opened since launch.
When to check
After any site move, redesign or platform change, without waiting for a reason to suspect something. It takes two minutes and occasionally explains months of confusion that everyone had attributed to something else.
Explore More Free Tools & Services
Frequently Asked Questions
Can this remove a page from Google?
Not reliably. It blocks crawling. A blocked page can still appear without a description. Noindex is the right instrument for exclusion.
What is the most common mistake?
A disallow all rule carried over from a staging site. It hides everything and nothing else looks wrong.
Do I need a robots file?
Not necessarily. Having none is safer than having a wrong one.
Should I block AI crawlers?
That is a business decision rather than an SEO one. Blocking them keeps you out of answers those tools give about you.
Two minutes, occasionally decisive
Most robots files are fine. The ones that are not tend to be quietly catastrophic, and nothing else on the site looks wrong while it is happening.
Browse All Free Tools →Check My RankingRobots.txt Tester
The robots file tells crawlers where they may go. It is a few lines of text and it is capable of removing a whole site from search with one of them.
The classic failure is a staging site rule that goes live with the site. Everything looks fine, nothing gets indexed, and the cause sits in a file nobody has opened since launch.
It is worth being clear about what it does not do. Blocking a page stops it being crawled, not indexed. A blocked page can still appear in results, with no description, which is the worst of both.
How to use it
- Load your robots file and read it as it stands.
- Test the URLs that matter. Home, services, location pages.
- Look for a blanket disallow rule left over from development.
- Use noindex, not robots, for pages you want kept out of results.
Why it matters
This is a two minute check that occasionally explains months of confusion. It is worth doing after any site move or redesign, without waiting for a reason.
Questions people ask
Can robots.txt remove a page from Google?
Not reliably. It blocks crawling. A blocked page can still be listed without a description. Use noindex to keep something out.
What is the most common mistake?
A disallow all rule from a staging site shipped to production. It hides everything.
Do I need one at all?
Not necessarily. An absent file is safer than a wrong one.