robots blocks all

critical warning robots-blocks-all robots.txt

Disallow: / for every agent closes the entire site to every search engine. It is almost always a staging file that was deployed to production. This is the single most expensive finding in the package, and the easiest to miss: the site keeps building, deploying and passing every other check while earning nothing at all. It is reported once, at site level, because there is exactly one file to fix and reporting it per page would bury it under everything else.

What indexgap reports

The finding reads:

critical  robots-blocks-all  Disallow: / for every agent except … — other search engines and AI search cannot see the site

How to fix it

Remove `Disallow: /` — right now the site is closed to every search engine.

How it behaves at scale

This is a site-level finding: it belongs to the site, not to any one page, and is reported once however many pages are audited.

Finding it

pipx install indexgap
cd your-project
indexgap init
indexgap check

Then indexgap brief turns every finding on this page into a work order in indexgap-briefs/, with the thresholds the result has to satisfy and the command to check it.

Related checks in robots.txt

All checks · robots.txt