robots blocks page

critical robots-blocks-page robots.txt

A page in the build that robots.txt does not let Googlebot crawl. The rules are matched the way Google matches them: the agent's own group beats `*`, the longest matching rule wins, `Allow` wins a tie, `*` and `$` work. Such a page cannot be read, so its text, canonical and even its noindex are invisible — it can still be indexed as a bare URL from links. Pages already closed with noindex and a whole-site block (reported once as `robots-blocks-all`) are not repeated here.

What indexgap reports

The finding reads:

critical  robots-blocks-page  robots.txt does not let the search engine into the page (rule “Disallow: …”) — it will not read the text; the links and the sitemap lead here for nothing

How to fix it

Find the Disallow rule that closes this section and decide: if the section belongs in search, remove the rule; if not, take the pages out of the sitemap and put noindex on them (the engine sees it only if crawling is allowed).

How it behaves at scale

Collapsed when it fires template-wide. If it appears on 90% or more of the pages — or of one language version — of a site with at least 20 pages, it is a property of the template and is reported once, with the count, instead of once per page.

Finding it

pipx install indexgap
cd your-project
indexgap init
indexgap check

Then indexgap brief turns every finding on this page into a work order in indexgap-briefs/, with the thresholds the result has to satisfy and the command to check it.

Related checks in robots.txt

All checks · robots.txt