robots unreadable

warning robots-unreadable robots.txt

A robots.txt that cannot be read is not a permissive robots.txt — crawler behaviour in that situation is not something you get to choose. An unreadable robots.txt is treated differently by different crawlers: some assume everything is allowed, others back off entirely. The problem is not which of those you get — it is that you do not get to choose, and the behaviour can differ between engines on the same day.

What indexgap reports

The finding reads:

warning  robots-unreadable  robots.txt was not read: …

How to fix it

Put robots.txt in the site root, or pass its path with --robots. While the file is missing, the crawler-access checks have not run — which does not mean access is open.

How it behaves at scale

This is a site-level finding: it belongs to the site, not to any one page, and is reported once however many pages are audited.

Finding it

pipx install indexgap
cd your-project
indexgap init
indexgap check

Then indexgap brief turns every finding on this page into a work order in indexgap-briefs/, with the thresholds the result has to satisfy and the command to check it.

Related checks in robots.txt

All checks · robots.txt