ai crawler blocked

The crawlers behind AI answers are blocked in robots.txt. That may well be deliberate — the point is that it should be a decision, taken knowingly, rather than a line inherited from a template.

What indexgap reports

The finding reads:

critical  ai-crawler-blocked  OAI-SearchBot is closed: … (critical for the AI-search crawlers, info for training crawlers)

How to fix it

How it behaves at scale

This is a site-level finding: it belongs to the site, not to any one page, and is reported once however many pages are audited.

Finding it

pipx install indexgap
cd your-project
indexgap init
indexgap check

Then indexgap brief turns every finding on this page into a work order in indexgap-briefs/, with the thresholds the result has to satisfy and the command to check it.

Related checks in ai search and answer extraction

• no-question-headingsno question headings
• no-answerno answer
• answer-shortanswer short
• answer-longanswer long
• answer-preambleanswer preamble
• jsonld-brokenjsonld broken
• jsonld-no-typejsonld no type
• jsonld-faq-invisiblejsonld faq invisible
• no-authorno author
• no-dateno date

All checks · AI search and answer extraction