missing content
Pages with no extractable content provide no value to search engines or users
What it is
crawler.sh flags a page as having missing content when the content extraction process finds no readable text - the word_count field is null. This is check #7 in the SEO analysis.
Note: This check requires content extraction to be enabled. If you crawled with
--no-extract, content checks won’t run.
Why it matters for SEO
Pages with no extractable content are problematic for search engines:
- Nothing to index - Search engines can’t rank a page if there’s no text content to analyze for relevance.
- Thin content penalties - Sites with many contentless pages may be flagged for thin content, lowering overall site quality scores.
- Crawl budget waste - Search engines spend resources crawling pages that offer no indexable value.
Why it matters for AEO
AI answer engines rely on page content to generate answers. A page with no extractable text cannot be used as a source, regardless of how well its title and description are optimized. Pages that are purely visual (images, videos, or interactive elements without text) are invisible to AI systems that primarily process text.
How to fix it
Identify why the page has no extractable content:
- JavaScript-rendered content - If the page relies entirely on client-side rendering, the crawler (and search engines) may see an empty page. Add server-side rendering or pre-rendering.
- Image-only pages - Pages with content embedded in images should include text alternatives, captions, or transcriptions.
- Redirect or error pages - These legitimately have no content. Consider whether they should exist as crawlable pages.
- Login-gated pages - If content is behind authentication, it’s expected to have no extractable content for anonymous crawlers.
What crawler.sh reports
In the CLI, missing content pages appear under the “Missing content” section of crawler seo output. Each affected URL is listed. In the desktop app, they appear in the SEO Issues card.
