Thin content - pages with fewer than 200 words - rarely ranks well in search engines. These pages do not provide enough depth to satisfy search intent or demonstrate expertise. Google’s helpful content guidelines explicitly favor comprehensive, valuable content over shallow pages.

This guide shows you how to find every thin content page on your website using the crawler.sh CLI.

Step 1: Install crawler.sh CLI

Install the CLI with a single command:

curl -fsSL https://install.crawler.sh | sh

This downloads the correct binary for your operating system and architecture, places it in ~/.crawler/bin/, and adds it to your PATH. Restart your terminal or run source ~/.bashrc (or ~/.zshrc) to pick up the new PATH entry.

Verify the installation:

crawler --version

Step 2: Crawl the target website

Run a crawl with content extraction enabled:

crawler crawl https://example.com --extract-content

The --extract-content flag enables word count analysis for each page. Results are saved as an NDJSON file (.crawl) in the current directory.

Step 3: Run SEO audit

Run the SEO analysis on your crawl data:

crawler seo example-com.crawl

The short content check flags any page with fewer than 200 words of extracted content.

Step 4: Identify short content

Look for the Short Content section in the SEO report. Common types of thin pages:

Category or tag pages with only a list of links
Product pages with minimal descriptions
FAQ pages with one-sentence answers
Blog posts that were published too early or never finished
Landing pages with mostly images and little text

Step 5: Fix and re-crawl

For each flagged page, consider:

Expand the content to cover the topic more thoroughly - aim for at least 300 words on informational pages
Consolidate multiple thin pages on similar topics into one comprehensive page
Noindex pages that are intentionally brief (utility pages, thank-you pages)
Add structured content like descriptions to product and category pages

After fixing, re-crawl to verify:

crawler crawl https://example.com --extract-content
crawler seo example-com.crawl

Why short content matters for SEO

Search engines use content depth as a quality signal. Pages under 200 words struggle to rank because they cannot adequately cover a topic, include relevant keywords naturally, or demonstrate expertise. Thin content can also trigger quality filters that affect your entire site. Expanding or consolidating thin pages is one of the most effective ways to improve organic search performance.