v0.2.5: Content Extraction Fixes & SEO Glossary

What’s New in v0.2.5

Content Extraction Improvements

Fixed several edge cases in the content extraction pipeline where certain HTML structures would produce broken or incomplete markdown output. Pages with complex nested layouts, tables, and embedded media now convert more reliably.

SEO Glossary

The documentation now includes an SEO glossary covering all 18 check categories in the SEO audit - from missing titles and short descriptions to noindex pages and non-self canonicals. Each entry explains what the issue is, why it matters, and how to fix it.

Bug Fixes

• Discovered links now followed correctly. Earlier versions could skip over some internal links surfaced during the initial discovery pass, leaving sub-pages out of the crawl. The link-tracking logic was tightened so every discovered URL is queued and visited as expected (see v0.2.4: Improved Link Discovery Tracking).
• Updated llms.txt reference documentation. The reference page describing how crawler.sh detects, validates, and reports on /llms.txt and /llms-full.txt was rewritten to match the current behavior, with clearer examples of valid file structures.

How to upgrade

Existing CLI users can upgrade in place by running crawler update from any terminal. Desktop users will be prompted to update on next launch, or you can grab the latest build from the download page.

Related

These resources cover the same content and link-discovery topics this release improved:

• Thin content
• Web crawler
• v0.4.0: Better Content Extraction
• How to find short content