v0.6.1: Broken Link Header Fix
Fixed broken link checking to properly handle HTTP header responses, plus improved Content card details in the desktop app.
What’s New in v0.6.1
Improved Broken Link Detection
Fixed an issue where broken link checking could misidentify links as broken when HEAD requests returned unexpected content types or missing headers. The checker now handles edge cases in HTTP header responses more reliably, reducing false positives in the broken links report.
Desktop Content Card
The Content card in the desktop app now ships with a more detailed view of every crawled page. Each row shows the extracted title, byline, word count, and a preview of the Markdown body, and clicking through opens the full extracted article alongside the page metadata. Browsing through hundreds of pages of content after a crawl is now noticeably faster.
Before and after
Previously, the broken-link checker treated any non-2xx response from a HEAD request as a broken link. Some servers return non-standard status codes for HEAD (or refuse it entirely), even when a normal GET to the same URL succeeds.
HEAD /article -> 405 Method Not Allowed (flagged as broken in 0.6.0)GET /article -> 200 OK (not actually broken)In v0.6.1, the checker falls back to GET when a HEAD response looks unreliable, so links like the one above are now correctly classified as healthy.
Who Benefits
- SEO professionals get more accurate broken link reports with fewer false positives
- Content teams can review page content more easily in the desktop app
- Developers can trust broken link data in
.crawlexports for automated workflows
Related
Further reading on broken links and internal linking, plus adjacent releases:
About crawler.sh
crawler.sh is a fast Rust-based web crawler and SEO auditing tool that runs entirely on your own machine. Use the CLI for automation, scripts, and CI pipelines, or the desktop app for a visual dashboard with live crawl progress, SEO issue charts, and one-click exports.
Every release ships across both the CLI and the desktop app.
Download the latest version
or run crawler update
from the terminal to upgrade an existing install.