v0.3.0: Smart Retry and Exponential Backoff
Crawler now automatically retries failed requests with exponential backoff and uses browser-like TLS fingerprints for better compatibility with protected sites.
What’s New in v0.3.0
Crawling real-world sites means dealing with flaky responses - rate limits, temporary server errors, and connection resets. Previously, Crawler treated every failure as final and moved on.
Now, failed requests are automatically retried with exponential backoff. This means transient 429 (Too Many Requests) and 5xx errors are handled gracefully without you having to re-run the entire crawl.
Retry Behavior
- Failed requests are retried up to 3 times
- Each retry waits progressively longer before trying again
- Only transient errors trigger retries - permanent failures like 404 are skipped immediately
Browser-Like Request Headers
Requests now use browser-compatible TLS fingerprints and Chrome Browser headers. This significantly improves success rates on sites that block non-browser traffic based on TLS handshake characteristics.
Who Benefits
- SEO professionals crawling client sites behind CDNs with aggressive bot protection
- Content teams archiving sites that rate-limit automated requests
- Developers testing against staging environments with intermittent connectivity
Related
About crawler.sh
crawler.sh is a fast Rust-based web crawler and SEO auditing tool that runs entirely on your own machine. Use the CLI for automation, scripts, and CI pipelines, or the desktop app for a visual dashboard with live crawl progress, SEO issue charts, and one-click exports.
Every release ships across both the CLI and the desktop app.
Download the latest version
or run crawler update
from the terminal to upgrade an existing install.
