v0.1.8: Pro Subscriptions & Content Archive
Introducing Pro subscriptions with 10,000 page crawl limits and Content Archive - download all extracted page content as a Markdown ZIP.
What’s New in v0.1.8
Pro Subscriptions
Crawler now offers a Pro tier at $99/year. Pro unlocks:
- 10,000 pages per crawl (up from 1,000 for free signed-in users)
- Content Archive export - download all extracted page content as a ZIP of Markdown files
Free users still get generous limits: 1,000 pages per crawl when signed in, or 600 pages without an account.
Content Archive (Pro)
After crawling a site, Pro users can export all extracted content as a ZIP file. Each page becomes a Markdown file, making it easy to:
- Archive website content for offline reference
- Feed content into other tools and workflows
- Create backups of important pages
Authentication
Sign in with your email using a one-time password - no traditional passwords to manage. Authentication works in both CLI and desktop:
- CLI:
crawler auth login- credentials stored locally - Desktop: Sign in via the Account card - stored in your browser
Bug Fixes
- Fixed URL resolution for crawl starting URLs
- Improved dark mode consistency in the desktop app
Related
About crawler.sh
crawler.sh is a fast Rust-based web crawler and SEO auditing tool that runs entirely on your own machine. Use the CLI for automation, scripts, and CI pipelines, or the desktop app for a visual dashboard with live crawl progress, SEO issue charts, and one-click exports.
Every release ships across both the CLI and the desktop app.
Download the latest version
or run crawler update
from the terminal to upgrade an existing install.
