v0.1.6: Crawl History & Download Page
Browse past crawls by site in the desktop app, download Crawler from a dedicated page, and explore new documentation.
What’s New in v0.1.6
Crawl History by Site
The desktop app now groups and displays your past crawls organized by site. Instead of scrolling through a flat list of crawl files, you can browse by domain and quickly locate results from previous sessions. Each entry shows the date, page count, and crawl duration so you can compare runs over time. This is especially useful when monitoring the same site on a regular basis - for example, tracking how a site’s SEO health changes after deploying content updates or fixing broken links.
Download Page
A dedicated download page at crawler.sh/download makes it easy to get started. The page automatically detects your operating system and architecture, then presents the correct download option - whether that’s the macOS desktop app, a Linux .deb package, or the CLI install command. No more guessing which binary to grab.
Authentication Flow Update
Switched from magic link to auth code verification for a faster, more reliable sign-in experience. The previous magic link approach had delivery delays with some email providers and occasionally landed in spam folders. The new flow is straightforward: enter your email, receive a short numeric code, and paste it in. Authentication completes in seconds and works consistently across all email providers.
Documentation Updates
- Expanded CLI and desktop documentation with detailed usage examples for every subcommand
- Added structured data (schema.org) to web pages for better search engine visibility
- Improved getting-started guides for first-time users
Related
Wrap-up
A CMS shouldn't slow you down. Crawler aims to expand into your workflow — whether you're coding content models, collaborating on product copy, or launching updates at 2am.
If that sounds like the kind of tooling you want to use — try Crawler .
