Roadmap
Windows support
Bring the CLI and desktop app to Windows with native builds and installer support.
Crawl diffing
Compare two .crawl files to detect added, removed, and changed pages between crawl runs.
Custom crawl rules
Define include/exclude URL patterns, respect robots.txt, and configure custom request headers.
API server mode
Run crawler.sh as a local HTTP server with REST endpoints for triggering crawls and retrieving results programmatically.
SEO analysis CLI
The `crawler seo` subcommand with 16 check categories, CSV and TXT export, and canonical-aware duplicate detection.
Content extraction
Content extraction with Markdown output, word count, byline, and excerpt for every crawled page.
Desktop app v0.1
Native desktop app with 8 dashboard cards, real-time crawl feed, SEO analysis panel, status code charts, and multi-format export.
CLI v0.1
Initial release with crawl, info, and export subcommands. NDJSON, JSON, and Sitemap XML output formats. Configurable concurrency, depth, and delay.