Roadmap

In Progress
Q1 2026

Windows support

Bring the CLI and desktop app to Windows with native builds and installer support.

Core TeamCore Team
In Progress
Q1 2026

Crawl diffing

Compare two .crawl files to detect added, removed, and changed pages between crawl runs.

Core TeamCore Team
Planned
Q2 2026

Custom crawl rules

Define include/exclude URL patterns, respect robots.txt, and configure custom request headers.

Core TeamCore Team
Planned
Q3 2026

API server mode

Run crawler.sh as a local HTTP server with REST endpoints for triggering crawls and retrieving results programmatically.

Core TeamCore Team
Shipped
February 2026

SEO analysis CLI

The `crawler seo` subcommand with 16 check categories, CSV and TXT export, and canonical-aware duplicate detection.

Core TeamCore Team
Shipped
Feb 2026

Content extraction

Content extraction with Markdown output, word count, byline, and excerpt for every crawled page.

Core TeamCore Team
Shipped
Feb 2026

Desktop app v0.1

Native desktop app with 8 dashboard cards, real-time crawl feed, SEO analysis panel, status code charts, and multi-format export.

Core TeamCore Team
Shipped
Feb 2026

CLI v0.1

Initial release with crawl, info, and export subcommands. NDJSON, JSON, and Sitemap XML output formats. Configurable concurrency, depth, and delay.

Core TeamCore Team

Community engagement

We build in public. Want to request something or upvote a feature? Join the conversation on:

Crawler.sh - SEO Spider, Technical SEO, Markdown Content Extractor for AI | Product Hunt