Overview - Getting Started with crawler.sh

What is crawler.sh?

crawler.sh is a fast, concurrent web crawler. It provides a CLI tool, a native desktop app, and an upcoming cloud API for crawling websites, extracting content, analyzing SEO issues, and exporting results in multiple formats.

Key capabilities:

• Concurrent crawling with configurable depth, page limits, and delay
• Content extraction - converts HTML to clean Markdown
• SEO analysis - 16 automated checks for common issues
• Multiple output formats - NDJSON, JSON, Sitemap XML, SEO CSV/TXT
• Real-time progress - event-driven architecture with live streaming

Products

Crawler CLI

Command-line interface with four subcommands: crawl, info, export, and seo. Features progress bars, color-coded output, and auto-generated filenames. Get started →

Crawler Desktop

Native desktop application with an interactive dashboard. Eight cards for live feed, SEO analysis, status charts, content browsing, and exports. Get started →

Crawler Cloud

Hosted crawling API with scheduled crawls, webhooks, and a web dashboard. Coming soon. Learn more →

Quick Start

CLI

1. Install the CLI

   curl -fsSL https://install.crawler.sh | sh

   Downloads the correct binary for your platform and adds it to your PATH.

2. Run your first crawl

   crawler crawl https://example.com

   This crawls the site (up to 100 pages by default) and saves results to example-com.crawl.

3. Inspect the results

   crawler info example-com.crawl

   View summary statistics: page count, status code distribution, and response times.

4. Analyze SEO issues

   crawler seo example-com.crawl

   Run 16 automated SEO checks and see issues grouped by category.

Desktop

1. Download the app

   Download the macOS DMG (universal binary - works on both Apple Silicon and Intel).

2. Install

   Open the .dmg file and drag Crawler to your Applications folder.

3. Start a crawl

   Launch Crawler, enter a URL in the top input bar, configure settings, and click Start Crawl.

4. Explore results

   Browse the dashboard cards: Live Feed for real-time progress, SEO Issues for automated analysis, Content for extracted Markdown, and Downloads for exports.

Next Steps

• Installation guide - install, uninstall, and system requirements
• Crawler CLI - command-line workflows, examples, and tutorials
• Crawler Desktop - dashboard walkthrough and features
• Crawler Cloud - upcoming cloud API