Crawl, Analyze & Export Any Website in Seconds

Desktop app and CLI for crawling websites blazing fast, analyzing SEO issues, visualizing HTTP status codes, and exporting results as JSON, Sitemap XML, or Markdown.

crawler.sh desktop app overview

Site
Crawling

Crawl entire sites in seconds.

Crawl any website while staying within the same domain. Configurable concurrency, depth limits, and polite delay between requests — fast enough for thousands of pages.

CLI crawling

Content Extraction

Readable content as clean Markdown.

Extract the main article content from any page and convert it to clean Markdown automatically. Includes word count, author byline, and excerpt for every page.

Content Extraction

SEO Analysis

16 automated checks on every page.

Detect missing titles, duplicate meta descriptions, noindex directives, thin content, long URLs, and more. Export issues as CSV or TXT for your team.

SEO Analysis

Multiple Output Formats

NDJSON, JSON, Sitemap XML, and more.

Stream results as NDJSON during the crawl, or export to JSON arrays and W3C-compliant Sitemap XML. SEO reports export as CSV or human-readable TXT.

Dashboard overview

Output Formats

Export crawl data in the format you need:

Loading...

Built for Every Workflow

SEO Auditing

Run 16 automated checks across every page — find missing titles, duplicate descriptions, thin content, and more before they hurt your rankings.

Content Archiving

Extract readable content from any website as clean Markdown. Perfect for backups, migrations, or feeding content into other tools.

Sitemap Generation

Generate W3C-compliant Sitemap XML from a live crawl. Keep your sitemaps accurate and up to date without manual maintenance.

Site Monitoring

Crawl your site regularly to catch broken links, missing pages, and status code changes before your visitors do.

Crawl any website, find every issue, and export the data you need — all from your own machine.

crawler.shFast, local-first, and privacy-friendly
Crawler.sh - SEO Spider, Technical SEO, Markdown Content Extractor for AI | Product Hunt