Question 1

How do I install crawler.sh?

Accepted Answer

Install the CLI with a single command: curl -fsSL https://install.crawler.sh | sh. The desktop app is available as a direct download from the product page. See the documentation for full details.

Question 2

What platforms does crawler.sh support?

Accepted Answer

The CLI runs on macOS (Apple Silicon and Intel) and Linux (x86_64). The desktop app is available for macOS (Apple Silicon and Intel) and Linux. Windows support is on the roadmap.

Question 3

Do I need to create an account?

Accepted Answer

No account is required to use the CLI or desktop app. Signing in unlocks a higher page limit (1,000 pages per crawl vs 600 for free). Everything runs locally on your machine.

Question 4

What subcommands does the CLI provide?

Accepted Answer

The CLI has 4 subcommands: crawl (crawl a website and save results), info (inspect and analyze a .crawl file), export (convert a .crawl file to JSON or Sitemap XML), and seo (analyze SEO issues across 23 check categories).

Question 5

What output formats are supported?

Accepted Answer

The default crawl output is NDJSON (.crawl extension) — streamable, one JSON object per line. You can also export to JSON arrays, W3C-compliant Sitemap XML, SEO CSV (two columns: Issue Type and URL), and SEO TXT (human-readable grouped report).

Question 6

How does content extraction work?

Accepted Answer

When enabled (on by default), the crawler extracts the main article content from HTML pages and converts it to clean Markdown. The result includes the Markdown text, word count, author byline, and excerpt. Use --no-extract to disable it for faster crawls.

Question 7

What does the SEO analysis check?

Accepted Answer

The seo subcommand checks 23 categories: missing/short/long titles, missing/short/long meta descriptions, missing/short/long content, long URLs, noindex pages, nofollow pages, non-self canonicals, paginated pages, duplicate titles, duplicate descriptions, missing/multiple/empty/long/short/duplicate H1 tags, and broken outgoing links. Only 2xx HTML pages are analyzed.

Question 8

What does the desktop app include?

Accepted Answer

The desktop app features 8 interactive dashboard cards: Live Feed (real-time crawled URLs), SEO Issues (automated analysis), Page Status (status code charts), Settings (crawl configuration), Downloads (export results), Content (Markdown viewer), Newsletter (email signup), and Premium (upgrade prompt).

Question 9

How is the desktop app different from the CLI?

Accepted Answer

Both share the same crawling engine. The CLI is designed for automation and scripting (pipe output, run in CI, etc.), while the desktop app provides a visual dashboard with charts, interactive cards, and real-time crawl monitoring.

Question 10

What is included in the premium tier?

Accepted Answer

The premium tier ($99 / year) unlocks the Content Archive export — full page content as clean Markdown files in a ZIP archive. All other features (crawling, SEO analysis, JSON/Sitemap export) are available in the free tier.

Question 11

Where can I get help with crawler.sh?

Accepted Answer

Reach out via email at hello@crawler.sh for any questions or support. The documentation covers installation, usage, and all CLI options.

Frequently Asked Questions

How do I install crawler.sh?

What platforms does crawler.sh support?

Do I need to create an account?