Comparison

crawler.sh vs Firecrawl

Cloud scraping API vs local-first desktop and CLI for the same job: turning websites into clean Markdown for AI.

Firecrawl is a cloud API for crawling and converting pages to Markdown, billed per credit. crawler.sh does the same job but the binary runs on your laptop, so the marginal cost per page is zero and pages never leave your machine. Both produce Markdown that drops cleanly into a RAG or fine-tuning pipeline.

Download crawler.sh Markdown for AI

Side by side

Honest comparison on the axes that decide which tool fits your workflow.

Property

Firecrawl

crawler.sh

Where it runs

Vendor cloud, called over HTTPS.

Local binary on your laptop or server.

Pricing model

Per-credit subscription. Free tier is small; paid tiers start around $19 a month.

Free up to 1,000 pages per crawl. $99 a year unlocks 10,000 pages and Markdown archive export. Flat, not per-page.

JavaScript rendering

Headless Chrome in the cloud.

Custom JavaScript engine, no headless Chrome dependency. Auto-detected per site.

TLS fingerprint

Vendor controls.

Chrome 131 TLS fingerprint applied by default. Many sites that block headless Chrome let crawler.sh through.

Markdown output

Yes, core feature. Returned per request via API or webhook.

Yes. NDJSON stream during the crawl, or bulk Markdown archive with YAML frontmatter on Pro.

robots.txt by default

Configurable.

On by default. Disallow, Allow, and Crawl-delay honored. --ignore-robots to opt out.

Data path

Every URL round-trips through vendor infrastructure.

Pages are fetched directly from the source by your machine. Nothing routes through us.

API key required to call it

Yes.

No. Local binary.

Desktop app

No.

Yes, plus the CLI. Same engine under both.

Pricing and feature notes reflect publicly listed information at the time of writing. Vendor pricing changes; verify against the source before deciding.

When to pick which

Both tools solve real problems. Pick based on where the work actually runs and what you are billing against.

Pick Firecrawl when

You want a hosted API your backend can call from anywhere, you are fine paying per page, and you want the vendor to handle proxies, retries, and infrastructure for you. Firecrawl is good at that and is well-integrated with several AI stacks.

Pick crawler.sh when

You want crawls to run locally for cost or privacy reasons, you do not want a per-page bill, you need a desktop GUI as well as a CLI, or you want a flat-fee tier that covers a large crawl without surprises. crawler.sh ships with politeness, robots.txt support, and JavaScript rendering built in.

Try the local-first path

Install in one command. Crawl any site into clean Markdown in seconds. Free up to 1,000 pages, $99 a year for 10,000.

Download Read more