Comparison

crawler.sh vs Playwright

Single-binary crawler with built-in orchestration vs a browser automation library you wire yourself.

Playwright is a browser automation library for testing and scraping. It gives you fine-grained control over headless Chrome but you write the crawl loop, the politeness logic, and the output formatting yourself. crawler.sh is a dedicated crawler: BFS traversal, robots.txt handling, and Markdown export are built in. Both can render JavaScript; crawler.sh does it without a separate Chrome process.

Side by side

Honest comparison on the axes that decide which tool fits your workflow.

What it is
Browser automation library for testing and scraping.
Dedicated web crawler with built-in orchestration and export.
Setup
Node.js or Python install, plus browser binaries.
One command installs a single static binary.
JavaScript rendering
Real headless Chrome. One process per browser context.
Custom JavaScript engine. No separate Chrome process.
Memory footprint
High: each page may spin up a Chrome renderer.
Low: in-process engine with a smaller footprint.
Crawl orchestration
You write the BFS loop, deduplication, and rate limiting.
Built-in BFS, depth limits, concurrency control, and adaptive backoff.
SEO analysis
None. You parse and analyze HTML yourself.
Built-in: titles, meta descriptions, h1, redirects, broken links, and more.
Output
Raw HTML or page screenshots. You format the rest.
NDJSON, sitemap XML, or Markdown archive with YAML frontmatter out of the box.
robots.txt / politeness
None by default. You implement it yourself.
On by default. Per-host adaptive backoff on 429 and 403.
Desktop app
No.
Yes.

Feature notes reflect publicly listed information at the time of writing.

When to pick which

Both tools solve real problems. Pick based on where the work actually runs and what you are billing against.

Pick Playwright when

You need to interact with pages (clicking, form filling, taking screenshots), you are building a test suite, or you want full programmatic control over a real browser for complex scraping logic.

Pick crawler.sh when

You want to crawl a whole site without writing a loop, you need SEO analysis and Markdown export built in, you want lower memory usage than headless Chrome, or you need a desktop app for non-engineers.

Try the local-first path

Install in one command. Crawl any site into clean Markdown in seconds. Free up to 1,000 pages, $99 a year for 10,000.

Crawler.sh - Free Local AEO & SEO Spider and a Markdown content extractor | Product Hunt