Install the Crawler MCP Server

crawler-mcp is a local binary. Once installed it lives in ~/.crawler/bin/, alongside the crawler CLI, so they share one PATH entry.

Install

curl -fsSL https://install.crawler.sh/install-mcp.sh | sh

Downloads the correct binary for your platform and drops it at ~/.crawler/bin/crawler-mcp.

CRAWLER_INSTALL_DIR=/opt/crawler curl -fsSL https://install.crawler.sh/install-mcp.sh | sh

Set CRAWLER_INSTALL_DIR before piping into sh to install somewhere other than ~/.crawler.

Pick the binary for your platform:

macOS Apple Silicon: crawler-mcp-macos-arm64
macOS Intel: crawler-mcp-macos-x64
Linux ARM64: crawler-mcp-linux-arm64
Linux x64: crawler-mcp-linux-x64

Then:

chmod +x crawler-mcp-*
mv crawler-mcp-* ~/.crawler/bin/crawler-mcp

Verify

which crawler-mcp

Should print an absolute path. The server only speaks JSON-RPC over stdio, so running it directly will look idle. That is correct: it waits for a client to talk to it.

Supported platforms

macOS 12+ (Apple Silicon and Intel)
Linux x64 and ARM64 (glibc 2.31+)

Windows is not yet shipped as a precompiled binary. If you build from source via make build-mcp it should work on WSL.

Wire it into a client

Pick the client you use:

Each client wants an absolute path to the binary. Run which crawler-mcp to print the value you need.

Tier caps

The server enforces a hard cap on pages per crawl, mirroring the crawler.sh tier rules:

No CRAWLER_TOKEN env var set: 50 pages per crawl.
CRAWLER_TOKEN set, free account: 400 pages per crawl.
CRAWLER_TOKEN set, Pro subscriber: 10,000 pages per crawl.

Set the token in the env block of your client config:

{
  "mcpServers": {
    "crawler-sh": {
      "command": "/Users/you/.crawler/bin/crawler-mcp",
      "env": { "CRAWLER_TOKEN": "your-token" }
    }
  }
}

The MCP server verifies your tier with the API once at startup, so the cap reflects your actual subscription state with no per-request network hop.

Politeness defaults

robots.txt Disallow rules are honored by default. Pass respect_robots: false to a tool call only for sites you own or have explicit authorization to crawl without restriction.
Crawl-delay is always honored, regardless of respect_robots.
Per-host adaptive backoff: a 429, 403, or 503 response doubles the host’s delay (starting at 250 ms, capped at 10 s). Five consecutive successes halve it back down.

Update

The MCP binary is not part of the signed version.json updater manifest yet. To upgrade, re-run the install script:

curl -fsSL https://install.crawler.sh/install-mcp.sh | sh

Uninstall

rm ~/.crawler/bin/crawler-mcp

Then remove the crawler-sh entry from each client config file you added.