v0.8.1: Model Context Protocol Server Release
New Model Context Protocol server for Claude Code, Cursor, Zed, and other MCP clients. Crawl, fetch, and discover links from chat.
What’s New in v0.8.1
This release ships a Model Context Protocol (MCP) server that exposes crawler.sh as a set of tools inside any MCP-compatible client. You can now ask Claude Code to crawl a whole site, fetch a single page, or discover links without leaving the terminal. The server runs locally and reuses the same Rust engine that powers the CLI and desktop app.
MCP Server
The MCP server is a standalone binary called crawler-mcp. It speaks JSON-RPC 2.0 over stdio and exposes three tools:
crawl_site- Crawl an entire domain and return every page with its URL, status code, title, markdown content, word count, and crawl depth.fetch_page- Fetch a single page and return the same fields ascrawl_site.discover_links- Return only the URLs found on a page, without fetching the full body. Useful for mapping site structure quickly.
All three tools respect robots.txt by default and honor the same tier-based page limits as the CLI.
Installation
Install the MCP server with a single command:
curl -fsSL https://install.crawler.sh/install-mcp.sh | shThe script detects your OS and architecture, downloads the matching binary, and places it on your PATH. There is no separate updater yet; rerun the install script to update.
How It Works
The server is built as a thin shim over crawler-core. It constructs a single shared JS rendering engine at startup so long-running MCP sessions do not leak QuickJS runtimes. The hand-rolled JSON-RPC 2.0 implementation means the build has zero external MCP SDK dependencies - protocol version drift in upstream crates cannot break the binary.
Tier Limits
The same page caps apply inside MCP clients:
| Tier | Max pages per crawl |
|---|---|
| Not signed in | 50 |
| Signed in (free) | 400 |
| Pro subscriber | 10,000 |
Set the CRAWLER_TOKEN environment variable to raise the cap. Free and Pro limits are enforced locally.
These caps were updated in v0.8.2 (the 0.8.1 release shipped with 600 / 1,000 / 10,000). See v0.8.2 for the rationale.
Supported Clients
The server has been tested with Claude Code, Claude Desktop, Cursor, Zed, and Opencode. Setup instructions for each client are in the documentation.
Related
About crawler.sh
crawler.sh is a fast Rust-based web crawler and SEO auditing tool that runs entirely on your own machine. Use the CLI for automation, scripts, and CI pipelines, or the desktop app for a visual dashboard with live crawl progress, SEO issue charts, and one-click exports.
Every release ships across both the CLI and the desktop app.
Download the latest version
or run crawler update
from the terminal to upgrade an existing install.
