Use Crawler MCP with Claude Code

Claude Code is Anthropic’s terminal agent. Once crawler-mcp is installed (guide), wire it in with one command.

Quick add

Register the server
Section titled “Register the server”
```
claude mcp add crawler-sh -- $(which crawler-mcp)
```
claude mcp add writes the entry to Claude Code’s config for you. The -- separates Claude Code’s flags from the command to run.
Verify
Section titled “Verify”
```
claude mcp list
```
You should see crawler-sh in the list. Open a new Claude Code session and the three tools (crawl_site, fetch_page, discover_links) will show up under /mcp.

Manual config

If you prefer to edit JSON directly, or want to commit the config so a team shares it, drop a .mcp.json file at the project root:

{
  "mcpServers": {
    "crawler-sh": {
      "command": "/Users/you/.crawler/bin/crawler-mcp",
      "args": [],
      "env": {}
    }
  }
}

Replace /Users/you/ with your home directory. Use which crawler-mcp to print the absolute path.

Raise the page cap

Set CRAWLER_TOKEN in the env block to raise the per-crawl cap from 50 to 400 pages (or 10,000 with a Pro subscription). See Tier caps.

{
  "mcpServers": {
    "crawler-sh": {
      "command": "/Users/you/.crawler/bin/crawler-mcp",
      "env": { "CRAWLER_TOKEN": "your-token" }
    }
  }
}

Try it

Open Claude Code and ask:

Use crawler-sh to fetch https://crawler.sh and summarise it.

Claude Code will call fetch_page and reason over the returned Markdown.

For deeper crawls:

Use crawler-sh to crawl https://example.com to depth 2 and list the top 10 internal links by inbound count.

Remove

claude mcp remove crawler-sh

Or delete the crawler-sh block from .mcp.json if you went the manual route.