Skip to content
Crawler.sh Logo Crawler.sh Logo Crawler

Use Crawler MCP with Claude Desktop

The Claude desktop app reads MCP servers from a JSON config file. Once crawler-mcp is installed (guide), add it to that file and restart the app.

Locate the config file

Section titled “Locate the config file”
~/Library/Application Support/Claude/claude_desktop_config.json

Open it from a terminal:

open "$HOME/Library/Application Support/Claude/claude_desktop_config.json"

If the file does not exist, create it. If it exists but is empty, start with {}.

Add the server entry

Section titled “Add the server entry”

  1. Find the absolute path to the binary

    Section titled “Find the absolute path to the binary”
    which crawler-mcp

    Copy the output. Claude Desktop will not expand ~ or $HOME.

  2. Edit the config

    Section titled “Edit the config”
    {
      "mcpServers": {
        "crawler-sh": {
          "command": "/Users/you/.crawler/bin/crawler-mcp",
          "args": [],
          "env": {}
        }
      }
    }

    Replace /Users/you/ with the path from step 1. If you already have other mcpServers entries, add "crawler-sh" alongside them.

  3. Restart Claude Desktop

    Section titled “Restart Claude Desktop”

    Quit completely (Cmd+Q on macOS, not just close the window) and reopen.

  4. Verify the tools

    Section titled “Verify the tools”

    Open a new conversation. Click the search and tools icon in the composer. You should see crawl_site, fetch_page, and discover_links listed under the crawler-sh server.

Raise the page cap

Section titled “Raise the page cap”

Set CRAWLER_TOKEN in the env block to raise the per-crawl cap from 50 to 400 pages (or 10,000 with a Pro subscription). See Tier caps.

{
  "mcpServers": {
    "crawler-sh": {
      "command": "/Users/you/.crawler/bin/crawler-mcp",
      "env": { "CRAWLER_TOKEN": "your-token" }
    }
  }
}

Try it

Section titled “Try it”

Ask Claude:

Crawl https://example.com to depth 2 with crawler-sh and tell me what the site is about.

Claude will request permission to call crawl_site, then summarise the returned pages.

Troubleshooting

Section titled “Troubleshooting”
  • Tools do not appear: Check the config file is valid JSON (no trailing commas). Quit Claude Desktop completely and reopen.
  • “command not found”: The path in command is wrong. Run which crawler-mcp again and paste the exact output.
  • Permission denied on macOS: Run chmod +x ~/.crawler/bin/crawler-mcp.

See also

Section titled “See also”