How to Raise the MCP Page Cap with CRAWLER_TOKEN

crawler-mcp enforces a hard cap on pages per crawl. Without a token, the limit is 50 pages. With CRAWLER_TOKEN set, it rises to 400 for free accounts, or 10,000 for Pro subscribers. This guide shows how to configure the token in every supported client.

Step 1: Get your token

Log in to crawler.sh via the CLI:

crawler auth login

Enter your email, paste the OTP code, and the CLI saves your session token to ~/.crawler/auth.json.

Copy the token value for the next step.

Step 2: Set CRAWLER_TOKEN in Claude Code

If you registered with claude mcp add, remove and re-add with the token:

claude mcp remove crawler-sh
claude mcp add crawler-sh -- env CRAWLER_TOKEN=your-token $(which crawler-mcp)

Or edit your .mcp.json and add the env block:

{
  "mcpServers": {
    "crawler-sh": {
      "command": "/Users/you/.crawler/bin/crawler-mcp",
      "env": { "CRAWLER_TOKEN": "your-token" }
    }
  }
}

Step 3: Set CRAWLER_TOKEN in Claude Desktop

Edit ~/Library/Application Support/Claude/claude_desktop_config.json (macOS) or %APPDATA%\Claude\claude_desktop_config.json (Windows):

{
  "mcpServers": {
    "crawler-sh": {
      "command": "/Users/you/.crawler/bin/crawler-mcp",
      "env": { "CRAWLER_TOKEN": "your-token" }
    }
  }
}

Quit Claude Desktop completely and reopen.

Step 4: Set CRAWLER_TOKEN in Cursor

Edit ~/.cursor/mcp.json or .cursor/mcp.json:

{
  "mcpServers": {
    "crawler-sh": {
      "command": "/Users/you/.crawler/bin/crawler-mcp",
      "env": { "CRAWLER_TOKEN": "your-token" }
    }
  }
}

Toggle the server off and on in Cursor Settings -> MCP.

Step 5: Set CRAWLER_TOKEN in OpenCode

Edit ~/.config/opencode/opencode.json or opencode.json:

{
  "mcp": {
    "crawler-sh": {
      "type": "local",
      "command": ["/Users/you/.crawler/bin/crawler-mcp"],
      "environment": { "CRAWLER_TOKEN": "your-token" }
    }
  }
}

Restart OpenCode.

Step 6: Set CRAWLER_TOKEN in Zed

Edit ~/.config/zed/settings.json:

{
  "context_servers": {
    "crawler-sh": {
      "command": {
        "path": "/Users/you/.crawler/bin/crawler-mcp",
        "env": { "CRAWLER_TOKEN": "your-token" }
      }
    }
  }
}

Save the file. Zed picks up the change immediately.

Step 7: Verify the new cap

Run a crawl and check the response:

Use crawler-sh to crawl https://example.com with max_pages 10.

Look at the cap_max field in the response. It should show 400 (free account) or 10000 (Pro subscriber) instead of 50.

Related

• Crawler MCP overview
• Tool reference
• Installation