Use Crawler MCP with Cursor

Cursor reads MCP servers from a JSON config file. Once crawler-mcp is installed (guide), add it to the file Cursor watches.

Pick a scope

Global (all projects)
Per-project

~/.cursor/mcp.json

Available to every project you open in Cursor.

.cursor/mcp.json

At the project root. Commit it so collaborators get the server automatically.

Add the server entry

Find the absolute path
Section titled “Find the absolute path”
```
which crawler-mcp
```

Edit the config

{
  "mcpServers": {
    "crawler-sh": {
      "command": "/Users/you/.crawler/bin/crawler-mcp",
      "args": [],
      "env": {}
    }
  }
}

Use the absolute path from step 1. Cursor will not expand ~.

Enable the server
Section titled “Enable the server”

Open Cursor Settings -> MCP. Toggle crawler-sh on if it is not already. The three tools (crawl_site, fetch_page, discover_links) appear as the server starts.

Raise the page cap

Set CRAWLER_TOKEN in the env block to raise the per-crawl cap from 50 to 400 pages (or 10,000 with a Pro subscription). See Tier caps.

{
  "mcpServers": {
    "crawler-sh": {
      "command": "/Users/you/.crawler/bin/crawler-mcp",
      "env": { "CRAWLER_TOKEN": "your-token" }
    }
  }
}

Try it

In Cursor’s chat panel, with agent mode on:

Use crawler-sh to crawl https://example.com to depth 2 and list the page titles.

Cursor will ask permission before calling the tool, then act on the returned Markdown.