How to Crawl a Website with Cursor

Cursor’s agent mode can call MCP tools during a chat session. With crawler-mcp connected, you can ask Cursor to fetch documentation, map out a site’s pages, or check links - all without leaving your editor.

This guide shows how to wire it in and start using it.

Step 1: Install crawler-mcp

Run the install script:

curl -fsSL https://install.crawler.sh/install-mcp.sh | sh

This downloads the correct binary for your platform to ~/.crawler/bin/crawler-mcp.

For more detail, see the installation guide.

Step 2: Wire it into Cursor

Create or edit ~/.cursor/mcp.json (global) or .cursor/mcp.json (per-project):

{
  "mcpServers": {
    "crawler-sh": {
      "command": "/Users/you/.crawler/bin/crawler-mcp",
      "args": [],
      "env": {}
    }
  }
}

Use the absolute path from which crawler-mcp. Open Cursor Settings -> MCP and toggle crawler-sh on.

For full setup details, see the Cursor docs.

Step 3: Fetch docs while coding

In Cursor’s chat panel with agent mode on, ask:

Use crawler-sh to fetch https://docs.example.com/api-reference and summarise the authentication section.

Cursor calls fetch_page and returns a summary you can reference while writing code.

Step 4: Discover a site’s structure

To map out a documentation site before diving in:

Use crawler-sh to discover_links on https://docs.example.com to depth 2.

Cursor returns a list of URLs with titles, which you can use to plan which pages to read.

Related

• Crawler MCP overview
• Tool reference
• How to crawl a website with Claude Code
• How to discover a site’s structure before scraping with MCP