How to Crawl a Website with Zed

Zed’s Agent Panel can call MCP tools during a conversation. With crawler-mcp connected, you can ask Zed to crawl a site, fetch a page, or discover links - all from inside your editor.

This guide shows how to set it up and start exploring.

Step 1: Install crawler-mcp

Run the install script:

curl -fsSL https://install.crawler.sh/install-mcp.sh | sh

This downloads the correct binary for your platform to ~/.crawler/bin/crawler-mcp.

For more detail, see the installation guide.

Step 2: Wire it into Zed

Open Zed settings (zed: open settings, Cmd+, on macOS) and add:

{
  "context_servers": {
    "crawler-sh": {
      "command": {
        "path": "/Users/you/.crawler/bin/crawler-mcp",
        "args": [],
        "env": {}
      }
    }
  }
}

Use the absolute path from which crawler-mcp. Save the file and open the Agent Panel to confirm crawler-sh appears.

For full setup details, see the Zed docs.

Step 3: Crawl a website

In the Agent Panel, ask:

Use crawler-sh to crawl https://example.com to depth 2 and list the page titles.

Zed will prompt you to allow the tool call, then act on the returned Markdown.

Step 4: Fetch a single page

For one-off lookups:

Use crawler-sh to fetch https://example.com/pricing and tell me the plans.

Zed calls fetch_page and returns the page content.

Step 5: Discover links

To map a site’s structure:

Use crawler-sh to discover_links on https://example.com to depth 1.

Zed returns URLs with titles so you can decide which pages to explore.

Related

• Crawler MCP overview
• Tool reference
• How to crawl a website with Claude Code
• How to crawl a website with Cursor