Do I need a paid subscription to use crawler-mcp with Claude Code?

No. The free tier covers 50 pages per crawl. If you set a CRAWLER_TOKEN tied to a free account, the cap rises to 400 pages. Pro subscribers get 10,000 pages.

Can I crawl any website with Claude Code?

crawler-mcp respects robots.txt Disallow rules by default, so it will skip pages that the site owner has asked crawlers not to visit. You can override this for sites you own by passing respect_robots: false in the tool call.

What happens if a crawl hits the page cap?

The tool returns all pages found so far and includes a cap_applied flag in the response. You can then raise the cap by setting CRAWLER_TOKEN or reducing max_pages to stay within the limit.

How to Crawl a Website with Claude Code

Claude Code is a terminal agent that can call tools on your behalf. With crawler-mcp installed, you can ask Claude to crawl websites, fetch single pages, and discover links - all without leaving your terminal.

This guide walks through installing the Crawler MCP server and using it from Claude Code.

Step 1: Install crawler-mcp

Run the install script:

curl -fsSL https://install.crawler.sh/install-mcp.sh | sh

This downloads the correct binary for your platform and places it in ~/.crawler/bin/crawler-mcp.

For more detail, see the installation guide.

Step 2: Wire it into Claude Code

claude mcp add crawler-sh -- $(which crawler-mcp)

Verify it is registered:

claude mcp list

For manual config or per-project setup, see the Claude Code docs.

Step 3: Crawl a website

Open Claude Code and ask:

Use crawler-sh to crawl https://example.com to depth 2 and give me a summary of what the site covers.

Claude Code will call crawl_site, receive Markdown and metadata for every page, and reason over the results to produce a summary.

You can refine the crawl with parameters:

Use crawler-sh to crawl https://example.com with max_pages 20 and delay_ms 500.

Step 4: Fetch a single page

When you only need one page, use fetch_page instead of a full crawl:

Use crawler-sh to fetch https://example.com/pricing and tell me the key features.

This is faster and avoids unnecessary requests.

Step 5: Discover links before crawling

To understand a site’s structure before committing to a deep crawl:

Use crawler-sh to discover_links on https://example.com to depth 1.

Claude will return a list of URLs with titles, which you can use to decide which sections to crawl in depth.