Is it legal to crawl a competitor website?

Crawling publicly available pages is generally legal for research purposes, but you should respect robots.txt and terms of service. Never crawl password-protected areas or scrape personal data.

Can I crawl a competitor site that blocks bots?

crawler-mcp uses a TLS fingerprint that matches current Chrome, which reduces detection. However, if a site actively blocks crawlers, you may see 403 responses. Respect the site defenses and do not try to bypass them.

How deep should I crawl for competitor research?

Start with depth 2 and max_pages 50. This covers the homepage, top-level sections, and their subpages. Deeper crawls are rarely needed for strategic analysis and risk hitting rate limits.

How to Research a Competitor Website with Claude and MCP

Competitor research usually means opening dozens of tabs and taking notes. With crawler-mcp, you can have Claude do the legwork: discover the site structure, fetch the pages that matter, and summarise findings in one conversation.

This guide shows a three-phase workflow using discover_links and fetch_page.

Step 1: Install crawler-mcp

Run the install script:

curl -fsSL https://install.crawler.sh/install-mcp.sh | sh

This downloads the correct binary for your platform to ~/.crawler/bin/crawler-mcp.

For more detail, see the installation guide.

Step 2: Wire it into Claude Desktop

Follow the Claude Desktop setup guide to add crawler-sh to your config file and restart the app.

Step 3: Discover the site structure

Start by mapping the competitor’s site:

Use crawler-sh to discover_links on https://competitor.com to depth 2.

Claude returns a list of URLs with titles. Ask Claude to organise them by category:

Group those links by section - product pages, blog posts, pricing, support, etc.

Step 4: Fetch key pages

Pick the pages that matter most and fetch them:

Use crawler-sh to fetch https://competitor.com/pricing and https://competitor.com/features, then compare their positioning to ours.

Claude calls fetch_page for each URL and produces a side-by-side comparison.

Step 5: Summarise findings

After fetching the key pages, ask Claude to synthesise:

Based on the pages we fetched, summarise their messaging strategy, pricing tiers, and key differentiators.

Claude reasons over the fetched Markdown and gives you a structured competitor analysis.