Can I run this in CI after every deploy?

Yes. Install crawler-mcp on your CI runner, set CRAWLER_TOKEN, and run a Claude Code script that crawls the staging URL and asserts no 4xx or 5xx responses. Exit non-zero if issues are found.

Should I crawl staging with the same settings as production?

Yes. Use the same max_pages, max_depth, and concurrency so the comparison is fair. Differences in settings can produce false positives or hide real regressions.

What if staging is behind basic auth or a VPN?

crawler-mcp runs locally and uses your machine network, so VPN access works. For basic auth, pass the credentials in the URL like https://user:pass@staging.example.com. For complex auth, use the CLI instead.

How to Monitor a Staging Site After Deploy with MCP

After a deployment, you need to know that pages still load, redirects still work, and titles have not changed unexpectedly. With crawler-mcp, an agent can crawl your staging site and compare it against a baseline or checklist - all from a prompt.

This guide shows how to validate a staging deployment using crawl_site.

Step 1: Install crawler-mcp

Run the install script:

curl -fsSL https://install.crawler.sh/install-mcp.sh | sh

This downloads the correct binary for your platform to ~/.crawler/bin/crawler-mcp.

For more detail, see the installation guide.

Step 2: Wire it into Claude Code

Follow the Claude Code setup guide to register crawler-sh.

This workflow works well in CI-adjacent contexts where Claude Code runs in a terminal alongside deploy scripts.

Step 3: Crawl the staging site

After deploy, ask Claude to validate the site:

Use crawler-sh to crawl https://staging.example.com to depth 2 with max_pages 50. Report any pages with status codes other than 200.

Claude calls crawl_site and flags any errors.

Step 4: Check critical pages

Ask Claude to verify specific pages:

Fetch https://staging.example.com/pricing and confirm the title is “Pricing - Example” and the H1 is “Pricing Plans”.

Claude calls fetch_page and compares the live metadata against your expectations.

Step 5: Compare to production

If you have a previous crawl saved, ask Claude to compare:

Compare this crawl to the previous one. Did any titles, descriptions, or status codes change?

Claude reasons over the before-and-after data and highlights regressions.