v0.8.2: New Tier Limits and MCP Pro Verification
Updated free-tier page caps (50 anonymous, 400 signed-in) and the MCP server now verifies Pro subscriptions, so Pro users get the full 10,000-page cap inside Claude Code, Cursor, Zed, and other MCP clients.
What’s New in v0.8.2
This release updates the free-tier page caps and finishes wiring Pro-tier verification into the MCP server. If you were running into the old 1,000-page MCP ceiling as a Pro subscriber, this release fixes that.
New Tier Caps
The per-crawl page limits have changed:
| Tier | Old | New |
|---|---|---|
| Not signed in | 600 | 50 |
| Signed in (free) | 1,000 | 400 |
| Pro subscriber | 10,000 | 10,000 |
The Pro cap is unchanged. The lower free-tier caps apply to the CLI, the desktop app, and the MCP server.
If a crawl exceeds your tier cap, the CLI and MCP server silently cap the request and surface the effective number in their output. Nothing fails; the run just stops at the cap.
MCP Now Verifies Pro
The previous MCP release shipped with a hard CAP_TOKEN = 1,000 ceiling for any CRAWLER_TOKEN, regardless of subscription state. We needed a way to confirm a token belonged to a Pro account without making a network call on every tool invocation.
In v0.8.2, crawler-mcp calls GET /auth/me once at startup when CRAWLER_TOKEN is set, resolves the cap for the session, and holds it for the lifetime of the process. Pro subscribers get the full 10,000-page cap from inside Claude Code, Claude Desktop, Cursor, Zed, and OpenCode. Free signed-in users get 400. Without a token, you stay at the anonymous 50. On any verification error the cap falls back to 400, so a transient API blip never demotes a signed-in user to anonymous.
There is no per-request network hop; the cap is resolved once when the MCP client launches the server. Restart your MCP client after upgrading or after changing your subscription state for the new cap to take effect.
Single Source of Truth for Constants
Internally, the tier-cap constants are now centralized: one Rust module (crates/crawler-core/src/tier.rs) for the CLI, MCP, and core; one TypeScript module (api/src/auth/tier.ts) for the API. The desktop store trusts whatever maxPagesLimit the API returns instead of re-deriving it from the tier string, which removes a class of subtle bug where a custom server-issued limit would be silently overwritten by a client-side default.
This is invisible to users but makes future cap changes a one-file edit per language.
Upgrade
CLI and MCP self-install:
curl -fsSL https://install.crawler.sh | shcurl -fsSL https://install.crawler.sh/install-mcp.sh | shThe desktop app updates automatically on launch.
Related
About crawler.sh
crawler.sh is a fast Rust-based web crawler and SEO auditing tool that runs entirely on your own machine. Use the CLI for automation, scripts, and CI pipelines, or the desktop app for a visual dashboard with live crawl progress, SEO issue charts, and one-click exports.
Every release ships across both the CLI and the desktop app.
Download the latest version
or run crawler update
from the terminal to upgrade an existing install.
