# crawler.sh - Full Documentation

> A fast, local-first web crawler and SEO analysis tool. Crawl any website in seconds, run 23 automated SEO checks, extract content as clean Markdown, and export to JSON, CSV, or Sitemap XML - from the terminal or the desktop app. Runs entirely on your machine.

## Project details

- **Name**: crawler.sh
- **URL**: https://crawler.sh
- **Category**: Developer Tools / SEO
- **License**: Proprietary (freemium)
- **Platforms**: macOS (Universal - Apple Silicon and Intel), Linux (x64 and ARM64, .deb). Windows coming soon.
- **Install**: `curl -fsSL https://install.crawler.sh | sh`
- **Pricing**: Free (50 pages/crawl anonymous, 400 signed-in, all features) · Pro $99/year (10,000 pages/crawl, Content Archive export)
- **Author**: Mehmet Kose
- **Contact**: https://crawler.sh/contact/

## Installation

### CLI - Install script (recommended)

```bash
curl -fsSL https://install.crawler.sh | sh
```

Downloads the correct binary for your architecture (macOS ARM64 or x64), installs to `~/.crawler/bin/`, and adds the directory to your PATH in `.zshrc`, `.bashrc`, or equivalent. Restart your shell or run:

```bash
export PATH="$HOME/.crawler/bin:$PATH"
```

Custom install directory:

```bash
CRAWLER_INSTALL_DIR=/opt/crawler curl -fsSL https://install.crawler.sh | sh
```

Verify:

```bash
crawler --help
```

### CLI - Uninstall

```bash
rm -rf ~/.crawler
```

Then remove the `# crawler.sh` PATH block from your shell config (`~/.zshrc`, `~/.bashrc`, `~/.bash_profile`, `~/.profile`, or `~/.config/fish/config.fish`).

### Desktop app

Download the universal macOS DMG from https://crawler.sh/download/ and drag Crawler to your Applications folder. Uninstall by dragging it to Trash.

---

## CLI Usage

The CLI has four subcommands: `crawl`, `info`, `seo`, and `export`.

### crawler crawl

Crawl a website using breadth-first search. Stays within the same domain. Records status codes, response times, titles, meta descriptions, canonical URLs, content, and more.

```
crawler crawl <URL> [OPTIONS]
```

**Flags:**

| Flag | Short | Default | Description |
|------|-------|---------|-------------|
| `--output <PATH>` | `-o` | auto-generated | Output file path |
| `--format <FMT>` | `-f` | `ndjson` | Output format: ndjson, json, sitemap |
| `--max-pages <N>` | `-p` | `100` | Maximum pages to crawl |
| `--max-depth <N>` | `-d` | `10` | Maximum crawl depth |
| `--concurrency <N>` | `-c` | `5` | Concurrent requests |
| `--delay <MS>` | | `200` | Delay between requests in ms |
| `--no-extract` | | `false` | Disable content extraction |
| `--verbose` | `-v` | `false` | Verbose logging |
| `--quiet` | `-q` | `false` | Suppress all output except errors |

**Examples:**

```bash
# Basic crawl (saves to example-com.crawl)
crawler crawl https://example.com

# URL prefix is optional
crawler crawl example.com

# Custom limits: 500 pages, depth 5, 10 concurrent requests
crawler crawl https://example.com -p 500 -d 5 -c 10

# Quick surface crawl
crawler crawl https://example.com -p 20 -d 2

# Output as JSON
crawler crawl https://example.com -f json -o site.json

# Generate a sitemap directly
crawler crawl https://example.com -f sitemap

# Fast crawl: skip content extraction, reduce delay
crawler crawl https://example.com --no-extract --delay 50 -c 10 -p 500
```

**Crawl engine behavior:**

- BFS traversal from the start URL
- Domain-constrained: only same-host links are followed
- URL normalization: fragments stripped, trailing slashes removed, duplicates skipped
- Link filtering: ignores `javascript:`, `mailto:`, `tel:`, and anchor-only links
- Polite crawling: configurable delay between requests (default 200ms)
- User-Agent: `crawler.sh/0.1`
- HTTP timeout: 30 seconds
- Max redirects: 10

### crawler info

Analyze a `.crawl` (NDJSON) file and display summary statistics: domain, page count, file size, HTTP status distribution, and response time stats.

```
crawler info <FILE>
```

**Example:**

```bash
crawler info example-com.crawl
```

No additional flags.

### crawler seo

Run 23 automated SEO checks against a `.crawl` file. Only analyzes pages with 2xx status and `text/html` content type. Duplicate detection is canonical-aware.

```
crawler seo <FILE> [OPTIONS]
```

**Flags:**

| Flag | Short | Default | Description |
|------|-------|---------|-------------|
| `--export <FMT>` | | | Export format: csv, txt |
| `--output <PATH>` | `-o` | auto-generated | Output file path for export |

**Examples:**

```bash
# Display results in terminal
crawler seo example-com.crawl

# Export as CSV
crawler seo example-com.crawl --export csv

# Export as TXT with custom filename
crawler seo example-com.crawl --export txt -o report.txt
```

**The 23 SEO checks:**

| # | Check | Rule |
|---|-------|------|
| 1 | Missing titles | title is empty or absent |
| 2 | Titles too short | title < 30 characters |
| 3 | Titles too long | title > 60 characters |
| 4 | Duplicate titles | multiple pages share the same title (grouped by canonical) |
| 5 | Missing meta descriptions | meta_description is empty or absent |
| 6 | Descriptions too short | meta_description < 50 characters |
| 7 | Descriptions too long | meta_description > 160 characters |
| 8 | Duplicate descriptions | multiple pages share the same description (grouped by canonical) |
| 9 | Missing H1 tags | page has no H1 element |
| 10 | Missing content | word_count is null |
| 11 | Short content (thin) | word_count < 200 |
| 12 | Long content | word_count > 5,000 |
| 13 | Long URLs | URL > 120 characters |
| 14 | Noindex pages | meta_robots or X-Robots-Tag contains noindex |
| 15 | Nofollow pages | meta_robots or X-Robots-Tag contains nofollow |
| 16 | Non-self canonicals | canonical_url differs from page URL |
| 17 | Paginated pages | page has rel="next" or rel="prev" |
| 18 | Multiple H1 tags | page has more than one H1 element |
| 19 | Empty H1 | H1 element contains no text |
| 20 | Long H1 | H1 text > 70 characters |
| 21 | Short H1 | H1 text < 10 characters |
| 22 | Duplicate H1 | multiple pages share the same H1 text |
| 23 | Broken links | outgoing links or images return 404/410/5xx or fail to connect |

### crawler export

Convert a `.crawl` file to JSON or Sitemap XML.

```
crawler export <FILE> --format <FMT> [OPTIONS]
```

**Flags:**

| Flag | Short | Default | Description |
|------|-------|---------|-------------|
| `--format <FMT>` | `-f` | required | Target format: json or sitemap |
| `--output <PATH>` | `-o` | auto-generated | Output file path |

**Examples:**

```bash
# Convert to JSON
crawler export example-com.crawl -f json

# Convert to sitemap
crawler export example-com.crawl -f sitemap

# Custom output path
crawler export example-com.crawl -f sitemap -o sitemap.xml
```

### Common workflows

**Full audit workflow:**

```bash
crawler crawl https://example.com -p 200
crawler info example-com.crawl
crawler seo example-com.crawl
crawler seo example-com.crawl --export csv
```

**Quick sitemap generation:**

```bash
crawler crawl https://example.com -f sitemap
```

**Deep crawl with content, then convert:**

```bash
crawler crawl https://example.com -p 400 -d 20 -c 10
crawler export example-com.crawl -f json
```

**SEO report for a team:**

```bash
crawler crawl https://example.com -p 200
crawler seo example-com.crawl --export csv
crawler seo example-com.crawl --export txt -o seo-report.txt
```

---

## Desktop App Usage

macOS desktop application with an interactive dashboard for crawling, SEO analysis, content browsing, and exporting.

### Starting a crawl

1. Open Crawler from Applications
2. Enter a URL in the top bar
3. Optionally configure settings before starting
4. Click **Start Crawl**

### Settings

Configure crawl parameters before starting. Settings are disabled during an active crawl.

| Parameter | Range | Default | Description |
|-----------|-------|---------|-------------|
| Max Pages | 1 – 400 | 20 | Maximum pages to crawl |
| Max Depth | 1 – 50 | 10 | Maximum depth from start URL |
| Concurrency | 1 – 20 | 5 | Concurrent HTTP requests |
| Delay (ms) | 0 – 5,000 | 50 | Delay between requests |
| Extract Content | on / off | on | Enable content-to-Markdown extraction |

Dark mode toggle is also available in Settings and persists across sessions.

### Dashboard cards

The desktop app has eight dashboard cards. Click any card to expand it. Press Escape or click the backdrop to close.

**1. Live Feed** - Real-time stream of crawled URLs with color-coded status badges (2xx green, 3xx yellow, 4xx/5xx red). Grid preview shows the last 14 entries. Expanded view shows the full feed up to 500 entries. Hover actions: copy URL, open in browser.

**2. SEO Issues** - Runs the same 23 automated checks as the CLI against all 2xx HTML pages. Grid preview shows the first 4 issue categories with counts. Expanded view shows an accordion with per-URL details for every category. Export buttons for CSV and TXT.

**3. Page Status** - HTTP status code distribution. Grid view shows a donut chart. Expanded view adds a horizontal bar chart and an accordion listing all URLs grouped by status code. Hover actions: copy URL, open in browser.

**4. Settings** - Crawl configuration form with the 5 parameters above plus dark mode toggle.

**5. Downloads** - Export crawl results in three formats:
- **JSON Archive** - Complete crawl data with metadata
- **Sitemap XML** - Standard XML sitemap (2xx pages only)
- **Content Archive** - Markdown files of extracted content (Pro only)

Export buttons are disabled until at least one page has been crawled.

**6. Content** - Split-panel content viewer. Left panel lists pages that have extracted Markdown, showing title, URL, word count, and byline. Right panel renders the selected page's Markdown with GitHub Flavored Markdown support. Requires Extract Content to be enabled in Settings.

**7. Newsletter** - Email subscription form for product updates.

**8. Premium** - Upgrade prompt with Pro pricing and purchase button.

### Keyboard shortcuts

- **Escape** - Close expanded card overlay

### Exporting from the desktop app

**From Downloads card:**
- JSON Archive: complete crawl data as a JSON file
- Sitemap XML: W3C-compliant sitemap with 2xx pages only
- Content Archive (Pro): ZIP of Markdown files

**From SEO Issues card:**
- CSV: two columns (Issue Type, URL)
- TXT: human-readable report grouped by issue category

Filenames follow the pattern: `{domain}-{label}-{timestamp}.{ext}` (e.g., `example-com-crawl-results-2026-02-22-14-30-00.json`).

---

## Output Formats

### NDJSON (.crawl)

Default CLI format. First line is a metadata header with crawl config. Subsequent lines are one JSON object per page. Streamable - valid even if the crawl is interrupted.

### JSON

Array of page objects written at the end of the crawl. Each object includes all page fields.

### Sitemap XML

W3C-compliant XML sitemap. Only includes pages with 2xx status codes. Limited to 50,000 URLs per sitemap.

### SEO CSV

Two columns: Issue Type and URL. One row per issue per page.

### SEO TXT

Human-readable report grouped by issue category with indented URLs under each heading.

### Filename conventions

| Format | CLI pattern | Example |
|--------|-------------|---------|
| NDJSON | `{domain}.crawl` | `example-com.crawl` |
| JSON | `{domain}.json` | `example-com.json` |
| Sitemap | `{domain}-sitemap.xml` | `example-com-sitemap.xml` |
| SEO CSV | `{domain}-seo.csv` | `example-com-seo.csv` |
| SEO TXT | `{domain}-seo.txt` | `example-com-seo.txt` |

---

## Page Data Model

Every crawled page records the following fields:

| Field | Type | Description |
|-------|------|-------------|
| url | String | Full URL of the page |
| status_code | Number | HTTP status code |
| content_type | String? | Content-Type header value |
| title | String? | HTML `<title>` text |
| meta_description | String? | `<meta name="description">` content |
| canonical_url | String? | `<link rel="canonical">` href |
| discovered_from | String? | Parent URL that linked to this page |
| links_found | Number | Number of new same-domain links discovered |
| depth | Number | Crawl depth from start URL |
| response_time_ms | Number | HTTP response time in milliseconds |
| markdown | String? | Extracted content as Markdown (when enabled) |
| word_count | Number? | Word count of extracted content |
| byline | String? | Author byline from content extraction |
| excerpt | String? | Article excerpt from content extraction |
| meta_robots | String? | `<meta name="robots">` content |
| x_robots_tag | String? | X-Robots-Tag HTTP header value |
| rel_next | String? | `<link rel="next">` href (pagination) |
| rel_prev | String? | `<link rel="prev">` href (pagination) |

---

## Crawl Events

The engine emits real-time events during a crawl. The CLI uses these for progress bars and output. The desktop app uses them for the live feed and status charts.

| Event | Data | Description |
|-------|------|-------------|
| Started | url | Crawl has begun |
| Discovered | url, depth | New URL found and enqueued |
| PageCrawled | page object | Page successfully fetched and parsed |
| PageError | url, error | Failed to fetch a page |
| Progress | crawled, total_discovered | Periodic progress update |
| Completed | total_pages, total_errors | Crawl finished |

---

## Content Extraction

Enabled by default. Extracts the main article content from each HTML page, strips navigation, sidebars, footers, and scripts, and converts to clean Markdown. Includes word count, author byline, and excerpt for every page.

Disable with `--no-extract` (CLI) or the Extract Content toggle (desktop) for faster crawling when content is not needed.

---

## Documentation links

- [Getting Started](https://crawler.sh/docs/)
- [Installation](https://crawler.sh/docs/installation/)
- [CLI Overview](https://crawler.sh/docs/cli/)
- [CLI Features](https://crawler.sh/docs/cli/features/)
- [CLI Reference](https://crawler.sh/docs/cli/reference/)
- [Desktop Overview](https://crawler.sh/docs/desktop/)
- [Desktop Features](https://crawler.sh/docs/desktop/features/)
- [Desktop Reference](https://crawler.sh/docs/desktop/reference/)
- [Product](https://crawler.sh/product/)
- [Technical SEO Audit](https://crawler.sh/technical-seo-audit/)
- [Download](https://crawler.sh/download/)
- [Roadmap](https://crawler.sh/roadmap/)
- [FAQ](https://crawler.sh/faq/)
- [Blog](https://crawler.sh/blog/)
- [Changelog](https://crawler.sh/changelog/)
- [About](https://crawler.sh/about/)
- [Contact](https://crawler.sh/contact/)
- [Privacy Policy](https://crawler.sh/privacy-policy/)
- [Terms of Service](https://crawler.sh/terms-of-service/)