v0.1.7: llms.txt Support & SEO CSV Export
Crawler now detects and downloads llms.txt files, validates their format, and lets you export SEO audit results as CSV.
What’s New in v0.1.7
llms.txt Detection & Download
Crawler now checks whether crawled sites have an llms.txt file - the emerging standard for telling LLMs what content to use from your site. During a crawl, Crawler will:
- Detect if
llms.txtexists at the site root - Report it in the site info summary
- Let you download it directly
This is useful for auditing whether your site is prepared for AI-powered search and content consumption.
SEO Audit CSV Export
The seo command can now export results as CSV in addition to the existing TXT format. This makes it easy to import audit results into spreadsheets or other analysis tools.
crawler seo site.crawl --format csv -o audit.csvCopy SEO Audit to Clipboard
In the desktop app, you can now copy the full SEO audit to your clipboard with one click - useful for sharing results in tickets, docs, or messages.
Other Improvements
- Sitemap and robots.txt parsing improvements
- CDN-specific URLs (like
cdn-cgi) are now ignored during crawls
Related
About crawler.sh
crawler.sh is a fast Rust-based web crawler and SEO auditing tool that runs entirely on your own machine. Use the CLI for automation, scripts, and CI pipelines, or the desktop app for a visual dashboard with live crawl progress, SEO issue charts, and one-click exports.
Every release ships across both the CLI and the desktop app.
Download the latest version
or run crawler update
from the terminal to upgrade an existing install.
