v0.1.7: llms.txt Support & SEO CSV Export
Crawler now detects and downloads llms.txt files, validates their format, and lets you export SEO audit results as CSV.
What’s New in v0.1.7
llms.txt Detection & Download
Crawler now checks whether crawled sites have an llms.txt file - the emerging standard for telling LLMs what content to use from your site. During a crawl, Crawler will:
- Detect if
llms.txtexists at the site root - Report it in the site info summary
- Let you download it directly
This is useful for auditing whether your site is prepared for AI-powered search and content consumption.
SEO Audit CSV Export
The seo command can now export results as CSV in addition to the existing TXT format. This makes it easy to import audit results into spreadsheets or other analysis tools.
crawler seo site.crawl --format csv -o audit.csvCopy SEO Audit to Clipboard
In the desktop app, you can now copy the full SEO audit to your clipboard with one click - useful for sharing results in tickets, docs, or messages.
Other Improvements
- Sitemap and robots.txt parsing improvements
- CDN-specific URLs (like
cdn-cgi) are now ignored during crawls
Related
Wrap-up
A CMS shouldn't slow you down. Crawler aims to expand into your workflow — whether you're coding content models, collaborating on product copy, or launching updates at 2am.
If that sounds like the kind of tooling you want to use — try Crawler .
