v0.2.2: Automated Robots.txt Checker
Crawler now checks for robots.txt during every crawl and reports whether the site has one, including it in SEO audit results.
What’s New in v0.2.2
Robots.txt Detection
Every crawl now automatically checks whether the target site has a robots.txt file. The result appears in the site info summary and as a site-level SEO check.
If no robots.txt is found, the SEO audit flags it as an issue. Search engines rely on this file to understand which pages should and shouldn’t be indexed. A missing robots.txt does not block crawling, but it signals to search engines that the site owner has not configured crawl directives - and it means you are missing an opportunity to point crawlers toward your sitemap.
Why Robots.txt Matters
The robots.txt file sits at the root of every website and serves as the first point of communication between a site and search engine crawlers. It tells bots which directories or pages to skip, which crawl rate to use, and where to find the XML sitemap. Sites without one may see search engines wasting crawl budget on low-value pages like admin panels, staging URLs, or duplicate filtered views. Adding a well-structured robots.txt is one of the simplest and most impactful technical SEO improvements you can make.
How It Works
At the start of each crawl, Crawler fetches <site>/robots.txt and reports:
- Whether the file exists
- Whether a sitemap is referenced in it
- The number of sitemap URLs discovered
This information shows up in:
- The SiteInfo event in CLI output
- The SEO Issues card in the desktop app
- CSV and TXT exports from the
seocommand
If the robots.txt references a sitemap, Crawler parses it and adds any discovered URLs to the crawl queue. This means you get broader coverage even on sites with deep or orphaned pages that might not be reachable through internal links alone.
Related
Wrap-up
A CMS shouldn't slow you down. Crawler aims to expand into your workflow — whether you're coding content models, collaborating on product copy, or launching updates at 2am.
If that sounds like the kind of tooling you want to use — try Crawler .
