v0.2.2: Automated Robots.txt Checker

What’s New in v0.2.2

Robots.txt Detection

Every crawl now automatically checks whether the target site has a robots.txt file. The result appears in the site info summary and as a site-level SEO check.

If no robots.txt is found, the SEO audit flags it as an issue. Search engines rely on this file to understand which pages should and shouldn’t be indexed. A missing robots.txt does not block crawling, but it signals to search engines that the site owner has not configured crawl directives - and it means you are missing an opportunity to point crawlers toward your sitemap.

Why Robots.txt Matters

The robots.txt file sits at the root of every website and serves as the first point of communication between a site and search engine crawlers. It tells bots which directories or pages to skip, which crawl rate to use, and where to find the XML sitemap. Sites without one may see search engines wasting crawl budget on low-value pages like admin panels, staging URLs, or duplicate filtered views. Adding a well-structured robots.txt is one of the simplest and most impactful technical SEO improvements you can make.

How It Works

At the start of each crawl, Crawler fetches <site>/robots.txt and reports:

• Whether the file exists
• Whether a sitemap is referenced in it
• The number of sitemap URLs discovered

This information shows up in:

• The SiteInfo event in CLI output
• The SEO Issues card in the desktop app
• CSV and TXT exports from the seo command

If the robots.txt references a sitemap, Crawler parses it and adds any discovered URLs to the crawl queue. This means you get broader coverage even on sites with deep or orphaned pages that might not be reachable through internal links alone.

Related

• Robots.txt
• Crawl budget
• How to create sitemap.xml
• v0.1.7: llms.txt & SEO CSV