noindex pages
Pages with noindex directives are excluded from search engine indexes
What it is
crawler.sh flags a page as noindex when it contains a noindex directive in either the <meta name="robots"> tag or the X-Robots-Tag HTTP header. This is check #11 in the SEO analysis.
Why it matters for SEO
A noindex directive tells search engines not to include the page in their index:
- Intentional exclusion - Some pages should be noindexed: thank-you pages, internal search results, admin pages, or staging environments.
- Accidental exclusion - A noindex tag left from development or applied site-wide by mistake can remove important pages from search results entirely.
- Crawl budget impact - Search engines still crawl noindexed pages but won’t show them in results. If many pages are unnecessarily noindexed, crawl budget is wasted.
Caution: An accidental noindex on key pages can cause significant traffic loss. Always verify that noindex directives are intentional.
Why it matters for AEO
AI answer engines respect noindex directives differently than traditional search engines. Some AI crawlers may still access and learn from noindexed content, while others skip it entirely. If you’re intentionally blocking pages, verify that your approach works across both traditional and AI search systems.
How to fix it
Review each noindexed page and decide whether the directive is intentional:
Pages that should typically be noindexed:
- Thank-you / confirmation pages
- Internal search results pages
- Login and account pages
- Duplicate content (filtered/sorted views)
- Staging or preview environments
Pages that should NOT be noindexed:
- Main content pages (blog posts, product pages, landing pages)
- Category and tag pages (unless duplicating other pages)
- Any page you want to appear in search results
To remove a noindex directive, delete the meta tag or HTTP header:
<!-- Remove this --><meta name="robots" content="noindex">What crawler.sh reports
In the CLI, noindex pages appear under the “Noindex pages” section of crawler seo output. Each affected URL is listed. In the desktop app, they appear in the SEO Issues card. The report surfaces both meta_robots and X-Robots-Tag sources.
