A nofollow directive on a page tells search engines not to follow any links on that page. This means no link equity (PageRank) flows from the nofollowed page to the pages it links to. While nofollow has legitimate uses - like on user-generated content pages - applying it to important pages can block the flow of authority through your site.

This guide shows you how to find every page with a nofollow directive using the crawler.sh CLI.

Step 1: Install crawler.sh CLI

Install the CLI with a single command:

curl -fsSL https://install.crawler.sh | sh

This downloads the correct binary for your operating system and architecture, places it in ~/.crawler/bin/, and adds it to your PATH. Restart your terminal or run source ~/.bashrc (or ~/.zshrc) to pick up the new PATH entry.

Verify the installation:

crawler --version

Step 2: Crawl the target website

Run a full crawl of the website you want to audit:

crawler crawl https://example.com

The crawler checks both the <meta name="robots"> tag and the X-Robots-Tag HTTP header for nofollow directives. Results are saved as an NDJSON file (.crawl) in the current directory.

Step 3: Run SEO audit

Run the SEO analysis on your crawl data:

crawler seo example-com.crawl

The nofollow pages check flags every page that contains a page-level nofollow directive.

Step 4: Identify nofollow pages

Look for the Nofollow Pages section in the SEO report. Review each flagged page to understand the impact. Common scenarios:

CMS plugins that add nofollow to archive or tag pages
Pages with user-generated content where nofollow was applied too broadly (page-level instead of link-level)
Security-focused settings that add nofollow to all non-core pages
Inherited directives from staging or development configurations

Step 5: Fix and re-crawl

For each flagged page:

Remove page-level nofollow if the page links to important internal pages
Use link-level nofollow (rel="nofollow") on individual external or untrusted links instead of nofollowing the entire page
Keep page-level nofollow only on pages where you genuinely want to prevent all link equity from flowing (rare in practice)

After fixing, re-crawl to verify:

crawler crawl https://example.com
crawler seo example-com.crawl

Why nofollow pages matter for SEO

Internal link equity is how search engines discover and prioritize your most important pages. When a page has a nofollow directive, all of its outgoing internal links become dead ends for authority flow. This can leave important deeper pages without enough authority to rank well. Auditing nofollow directives ensures your internal linking structure works as intended and that authority flows to the pages that need it most.