Guides & Tutorials
Step-by-step tutorials for getting the most out of crawler.sh.
How to Crawl Data to Train AI Model with CLI
Learn how to crawl website content and extract clean Markdown for AI training datasets using crawler.sh CLI. Export structured data for LLM fine-tuning.
How to Find Broken Links of a Website with CLI
Learn how to detect broken links and dead pages on any website using crawler.sh CLI. Crawl your site, identify 4xx/5xx errors, and export a report.
How to Find Empty H1 Tags with CLI
Learn how to detect pages with empty H1 tags using crawler.sh CLI. Find headings that contain no text and fix them to improve SEO and page structure.
How to Find Duplicate Descriptions with CLI
Detect pages sharing the same meta description using crawler.sh CLI. Find duplicates and write unique snippets for better search visibility.
How to Find Duplicate Titles with CLI
Learn how to detect pages sharing the same title tag using crawler.sh CLI. Find duplicate titles that confuse search engines and dilute your rankings.
How to Find Long Content with CLI
Learn how to detect pages with over 5,000 words using crawler.sh CLI. Find excessively long pages that may need to be split for better user experience and SEO.
How to Find Duplicate H1 with CLI
Learn how to detect pages sharing the same H1 tag using crawler.sh CLI. Find duplicate headings that confuse search engines and differentiate your page topics.
How to Find Long Descriptions with CLI
Learn how to detect pages with long meta descriptions (over 160 characters) using crawler.sh CLI. Find descriptions that get truncated in search results.
Showing 8 of 26 guides
