MTLD (Measure of Textual Lexical Diversity) measures lexical diversity independent of text length. Unlike Type-Token Ratio (TTR), which drops as text grows, MTLD stays stable whether you analyze a 200-word paragraph or a 10,000-word article.
This makes MTLD the preferred metric for comparing content of different lengths - the typical scenario in SEO and content analysis.
How MTLD works
MTLD measures the average length of sequential word strings in a text that maintain lexical diversity. The algorithm reads through the text word by word, tracking the running TTR. When it drops below a threshold (typically 0.72), that segment counts as one “factor.” A new segment then begins.
The final MTLD score is the total number of words divided by the number of factors. A higher score means the text maintains diverse vocabulary for longer stretches before becoming repetitive.
For example:
- A text where vocabulary stays varied for long stretches before repeating might score an MTLD of 100+
- A text that quickly becomes repetitive might score an MTLD of 40-50
- The score is calculated in both forward and reverse directions, then averaged for stability
Why MTLD solves the length problem
The core issue with TTR is straightforward: longer texts reuse common words more, so the ratio drops as length increases. This makes fair comparison between a short product description and a long-form guide impossible.
MTLD avoids this because it doesn’t measure the ratio of unique words to total words. Instead, it measures how far the author can write before running out of new vocabulary. A skilled writer covering a topic in depth will maintain varied language for longer stretches, regardless of total article length.
Research confirms MTLD as one of the most reliable lexical diversity metrics, with minimal sensitivity to text length across samples from a few hundred to tens of thousands of words.
MTLD in content quality assessment
MTLD is particularly useful for:
- Cross-page comparison - Compare lexical diversity across pages of any length on your site. Identify which pages use repetitive language regardless of their word count.
- Competitor benchmarking - Measure your content’s vocabulary richness against competing pages that rank for the same queries, even when content lengths differ significantly.
- Content depth analysis - Low MTLD scores often correlate with shallow content that restates the same points. High scores suggest the author explored the topic from multiple angles using precise, varied language.
- Detecting generated content - Unedited AI-generated text often shows characteristic MTLD patterns, with certain vocabulary ranges over-represented while others are absent.
- AEO optimization - Answer engines extract content based on how well it addresses diverse phrasings of a query. Content with high MTLD naturally covers more semantic ground, making it more likely to be selected as a source.
MTLD vs other metrics
| Metric | Length-sensitive | Calculation | Best for |
|---|---|---|---|
| TTR | Yes (strongly) | Unique words / total words | Comparing texts of equal length |
| Standardized TTR | Partially | Average TTR over fixed-size windows | Medium-length comparisons |
| Root TTR | Partially | Types / sqrt(tokens) | Quick approximation |
| MTLD | No | Avg. words per diversity factor | Comparing texts of any length |
Practical considerations
MTLD requires a minimum text length to produce meaningful results - generally at least 100 words. Very short texts lack enough data for the algorithm to identify valid factors.
The standard threshold of 0.72 works well for English text, but different languages may need adjusted thresholds due to differences in morphology and word frequency distributions.
Most computational linguistics libraries (NLTK, spaCy, LexicalRichness for Python) include MTLD implementations. Calculating scores at scale across a crawled site is straightforward.
How crawler.sh helps
The crawler crawl --extract-content flag extracts clean Markdown content from every crawled page, providing the text input needed to calculate MTLD scores across your entire site. Use this alongside the crawler seo command to correlate lexical diversity metrics with other content quality signals like word count, missing content flags, and thin content warnings.
