llms.txt is a proposed standard for a text file that website owners can place at the root of their domain to provide structured information about their site to large language models (LLMs) and AI crawlers. Similar to how robots.txt guides traditional web crawlers, llms.txt aims to help AI systems understand what a site offers and how to interpret its content.
As AI-powered search and chat interfaces become more common, website owners need a mechanism to communicate with these new types of crawlers. Traditional robots.txt was designed for search engine indexing, not for training data collection or AI summarization. llms.txt fills this gap.
Purpose of llms.txt
As AI crawlers become more common, website owners want a way to:
- Indicate whether AI training on their content is permitted
- Provide context about the site that helps AI systems summarize or reference it accurately
- Set boundaries on how AI systems should use the content, such as requiring attribution or prohibiting verbatim reproduction
- Offer preferred contact or attribution information for AI companies
- Specify which parts of the site are appropriate for AI consumption versus which are private or time-sensitive
Some publishers welcome AI training because it drives discovery and traffic. Others worry about uncompensated use of their intellectual property. llms.txt attempts to give both groups a standardized way to express their preferences.
How llms.txt works
The file lives at /.well-known/llms.txt or /llms.txt and uses a simple text format. It may include sections like:
- Overview - A summary of the site and its purpose
- Opt-in / opt-out - Whether AI training is allowed, disallowed, or allowed with conditions
- Attribution - How the site should be credited when referenced by AI systems
- Contact - How to reach the site owner for AI-related questions or licensing inquiries
- Exclusions - Specific paths or content types that should not be used for training
Here is a hypothetical example:
# llms.txt for example.com
## OverviewExample.com publishes in-depth tutorials on web development and SEO.
## AI Usage- Training: allowed- Attribution: required- Commercial use: allowed with attribution
## Contactai-contact@example.com
## Exclusions/private//staging/llms.txt vs robots.txt
| robots.txt | llms.txt |
|---|---|
| Targets traditional search crawlers | Targets AI and LLM crawlers |
| Focuses on crawl access and frequency | Focuses on usage rights and attribution |
| Established standard (1994) | Emerging convention (2024+) |
Uses Disallow for paths | Uses opt-in/opt-out and usage guidelines |
| Enforced by well-behaved crawlers | Respected on a voluntary basis |
| Single purpose: crawling control | Multiple purposes: training, attribution, contact |
These files are complementary, not competing. A site might use robots.txt to block /admin/ from all crawlers while using llms.txt to allow AI training on /blog/ but not /paid-reports/.
Current adoption
llms.txt is an emerging standard rather than a formal specification. Some AI companies have stated they will respect it, but adoption is not universal. Website owners who implement it should not rely on it as the sole protection against unwanted AI crawling. Complementary measures may still be needed:
- Blocking specific AI user agents in
robots.txt(OpenAI-GPT, anthropic-ai, etc.) - Terms of service clauses that prohibit scraping for AI training
- Technical measures like rate limiting and CAPTCHA for suspicious traffic patterns
- Legal frameworks such as the robots.txt protocol combined with terms of service
The situation is evolving rapidly. What is best practice today may change as standards bodies and courts weigh in on AI training data usage.
Should you implement llms.txt?
Consider implementing llms.txt if:
- You publish original research or creative content that AI systems might train on
- You want to set clear attribution requirements for AI-generated summaries of your work
- You operate in a jurisdiction with emerging AI copyright regulations
- You want to signal professionalism to AI companies and standards bodies
- You have mixed feelings about AI training and want granular control
You may skip it if:
- Your site is primarily transactional or functional (login portals, dashboards)
- You do not create original content worth protecting
- You are satisfied with robots.txt alone for crawler management
How crawler.sh checks for llms.txt
crawler.sh detects the presence of llms.txt during a crawl and reports it in the site info summary. The crawler seo command includes a site-level check that flags whether the file exists, helping you verify your AI opt-in/opt-out signaling is in place.
This is useful for auditing your own sites or checking whether competitors have adopted the standard. The crawl output records the HTTP status code (200, 404, etc.) and response size so you can see whether the file is properly configured or returns an error.
