AI SEO
llms.txt Explained: Should Your Brand Add One for AI Crawlers in 2026?
Key takeaways
- Four independent studies across 500,000+ domains found no measurable visibility benefit from llms.txt.
- No major AI provider has committed to parsing it u2014 the one lab reading it at any scale is Meta.
- Google’s Lighthouse check is about agent efficiency, not search: u201cit’s not done for searchu201d (John Mueller).
- Worth having for developer tools, technical SaaS and API products; skip it for local, e-commerce, publishers and standard B2B.
As AI crawlers reshape how people find information, llms.txt has become one of the most confidently recommended tactics in marketing — and one of the least tested. Add the file, the advice goes, and AI tools will understand your site better.
That advice is now old enough to check. Four independent studies covering more than 500,000 domains have measured whether AI systems actually read llms.txt files. The spec itself was revised in August 2026. Google added a check for it to Lighthouse, then clarified it has nothing to do with search.
This guide covers what the file is, how to build one, what the evidence actually shows, and which businesses should bother. The conclusion is more specific than yes or no — it depends heavily on what kind of site you run.
What is llms.txt?
An llms.txt file is a plain text file, written in Markdown, that sits at the root of your domain:
yourdomain.com/llms.txtIts job is to give AI systems a curated guide to your site. Instead of leaving a language model to crawl hundreds of pages and work out what matters, you hand it a short, structured summary: who you are, what you do, and which pages are worth reading.
The format was proposed in September 2024 by Jeremy Howard, co-founder of Answer.AI and fast.ai, as a community-led standard.
The reasoning behind it is sound. Language models work within a limited context window, and a modern web page is mostly overhead — navigation, cookie banners, tracking scripts, interface chrome. Very little of it carries meaning. A clean Markdown index strips all that away and delivers only substance.
A typical llms.txt contains:
- Your brand or site name
- A one-line description of the business
- Grouped links to important pages
- A short description beside each link
- Optional topic categories
It’s worth being precise about what the file does not do. It does not control crawler access — that’s robots.txt. It does not list every URL for indexing — that’s your sitemap. And it carries no enforcement of any kind. It is a suggestion, and nothing obliges any AI system to read it.
llms.txt vs robots.txt vs sitemap.xml
This is the most common point of confusion, and the differences matter more than they first appear:
| robots.txt | sitemap.xml | llms.txt | |
|---|---|---|---|
| Purpose | Controls crawler access | Lists URLs for indexing | Curates key content for AI systems |
| Format | Plain text | XML | Markdown |
| Status | RFC 9309, formal standard | Long-established standard | Community proposal |
| Honored by | Effectively all major crawlers | All major search engines | No major AI provider has committed |
| Enforcement | Voluntary but near-universal | N/A | None |
| Legal weight (EU) | Yes — TDM opt-out signals carry copyright significance under the EU AI Act | No | No |
The last row is the one most guides miss. Since the EU AI Act became fully enforceable on 2 August 2026, machine-readable rights reservations expressed through robots.txt and TDM signals carry genuine legal weight, with penalties reaching €15M or 3% of global turnover. llms.txt carries none.
If your actual question is “how do I control what AI systems do with my content?” — llms.txt is the wrong file entirely. You want robots.txt, and depending on your setup, CDN-level controls.
llms.txt answers a different question: “how do I help an AI system that’s already reading my site understand it faster?” That’s a real question. It’s just a much smaller one.
What changed in spec v2 (August 2026)
The specification was revised to version 2 on 10 August 2026. Most published guidance — including most articles written this year — still describes the original 2024 proposal. If you’re implementing from a tutorial, check its date.
The meaningful changes:
- Declared discovery. You can now point agents to your file using standard link relations rather than hoping they guess the path. Use
rel="describedby"for the llms.txt itself, andrel="alternate" type="text/markdown"for Markdown versions of individual pages — as either an HTML<link>element or an HTTP header. - Flexible Markdown URLs. Both conventions are now valid: appending
.mdto a full URL, or replacing the existing extension. This accommodates how real publishing tools behave. - Subpath support. llms.txt files can now live in subdirectories, with the most specific file taking precedence. If you control only part of a domain, you can still participate.
- A simpler consumption model. V2 drops the original assumption that a dedicated tool would expand the file into model context. The expectation now is straightforward: an agent reads the file and follows whichever links look relevant.
Optionallost its special meaning. The section is now purely descriptive, with no mechanical behaviour attached.
The spec improved. Adoption by AI providers did not.
How to create an llms.txt (and llms-full.txt)
The build is genuinely simple — no development skills required. If your documentation platform generates one automatically, skip this section entirely and let it.
Step 1: Open a plain text editor
Notepad, TextEdit, VS Code — anything that saves plain text. Save the file as llms.txt. Not .docx, not .rtf.
Step 2: Write the header block
Start with an H1 for your brand name and a blockquote summarising what you do. This is the context an AI system reads first:
# Your Brand Name
> One or two sentences describing what your business does,
> who it serves, and what makes it distinct.Step 3: Add grouped links with descriptions
List your most important pages under H2 category headings, each with a short factual description. The description does the work — a bare URL tells a model nothing:
## Services
- [Technical SEO](https://yourdomain.com/technical-seo): Site architecture,
crawl efficiency, and Core Web Vitals for large sites.
- [AI Visibility](https://yourdomain.com/ai-visibility): Measuring and
improving how AI assistants describe your brand.
## Resources
- [2026 Search Trends Report](https://yourdomain.com/report): Annual
analysis of query behaviour across 12 industries.Keep it to genuinely valuable pages. A file listing every URL you own defeats the purpose — you’ve just rebuilt your sitemap in a format nothing reads.
Step 4: Add the v2 discovery link
New in v2, and skipped by most tutorials. Add this to your <head>:
<link rel="describedby" type="text/markdown" href="/llms.txt">Step 5: Upload to your root directory
The file must resolve at yourdomain.com/llms.txt. Placement in a subfolder means nothing will find it. Verify by loading the URL in a browser.
What about llms-full.txt?
llms-full.txt is the expanded variant — instead of links and descriptions, it contains the actual page content inline, so an AI system gets everything in a single fetch without following any links.
For most brands, skip it. It’s substantially larger, harder to keep current, and the maintenance cost scales with your content. It makes sense for documentation sites where the content is already generated from source. It rarely makes sense for a marketing site.
Automate generation if you possibly can. A hand-maintained llms.txt goes stale within months, and nothing warns you when it does. A file that contradicts your live pages is worse than no file at all. Platforms including Mintlify, GitBook, Docusaurus and ReadMe generate and refresh it as part of your build; Squarespace and Shopify now produce one natively.
Getting file placement, headers, and site structure right sits alongside the rest of your technical foundations — see our SEO services for how this fits a broader setup.
Does llms.txt actually work? What four studies found
This is the question that matters, and unlike a year ago, it now has real answers. Not opinions — server logs.
Ahrefs: 137,000 domains
Published June 2026. Adoption was healthier than expected at 28% of domains carrying a valid file. Then the number that should end most of these conversations: 97% of those files received zero requests during the study month. Not low traffic. Zero.
Ryan Law’s conclusion was unambiguous: “There’s no evidence that llms.txt improves AI retrieval, boosts traffic, or enhances model accuracy. And no provider has committed to parsing it.”
SE Ranking: ~300,000 domains
A larger sample, modelling whether the presence of llms.txt predicted AI citations. Removing llms.txt from the model improved its predictive accuracy. In statistical terms, the file behaved as noise rather than signal.
EZY.AI: 83 sites, server logs, April–July 2026
Deployed llms.txt across 83 real sites and watched the logs for roughly three months.
| Crawler | llms.txt fetches | robots.txt fetches (same window) |
|---|---|---|
| Meta | 193 | — |
| Googlebot | 67 | 5,125 |
| Anthropic | 9 | 3,120 |
| OpenAI | 7 | 3,990 |
| Perplexity | 0 | 775 |
Two things stand out. Every lab fetches robots.txt hundreds of times more often than llms.txt. And the one AI company reading llms.txt at any scale is Meta — the lab almost nobody optimises for.
Otterly.ai: 62,100+ AI bot visits over 90 days
Just 84 visits — roughly 0.1% — went to /llms.txt. The average content page received around 265.
Four studies. Different samples, different methods, same finding.
Why Google added it to Lighthouse
Here’s where most of 2026’s confusion originates.
In May 2026, Google added an Agentic Browsing category to Lighthouse, and one of its checks looks for a machine-readable summary at the domain root. Audit tools started flagging its absence, and a lot of brands started being told they had a problem.
Google’s stated rationale is narrower than it looks: “Without llms.txt, agents may spend more time crawling the site to understand its high-level structure and primary content.”
When Lily Ray pressed on the apparent contradiction — Google saying you don’t need these files for AI search, then auditing for one — John Mueller answered directly on 20 May 2026:
“The short answer is that it’s not done for search.”
The distinction he drew is the one to hold onto. Search visibility is about discovery — being found. The Lighthouse check is about functionality — helping an agent work efficiently once it’s already on your site, usually mid-task.
Mueller was blunter still about audience: “For non-developer sites, I don’t think this makes much sense.” And on priorities: your site has “much more important things to do for SEO than to prepare for a potential future situation that may or may not come.”
A Lighthouse flag is a checklist item in an agent-workflow category. It is not a ranking signal, and it is not a penalty.
Should your brand add one in 2026?
Adoption data makes the answer’s shape obvious before you read a word of analysis. Developer tools sit near 69% adoption. SaaS near 67%. Media sits at 0%. This is not a question with one answer.
Where llms.txt stands today
| Factor | Assessment |
|---|---|
| Effort required | Low |
| Cost | Minimal |
| Technical complexity | Low |
| Implementation risk | Low |
| Measured visibility impact | None found across 500,000+ domains |
| Provider commitment | None from any major AI lab |
| Reversibility | Complete — delete the file |
| Best-evidenced use case | Coding agents reading product documentation |
Worth doing
- Developer tools and technical SaaS. This is the real use case, and it is well-evidenced. Coding assistants and IDE agents genuinely do fetch documentation mid-task. Clean Markdown reference material makes your product easier to build against — which affects adoption, not rankings.
- API and integration-led businesses. Same logic. Your buyers evaluate you through an AI assistant while writing code.
- Anyone whose platform generates it automatically. If Mintlify or GitBook produces the file as part of your build, there is no argument for turning it off. Zero marginal cost, non-zero optionality.
Not worth prioritising
- Local and service businesses. No evidence of benefit. Your Google Business Profile, review velocity, and structured data all matter enormously more — see local SEO.
- E-commerce. Product data reaches AI systems through merchant feeds and schema markup, not a Markdown index.
- Publishers and media. 0% adoption, for sound reasons. Your real exposure question is licensing and blocking.
- Most B2B marketing sites. If your site is forty pages of positioning copy, AI systems parse it perfectly well already.
For enterprise teams
At enterprise scale the file itself is trivial; the governance around it isn’t. If you proceed, the work is deciding which content hubs are authoritative, assigning clear ownership for updates, and building regeneration into your existing publishing pipeline rather than treating it as a one-off task.
Be realistic about opportunity cost. An enterprise SEO team has finite capacity, and hours spent curating a file with no measured impact are hours not spent on the items below.
What actually moves the needle instead
If your underlying goal is AI visibility, three 2026 developments deserve your attention well before llms.txt does.
Cloudflare’s 15 September 2026 deadline. Cloudflare will begin default-blocking “mixed-use” AI crawlers — bots blending search indexing with AI training — on monetised pages, starting with new customers, new sites, and free-tier users. If you’re on Cloudflare, this changes who reaches your content automatically, whether or not you act. Check your configuration.
The EU AI Act, fully enforceable since 2 August 2026. Machine-readable rights reservations now carry copyright significance. Your robots.txt became a legal instrument. Most brands have not revisited theirs.
The standards that actually got backing. Agentic Resource Discovery (ARD) launched in June 2026 under a Linux Foundation working group with Google, Microsoft, GitHub, NVIDIA, Salesforce and Snowflake behind it — solving roughly llms.txt’s problem, with real governance. WebMCP reached a Chrome origin trial, exposing site functions as callable tools rather than describing content. That’s where vendor investment went in 2026.
And the fundamentals have not moved: content quality, structured data, topical authority, site health, and brand trust signals still determine whether AI systems cite you. A single file does not. If you want a clear read on where your site actually stands, a free AI visibility audit is a more useful starting point than any checklist item.
Three risks worth naming
Maintenance debt. Every hand-written llms.txt is a second version of your site to keep current. Most rot within months, silently.
Prompt injection surface. A file written specifically to be ingested by AI agents is, by design, untrusted input to a language model. If anyone but you can influence its contents, treat that as a security consideration.
A competitive content map. You’re publishing a tidy, structured guide to your most valuable pages. Harmless for most brands. Worth thinking through if your content architecture is a genuine moat.
Conclusion
llms.txt is a well-designed answer to a real problem that the companies building AI systems have, so far, declined to adopt. Four independent studies across more than 500,000 domains found no measurable benefit. No major provider has committed to parsing it. The one lab reading it consistently is Meta.
That could change. If OpenAI or Google publicly commits to parsing llms.txt, the calculus flips overnight — and because implementation is cheap and completely reversible, being late would cost you very little. Hold it as a watch item, not a work item.
One warning about your other sources. A striking amount of 2026 llms.txt content is AI-generated and contains fabricated vendor confirmations — claims that Anthropic or Perplexity have “confirmed support,” which server logs flatly contradict. A widely circulated claim about a “June 2026 W3C proposal to standardise llms.txt” traces back to an investigation ticket opened in April 2025 that never advanced.
If an article tells you a major AI lab officially honours llms.txt, ask where that’s documented. So far, nowhere.
Working out which AI visibility tactics deserve budget and which are noise is the harder half of the job. Talk to us about your AI search strategy, or read how we approach LLM SEO.
Frequently asked questions
Is llms.txt an official web standard?
No. It’s a community proposal, published by Jeremy Howard of Answer.AI in September 2024 and revised to version 2 in August 2026. It has no standards body behind it. Support is entirely voluntary, and no major AI provider has committed to it.
Does ChatGPT read llms.txt?
There’s no evidence it does at any meaningful scale. In a three-month server-log study across 83 sites, OpenAI’s crawlers fetched llms.txt seven times total, while fetching robots.txt nearly 4,000 times over the same window. OpenAI has never publicly committed to parsing the file.
Does Google use llms.txt?
Not for search. Google added an llms.txt check to Lighthouse’s Agentic Browsing category in May 2026, but John Mueller clarified that “it’s not done for search” — it’s aimed at helping agents complete tasks on a site, not at discovery or ranking.
Do I need an llms.txt file?
Most brands don’t. If you run developer tools, technical SaaS, or an API-led product with substantial documentation, it’s worth having — coding agents genuinely fetch docs mid-task. For local businesses, e-commerce, publishers and standard B2B marketing sites, there’s no measured benefit. If your platform generates one automatically, keep it; the cost is zero.
What’s the difference between llms.txt and robots.txt?
robots.txt controls crawler access and is a formal standard honoured by effectively every major crawler — and since August 2026 it carries legal weight under the EU AI Act. llms.txt curates content for AI systems, has no enforcement mechanism, and no major provider has committed to reading it. They solve different problems, and only one of them is load-bearing.
What is llms-full.txt?
An expanded variant containing full page content inline rather than links and descriptions, so an AI system gets everything in one fetch. It’s much larger and harder to maintain. Sensible for documentation sites where content is generated from source; rarely worth it for a marketing site.
Will adding llms.txt improve my AI search rankings?
There’s no evidence that it will. Across Ahrefs’ 137,000-domain study and SE Ranking’s 300,000-domain analysis, no measurable citation or visibility benefit was found — SE Ranking’s model actually performed better with the llms.txt variable removed.
