There is currently no public evidence that llms.txt improves AI search visibility. Google has stated repeatedly that no AI system reads it, Gary Illyes confirmed Google has no plans to support it, and no major AI provider has committed to honouring it. It has one genuine use case, developer documentation for AI coding tools, which is what its creator designed it for and is not the same thing as GEO.
If you have received a proposal recently with “llms.txt implementation” as a line item, this post is the due diligence. It is not an argument that the idea is stupid. It is a summary of what has actually been said and measured, so you can decide with the evidence rather than the pitch.
What llms.txt actually is
llms.txt is a plain Markdown file placed at the root of a website. It provides a curated, low-noise index of the pages the site owner considers most important, with brief summaries. The idea is that instead of forcing a model to parse hundreds of HTML pages, you hand it a map.
It is worth being precise about what it is not:
- It is not a blocking tool. It cannot restrict any crawler or prevent any AI system from reading your site. That is robots.txt.
- It is not a ranking factor. It has no effect on Google Search rankings.
- It is not an official standard. It is a community proposal, not something published by Google, OpenAI or Anthropic.
Where it came from, and what it was for
This is the part that explains most of the confusion.
llms.txt was proposed on 3 September 2024 by Jeremy Howard, co-founder of Answer.AI and fast.ai, and published at llmstxt.org. Howard was not trying to help marketers rank in AI search. He was solving a specific technical problem: LLM context windows were too small to hold entire websites, and converting HTML full of navigation, advertising and JavaScript into clean text was wasteful and laborious.
The target user was an AI coding assistant trying to read API documentation. If a model burns its context window on menu markup before reaching the actual reference material, the documentation is useless to it. A curated Markdown index solves that.
Adoption grew in that context. Mintlify, a documentation hosting platform, rolled out support across its hosted docs in November 2024, and other documentation sites followed. Then the SEO industry noticed the file and began repackaging it as an AI search opportunity.
What Google has actually said
Google’s position has been consistent and repeatedly stated. The dates matter, because this is not one offhand comment.
Two additional points from Google are worth quoting in substance.
On the redundancy problem, Mueller made the observation that if you use an LLM to generate your llms.txt file, the LLM could evidently have produced that summary itself, which rather undermines the argument that it needs the file.
On the format argument, Mueller and Martin Splitt addressed Markdown directly on Google’s Search Off the Record podcast. HTML has been the web standard for far longer than Markdown has existed, every crawler is built around it, and converting HTML to text is trivial. Splitt added a point that cuts deeper: Markdown strips out the link structure, navigation and heading relationships that models use to understand context. On that argument the file is not neutral, it discards signal.
What the data shows
Opinion from Google is one thing. Measurement is better.
The largest study on this question to date came from SE Ranking in November 2025, analysing around 300,000 domains. It found no clear effect of llms.txt on AI citations. Sites with the file were not measurably more likely to be cited than sites without it.
Common Crawl, the corpus behind a large share of LLM training data, ingests raw HTML. Not Markdown, and not llms.txt. Research published at ACM FAccT 2024 found Common Crawl fed at least 64% of all LLMs released between 2019 and late 2023. If the training pipeline is reading HTML, a Markdown file sitting alongside it has no obvious route into the model.
The fair case for adding one anyway
An honest post has to put the other side properly, and there are three arguments worth taking seriously.
1. The cost is close to zero
Yoast and Rank Math both generate the file with one click. If it takes five minutes and does nothing, the downside is five minutes. That is a defensible position, and it is a different argument from claiming it works.
2. Standards need early adopters
Sitemaps were a proposal before they were a standard. If enough sites publish llms.txt, providers may eventually find it worth reading. Someone has to go first for any convention to establish itself.
3. It may enter training data as content
Some practitioners argue that even if no system reads llms.txt as a protocol, the file is still text on your domain and could be ingested like any other page. This is speculative and unmeasured, but it is not absurd.
Two caveats. First, opportunity cost: if llms.txt is a line item on an invoice, you are paying for it, and that budget could go to work with evidence behind it. Second, Mueller’s comparison to the keywords meta tag is pointed. That tag died because self-declared signals with no verification are trivially gamed. Any future llms.txt support would face exactly the same problem.
So should you add one?
What to do with the budget instead
The reason this matters is not that llms.txt is harmful. It is that attention spent on it is attention not spent on things with evidence behind them.
- Get named in third party sources. Models weight independent mentions heavily. Roundups, directories, review platforms and press coverage move citation rates in a way a root-level file does not.
- Add statistics and quote credible sources. The peer-reviewed GEO research found these among the highest performing content changes, with gains up to 40%. Covered in our post on what GEO, AEO and SEO actually mean.
- Write clearly. The same research found improving fluency alone, adding no new information, produced roughly 28% gains.
- Fix entity consistency. One canonical name, address, phone number and description everywhere, so models can connect mentions to one company.
- Keep the SEO foundation strong. Ranking position remains one of the strongest predictors of citation.
Getting a GEO proposal? Bring us the questions.
We will tell you which parts of any AI visibility proposal have evidence behind them and which are trend-chasing, including ours. Then we will show you where your actual gap is.
Explore AI SEO, GEO & AEO servicesFrequently asked questions
Does llms.txt improve AI search visibility?
There is currently no public evidence that it does. SE Ranking’s November 2025 analysis of around 300,000 domains found no clear effect of llms.txt on AI citations. Google’s John Mueller has stated that no AI system uses the file, and no major AI provider including OpenAI, Anthropic, Google or Perplexity has publicly committed to reading it.
Does Google use llms.txt?
No. Gary Illyes confirmed at Google Search Central Live in July 2025 that Google does not support llms.txt and has no plans to. John Mueller stated in June 2025 that no AI system currently uses it, citing server log evidence, and in 2026 described it as purely speculative. Google’s published guidance on AI features separately states that no special file formats are required.
Who created llms.txt and why?
Jeremy Howard, co-founder of Answer.AI and fast.ai, proposed it on 3 September 2024. His goal was not AI search optimisation. He was solving a technical problem in developer documentation, where LLM context windows were too small to hold entire websites and parsing HTML full of navigation and scripts wasted the available context before reaching the actual reference material.
Is there any situation where llms.txt is genuinely useful?
Yes. If you publish extensive developer or API documentation and your users work with AI coding assistants such as Cursor, GitHub Copilot or Claude in an IDE, a well structured llms.txt gives those tools an efficient entry point to your reference material. That is the use case it was designed for. It is developer relations rather than SEO, and it applies to a small fraction of websites.
Should I pay an agency to implement llms.txt?
Yoast and Rank Math generate the file with one click, so paying a meaningful fee for implementation is hard to justify. If llms.txt appears as a priced deliverable in a GEO proposal, a reasonable question is to ask the agency to name a single AI provider that has confirmed reading it, and to reconcile the deliverable with Google’s published position that no special file formats are needed.
Could llms.txt work in the future?
Possibly. It is a community proposal and standards do sometimes gain adoption after a slow start. However, Mueller has drawn a comparison to the keywords meta tag, which search engines abandoned because self-declared signals with no independent verification are easy to abuse. Any future adoption would need to solve that problem. Mueller has separately expressed more interest in the WebMCP approach, which gives agents defined tasks and processes rather than a list of preferred pages.
Sources
- Howard, J. “The llms.txt proposal”, Answer.AI and llmstxt.org, 3 September 2024.
- Mueller, J. Public statement comparing llms.txt to the keywords meta tag, April 2025. Reported via Search Engine Journal.
- Mueller, J. “No AI system currently uses llms.txt”, public post, 17 June 2025. Reported via Search Engine Roundtable.
- Illyes, G. Statement at Google Search Central Live, July 2025, confirming no support and no plans to add it.
- Mueller, J. “Purely speculative for now”, 2026. Reported via Search Engine Journal.
- Mueller, J. and Splitt, M. Search Off the Record podcast, discussion of Markdown versus HTML for crawlers.
- SE Ranking, “LLMs.txt shows no clear effect on AI citations”, analysis of approximately 300,000 domains, November 2025.
- Baack, S. “A Critical Analysis of the Largest Source for Generative AI Training Data: Common Crawl”, ACM FAccT 2024. doi:10.1145/3630106.3659033.
- Aggarwal, P. et al. “GEO: Generative Engine Optimization”, KDD 2024.
- Google Search Central, published guidance on AI features and file format requirements.
Note on sourcing: several Google statements above were made in public posts, podcasts or at live events and are cited here via the industry publications that reported them. Where possible, verify against the original before quoting in your own work.
Muhammad Asjad Khan is a seasoned digital marketer and the Chief Operating Officer at Digital Age, where he leads strategy, innovation, and client success. With years of experience in SEO, content marketing, and performance-driven digital campaigns, Asjad is passionate about helping brands grow their online presence and achieve measurable results. When he’s not optimizing websites or scaling marketing strategies, he’s exploring the latest trends in tech and digital media
