llms.txt: What It Is and Whether It's Worth It

Google names llms.txt in its own documentation and says Search ignores it, which should have ended the debate a year ago. It did not, because the file gets sold as an AI-visibility tactic and the thing it was actually designed for is unglamorous. This guide separates the two, using what the engines and the proposal's own author put in writing.

The short version: if you publish technical documentation, an llms.txt file is worth about twenty minutes. If you run a service business, a shop or a brochure site, it does nothing for you, and anyone charging for one is selling you the meta keywords tag of 2026.

What Google actually says

There is no ambiguity to interpret here. Google's generative AI optimization guide, last updated 10 July 2026, carries a section titled "Mythbusting generative AI search: what you don't need to do." It states:

"You don't need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search (including its generative AI capabilities), as Google Search itself doesn't use them."

And, naming the file directly:

"It's completely fine if you decide to create and maintain LLMS.txt files (or other similar files) for other services or systems that use these files. Doing so will neither harm nor help your site's visibility or rankings in Google Search, as Google Search ignores them."

The same section adds that "Structured data isn't required for generative AI search, and there's no special schema.org markup you need to add." That sentence is worth holding onto, because the market for AI-specific files and AI-specific markup does not shrink just because the largest engine has published a page saying it does not read them. We made the same point in our GEO guide, and Google has since made it more plainly than we did.

What the proposal actually claimed

The second surprise is that the specification never promised what it is being sold as. llmstxt.org, published by Jeremy Howard in September 2024, describes a different problem entirely: an agent trying to use a website while it works.

"Web pages are built for people. An HTML page wraps its information in navigation, ads, and JavaScript, and converting it back into clean text is difficult and imprecise."

Its stated expectation is that the file "would mainly be useful for inference rather than training" — in other words, for the moment an assistant needs to look something up, not for the moment a model is built. And it draws the line with sitemaps itself: "While sitemaps list all pages for search engines, llms.txt offers a curated overview for LLMs."

So the gap is not between the file and the engines. It is between the file and the story told about it. A curated reading list for an agent that is already on your site is a reasonable idea. It is simply not a ranking or citation mechanism, and it was never presented as one.

Who publishes one, and why that tells you the answer

Look at which sites carry an llms.txt today and the intended use becomes obvious. OpenAI, Anthropic, Perplexity and Cloudflare all publish one — and all four publish it for their developer documentation, not their marketing site. The companies building the models use the file the way the proposal describes: to help a coding agent find the right page in a large technical corpus.

That is also the tell for your own decision. If an agent would plausibly be pointed at your site mid-task — you publish an API, an SDK, a technical reference, a long product manual — the file has a job. If the realistic reader of your site is a person deciding whether to hire you, there is no agent to serve, and a reading list at your domain root is a file nobody requests.

Note also what OpenAI's crawler documentation tells site owners to do. It describes four crawlers, and it explains control through robots.txt, user-agent strings and published IP ranges. It does not ask you to publish an llms.txt for its bots to read. If you want to decide which of those crawlers to allow in the first place, we worked through that separately in AI crawlers: allow or block.

The thing that works instead

Cloudflare built an agent-readiness score and published what it measures, which makes it the most useful public check on this whole category — with the obvious caveat that Cloudflare sells tooling in it.

The revealing detail is what its scan does by default. From the announcement, dated 17 April 2026: "By default, we only check whether the site correctly handles Markdown content negotiation, and do not check for llms.txt. You can customize the scan to include llms.txt if you choose to."

Markdown content negotiation means that when a request arrives with an Accept: text/markdown header, the server returns a clean Markdown version of that page instead of HTML. Cloudflare measured "up to 80% token reduction in some cases." It works on every URL you already have, it needs no second file to keep in sync, and it degrades to normal HTML for everyone else.

Two numbers from that chart change how you should prioritise. Sitemaps are effectively universal at 78%, so they are table stakes rather than an edge. Markdown content negotiation passes on 3.9% of sites — which means doing it puts you in a small minority, for work that is a header check and a rendering path rather than a new artefact to maintain.

One honest limit before you build it. Cloudflare reports that as of February 2026, "of 7 agents tested, only Claude Code, OpenCode, and Cursor request content with the Accept: text/markdown header by default." Those are coding agents. So Markdown negotiation is better-targeted than llms.txt, but it is aimed at the same narrow audience, and neither one is a consumer-search tactic.

The one measured benefit, and who measured it

There is a published figure, and it deserves its caveat attached. Cloudflare pointed an agent — Kimi-k2.5 running in OpenCode — at large technical documentation sites' llms.txt files and asked highly specific technical questions. It reports that "the agent pointed at Cloudflare's documentation consumed 31% fewer tokens and arrived at the correct answer 66% faster than the average site that is not refined for agents."

That is a real measurement with a real method, and it is a vendor measuring its own documentation against unnamed comparators. Read it as evidence that a well-structured documentation site is cheaper for an agent to use — which is plausible and consistent with the proposal — and not as evidence that adding a file to a site without documentation produces anything.

What we do for clients

We add an llms.txt when a client publishes documentation, and we generate it from the same source the site is built from, so it cannot drift. We do not add one to a marketing site, a shop or a brochure, and when it is requested we say why in one sentence rather than billing for it.

What we do instead is the unglamorous list that Google's own guidance points at: content that is crawlable without executing JavaScript, pages that answer the question they are titled with, structured data where a rich result depends on it, and clean URLs that return the same content to a person and a machine. Our GEO service page sets out how we scope that work, and our guide to Google AI Overviews covers the measurement side, which is where most of this category's honesty problems actually live.

Deciding in five minutes

1. Do you publish documentation an agent would be pointed at mid-task? If no, stop here. You do not need the file.
2. If yes, can you generate it automatically? A hand-maintained list becomes wrong within one release cycle, and a wrong map is worse than none.
3. Have you done Markdown content negotiation? It applies to every URL, needs no second artefact, and 96% of sites have not done it.
4. Is anyone telling you this affects Google? Google has published a page saying it does not. That claim is a reliable signal about the person making it.
5. Is your crawlability actually fine? Almost every real AI-visibility problem we are asked to fix turns out to be a page that does not render without JavaScript, not a missing text file. And before any of it, check what your analytics can actually attribute.

If you want a straight read on which of these your site actually needs, tell us what you publish and we will say which ones apply — including when the answer is none of them.