llms.txt Is Not a Ranking Signal. Here's What Is.
Quick answer: No major AI provider has committed to consuming /llms.txt, and Google has said outright that it does not support the format and does not plan to. The file is cheap enough to ship and it has one legitimate use — a curated entry point for agents and tools you or your users point at it. It is not a visibility lever, and treating it as one has been absorbing budget that belongs somewhere else.
Two years into the llms.txt discourse, the format has an unusual profile: enormous mindshare, near-zero confirmed consumption. That gap is worth understanding precisely, because the useful conclusion is neither "ship it, it's the new sitemap" nor "ignore it, it's snake oil."
What the proposal actually says
Jeremy Howard published the llms.txt proposal on September 3, 2024. The spec is small: a markdown file at the site root, one required H1 title, an optional blockquote summary, and H2-delimited lists of links. A companion convention suggests serving a .md version of each page at the page URL plus .md.
The stated problem is real. A model assembling context from a website has to decide what to pull — sitemap, external links, source code — and the proposal's argument is that site authors know best and should curate that themselves. That is a sound engineering instinct. It is also, importantly, a proposal about context assembly for tools, not about ranking or citation. The original post makes no claim that any provider ingests the file.
The SEO industry supplied that claim on its own.
Who consumes it: the honest answer
Google has been explicit. At the Search Central Deep Dive event in Asia Pacific on July 23, 2025, Gary Illyes stated that Google does not support llms.txt and is not planning to — reported by Search Engine Land the following day. The same session produced the blunter line that appearing in AI Overviews takes normal SEO, not GEO or LLMO.
Google's own documentation has since said the same thing in writing. The AI features page states that you need "no new machine readable files, AI text files, or markup to appear in these features," and that there are "no additional requirements to appear in AI Overviews or AI Mode." That sentence was written to end this exact debate.
The awkward detail people cite as counter-evidence: several AI companies publish an llms.txt for their own documentation. That is true and it proves nothing about retrieval. Publishing a curated index of your docs so that developers can paste it into a model is a documentation decision. It is not the same as your crawler fetching other people's llms.txt files and weighting them. Nobody has documented the latter.
The crawl economics nobody puts in the pitch deck
While the industry argued about a 2KB markdown file, the actual traffic relationship shifted hard. Cloudflare published crawl-to-refer ratios — HTML pages crawled per HTML page referral sent back — and for the week of June 19 to 26, 2025 the spread across AI platforms ran from roughly 70,900:1 at the top down to 0.1:1 at the bottom. Those ratios move week to week and that snapshot is now over a year old, so treat the exact figures as a period reading rather than a constant. The direction — more crawling, fewer referrals — has been consistent since.
That reframes the question. If a platform fetches tens of thousands of your pages per visitor it returns, your leverage is not in publishing an extra file to help it. Your leverage is in deciding, deliberately, which crawlers get access to what — which is a robots.txt decision, and it is enforced, unlike llms.txt, which is a suggestion nobody promised to read.
Should you ship one anyway?
Yes, if it takes you under an hour and you are honest about why. No, if it is displacing work that changes outcomes.
| If you are… | Worth it? | Why |
|---|---|---|
| A docs or API site | Yes | Developers paste it into models by hand. Real usage today. |
| Shipping an agent or MCP server | Yes | You control the client, so you control whether it reads the file. |
| A content or marketing site | Marginal | Cheap to ship, no confirmed consumer. Do it last. |
| Paying an agency for it | No | It is a generated file. The generator is free. |
If you do ship one, ship it as a real index — your genuinely important pages with one-line descriptions — not a dump of every URL in your sitemap. A curated file has a plausible reader. A 4,000-line file has none. The same applies to ai.txt if you maintain one for crawler policy.
What actually determines whether you get cited
Google's guidance names the real gate: a page must be indexed and eligible to be shown in Search with a snippet. Everything downstream of that is a retrieval problem, and retrieval systems reward the same three properties regardless of which model is doing the retrieving.
1. The answer survives being cut out of the page
Retrieval pulls passages, not documents. A paragraph that reads correctly in isolation — no "as mentioned above," no pronoun pointing three sections back — is a paragraph that can be quoted. Run a page through a passage extractor and read what comes out. If the extracted chunks are incoherent, no file at your site root fixes that.
2. The structure matches the content
Chunking follows heading boundaries in most pipelines. Skipped levels, decorative headings, and two H1s all produce chunks that straddle topics. A heading structure check plus a look at how the page splits into chunks catches most of it in minutes.
3. The specifics are on the page
Models cite pages that contain the concrete thing — the version number, the pricing tier, the benchmark figure, the exact command. Pages that gesture at a topic without committing to specifics get read and discarded. An AI readability score is a rough proxy, but the honest test is simpler: can a reader answer their question without leaving the page?
None of this is new. It is the same work that makes a page rank, which is precisely what Google said. The distribution side — being the page other people reference when they explain the thing — is the part that does not automate, and we argued that case separately in why solo SEO is dead.
The one thing llms.txt is genuinely good at
Strip the SEO framing and a use case survives. If you build developer tools, your users are already pasting your docs into models, and a curated markdown index is a better paste than a scraped HTML page. That is a documentation quality improvement with an immediate, observable beneficiary — the developer evaluating your product at 11pm. Plenty of the projects in the AI tools and platforms directory ship one for exactly that reason, and it is a defensible reason.
Ship it for that. Do not ship it and then report it upward as an AI visibility initiative.
FAQ
Does Google use llms.txt?
No. Gary Illyes said at Google's Search Central Deep Dive in Asia Pacific on July 23, 2025 that Google does not support llms.txt and is not planning to. Google's AI features documentation separately states that no new machine-readable files or AI text files are needed to appear in AI Overviews or AI Mode.
Will adding llms.txt hurt my site?
Not directly. It is a static file at your root and search engines that ignore it simply ignore it. The cost is opportunity cost: time spent maintaining a file with no confirmed consumer is time not spent on indexing, structure, or content specificity. Some practitioners also prefer to noindex the file so it does not appear as a search result itself.
Is llms.txt the same as robots.txt for AI?
No, and conflating them is the most common mistake. robots.txt is an access control convention that major crawlers honour. llms.txt is a content curation proposal with no enforcement and no committed consumers. If your goal is controlling which AI crawlers reach your content, robots.txt is the file that does that.
What format does llms.txt use?
Markdown at the site root. The proposal requires an H1 title, then allows an optional blockquote summary, optional detail paragraphs, and H2-delimited sections containing lists of links with short descriptions. A companion convention serves a markdown version of each page at the page URL with .md appended.
What should I do first if I want AI visibility?
Confirm the pages you care about are indexed and snippet-eligible. Then make each page answer one question completely, with self-contained paragraphs and headings that match the content structure. Then check what your pages look like after chunking. llms.txt comes after all of that, if at all.
Why do AI companies publish llms.txt if they don't read it?
Because publishing a curated markdown index of your own documentation helps developers who paste it into a model manually. That is a documentation decision about their own site. It says nothing about whether their crawlers fetch and weight llms.txt files on other people's sites.
A file nobody has promised to read is not a strategy. It is a twenty-minute task with a good excuse attached.