llms.txt: What It Does and What It Does Not

Every few months a new file is supposed to fix AI visibility. The current one is llms.txt: a plain Markdown file at the root of your domain that summarises who you are and points to the pages that matter. It was proposed in September 2024 by Jeremy Howard of Answer.AI, and within a year it had become a line item in agency proposals.

We publish one ourselves, so this is not an argument against the file. It is an argument against paying for it as if it were a strategy.

What the file actually is

A proposal, not a standard. There is no standards body behind it, no specification any AI provider has committed to, and no equivalent of the decades of crawler behaviour that make robots.txt dependable. It is a convention a site owner can follow in the hope that a machine on the other end chooses to look.

That matters because the pitch usually borrows robots.txt's authority. "The robots.txt of AI" implies that assistants check for the file before reading your site. Nothing published by the companies that run those assistants supports that.

Who reads it today

• Google has said publicly that it does not use llms.txt for Search or AI Overviews and has no plans to.
• OpenAI documents its crawlers and points site owners at robots.txt. llms.txt is not mentioned.
• Anthropic publishes an llms.txt for its own documentation, but has not said that Claude's retrieval reads the file on third-party sites.
• Coding agents and developer tools are the clearest real consumers. Point one at a documentation domain and it will often fetch the file first, which is what the format was designed for.

The log-file studies published so far point the same way: most llms.txt files are requested rarely or never by AI crawlers. You do not need to take anyone's word for that, including ours. Your server logs will tell you in five minutes whether anything has asked for yours.

What it is good for

Three things, none of them glamorous.

• A forced editorial exercise. Writing one page that states what you sell, to whom, at what price and on what terms exposes every place your site is vague. That clarity is worth more than the file.
• A clean source for tools that do read it. If your buyers or their developers point an agent at your domain, a tidy summary beats a crawl of your navigation.
• Cheap insurance. If a major assistant adopts the convention later, you already have one. The cost of being early is about an hour.

What it is not

It is not a ranking factor, a citation lever or a substitute for the work that does earn citations: structured data that agrees with the visible copy, an organisation entity a model can resolve without guessing, and third-party sources that describe you the same way you describe yourself. We set out how to test an agency on exactly those points in seven questions for evaluating an AEO agency.

The real risk is drift

The failure mode nobody sells against: an llms.txt is a second copy of your facts, and second copies go stale. We learned this on our own file. An audit in August found it describing a service we no longer sold and a price we no longer charged, while every visible page was correct. A file written to be quoted verbatim by machines was the least accurate page on the site.

If you keep one, treat it as a price surface. It changes in the same pass as the page it summarises, or it does not exist.

What we would do with the hour

Write the file, check it against your live pages, put a date on it, and add it to whatever checklist governs price changes. Then spend the budget somebody wanted for "llms.txt optimisation" on schema and corroboration, which is where the evidence points.

If you want to see what assistants say about you now, before anything changes, the Competitor Snapshot runs the same prompts against you and three named competitors.