What is llms.txt, and do you need it?

llms.txt is having a moment. A steady stream of posts insists you need one immediately or you will vanish from AI search, and the claim is compelling enough that a lot of marketing teams have added the file without ever asking what it actually does. The honest picture is more useful than the hype, and it is worth having before you spend time on it.

llms.txt is a plain-text Markdown file at your domain root that gives AI models a curated map of your most important pages. It is cheap to add and low risk, but be honest about the payoff: Google has said it does not affect Search or AI Overviews, and there is little evidence yet that AI crawlers read it widely.

Its clearest use today is documentation. Add it if you like, but do not expect it to move rankings. This piece explains what the file is, how to create one, and where it genuinely helps, without the urgency that usually surrounds it.

What is llms.txt?

llms.txt is a proposed convention, not an official standard, for a file placed at the root of your domain: yourdomain.com/llms.txt. Written in Markdown, it presents a short, curated list of your key pages with links and one-line descriptions.

The idea is to give a large language model a clean, distilled map of what matters on your site, so it can find the important content quickly rather than crawling and parsing everything. Think of it as a hand-picked table of contents aimed at AI, distinct from the machine-exhaustive list a sitemap provides.

It is important to be clear about its status. No search engine mandates it, no major AI provider has committed to ranking or citing based on it, and it emerged from the community rather than from Google or OpenAI. That does not make it useless, but it does mean the confident "you need this or else" framing runs well ahead of the evidence.

What goes in an llms.txt file, and how to create one

The format is deliberately simple. You open with the site name and a short summary, then group your most important pages under a few Markdown headings, each page a link with a brief description.

A typical file lists the pages that best represent what you do: your core service or product pages, your best guides, your key articles. The goal is curation, not coverage, so you are choosing the handful of pages you would most want an AI to understand, not dumping every URL.

Creating one by hand is easy but goes stale, because you have to remember to update it every time you publish. The better approach is to generate it from your content automatically, the same way a good sitemap is generated, so a new article appears in it the moment it ships.

This site does exactly that: it publishes a generated file at /llms.txt that updates on every publish, as a working example of the convention rather than a static artefact. If you are going to add the file at all, generating it is the version worth doing, because a stale curated map is worse than none.

Do you need llms.txt? The honest answer

Here is the part the enthusiastic posts tend to skip. Google has stated in its own documentation that llms.txt has no impact on Search rankings or AI Overviews, and John Mueller has clarified that the file is about agent functionality, helping an AI agent complete a task, rather than about search discovery.

Independent research across tens of thousands of domains has found that almost none of these files are actually read by AI crawlers. So the mechanism by which llms.txt is supposed to improve your AI visibility mostly is not happening yet, however sensible the idea sounds in principle.

That does not mean you should avoid it. Adding the file is low effort and carries little risk, and if adoption grows you are already positioned. But it does mean you should hold realistic expectations and not treat it as a fix for AI visibility.

The real priority for showing up in AI answers is the same as it has always been: crawlable, well-structured content, strong technical foundations, credible third-party citations, and genuine expertise. llms.txt is housekeeping at the edge of that work, not a substitute for any of it.

llms.txt vs robots.txt and sitemaps

Because the three files sit at the root and all speak to crawlers, they get confused, so it helps to separate them cleanly.

robots.txt is about permission: it tells crawlers, including AI crawlers, what they are and are not allowed to access. Getting it wrong can quietly remove you from AI answers entirely, which is why an accidental block is worth ruling out, as blocking AI crawlers covers.

A sitemap is about discovery: it lists every indexable URL so a search engine can find them all. It is exhaustive by design.

llms.txt is about curation: a short, opinionated shortlist of your best content aimed at AI models. It does not replace either of the others, and it carries none of robots.txt's power to help or hurt. If you only have time for one of the three, robots.txt correctness matters most, because it can cost you visibility if it is wrong.

Where llms.txt genuinely helps today

There is one setting where the file has a clear, practical payoff right now: documentation. AI coding assistants and agents working with a product's docs benefit from a short, curated map of the technical pages, because it helps them find the right reference quickly and complete a task accurately.

If you run developer documentation, a knowledge base, or a product that AI agents interact with, llms.txt earns its place, because the agent-functionality use case Mueller described is real and active. That is a genuine reason to add it, distinct from the vaguer promise of better rankings.

For a typical marketing site, the file is closer to a reasonable bet than a necessity. Add it, generate it so it stays current, and then forget about it and spend your attention on the things that actually drive AI citation. The mistake is not adding llms.txt; the mistake is believing it does more than it does and neglecting the work that matters while you tidy a file almost nobody reads.

The takeaway

llms.txt is a simple, curated Markdown map of your key pages, served at your domain root for AI models. It is low cost and low risk, and there is a real case for it in documentation and agent settings. But Google has confirmed it does not affect Search or AI Overviews, and AI crawlers rarely read it today, so it is not a ranking lever and should not be sold as one.

Add it if you want, generate it so it stays current, and keep your real effort on crawlability, structure, authority, and originality, which are what get you cited. The file is housekeeping, not strategy.

If you want a clear read on what is actually driving or blocking your visibility in AI answers, beyond a file at the root, that is what an AI search visibility audit provides.

This article is part of the SEO in the AI Era: The Complete Guide guide.