llms.txt Explained: What It Is, What Goes In It, and What It Cannot Do

Quick Answer: llms.txt is a plain Markdown file at the root of your site that tells a large language model, in a few hundred words, what your site is for and which pages matter most. It is a proposed convention rather than a ratified standard, and no AI assistant guarantees it will read one — so treat it as a cheap, useful summary of your site, not as a ranking lever.

What problem does llms.txt solve?

A model working on a question has a finite context window and very little patience for navigation. Handing it a marketing homepage full of components, carousels and cookie banners is a poor way to explain what your business does. Handing it one Markdown page that says "this is what we are, these are the ten URLs that matter, here is what each one covers" is a much better one.

That is the entire idea. llms.txt is a curated, machine-friendly table of contents. It doesn't replace your sitemap, which exists so crawlers can find every URL. It does the opposite job: it tells a reader which few URLs are worth reading first.

How is llms.txt different from robots.txt and sitemap.xml?

The three files are complementary and answer different questions. robots.txt answers "what are you allowed to fetch" — it is a permissions file, and it is the only one of the three that is a genuine, widely honoured standard. sitemap.xml answers "what URLs exist" — a complete inventory for discovery. llms.txt answers "what should you read, and what does it mean" — an editorial summary with priorities.

The practical consequence is that llms.txt cannot rescue a page that robots.txt blocks, and it cannot substitute for an XML sitemap. If you only have budget for one of the three, fix robots.txt first.

What goes in a good llms.txt file?

The convention is deliberately simple: an H1 with the site or project name, an optional blockquote summarising it in one or two sentences, then H2 sections containing lists of links, each with a short description after a colon. Beyond that structure, the content that earns its place looks like this:

A one-paragraph definition of what the site is and who it serves, written without adjectives.
Your ten to twenty most important URLs, each with a one-line description of what a reader will find there.
Section groupings that reflect how a stranger would look for things — documentation, pricing, guides, policies — rather than your internal org chart.
Key facts and entities: the official product names, what they do, and any terminology unique to you.
An explicit scope boundary. Saying what you do not do prevents a model from confidently placing you in the wrong category.
An optional section for links that are useful but secondary, so a model with limited budget knows what it can skip.

What should you leave out?

Keyword stuffing, sales copy, and claims you cannot support anywhere else on the site. A model that reads llms.txt will usually go on to read the pages it points at; a mismatch between the two is worse than no file at all. Leave out anything you would not want quoted verbatim, and leave out private or unlinked URLs — the file is public, and publishing a path is publishing it to everyone, not only to AI agents.

Also resist the urge to make it long. The value comes from compression. If your llms.txt runs to thousands of words, you have rebuilt your sitemap in Markdown and lost the point.

Where does llms-full.txt fit in?

Some sites publish a companion file that concatenates the actual content of their key pages into one long Markdown document, so a model can ingest the substance in a single fetch instead of following links. That is useful for documentation-heavy sites where the content is stable and text-first. It is a poor fit where content changes constantly, because a stale full-text dump will contradict your live pages, and contradiction is the one thing you most want to avoid.

If you publish one, regenerate it from the same source as the pages themselves, and put the generation into your build or deploy step rather than a manual process someone will forget.

Does llms.txt actually do anything?

Honest answer: nobody outside the assistant vendors can prove a citation lift from llms.txt alone, and no major assistant has committed publicly to reading it on every fetch. Anyone quoting you a percentage improvement from adding one is making it up.

What is defensible is the cost-benefit. The file takes an hour to write, costs nothing to serve, and carries no downside if it is accurate. It is also occasionally read by agents and tools built on top of the assistants. And the exercise itself has a real side effect: teams that write an honest llms.txt usually discover that they cannot describe several of their own pages in one line, which is a content problem worth fixing regardless of who reads the file.

How do you maintain it?

Generate it, do not hand-edit it forever. Drive it from the same source of truth as your navigation or sitemap so it cannot drift.
Review it whenever your positioning changes, not on a calendar. A stale summary is the main way this file goes wrong.
Check it is served as plain text at the site root and returns a 200, not a redirect into your single-page app.
Keep the descriptions honest. Every line should be something you would be happy to see quoted back at you in an AI answer.

Frequently asked questions

Is llms.txt an official standard?

No. It is a proposed convention that has been widely adopted by documentation and developer sites, but it has not been ratified by a standards body and no assistant vendor is obliged to honour it.

Where does the file go?

At the root of the domain, served as plain text or Markdown at /llms.txt, returning a 200 status. If it redirects or returns HTML from a JavaScript application shell, it will not be usable.

Does llms.txt control whether AI can use my content?

No. Permissions are handled by robots.txt and by your terms of use. llms.txt is descriptive, not restrictive, so do not rely on it to opt out of anything.

How long should llms.txt be?

Short enough to read in one pass. A concise site summary plus curated links with one-line descriptions is the intent; if it grows into thousands of words it stops being a summary.

Do I need llms-full.txt as well?

Only if your content is text-heavy, stable and worth ingesting whole, such as documentation. If it would go stale quickly, skip it, because a full-text dump that contradicts your live pages does more harm than good.

Will adding llms.txt get me cited by ChatGPT?

Not on its own. Crawler access, indexation and answer-first content structure do far more of the work. Treat llms.txt as a cheap complement to those, not a substitute.

Read the full AEO guide for the signals that matter most.