AGENTS.md vs llms.txt: What AI Agents Actually Read

A client asks the team to add "AGENTS.md and llms.txt" to the site because a competitor has both and their traffic from ChatGPT went up. The team publishes both files by the end of the day, then has no way to tell which one did anything, or whether either one mattered at all. AGENTS.md is a markdown file that gives AI coding agents instructions for working inside a software repository. llms.txt is a separate, root-level markdown index that summarizes a website's key pages for AI systems that fetch content. They were built for different readers, and treating them as a matched pair wastes the half of the effort spent on the wrong one.

What Is llms.txt For?

llms.txt is a plain markdown file placed at a site's root (/llms.txt) that lists a site's most important pages in a short, structured format an AI system can parse quickly. Answer.AI's Jeremy Howard proposed the convention in 2024 as a lightweight alternative to making an AI model crawl and parse full HTML pages, complete with navigation, scripts, and ads, just to find the three paragraphs that answer a question.

The format is simple: an H1 with the site name, a one-line summary, then markdown links grouped under headings like "Docs" or "Pricing." Tools that support the convention, mostly AI coding assistants and a handful of retrieval-augmented chat products, can fetch this one file instead of scraping the whole site.

What Is AGENTS.md For?

AGENTS.md is a markdown file placed in the root of a code repository that tells an AI coding agent how to build, test, and run the project: which package manager to use, which commands run the test suite, and which directories to leave alone. It reads like a README written for a machine instead of a new engineer.

AGENTS.md is read by coding agents such as OpenAI Codex, Cursor, and Claude Code while they work inside a repository, not by the crawlers that power AI search answers. A marketing site with no public codebase and no AI agent editing its files has no natural reader for this file. Publishing one on a content site is a bet on a future use case, not a fix for an existing one.

Do You Need Both Files on a Marketing Site?

For most SEO and content teams, the honest answer is: one of them, maybe, and it is llms.txt. AGENTS.md solves a software engineering problem. If your site is a WordPress blog, a SaaS marketing page, or an ecommerce storefront with no public repository that an AI agent touches, there is no coding agent that will ever open that file. Adding it because a competitor did costs a few minutes and changes nothing measurable.

llms.txt at least targets the right audience: AI systems trying to understand what your site covers. It will not out-rank a well-structured page with clean headings and FAQ schema, but it costs little to maintain once your page list is stable, and it gives an AI system a fast, unambiguous summary instead of forcing it to infer one from your navigation menu.

How to Publish Both Files Correctly

If you decide a file is worth adding, do it properly instead of copying a competitor's version with your logo swapped in.

1. List the five to ten pages an AI system would need to explain your product accurately: home, pricing, docs, and your best evergreen guides.
2. Write a one-sentence, plain description of what the business does, the same sentence you would want quoted back to a user.
3. Group the links under two or three headings in llms.txt and keep the whole file under one printed page.
4. Only add AGENTS.md if an AI coding agent actually works in your repository, and describe your real build and test commands, not a generic template.
5. Re-check both files after any URL change; a stale link in either file is worse than no file at all.

Neither file replaces the on-page work that AI search answers actually cite: clear headings, a direct definition near the top of the page, and FAQ markup an engine can lift without guessing at context. AskEO's audit checks whether AI crawlers can reach your pages at all and whether your structured data gives them something worth citing, which matters more than either root file on its own. Running a scan before adding either file shows whether crawler access or missing schema is the bigger gap; see the audit plans for what the full report covers.

Will Either File Actually Improve Your AI Search Visibility?

Neither Google, OpenAI, Perplexity, nor Anthropic has published documentation confirming that llms.txt or AGENTS.md is read as a ranking or citation input. Their crawlers, GPTBot, ClaudeBot, PerplexityBot, and Google-Extended, are documented to fetch and parse your normal HTML pages, not a special summary file. Any citation lift attributed to llms.txt so far is anecdotal, not measured against a control.

What is measurable is whether your pages are reachable at all: whether robots.txt allows the AI crawler user agents, whether the page returns real content on first load instead of a JavaScript shell, and whether your claims are stated plainly enough to quote. Those three technical facts move AI citations. A summary file is a convenience layer on top of content that already has to earn the citation on its own.

What If Adding These Files Turns Out to Do Nothing?

That is the most likely outcome for AGENTS.md on a typical marketing site, and a real possibility for llms.txt too. The honest case for adding llms.txt anyway is that the cost is near zero: one static file, a few minutes to write, no ongoing engineering work once your key pages stop changing weekly. The cost of skipping it is also near zero. Treat it as a low-effort hedge, not a strategy, and put the time you save into the parts of the page that are already proven to get quoted: a plain definition sentence, a real FAQ section, and headings phrased as the questions people actually ask.

Check What Your Site Is Missing Before You Add Another File

Run a free scan and see whether AI crawlers can reach your pages, whether your FAQ schema is structured correctly, and whether a root file would even be the highest-value fix for your specific site. Create a free AskEO account and scan your homepage; the report lists every crawler-access and schema issue by name, so you know which fix to make first instead of copying a competitor's file list.