llms.txt: what it is, how to create it, and who reads it

llms.txt is a Markdown file published at the root of a website (yourdomain.com/llms.txt) to give language models and AI agents a summary of the site and a curated list of its most useful pages. If you work in marketing, it has probably already reached you in some GEO checklist, on the same line as "add schema" and "write in 200-word chunks", with a ticket for the dev team.

Before you open that ticket, it's worth looking at the data. In May 2026, Ahrefs reviewed 137,210 domains and found that 97% of the published llms.txt files didn't receive a single request all month.

My take is that an llms.txt file is cheap to have and fine to keep if your platform generates it on its own, but today it gets requested mostly by audit tools and assistants working with technical documentation, so the decision to spend more than an afternoon on it should come from your logs, not from a checklist. In this guide I explain what it is, how it's built, who reads it, what Google says, and how to create one step by step.

What is the llms.txt file?

llms.txt is a proposed standard that Jeremy Howard published on September 3, 2024. The specification lives at llmstxt.org, and its version 2 was last modified on August 10, 2026.

A modern website is full of menus, banners, scripts and footers that get in a language model's way, and it's also too big to read in full at the moment of answering. llms.txt solves that with a short document, written in Markdown, that says "this is what we do and these are the pages that matter".

The analogy that works best for me is a restaurant menu. You can walk into the kitchen, open the fridge and work out what they serve, or you can read the menu. llms.txt wants to be the menu.

Something most guides skip: the specification was designed for inference time, when an agent is helping someone and needs to understand your site right then, and not to control who uses your content. Howard himself acknowledges it could also help with training models, but that isn't the focus.

What does an llms.txt look like inside?

According to the specification, the file follows a fixed order:

1. An H1 heading with the name of the site or project. It's the only required part.
2. A blockquote (the line starting with >) with a short summary.
3. Detail paragraphs or lists that explain how to read the rest.
4. H2 sections with lists of links, each in the name format and, optionally, a note after a colon.
5. An "Optional" section for secondary links that can be skipped when space is short.

An example for a made-up consultancy:

# Example Consulting

> SEO consultancy for mid-sized companies in Chile and Peru.

## Services
- [SEO audit](https://example.com/audit): what we review and how long it takes
- [AI visibility](https://example.com/ai): how we measure mentions in ChatGPT

## Optional
- [Press](https://example.com/press)

The proposal also asks for something few sites implement: a clean Markdown version of each important page, at the same URL with .md appended, and a link in the HTML pointing to it (rel="alternate" with type="text/markdown").

What are llms-full.txt and the .md versions?

llms-full.txt, a convention born on documentation platforms, puts the complete content into a single file instead of a list of links. It saves the assistant from going page by page, although on large sites it can get huge. The .md versions serve the same purpose at page level: plain text, without the design noise.

How is llms.txt different from robots.txt and the sitemap?

They look alike in name and they all live at the root, but they do different things:

Watch out for this one, because it gets around: you'll find guides describing llms.txt "directives" to allow or forbid uses, cap answer length or set an embargo. Those rules aren't in the specification. llms.txt neither grants nor removes access permissions: if you don't want a bot using your content, that's handled in robots.txt (I explain it in this guide).

Who reads llms.txt?

The Ahrefs study (June 2026) looked at 137,210 domains with traffic in May 2026, using its web analytics and bot log data:

• 28% of domains publish an llms.txt (a ceiling, according to Ahrefs, because its customers are more technical).
• 97% of those files received no requests during the month.
• The remaining 3%, about 1,100 domains, accounted for the roughly 22,000 requests measured. 96% came from bots.
• The biggest requesters were SEO audit tools (21.7%). All AI categories combined (assistants, agents and data-collection bots) reached 19.5%.
• Among AI bots, GPTBot (the crawler OpenAI uses to collect training data for its models) requested it the most. ClaudeBot came in at 0.8% and OAI-SearchBot, which powers ChatGPT search, at 0.74%.
• Bots collecting training data requested it almost five times more than bots fetching content to answer in real time.
• PerplexityBot requested it fewer times than Slackbot (yes, Slack's bot, the one behind link previews).

I mention it because there are guides claiming Perplexity already reads it.

The other big study comes from SE Ranking, covering about 300,000 domains: 10.13% had the file, and they found no relationship between having it and the number of citations in AI answers. In fact, their model predicted citations better when the llms.txt variable was removed.

What does Google say about llms.txt?

Google has been fairly consistent. Gary Illyes said in July 2025 that Google doesn't support llms.txt and has no plans to. And Google's guide to its generative AI features (updated July 10, 2026) spells it out: you don't need AI text files, special markup or Markdown to appear in Search, because Google Search ignores those files. It adds that it's completely fine if you decide to keep one for other services, and that doing so neither helps nor hurts you in Google. I covered that guide in what changed in Google's guidance and in AI Mode.

John Mueller, from Google, was more graphic on the Search Off the Record podcast: an llms.txt is basically telling these systems "I have the best website ever, and here are all of the pages that everyone must go to". A self-declared file doesn't help a system decide whom to show. Mueller did see a narrow use: when an agent is already on your site and needs to find something or complete a task.

The signal that confuses many teams comes from Chrome. Since May 2026, Lighthouse has an experimental agentic browsing category with an audit for llms.txt. If the file doesn't exist (a 404), the audit is marked "not applicable". It only flags a problem when the server returns an error when the file is requested. It has nothing to do with rankings or citations.

Does llms.txt help you show up in ChatGPT or AI Overviews?

With the evidence available today, I don't see how. Google says it ignores it, the bots that fetch content to answer barely request it, and the SE Ranking study found no relationship with citations.

llms.txt reminds me of schema, which I once called the horoscope of GEO: it's technical, you can tick it off as done and it gives a feeling of control. And since it costs little, it becomes the default answer to a hard question ("why does ChatGPT recommend my competitor?").

What I have seen move citations is content. In a study I did on 500 prompts for Previsible, queries with a specific industry, team size, year or use case received on average 3.4 times more citations than their generic versions. That work happens on the pages of your website, with specific answers and your own data (the details are in what is AEO and in SEO for ChatGPT, in Spanish).

And to know whether something works, you have to measure visibility. With our clients we track 20 to 30 fixed questions, the ones someone would ask before buying, and we run them every week in ChatGPT and Gemini. If you publish an llms.txt file and want to see whether anything changed, that kind of tracking will tell you. The full method is in how to measure visibility in LLMs (in Spanish).

Which sites is llms.txt useful for? Documentation, APIs and SaaS

Where llms.txt makes sense is technical material read by coding assistants:

• Product and API references. Someone integrating your API with AI help needs the assistant to quickly find the right reference and the current version. Claude Code (Anthropic's coding assistant) shows up among the AI bots that requested the file most in the Ahrefs data, and it's exactly the kind of tool that looks up documentation.
• SaaS and developer tools. Anthropic, Vercel and Hugging Face publish an llms.txt in their documentation, and platforms like GitBook or Mintlify generate it automatically.
• Large knowledge bases. Help centers with hundreds of articles where a support agent has to land on the right one.

For a blog, a services website or an ecommerce store, I'd treat it as low cost and low priority: if the platform generates it, fine; if it has to be built by hand, other things pay off more in the same time.

How do you know if anyone reads your llms.txt?

This is where I'd spend the 20 minutes you were going to give that ticket. Ahrefs recommends it in its study, and it's the method we use in audits of large websites: server logs.

1. Take 30 days of logs from the server or the CDN.
2. Filter requests to /llms.txt (and /llms-full.txt if you have one).
3. Group by user agent and verify that bots are who they say they are, because anyone can call itself GPTBot.
4. Compare against requests to robots.txt for the same period, for scale.
5. Look at what happens next: whether the bot that requested the file then visited the pages it links to.

An illustrative example: a website that gets 12,000 requests to robots.txt in a month and 9 to llms.txt, all from audit tools. That site doesn't need an llms.txt project. If instead you see a coding assistant requesting the file and then crawling your API reference, it's worth looking after. In the 503 error guide I explain how to filter logs by template, and in the one on ChatGPT's own search index, how to track OAI-SearchBot.

How to create an llms.txt step by step

If you decide to have one, here's how to create the llms.txt file without turning it into a project:

1. Pick the pages. The ones that explain what you do and for whom: services or products, pricing, current technical references and the guides that answer best. Leave out thin, duplicate or archived pages.
2. Write it in Markdown. An H1 with the name (the only required heading), a blockquote with the summary and H2 sections with links and a one-line description.
3. Publish it at the root (yourdomain.com/llms.txt) or on the documentation path if it only covers that part (yourdomain.com/docs/llms.txt).
4. Check that it returns 200 with a curl -I to the URL and that it shows as plain text in the browser. A server error is the only thing the Lighthouse audit flags as a problem.
5. Link to it from your help pages or from the instructions you give your assistants, because they request it when someone sends them there.
6. Keep it updated. Every time you reorganize the site or change versions, review it. Ideally, it's generated automatically from the CMS.

Which tools generate an llms.txt?

There are free generators, like Firecrawl's, that build a draft from your sitemap, and platforms like GitBook, Mintlify or some site builders that generate and update it on their own. My recommendation is the same as Ahrefs': let the platform do it and keep the effort minimal. My consultancy's site has one, with the main pages, the case studies and the tools, and I wouldn't spend more than an afternoon a year on it.

What are the risks of publishing an llms.txt?

It makes copying your best pages easier for a competitor, because you hand them the organized map. And since some systems read it and act on what it says, it's worth treating it like code: version it, limit who can edit it, get alerts if someone changes it and link only to resources you control.

Frequently asked questions about llms.txt

Is it mandatory? No. It's a proposal with no standards body behind it, and neither Google nor OpenAI has committed to reading it.

Does it improve SEO? In Google, no: its guide says it ignores the file and that it neither helps nor hurts. In other systems, there's no evidence that it increases citations.

Can it block training on my content? No. For that you need to block those bots in robots.txt.

What I'd do this week

If someone on your team has an open ticket to create an llms.txt, first ask them for the last 30 days of logs and check how many times the file was requested. If your site has documentation used by developers, create the file and let it generate itself. If it doesn't, close the ticket with a clear conscience and spend those hours improving the three pages that should show up most when someone asks ChatGPT about what you sell.