llms.txt: what the file does, who reads it and when it's worth it

llms.txt is a Markdown file at /llms.txt that gives language models and AI agents a short, curated overview of your website: its name, a summary and links to the pages that matter. It is a proposal by Jeremy Howard from September 2024, not an official standard. Google Search ignores it, and as of November 2026 none of the major AI vendors has publicly confirmed that their crawlers or products read it on other people’s websites.

That sounds like “skip it”. It isn’t quite that simple. The file takes little work, and there are cases where it does get read. Below is what can be verified, with links to the sources.

What llms.txt is and how the file is structured

Jeremy Howard published the proposal on llmstxt.org on 3 September 2024. Version 2 followed in August 2026. The idea: HTML pages are built for people, with navigation, scripts and ads. An agent looking something up for a user needs a short overview and clean text instead. llms.txt tells it where to find them.

The structure is fixed, in this order:

1. An H1 with the name of the site or project. This is the only required part.
2. A blockquote (>) with a short summary containing what’s needed to understand the rest of the file.
3. Optional paragraphs or lists without headings, with more detail and notes.
4. Sections under H2 headings, each holding a list of links. Every entry is a Markdown link, optionally followed by a colon and a short description.
5. An “Optional” section for links an agent can skip when it needs a shorter context. Since version 2 this is a convention only, with no technical special meaning.

Version 2 adds two things: the file may sit in a subfolder (/docs/llms.txt then covers everything under /docs/), and pages can point to their Markdown version with rel="alternate" type="text/markdown" and to the llms.txt that covers them with rel="describedby".

llms-full.txt is not part of the proposal. It is a convention from documentation platforms such as Mintlify, which use it to serve an entire documentation site as one file. A company website with 30 pages doesn’t need it.

Who reads the file today, and who doesn’t

The evidence is thinner than many sellers of “GEO packages” suggest. As of November 2026:

• Google Search: no. Google’s guide to optimizing for generative AI features (last updated 10 July 2026) says you don’t need to create new machine-readable files, AI text files, markup or Markdown to appear in Google Search, “as Google Search itself doesn’t use them”. It’s fine to create an llms.txt for other services, but doing so “will neither harm nor help your site’s visibility or rankings in Google Search, as Google Search ignores them.” The page “AI features and your website” (last updated 10 December 2025) says the same for AI Overviews and AI Mode.
• Chrome Lighthouse: only checks that it’s served without errors. Lighthouse has an llms.txt audit among its agentic browsing audits. It only flags a server error when fetching the file. If the file is missing (404), the audit is marked not applicable, because the file is “optional at the moment”. That’s a developer tool, not a ranking signal.
• OpenAI, Anthropic, Perplexity: no confirmation. The crawler documentation from OpenAI, Anthropic and Perplexity explains how to control their bots with robots.txt. None of it says that these bots, or ChatGPT, Claude and Perplexity, read the llms.txt of other websites.

A common mix-up: all three vendors publish an llms.txt for their own developer docs, for example Anthropic, OpenAI and Perplexity. That only means they prepare their documentation for agents. It does not mean their systems read your file.

So who does read it? According to llmstxt.org, mostly coding agents fetching the documentation of a software library. The proposal also expects agents that search a website on a user’s behalf to fetch the llms.txt first. I found no solid, publicly verifiable numbers on how often that happens on ordinary business websites. Your own server logs are the best source here.

llms.txt vs robots.txt vs sitemap.xml: three files, three jobs

File	Job	Who reads it
robots.txt	Sets which bots may fetch which parts of the site	Search engines and most AI crawlers, documented by the operators
sitemap.xml	Lists all indexable URLs for search engines	Google, Bing and other search engines
llms.txt	Gives a curated overview with a summary and the key links	Agents that fetch it on purpose; no major vendor has confirmed this for its products

llms.txt replaces neither of the other two. If you want to allow or block AI crawlers, robots.txt is the place. How that works in 2026 is covered in AI crawler controls. To check that your rules do what you expect, use the robots.txt Tester.

Two examples

Both examples are made up and shortened. They only show the structure.

Example 1: service business (heating installer)

# Example Heating Ltd

> Heating installation and heat pumps in Manchester and within 25 miles. Advice, installation, servicing and emergency call-outs for homeowners and property managers.

Quotes only after a site visit. We handle grant applications for our customers.

## Services
- [Heat pump installation](https://example.com/heat-pumps/): process, timeline, grants
- [Boiler servicing](https://example.com/servicing/): service plans and prices
- [Emergency call-out](https://example.com/emergency/): hours and coverage area

## Company
- [About us](https://example.com/about/)
- [Contact](https://example.com/contact/)

## Optional
- [Guides](https://example.com/guides/): articles about heating

Example 2: online shop (coffee)

# Example Roastery

> Online shop for coffee beans from our own roastery. Shipping across the UK and the EU.

## Products
- [Espresso](https://example-shop.com/espresso/): all espresso roasts
- [Filter coffee](https://example-shop.com/filter/)
- [Subscriptions](https://example-shop.com/subscriptions/): delivery intervals and cancellation

## Buying and service
- [Shipping and delivery times](https://example-shop.com/shipping/)
- [Returns](https://example-shop.com/returns/)
- [Payment methods](https://example-shop.com/payment/)

For a shop, list categories and service pages, not every product. Complete product data belongs in feeds and structured data.

For a real example, see this site’s llms.txt: an H1, a summary, a paragraph about me, then sections for services, free tools and articles, with the German articles under “Optional”.

When llms.txt is worth it, and when it isn’t

It’s worth it if:

• you offer software, an API or documentation. That’s where the file is most widespread, and coding agents are the users it was designed for.
• your site is large or hard to navigate and you want to give an agent a clear starting point.
• you can create it in a few minutes and keep it updated automatically from your CMS.

It isn’t worth it if:

• you expect better rankings or more mentions in AI Overviews from it. Google says plainly that Search ignores the file.
• it takes time away from things that demonstrably work: not blocking AI crawlers by accident, serving content in the HTML rather than only via JavaScript, answering questions directly. What the research says about this is summed up in What is GEO?
• someone wants to sell you its creation as a paid standalone service. It’s a text file with a dozen links.

My advice: create it if it’s quick, then stop thinking about it. An outdated llms.txt full of dead links is worse than none, so tie it to your normal publishing process.

Create your llms.txt in 5 minutes

1. Open the free llms.txt Generator and scroll down to the generator. Enter your site’s name, a two- or three-sentence summary and your most important pages.
2. Group the links into two to four sections, for example Services, Company, Guides. Anything secondary goes under “Optional”.
3. Upload the file as llms.txt to your root directory so it loads at https://yourdomain.com/llms.txt: status code 200, served as text/plain or text/markdown in UTF-8.
4. Then enter your domain in the checker at the top. It checks the structure against the llmstxt.org proposal and fetches up to 15 of the links, so you see dead links and redirects.

If you want to know whether you appear in ChatGPT, Perplexity, Gemini and AI Overviews at all, and why competitors get named instead of you, llms.txt is not the lever. The AI search visibility audit answers that for €990. For the file itself, the generator is enough.