llms.txt
Also called llms-full.txt, llms txt file
A plain-text file at a website’s root that tells AI tools what the site is about and which pages matter most.
llms.txt is a proposed standard, introduced in 2024 and documented at llmstxt.org, for a markdown file placed at the root of a website (yoursite.com/llms.txt). It gives language models and AI tools a concise, structured summary of the site: a title, a short description, and lists of links to the most useful pages, each with a one-line explanation.
The motivation is practical. Web pages are full of navigation, scripts and layout that make them awkward for AI tools to read, and context windows are limited. A clean markdown index helps an AI assistant, coding tool or agent quickly find the right page. Some sites also publish llms-full.txt, which contains the full text of key content in one file.
Adoption is growing among documentation sites, developer tools and content publishers, but it is not an official web standard, and major search and AI companies have not all committed to using it. It is cheap to create and harmless to publish, so many sites add one as a low-effort step. It does not control crawling; that is still the job of robots.txt.
Example
A software company's llms.txt starts with "# Acme API", a one-sentence summary, then sections such as "## Docs" with links to the quickstart, authentication guide and error reference, each with a short description. A coding assistant reads it and goes straight to the authentication guide.
Common mistake
Treating llms.txt as a way to control AI access or improve rankings. It is a helpful map, not a permission system or a ranking signal. Use robots.txt to manage crawlers, and keep llms.txt short, accurate and updated when key pages change.