Where llms.txt came from
Jeremy Howard of Answer.AI proposed llms.txt in September 2024. His point was that web pages are built for people, and turning HTML into clean text for a model is messy and imprecise.
It's an open proposal published at llmstxt.org, not a standard from a standards body. The current version of the proposal says thousands of sites now publish one, including AI labs for their own developer docs.
What goes in an llms.txt file
The format is plain Markdown in a set order. An H1 with the site or project name is the only required part. After it come a short blockquote summary, optional notes, then H2 sections listing links with a line of description each.
A section titled "Optional" holds links an agent can skip when space is tight. The proposal also suggests offering clean Markdown copies of useful pages at the same URL with .md added.
llms.txt vs robots.txt vs sitemap.xml
robots.txt tells crawlers what they may access. sitemap.xml lists the pages you want indexed. llms.txt is a short, hand-picked reading list for AI tools that are working on a user's question.
None of them replaces the others. The proposal describes llms.txt as working alongside robots.txt and sitemaps. It also notes that a sitemap usually covers far more than fits in a model's context window.
Does llms.txt help with AI search?
Not with Google. Google's guide to optimising for generative AI in Search says Google Search doesn't use llms.txt files. It adds that keeping one will neither harm nor help your rankings there.
The proposal itself says the files are used most heavily for software documentation, where coding agents follow them to API references. For a docs site, that's a real use. For a marketing site, treat it as a cheap extra and put most of the effort into clear pages and structured data.
How Gr8r handles llms.txt
We generate our own llms.txt from the site content on every build, so it never drifts from the pages it lists, and we do the same for clients who want one. We point it at the pages that answer buyer questions, like services, pricing, docs and policies.
We don't sell it as a ranking tactic. If you want AI assistants to work with your product more directly, a protocol like MCP is a different route. Gr8r runs a free MCP connector, Gr8r UX Advisor, that works this way: assistants call its tools instead of reading a file.
Example
A developer tools company publishes /llms.txt with its product name and a two-line summary. Its sections link to Markdown versions of the quickstart, API reference and pricing page. A coding assistant reads it to find the right API page.
Reviewed by Tanjim Islam, Founder & CEO · Updated 8 November 2026
Sources
