llms.txt is a proposed standard: a Markdown file placed at the root of a website, at /llms.txt, that gives large language models a short summary of the site and a curated list of its most useful pages. Implementing it means choosing the content to include, writing clear descriptions, publishing the file, optionally offering Markdown versions of pages, and maintaining it. It is low cost, but there is no confirmed ranking benefit.
The proposal was published by Jeremy Howard of Answer.AI in 2024. It was designed for a specific problem: language models working with websites have limited context, and full HTML pages contain navigation, scripts and layout that waste it. A concise, structured file helps a model or an AI tool find the right information quickly.
It is important to be clear about status. llms.txt is not an official web standard like robots.txt. Major AI search providers have not confirmed that they use it for ranking or citation, and Google representatives have said publicly that Google Search does not use it. Adoption has been strongest among developer documentation sites, where coding assistants can fetch it directly. Treat it as a helpful file for AI tools, not a visibility shortcut.
What does an llms.txt file contain?
The proposed format is simple Markdown with a fixed order.
- An H1 with the site or project name. This is the only required element.
- A short blockquote summarising what the site is about.
- Optional paragraphs with further context or guidance.
- H2 sections, each containing a list of links with a short description after each one.
- An optional section titled “Optional” for secondary links that can be skipped when context is short.
The proposal also suggests offering Markdown versions of key pages at the same URL with .md added, and some sites publish an llms-full.txt file containing the full text of important pages in one document.
Pilot checklist
The pilot stage is about deciding what is worth including and testing whether the file is useful.
- Decide the purpose: documentation for developers, service descriptions for prospects or policies for customers.
- List the twenty to fifty pages that best explain what you do, such as service pages, pricing, key guides, FAQ and contact.
- Exclude thin, outdated, duplicate and campaign pages.
- Write a one-sentence description for each page that says what a reader will learn, not a slogan.
- Group pages under clear H2 headings, such as Services, Guides, Policies and Company.
- Write the summary blockquote in plain, factual language.
- Validate the Markdown and check every link returns a 200 status.
- Test by pasting the file into an AI assistant and asking typical customer questions to see whether it points to the right pages.
Production checklist
Moving to production means publishing the file properly and making it part of normal site maintenance.
Publishing
- Serve the file at /llms.txt with a text/plain or text/markdown content type and UTF-8 encoding.
- Make sure robots.txt does not block it, and that caching or security plugins do not return an HTML error page.
- For multilingual sites, decide whether to list the main language only or include language versions, and keep descriptions in the same language as the linked pages.
Optional Markdown versions
- For documentation-heavy sites, generate clean Markdown versions of key pages.
- Keep these in sync with the HTML pages; an outdated Markdown copy is worse than none.
- Consider whether an llms-full.txt is useful; for large sites it can become too long to help.
Maintenance
- Assign an owner and review the file whenever key pages are added, renamed or removed.
- Add a check to your release or publishing process so changes to service pages trigger a review.
- Monitor server logs for requests to /llms.txt to see whether AI tools or crawlers fetch it.
What should llms.txt not replace?
The file does not replace the basics that AI search features actually rely on. Pages should be crawlable, with clear headings, accurate structured data, fast load times and content that answers questions directly. robots.txt rules for AI crawlers such as GPTBot, ClaudeBot, PerplexityBot and Google-Extended still control access. Brand mentions, reviews and coverage on other sites still shape how AI systems describe a business.
If those foundations are weak, llms.txt will not compensate. If they are strong, llms.txt is a small, sensible addition.
How should you measure it?
Measure modestly. Server logs show whether the file is requested and by which user agents. Referral traffic from AI assistants, such as ChatGPT, Perplexity, Copilot and Gemini, can be tracked in GA4 by source. Manual checks, asking assistants questions about your services, show whether answers point to the right pages.
Do not attribute changes in AI referrals to llms.txt alone; many factors change at once.
What does a short example look like?
For a small agency site, the file might start with an H1 of the company name, a one-line summary of its services and location, then an H2 for Services listing six service pages with one sentence each, an H2 for Guides with five in-depth articles, an H2 for Company with the about, team and contact pages, and an Optional section with the blog index. The whole file can fit on one screen.
How Alien Road applies llms.txt
In a typical engagement llms.txt is part of a wider generative engine optimisation review. Discovery covers how AI crawlers are treated in robots.txt, whether key pages are crawlable and well structured, and how AI assistants currently describe the business. llms.txt comes after that, as a supporting step.
We draft the file with the client: choosing pages, writing factual descriptions and grouping sections. It is tested by asking several assistants typical questions with and without the file provided, then published at the root with the correct content type. On WordPress sites we make sure caching and security plugins serve it correctly.
Measurement uses server logs for requests to the file, GA4 for AI assistant referrals and a monthly prompt check of how assistants describe the brand. Reports are cautious and present llms.txt as one factor among several.
Where to start this week
List the twenty pages that best explain your business and write one plain sentence about each. Put them into the llms.txt format under three or four headings with a short summary at the top.
Check the links, publish the file at /llms.txt and confirm it loads as plain text in a browser. Then add a reminder to review it whenever a key page changes.
Keep reading: GEO – AIO – AEO · GEO