llms.txt is a proposed plain-text Markdown file, placed at the root of a website, that gives large language models a short, curated guide to the site’s most useful content. It does not replace robots.txt, XML sitemaps or structured data, and no major search engine has said it uses the file for ranking. It is worth adding when your site has documentation or reference content that AI tools read; otherwise, traditional tools still matter more.
What is llms.txt, exactly?
The llms.txt format was proposed in 2024 by Jeremy Howard of Answer.AI. The idea is simple: websites are full of navigation, scripts and layout that make them hard for a language model to read inside a limited context window. A single Markdown file can list the pages that matter, with a one-line description each, so an AI tool can find the right content quickly.
A typical file has a title, a short summary in a blockquote, and sections of links. The proposal also suggests an optional llms-full.txt with complete content, and Markdown versions of key pages. Many documentation platforms and developer tool companies have added the file, and generators exist for common CMS platforms.
How does llms.txt compare with traditional tools?
Each file in the traditional SEO toolkit has a specific job and a known reader. llms.txt has a different job and, so far, a less certain audience.
- robots.txt: tells crawlers what they may access. It is a long-standing standard (formalised as RFC 9309) and respected by major search and AI crawlers such as Googlebot, GPTBot and ClaudeBot. llms.txt does not grant or block access.
- XML sitemap: lists every URL you want indexed, with update dates. Search engines use it for discovery. llms.txt lists a small, chosen subset with human-readable descriptions.
- Structured data (Schema.org): marks up facts on each page, such as products, prices, FAQs and authors. Google uses it for rich results. llms.txt carries no per-page facts.
- Clean HTML and good content: still the main way any system, AI or search, understands a page.
The short version: robots.txt controls access, sitemaps support discovery, structured data describes facts and llms.txt offers a reading guide. They do not compete.
Who actually reads llms.txt?
This is the honest weak point. At the time of writing, Google has stated that it does not use llms.txt for Search or AI Overviews, and that normal SEO practices apply. Other major AI providers have not published clear commitments to reading it during crawling.
Where the file does get used is in developer tooling. Coding assistants and documentation tools can be pointed at a library’s llms.txt so the model reads current docs instead of guessing. Some AI agents and browser tools check for it when asked about a site. That makes it valuable for software companies and less clearly valuable for a local restaurant.
When should you add llms.txt?
- You publish technical documentation, API references or a knowledge base that developers query with AI tools.
- Your site is large and the best answers are buried several clicks deep.
- You already have clean robots.txt, sitemaps and structured data, and want a low-cost extra.
- You can maintain it; an outdated llms.txt pointing to old pages is worse than none.
When is it not worth the effort?
If your technical SEO basics are broken, fix those first. A site with blocked CSS, missing canonicals or slow pages will gain nothing from an extra text file. The same goes for small brochure sites with ten pages; a model can read those directly.
It is also not a way to control AI training. If you want to limit how AI companies use your content, the relevant controls are robots.txt rules for specific user agents, such as GPTBot or Google-Extended, plus any opt-out mechanisms the providers offer.
How do you write a good llms.txt?
Keep it short and editorial. The file is a guide, not a dump of every URL.
- Start with the site name as an H1 and a two-sentence summary of what the organisation does.
- Group links under clear H2 sections: Services, Guides, Documentation, Policies.
- Write one plain line per link describing what the page answers.
- Link to canonical URLs only, and prefer pages with stable addresses.
- Add a review date to your content calendar so the file stays current.
If you run WordPress, Shopify or a headless CMS, decide where the file lives and who updates it. Some SEO plugins can generate it automatically, which keeps it in sync but tends to include too many pages. A hand-edited file with a quarterly review is usually cleaner, and the editorial choice of what to include is the part that gives it value.
How Alien Road applies llms.txt in GEO work
In a typical generative engine optimisation engagement, llms.txt is one item in a longer checklist, not the headline. Discovery starts with crawl access: we check robots.txt rules for AI user agents against the client’s wishes, then review server logs to see which AI crawlers actually visit.
Setup focuses on the content AI systems quote: clear answers near the top of key pages, author and organisation markup, and FAQ sections where they help users. Once that is in place, we draft an llms.txt with the client’s team, choosing the 20 to 50 pages that best represent their expertise.
Measurement tracks AI crawler hits in logs, referral traffic from AI assistants in analytics and whether the brand appears in answers for a fixed list of prompts tested each month. We report llms.txt as an experiment with uncertain impact, which is what it is.
Where to start this week
Open yoursite.com/robots.txt and check what it says about AI crawlers. Then search your server logs or CDN analytics for GPTBot, ClaudeBot, PerplexityBot and Google-Extended to see who is already reading your pages.
If the basics look healthy and you have reference content worth guiding readers to, write a first llms.txt with 20 links. It takes an afternoon, costs nothing to host and gives you a clean list of your most important pages, which is useful in itself. Keep a copy in your repository or CMS so changes are tracked.
Keep reading: GEO – AIO – AEO · GEO