An llms.txt file is a proposed standard: a Markdown file at the root of a website that gives large language models a short, curated guide to the site’s most useful content. At the time of writing it is not a ranking factor for Google, and major AI search providers have not confirmed that they use it. It is cheap to add, can help some AI tools read your documentation, and should not replace classic SEO work.
The idea was proposed in 2024 by Jeremy Howard of Answer.AI. The file lives at /llms.txt and uses plain Markdown: a title, a short description of the site, and lists of links to key pages with one-line explanations. A related convention suggests offering clean Markdown versions of pages, for example page.md alongside page.html, so models can read content without navigation, scripts and ads.
What problem does llms.txt try to solve?
Web pages are built for browsers. They contain menus, cookie banners, scripts and layout code that make up much of what a crawler downloads. Language models have limited context windows, so feeding them a whole site is wasteful. A short, structured file pointing to the important pages, in a format models handle well, gives them a map.
The use case is strongest for documentation-heavy sites, such as software products, APIs and knowledge bases, where developers paste documentation into AI coding tools.
How is llms.txt different from robots.txt and sitemaps?
The three files do different jobs:
- robots.txt tells crawlers what they may and may not access. It is a widely respected convention and the place to allow or block AI crawlers such as GPTBot, ClaudeBot or PerplexityBot.
- sitemap.xml lists URLs you want search engines to discover, with optional dates. Google and Bing read it.
- llms.txt describes and prioritises content for language models in human-readable form. It grants no permissions and blocks nothing.
If you want to stop AI crawlers, llms.txt will not help. If you want Google to find new pages, the sitemap remains the tool.
Does llms.txt affect Google rankings?
No evidence suggests it does. Google’s representatives have said publicly that Google does not use llms.txt, and Google Search uses its own crawling and indexing systems. Adding the file will not raise rankings or AI Overviews visibility.
Other AI providers have not made firm public commitments either. Some developer tools and AI coding assistants do read llms.txt files when users point them at a documentation site, and several documentation platforms generate the file automatically. That is the current, modest reality.
Who should add an llms.txt file?
It makes most sense for:
- SaaS companies and developer platforms with public documentation.
- Sites with large knowledge bases where a curated entry point helps.
- Companies that want a low-cost experiment with AI readability.
For a local business site with ten pages, the benefit is small. It does no harm, but other work, such as accurate Google Business Profile details and clear service pages, will matter far more.
How do you write a good llms.txt?
Keep it short and curated. The value is in choosing, not in listing everything.
- Start with an H1 title: the site or product name.
- Add a short blockquote summary describing what the site offers and for whom.
- Add a few H2 sections such as Docs, Guides, Pricing or Policies.
- Under each, list the most important pages as Markdown links with a one-line description.
- Optionally add an “Optional” section for secondary pages that can be skipped when context is limited.
- Upload it to the site root and check it loads at yourdomain.com/llms.txt as plain text.
Keep it current
An llms.txt that points to retired pages is worse than none. Add it to the checklist for site changes, or generate it automatically from your CMS or documentation tool.
What SEO work matters more for AI visibility?
The foundations that help answer engines are mostly classic SEO. Make sure AI and search crawlers can access the site through robots.txt and your CDN settings. Keep pages fast and readable without heavy scripts. Structure content with clear headings and direct answers. Mark up key facts with schema where it fits. Earn mentions from trusted third-party sites, which AI systems often cite.
A common misunderstanding
Some guides present llms.txt as a way to opt out of AI training or to control how models describe your brand. It does neither. Models that never request the file are unaffected by it, and a file cannot instruct a model to say particular things. Treat any service selling llms.txt as an AI ranking shortcut with caution; the honest claim is that it may make your content easier to read for tools that choose to use it.
If you have limited time, spend it there first. llms.txt is a small addition on top, not a strategy.
How Alien Road applies llms.txt in GEO work
In a typical GEO engagement, llms.txt is one item in a wider AI readability review. Discovery checks robots.txt rules for AI crawlers, CDN bot settings, rendering of key pages and whether important content is available as text.
When the site has documentation or a knowledge base, setup includes a curated llms.txt with the main sections and, where the CMS supports it, Markdown versions of key pages. For smaller sites we usually recommend skipping it until the basics are done. We document the file’s owner and update process so it does not go stale.
Where a CMS or documentation tool can generate the file automatically, we prefer that to a hand-edited version, because generated files stay in step with the site.
Measurement is honest about limits. Server logs show whether any AI agents request /llms.txt, and we report that alongside AI referral traffic in GA4 and citation checks in AI assistants, without attributing changes to the file that we cannot prove.
Where to start this week
Check your server or CDN logs for requests to /llms.txt over the past month. If you see none and your site has no documentation, put the effort into robots.txt and content structure instead. If you run a documentation site, draft a 20-line llms.txt with your ten most useful pages and publish it.
Keep reading: GEO – AIO – AEO · GEO