llms.txt is a proposed standard: a Markdown file placed at the root of a website that gives large language models a short, curated map of the site’s most useful content. It was suggested by Jeremy Howard of Answer.AI in 2024. It is not an access control file, and no major AI company has confirmed that it relies on it, so teams should publish it with clear rules and modest expectations.
What is llms.txt supposed to do?
The idea is simple. Websites are full of navigation, scripts and ads that make it hard for a model to find the important text within a limited context window. llms.txt offers a clean summary: a title, a short description and grouped lists of links to key pages, ideally with plain Markdown versions of those pages.
The proposal also describes an optional llms-full.txt, which contains the full text of the important pages in one file. Documentation sites for developer tools were among the first adopters, because programmers often paste documentation into AI assistants.
Some SEO plugins and site builders now generate the file automatically. That makes it easy to publish, and also easy to publish without thinking about what it contains.
Does llms.txt improve visibility in AI answers?
At the time of writing there is no public evidence that ChatGPT, Gemini, Claude or Perplexity use llms.txt when choosing sources for answers. Google representatives have said publicly that Google’s AI features do not use it. Server logs on many sites show few or no requests for the file from AI crawlers.
That does not make it useless. Developer-focused tools and some agent frameworks can read it, and it costs little to maintain. But it should not replace the work that does affect AI visibility: crawlable pages, clear structure, accurate facts, structured data and mentions on trusted third-party sites.
What are the real risks of publishing llms.txt?
The file is small, but it is public and it states what you consider important. That creates a few risks worth listing before anyone uploads it.
- Confusing it with robots.txt: llms.txt does not block anything. Crawler access is still controlled by robots.txt user agents such as GPTBot, ClaudeBot, Google-Extended and PerplexityBot.
- Exposing hidden pages: auto-generated files sometimes list staging, thank-you or gated pages.
- Stale content: a file that points to discontinued products or old prices sends models the wrong facts.
- Content duplication: llms-full.txt copies whole pages into another public URL, which other scrapers can take in one request.
- Over-investment: hours spent tuning the file instead of fixing pages that people and crawlers actually read.
Of these, stale content is the most common. A file written once during a website launch rarely gets updated when the product range changes.
Which governance rules should you set first?
Treat llms.txt like any other public statement about the company. It needs an owner, a review cycle and a clear scope.
Ownership and review
Assign one owner, usually in SEO or content. Review the file whenever the site navigation, product lineup or pricing structure changes, and at least once a quarter. Keep it in version control with the rest of the site so that changes are traceable.
Scope rules
List only public, indexable pages that you would be happy to see quoted. Exclude account areas, carts, legal drafts, internal tools and campaign landing pages that will expire. If a page carries a noindex tag, it does not belong in the file.
Consistency with robots.txt
Decide your AI crawler policy first, in robots.txt. If you block a crawler there, listing pages for it in llms.txt sends mixed signals. Write the two decisions on the same page of your policy document.
How do you write a good llms.txt file?
Keep it short and factual. Start with an H1 containing the company or site name, then a one-paragraph blockquote saying what the organisation does and for whom. Group links under H2 headings such as Services, Guides and Company, each with a one-line description.
- Use plain, accurate descriptions rather than marketing slogans.
- Link to the canonical URL of each page.
- Mention locations, languages and core services in the summary.
- Add an Optional section for secondary resources, as the proposal suggests.
- Check every link returns a 200 status before publishing.
How Alien Road applies llms.txt
Our GEO team treats llms.txt as a small part of a wider AI visibility audit. Discovery starts with server logs and robots.txt: which AI crawlers visit, what they request and what the client currently allows.
If the client wants the file, we draft it from the canonical sitemap, remove anything noindexed or temporary, and have the content owner approve the descriptions. It goes live with a quarterly review date in the project calendar.
Measurement is honest about uncertainty. We log requests for the file, track brand mentions in AI answers for a fixed set of prompts, and report both. If the file shows no measurable effect, the report says so and effort moves to the pages and citations that matter more.
How does llms.txt fit into a wider GEO plan?
Generative engine optimisation, or GEO, is about being found and cited in AI answers. The factors that matter most are the same ones that matter for search: pages that load and render without heavy JavaScript, clear headings, direct answers near the top, author and company information, and structured data that describes products, services and organisations.
Third-party mentions carry weight as well. AI answers often draw on review sites, industry directories, news coverage and forums. A brand that appears accurately in those places is easier for a model to describe correctly than one that only describes itself.
Seen in that order, llms.txt is a tidy final layer. It summarises what is already strong on the site. If the underlying pages are thin or out of date, the file only points models to weak material more efficiently.
A sensible budget split is therefore lopsided: the large majority of effort on content quality, crawl access and citations, and a small slice on llms.txt maintenance. If the standard gains confirmed support later, the file will already be in good shape.
Where to start this week
Open your robots.txt and write down your current position on each major AI crawler. Then search your server logs for requests to /llms.txt to see whether anything is already asking for it. If you decide to publish, draft the file by hand from your top twenty public pages, get one owner to approve it, and put a review date in the calendar before it goes live.
Keep reading: GEO – AIO – AEO · GEO