llms.txt is a proposed plain-text Markdown file, placed at a site’s root, that gives large language models a short guide to the site’s most useful content. It is cheap to create, often a few hours of work, but its return is uncertain because major AI search providers have not publicly committed to reading it at the time of writing. Budget it as a low-cost experiment and measure crawler access and AI referrals rather than expecting ranking gains.
What is llms.txt?
The proposal was published in 2024 by Jeremy Howard of Answer.AI. The idea is simple: websites are full of navigation, scripts and ads that make them hard for language models to read. A Markdown file at /llms.txt lists the site’s purpose, its key pages and short descriptions, so a model or AI tool can find the right content quickly.
The format suggests an H1 with the site name, a short summary in a blockquote, then sections of links with one-line descriptions. Some sites also publish /llms-full.txt, a single file with the full text of their documentation.
It is not a standard from any standards body, and it is not the same as robots.txt. It does not grant or block access; it only offers a map.
Who uses it today?
Adoption is strongest among developer documentation sites. Tools such as Mintlify and other documentation platforms generate llms.txt automatically, and several AI and software companies publish one. SEO plugins for WordPress, including Yoast, have added options to create it.
On the reading side, the picture is less clear. Some AI coding tools and agents fetch llms.txt when a user points them to a site. Google representatives have said publicly that Google Search does not use it. Other major providers have not documented using it for search answers. That gap is the main reason ROI is uncertain.
What does it cost to create and keep?
The cost depends on site size and how much you automate.
Budget lines
- Planning: choosing which pages belong in the file and writing one-line descriptions, a few hours for a small site
- Writing the summary and sections, often under a day
- Automation: a plugin or script to regenerate the file when content changes, if your CMS supports it
- Optional full-text file: more work, since it needs clean text from many pages
- Maintenance: a quarterly check that links work and the list still reflects priorities
For a typical service business with 20 to 50 key pages, the total is small. For a large catalog or publisher, the main cost is deciding what to include, because the file loses value if it lists everything.
How do you measure ROI?
Because no major provider promises to use the file, measure what you can observe directly. Do not attribute traffic changes to llms.txt unless the data points there.
- Server logs: requests to /llms.txt and /llms-full.txt, by user agent
- Which bots fetch it: GPTBot, ClaudeBot, PerplexityBot, coding assistants or generic fetchers
- Referral traffic from AI tools: chatgpt.com, perplexity.ai, claude.ai, gemini.google.com, copilot.microsoft.com
- Mentions of the brand in AI answers for a fixed prompt list, checked monthly
- Time spent creating and maintaining the file
If logs show no AI fetches after several months, the file is a low-cost bet that has not paid off yet. Keep it if maintenance is automatic; drop it if it needs manual work.
What gives better returns than llms.txt?
For AI visibility, other work usually matters more. AI search tools rely on their own crawlers and on traditional search indexes, so the basics come first.
- Allow the AI search crawlers you want in robots.txt
- Serve key content as clean HTML that loads without heavy JavaScript
- Use clear headings, direct answers near the top of pages and accurate structured data
- Keep facts such as prices, locations and policies on plain, linkable pages
- Earn mentions on trusted third-party sites, which AI answers often cite
Think of llms.txt as a small addition on top of this work, not a substitute for it.
Are there risks?
The risks are small but real. A file that lists outdated pages points tools to wrong information. A full-text file can expose content you meant to keep behind navigation, such as old pricing. And some site owners worry that making content easier for AI to read helps others reuse it without credit; that is a business decision about AI access in general, not specific to llms.txt.
- Keep the file in sync with the site
- Do not include private, staging or gated URLs
- Review the full-text version before publishing
What does a simple llms.txt look like?
For a service business, the file can be short. It starts with the business name as a heading, a one-paragraph summary of what the business does and where, then a section of key pages, each with a link and one line of description. A second section can list secondary pages, such as the blog index or case studies, that a tool may skip when space is tight.
Write the descriptions for a reader who knows nothing about you. “Services” is not useful; “Google Ads management for B2B companies in Istanbul, with pricing ranges” is.
How Alien Road applies llms.txt
In a typical GEO engagement, llms.txt comes after the basics. We first check crawler access, rendering, structured data and answer-first content on key pages, because those affect every AI and search system.
Once that is in place, we create llms.txt from the client’s priority page list, with plain one-line descriptions, and set up automatic regeneration where the CMS allows. We add log monitoring for the file and AI crawlers.
Reports show fetches of the file, AI crawler activity and AI referral traffic next to other GEO metrics. We are clear with clients that the file is an experiment, and we recommend keeping or dropping it based on what the logs show after a few months.
Where to start this week
Check your server logs for any requests to /llms.txt; some bots already request it even when the file is missing. Then write a first version with your top 20 pages and one line each, publish it at the root and note the date. Review the logs in 60 days. That small test answers the ROI question for your own site better than any general claim.
Keep reading: GEO – AIO – AEO · GEO