The most common llms.txt mistakes are treating the file as a ranking switch, copying the sitemap into it, linking to pages that are blocked or broken, and never updating it. The file is a proposed convention: a Markdown document at the site root that gives language models a short, curated map of your most useful content. Done carelessly, it adds nothing; done well, it is a tidy index for AI tools.
It helps to be honest about the status first. llms.txt was proposed in 2024 by Jeremy Howard of Answer.AI. At the time of writing it is not a formal standard, and major search and AI providers have not publicly committed to reading it in their main crawlers. Some developer tools and documentation platforms do use it. That makes it low-cost and low-risk, but not a substitute for good SEO.
What does a correct llms.txt file look like?
The proposal describes a simple structure. An H1 with the project or site name comes first, followed by a short blockquote summary. Then come optional paragraphs of context and H2 sections containing Markdown link lists, where each link can carry a brief note after a colon. A section titled Optional signals links that can be skipped when context is limited.
Some sites also publish an llms-full.txt with the full text of key pages, and Markdown versions of individual pages at the same URL with .md appended. These are companions, not requirements.
Mistake 1: Expecting it to improve rankings or citations directly
Teams publish the file, wait a month and report that nothing changed in Google AI Overviews or ChatGPT answers. That is the expected result. Google has said AI features in Search rely on the same crawling and indexing systems as regular results, and robots.txt and normal technical SEO remain the controls that matter there.
Agencies avoid this by setting the right expectation in the brief: llms.txt is a hygiene item for agent and developer use cases, measured by whether it is valid and fetched, not by traffic.
Mistake 2: Dumping the entire sitemap into the file
A file with four thousand links defeats the point. The value of llms.txt is curation, a short list that tells a model where the authoritative answers live. If every blog post and tag archive is listed, the signal disappears and the file can exceed what a tool is willing to load.
- Keep the main list to the pages you would hand a new employee on day one
- Group links by intent: services, pricing, documentation, policies, contact
- Move long-tail articles to an Optional section or leave them out
- Write one plain sentence per link explaining what the page answers
Mistake 3: Linking to pages crawlers cannot reach
We regularly see files that point to URLs blocked in robots.txt, pages marked noindex, staging domains or links that redirect twice. A model or agent following those links either fails or receives a different page than the one described.
The fix is a simple crawl of the file itself. Fetch every URL, confirm a 200 status, check that it is canonical, and make sure robots.txt allows the user agents you care about. If you have chosen to block GPTBot or ClaudeBot, listing pages for them in llms.txt sends mixed signals.
Mistake 4: Writing marketing copy instead of facts
Summaries like world-class solutions for every need tell a model nothing. Language models extract facts: what the company does, where it operates, what it costs, who it serves. Vague praise is skipped, or worse, repeated back without context.
- Name the services and the markets you actually serve
- State locations, languages and support hours where relevant
- Point to a pricing or packages page if one exists
- Link the canonical about page and any policies a buyer would check
Mistake 5: Wrong location, wrong format, wrong headers
The proposal places the file at /llms.txt on the root domain. Putting it in a subfolder, serving it as HTML, or returning it with a download header makes it harder to consume. Check that the server returns text/plain or text/markdown with UTF-8 encoding.
On WordPress, a static file in the web root is the simplest route. If a plugin generates it dynamically, confirm that page caching and CDN rules do not serve an old copy or strip line breaks. Multilingual sites should decide deliberately whether one file covers all languages or whether each language section gets its own list.
Mistake 6: Publishing once and forgetting it
Services change, pages get merged and URLs move. A file written during a launch can be half broken within a year. Because nothing visible breaks on the site, nobody notices.
Add the file to the same checklist used for redirects after any site change. A monthly automated check that fetches every link and flags non-200 responses takes minutes to set up and removes the problem.
How Alien Road applies llms.txt
In a typical engagement llms.txt is one line item in a wider generative engine optimisation audit, not a project of its own. Discovery starts with the questions buyers ask AI assistants about the client’s category, and which pages on the site actually answer them. That list becomes the core of the file.
During setup we write the summary and link notes in plain factual English, validate every URL against robots.txt and canonical tags, and confirm server headers. For measurement we review server logs for requests to /llms.txt and its linked Markdown pages, separating known AI user agents from ordinary bots. Reporting is candid: we show whether the file is being fetched and by whom, alongside the metrics that move citations, such as structured data coverage and content clarity.
A pre-launch checklist
- File lives at the root and returns 200 as plain text
- H1 name, blockquote summary and H2 link sections follow the proposal
- Fewer than roughly fifty core links, each with a one-line note
- Every link is canonical, indexable and allowed for the agents you target
- No staging, parameter or redirected URLs
- An owner and a review date are recorded internally
Where to start this week
Open yourdomain.com/llms.txt and see whether anything is there. If a file exists, run every link through a status checker and remove anything that is not a clean 200. If nothing exists, draft ten to twenty links to your most important answer pages with one factual sentence each, publish it at the root, and add a reminder to review it whenever site structure changes. Keep your effort proportionate: an hour now, a few minutes a month after.
Keep reading: GEO – AIO – AEO · GEO