{"id":29488,"date":"2026-09-20T10:33:51","date_gmt":"2026-09-20T07:33:51","guid":{"rendered":"https:\/\/alienroad.com\/the-llms-txt-file-telling-ai-crawlers-what-your-site-offers\/"},"modified":"2026-09-20T10:33:51","modified_gmt":"2026-09-20T07:33:51","slug":"the-llms-txt-file-telling-ai-crawlers-what-your-site-offers","status":"publish","type":"post","link":"https:\/\/alienroad.com\/the-llms-txt-file-telling-ai-crawlers-what-your-site-offers\/","title":{"rendered":"The llms.txt File: Telling AI Crawlers What Your Site Offers"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\">An <strong>llms txt file<\/strong> is a markdown document at the root of your domain that describes what the site is and points to the pages worth reading. The idea borrows its placement from robots.txt but not its purpose. Robots.txt tells crawlers what to avoid. This file tells a language model where the substance is, in a format it can read in one pass without wading through navigation, cookie banners and footers.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">It is a proposal, not a standard, and that is the first thing to be clear about with a client. No major assistant has committed to reading it, and publishing one does not get you cited. We still recommend it for most sites with real documentation or a deep content library, because it costs an hour, it cannot hurt, and the exercise of writing it surfaces problems worth fixing anyway.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img src=\"https:\/\/alienroad.com\/wp-content\/uploads\/blog-ici\/ar-5-1-48e062e8.png\" alt=\"The llms.txt File: Telling AI Crawlers What Your Site Offers \u2014 overview\" class=\"wp-image-29486\" width=\"800\" height=\"420\" loading=\"lazy\" decoding=\"async\" \/><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\">What goes in an llms txt file<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">The convention is simple markdown. A single H1 with the site or organisation name. A short blockquote summarising what you do in one or two sentences. Then H2 sections grouping links, each link followed by a colon and a short description of what the page contains. Keep it to the pages that carry information, not the whole sitemap. A hundred well chosen links beat a dump of every URL.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The descriptions are the part people rush and the part that matters. A line reading only the page title tells a reader nothing it could not get by fetching the page. A line saying what question the page answers and for whom lets a model decide whether to open it at all, which is the entire point of the document.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Writing one that is actually useful<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Think of it as a briefing for a competent stranger with limited time. What do they need to know about your organisation, and which ten or thirty documents would you hand them first? That framing produces a far better file than treating it as a technical artefact to generate.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img src=\"https:\/\/alienroad.com\/wp-content\/uploads\/blog-ici\/ar-5-2-48e062e8.png\" alt=\"The llms.txt File: Telling AI Crawlers What Your Site Offers \u2014 in practice\" class=\"wp-image-29487\" width=\"800\" height=\"420\" loading=\"lazy\" decoding=\"async\" \/><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">A few rules we apply on every one we write:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Serve it at the root as plain text or markdown, not behind a redirect or a rewrite.<\/li>\n<li>Use absolute URLs, so the file makes sense when read outside its origin.<\/li>\n<li>Describe each link in one clear sentence rather than repeating its title.<\/li>\n<li>Group links under headings a human would recognise: docs, pricing, guides, policies.<\/li>\n<li>Include only pages you would be happy to see quoted, and keep it under a few hundred lines.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">Some sites also publish expanded versions with the full text of key pages inlined. That is worth doing when your documentation is genuinely large and heavily cross linked. For a normal marketing site it adds a maintenance burden with no observable return, and a stale copy of your pricing page is worse than no copy at all.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">One practical detail catches people out. Many content management systems will not serve an arbitrary file from the root, and the workaround chosen is often a redirect from the root path to a page inside the site. A client fetching the root address then gets a redirect chain or an HTML page instead of markdown, which defeats the exercise. Check the response with a plain request and confirm you get the text back with a sensible content type, not your theme&#8217;s layout wrapped around it.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">What it does not do<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">It does not control access. Anything you list is a page that was already public, and anything you want kept out of training or retrieval has to be handled through robots.txt directives, authentication or the crawler specific controls the major providers publish. Treating this file as a permission layer is a mistake we have had to correct more than once.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">It also does not replace anything. Your sitemap still drives discovery, your structured data still describes entities, your robots rules still govern crawling. The llms txt file sits alongside them as a human readable summary of what matters, and if the assistants never read it, everything else on your site still works exactly as before.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Maintenance is the real cost<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Files like this rot. Links move, products get renamed, the page you featured gets merged into another. A file full of redirects and dead ends makes a worse impression than having none. Put it on the same review cycle as your sitemap and your navigation, check it when you ship a site change, and drop entries you no longer maintain rather than leaving them to decay.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Our recommendation is proportionate: write one, keep it short and honest, review it quarterly, and do not let it become a project. The value sits mostly in the side effects. Deciding which thirty pages represent your site, and writing an accurate sentence about each, tends to expose thin content, duplicated explanations and pages nobody could summarise. Those findings are worth the hour even if no crawler ever requests the file.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Keep reading:<\/strong> <a href=\"https:\/\/alienroad.com\/category\/geo\/\">GEO<\/a><\/p>\n\n\n","protected":false},"excerpt":{"rendered":"<p>What an llms txt file is, how to write one that is actually useful, and an honest read on what it does and does not do for your site.<\/p>\n","protected":false},"author":0,"featured_media":29489,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[45],"tags":[],"class_list":["post-29488","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-geo"],"_links":{"self":[{"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/posts\/29488","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/types\/post"}],"replies":[{"embeddable":true,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/comments?post=29488"}],"version-history":[{"count":0,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/posts\/29488\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/media\/29489"}],"wp:attachment":[{"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/media?parent=29488"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/categories?post=29488"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/tags?post=29488"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}