Thousands of sites now serve the file. No major AI engine has promised to read it.
llms.txt is a plain markdown file that sits at the root of your site (yourdomain.com/llms.txt) and gives AI systems a short, curated map of your most important pages. Jeremy Howard of Answer.AI proposed it in September 2024, it takes maybe twenty minutes to write, and right now it's carrying more hype per byte than anything else in SEO. It probably won't hurt you, it might eventually help you, and most of what's written about it oversells it badly.
The problem Howard was pointing at is real. When an AI assistant tries to use your website to answer a question, it works under two constraints. The first is the context window: a language model can only hold so much text at once, so it can't ingest your whole site and has to pick. The second is that web pages are hostile to machines. Navigation menus, cookie banners, sidebars, footers, and JavaScript that never renders server-side all sit between a crawler and the actual content. A human glances past that clutter without noticing. A bot frequently gets soup.
llms.txt is the proposed fix: one predictable location where a site publishes clean, structured markdown saying "here's who we are, and here are the pages that matter, each with a one-line description." The spec (published at llmstxt.org) is deliberately tiny. A single H1 with the site or project name, a blockquote summary, optional context paragraphs, then H2 sections containing link lists. Two companion conventions travel with it: markdown twins of important pages, served by appending .md to the page URL, and an llms-full.txt variant that inlines the full content of everything into one big file instead of linking out.
Adoption so far follows a clear pattern: developer documentation first, everything else barely. Anthropic serves an llms.txt for its own docs, several documentation platforms added one-click generation, and plugins now exist for most CMSes. Outside dev tools and docs-heavy SaaS, uptake is thin, and that unevenness is itself useful information about how settled this "standard" actually is.
llms.txt vs robots.txt vs sitemap.xml.
The three get lumped together because they all live at the site root and all talk to machines. They do completely different work, and only one of them is optional.
| robots.txt | sitemap.xml | llms.txt | |
|---|---|---|---|
| What it is | Access rules for crawlers | An inventory of your indexable URLs | A curated, annotated guide to your best content |
| Who it talks to | All bots, including AI crawlers like GPTBot and ClaudeBot | Search engine crawlers | LLMs and AI assistants, in theory |
| What it does | Allows or blocks crawling per bot and per path | Helps engines discover every page you want indexed | Suggests which pages matter and summarizes them in markdown |
| Is it honored? | Yes, respected by every legitimate crawler | Yes, consumed by all major search engines | Unconfirmed; no major AI provider documents using it |
| Standing | Decades old, formally standardized | Established protocol since 2006 | A proposal from late 2024, not a standard |
| Should you have one? | Yes | Yes | Optional; cheap insurance at best |
What llms.txt doesn't do.
Start with the confusion that causes actual damage: llms.txt does not control access. If you want to stop OpenAI or Anthropic from crawling your site, that's a robots.txt job (both companies document their user agents and state that they respect robots directives). An llms.txt file can't block anything, and it isn't a training opt-out either. The file makes suggestions; it has no power to forbid.
Second, and this is the part the hype skips: no major AI provider has publicly confirmed that its crawlers or assistants consume llms.txt. Google's John Mueller has compared it to the keywords meta tag, a file site owners fill out that the engines themselves ignore. You can run the test on your own infrastructure in five minutes: grep your server logs for requests to /llms.txt. On most sites the answer is a trickle of curious humans and SEO tools, not a parade of AI user agents.
Third, it isn't a ranking signal, in Google or inside AI answers. Getting mentioned and cited by ChatGPT or Perplexity is earned the unglamorous way: crawlable pages, direct answers near the top of the content, structured data, and a brand that shows up in the sources models already trust. That's the actual work of generative engine optimization, and a text file doesn't substitute for any of it. If your content can't get cited on its merits, llms.txt is a beautifully formatted menu for a restaurant with no food.
Write the file if you like. Just know that nobody on the other end has promised to read it.
So why write one at all? Three defensible reasons. The cost is trivial and the downside is roughly zero. The pattern it encodes (clean markdown versions of your key content) has value beyond the file itself, because agents and AI tools that fetch pages directly genuinely do better with markdown than with rendered HTML. And if any major provider does adopt the convention later, the sites that already serve it are done. That's the whole case: cheap insurance plus a useful side effect. Claims beyond that usually trace back to someone with a generator to sell.
How to write an llms.txt file that actually follows the spec.
The format is markdown with a fixed skeleton. Here's a complete, valid example for an imaginary analytics product.
# Acme Analytics
> Acme Analytics is self-serve product analytics
> for small SaaS teams. Event tracking, funnels,
> and retention reports without an SQL requirement.
## Docs
- [Quickstart](https://acme.com/docs/quickstart.md): install the snippet and send your first event
- [API reference](https://acme.com/docs/api.md): REST endpoints, auth, rate limits
## Guides
- [Funnel analysis](https://acme.com/guides/funnels.md): building and reading conversion funnels
## Optional
- [Changelog](https://acme.com/changelog.md): release notes by month
- Serve it at the root. The file lives at /llms.txt on your domain, plain text, UTF-8, publicly accessible. A file buried at /assets/llms.txt satisfies nobody, including the spec.
- One H1, and it's your name. The single required element is a top-level heading with the site or project name. Everything else is technically optional.
- Summarize in a blockquote. Directly under the H1, a short blockquote saying what the site is and who it's for. This is the elevator pitch a model would quote, so write it like one.
- Group links under H2 sections. Docs, Guides, Pricing, whatever fits your site. Each link on its own line: the linked title, a colon, then a one-line description that says what a reader gets there.
- Use an "Optional" section for the skippable stuff. The spec gives this heading special meaning: content a model can drop when its context is tight. Changelogs and archives go here, not your pricing page.
- Curate, don't dump. A 400-line llms.txt that mirrors your sitemap defeats the entire point, which is triage. Ten to thirty links covering the pages you'd want an AI to quote is the right shape for most sites.
- Point at markdown where you can. If your platform can serve .md twins of key pages, link those; clean markdown is the version of your content a model parses best.
- Keep it current. A file that still links three renamed products is worse than no file. Regenerate it whenever your core pages change, or wire it into your build.
Where llms.txt fits in an AI-crawler accessibility check.
If your real question is "can AI engines see and cite my site," llms.txt is the last item on the checklist, not the first. The full check has four parts, in descending order of consequence.
Robots rules for AI bots
Check robots.txt for accidental blocks on GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, and Google-Extended. Plenty of sites blanket-blocked AI crawlers in 2023 and forgot; that decision now silently removes you from AI answers.
Content that renders without JavaScript
Most AI crawlers don't execute JS the way Googlebot does. If your body copy only exists after a client-side render, models are citing your competitors' server-rendered pages instead. Curl your key URLs and read what comes back.
Structured data
Schema markup (Organization, Product, FAQPage, Article) gives machines unambiguous facts to lift: who you are, what it costs, what the page answers. It's the closest thing to a machine-readable channel that engines demonstrably consume.
llms.txt and markdown twins
Last, publish the curated map. It's the speculative layer of the stack, but it's also the cheapest, and serving clean markdown helps any agent that fetches your pages directly today.
This is the same discipline a technical SEO audit applies to Googlebot, pointed at a new set of user agents. Hypado bakes it into its AI visibility toolkit: the scan checks your AI-crawler accessibility, including whether llms.txt exists and whether your schema is in place, alongside the thing the file is supposed to influence, which is whether ChatGPT, Claude, Gemini, Perplexity, and Google AI Overviews actually mention and cite you when your customers' questions get asked. That pairing is the point. A checklist tells you the door is open; only AI visibility tracking tells you anyone walked through it, and because model answers vary run to run, that evidence is a trend across repeated scans, not a single number. The honest limit applies to every tool in this category, Hypado included: software can verify the file exists, parses, and isn't blocked, but nothing can force a model to read it.
One practical convenience if you go this route: Hypado is MCP-native, so you can connect it to the AI account you already pay for and ask about your accessibility findings and citation trends in plain English. It works with MCP-compatible assistants; ChatGPT, Claude, and z.ai are tested. MCP access starts on the $49/mo Starter plan, though the visibility tracking itself begins on the free tier.
llms.txt, quick answers.
Does Google use llms.txt?
There's no evidence it does. Google has never documented support, and John Mueller has publicly likened llms.txt to the keywords meta tag, which Google famously ignores. AI Overviews are built from Google's regular index, so the levers that work there are the ordinary ones: crawlable, well-structured, authoritative content.
Does ChatGPT read llms.txt?
OpenAI documents its crawlers (GPTBot for training, OAI-SearchBot for search) and their robots.txt behavior, but publishes no llms.txt support. An assistant can fetch and summarize the file if a user points it there directly, which is real but very different from models consulting it on their own.
What's the difference between llms.txt and llms-full.txt?
llms.txt is the short index: name, summary, and curated links with descriptions. llms-full.txt inlines the complete content of those pages into one large markdown file, so a tool can grab everything in a single request. Docs sites generate both; for most marketing sites the small index is plenty.
Is llms.txt worth creating in 2026?
If it takes under an hour, yes, as low-cost insurance, and sooner if you're a docs-heavy product whose users already paste your pages into AI tools. Just sequence it honestly: fix robots rules, rendering, and schema first, since those demonstrably affect whether AI systems can use your content, and treat the file as the optional final step.
Find out whether the machines can even read you.
Hypado's AI visibility scan checks your llms.txt, schema, and AI-crawler access, then tracks whether ChatGPT, Claude, Gemini, Perplexity, and AI Overviews actually cite you. Start free, no credit card.