llms.txt: The Proposed Standard for AI-Friendly Sites

What llms.txt is

llms.txt is a community-proposed convention, first put forward in 2024, for a markdown file that lives at the root of your domain — yoursite.com/llms.txt. Its job is narrow: give a language model a fast, structured map of your most valuable pages, so it does not have to guess by crawling. A sitemap lists every URL for crawlers; a homepage speaks to humans. llms.txt sits between the two — a short, curated index written specifically for machines that read with a limited context window.

The logic behind it: when a model lands on your site, it usually has room to read a handful of pages, not your whole archive. A file that says "these are the pages that matter, and here is what each one covers in one line" removes the guesswork from that choice.

The honest status: a proposal, not a standard

Be clear about what llms.txt is not. It is not an official standard ratified by any standards body. No major AI engine has publicly documented that it fetches or parses the file during normal operation, and informal tests so far have not produced a consistent answer on whether it changes citations. Anyone selling it as a proven ranking lever is ahead of the evidence.

So why consider it at all? Because the cost is nearly zero. It is a single static file of a few kilobytes. The worst realistic outcome is that you spent an hour writing an index nothing reads. The better outcome is that you handed every future model a direct channel into your best content on day one. That asymmetry — near-zero downside, plausibly real upside — is the entire argument.

The format, piece by piece

The proposed format is deliberately minimal, and every element is plain markdown:

One H1 with your site or product name. Required, and it must be the first line.
A blockquote directly below it: one sentence stating what the site does. Write it for a reader who will read nothing else.
H2 sections that group links by purpose — Tools, Guides, Reference, Pricing, whatever matches your site.
Markdown link lists under each section: the page title in brackets, the URL in parentheses, then an optional one-line description after the link.
Optional sections, which a model with a tight context window may skip. Use them for secondary material only.

You can also publish a companion llms-full.txt: a single markdown document containing the full text of the pages you listed, so a model that wants everything can take it in one fetch instead of twenty.

A complete example

Here is a full llms.txt for a site like this one — small enough to read in ten seconds:

# CrawlCheck

> CrawlCheck checks whether AI crawlers can access your site and helps you make your content citable by AI answer engines.

## Tools

- [AI Crawler Auditor](/#tool-auditor): Paste your robots.txt and get the verdict for 60+ AI crawlers in seconds
- [AI Visibility Scanner](/scanner/): Fetches your pages as each major AI bot and reports exactly which ones are blocked
- [llms.txt Generator](/#tool-generator): Build a valid llms.txt file from your best pages without hand-editing markdown

## Guides

- [What is GEO?](/guides/what-is-geo/): A practical introduction to generative engine optimization
- [robots.txt for AI crawlers](/guides/robots-txt-for-ai-crawlers/): Per-crawler policy with copy-paste rules
- [Measuring AI visibility](/guides/measure-ai-visibility/): The four metrics that matter and how to track them

Who is using it

Adoption is real but should be described conservatively. Several AI vendor documentation sites have adopted the format for their own products, and a number of docs and static-site platforms now ship first-party support for generating the file. That is a meaningful signal — the builders who think hardest about machine readability find it useful for their own documentation. It is not yet proof that ChatGPT, Perplexity or Copilot read the file in production at scale. Weigh it accordingly.

How much work is it

For most sites: one static file, thirty to sixty minutes. The discipline is in the editing, not the writing. Keep it to 10-25 links, ordered by what you most want a model to read. Keep every description under one line. Cut anything you would not defend as one of the most useful pages on your site. A file with 200 links is a sitemap in disguise, and it defeats the purpose.

How to decide

Sequence matters. llms.txt only pays off if a model that follows its links can actually read the pages — which makes it step three, not step one:

Fix access first. A model that follows your llms.txt links still has to get through robots.txt and your firewall. The robots.txt for AI crawlers guide covers per-crawler policy, and the free Auditor shows how your current file reads to each bot.
Make the target pages extractable. Clean headings, answer-first paragraphs and structured data decide whether a fetched page yields quotable text. Start with the schema markup guide and the GEO overview for the full picture.
Then write the file. Build it with the free llms.txt Generator in minutes, review it like an editor, and update it whenever a page earns its place. Afterward, track whether anything moved with the AI visibility metrics guide.

If your site is small, well-structured and already crawlable, llms.txt is a cheap, sensible addition. If your robots.txt currently blocks half the AI web, spend your hour there first — the file will still be there when access is fixed.

Frequently asked questions

Is llms.txt an official web standard?

No. It is a community proposal from 2024, not a standard ratified by any body, and no major AI engine has publicly committed to reading it. That is exactly why the honest framing is low cost, uncertain reward: the file costs almost nothing to publish, but no one can guarantee an effect yet.

Do AI crawlers actually fetch llms.txt?

There is no public documentation that GPTBot, PerplexityBot or ClaudeBot fetch the file during normal crawls, and anecdotal results vary. What is verified: several AI vendor documentation sites publish it, which keeps the format alive and plausible. Publish it if the hour it costs is cheap for you, and do not expect measurable movement on its own.

What is the difference between llms.txt and llms-full.txt?

llms.txt is the curated index — links with one-line descriptions, grouped by section. llms-full.txt is the same list but with the full text of those pages inline, so a model can consume everything in a single fetch. Small sites often need only llms.txt; large documentation sites benefit from both.

Does llms.txt replace sitemap.xml or robots.txt?

No, and it never should. robots.txt controls which crawlers may access your site, sitemap.xml tells crawlers every URL you have, and llms.txt is an opinionated shortlist of what is most worth reading. All three do different jobs, and the first two remain far more important.

Keep reading

How to Measure AI Visibility: Metrics That Actually Matter
AI visibility has no industry-standard score yet. Track citation share, referral traffic, bot access rate and answer accuracy instead — here is how to do it.
How to Get Cited by ChatGPT, Perplexity and Copilot
ChatGPT, Perplexity and Copilot cite pages through a fetch-retrieve-extract pipeline. What each stage rewards, how the engines differ, and a 10-point checklist.
Content Structure for AI: Pages That Get Extracted and Cited
AI answer engines extract pages in chunks, not wholes. Structure rules that make content quotable: question headings, answer-first paragraphs, tables, FAQs.