llms.txt & llms-full.txt: Teaching LLMs What Your Site Offers

Part of the Agent Readiness course — the web standards that decide whether an AI agent can actually read, understand, and act on your site. Measure any page against these with the Core Agent Vitals analyzer.

What it is

llms.txt is a single markdown file you publish at your site root — https://your-site.com/llms.txt. It gives LLMs and AI agents a curated, machine-readable map of your site: your name, a one-line description, and the handful of links that actually matter (docs, pricing, API, key articles), each with a short note on what it contains.

A companion file, llms-full.txt, goes further: it inlines the actual content of those key pages as clean markdown, so an agent can answer questions about your site from one fetch — no crawling, no HTML parsing, no JavaScript execution.

Think of llms.txt as the table of contents and llms-full.txt as the printed book. The spec lives at llmstxt.org.

Why agents need it

An AI agent works in a loop: observe → reason → act. The quality of everything downstream depends on that first observation. Without a machine-readable index, the agent's only option is to fetch your homepage HTML, strip the navigation and boilerplate, guess which links are relevant, fetch those, and repeat — spending tokens and time on markup it will throw away, and frequently landing on the wrong page.

llms.txt collapses that whole discovery phase into one cheap, high-signal read:

• Lower token cost. A 200-line markdown index costs a fraction of rendering and tokenizing full HTML pages. On the analyzer this shows up directly in Token Cost (TC).
• Fewer wrong answers. You decide what the agent sees first, so it stops inferring your structure from a cluttered DOM.
• Deterministic entry points. Agents that support the convention read /llms.txt before crawling — you get to curate the first impression.

How to implement

1. Create /llms.txt — an H1 with your name, a blockquote one-liner, then link sections:

> Developer documentation and API reference for the Acme platform.

## Docs
- [Quickstart](https://acme.com/docs/quickstart): Install and make your first call in 5 minutes.
- [Authentication](https://acme.com/docs/auth): API keys, OAuth, and scopes.

## API
- [REST reference](https://acme.com/api): All endpoints, params, and response shapes.

## Optional
- [Changelog](https://acme.com/changelog): Release notes.

2. (Recommended) Create /llms-full.txt — the same structure, but paste the clean markdown content of each key page inline. Generate it from your existing markdown/MDX at build time so it never drifts.

3. Serve both as text/plain at the root. On a static host, drop the files in public/. On a framework, add two routes that return the files with Content-Type: text/plain; charset=utf-8.

Validate

curl -s https://your-site.com/llms.txt | head

You should get markdown, HTTP 200, and a text/plain content type. Then run your site through the Core Agent Vitals analyzer — the Agent Discoverability panel checks for llms.txt at both /llms.txt and /.well-known/llms.txt, and flags it if the H1/blockquote structure is missing.

Common mistakes

• Publishing it as HTML. If your framework wraps it in a layout, agents get a web page, not markdown. Serve raw text/plain.
• Listing every URL. llms.txt is a curated index, not a sitemap. If it lists 400 links it's noise — link the 5–15 pages that matter and let the sitemap handle the long tail.
• No H1 or blockquote. Parsers key off the # Title and > one-liner. Skip them and tools treat the file as malformed — it fails silently.
• Letting llms-full.txt rot. If you hand-maintain the inlined content, it drifts from the real pages within weeks. Generate it from source at build time or don't ship it.
• Blocking the crawlers that would read it. An llms.txt behind a robots.txt disallow, a login wall, or a bot-challenge is invisible. Keep the file and the pages it points to publicly reachable.

---

Next in the course: Sitemaps for Agent Discovery — the table of contents for everything llms.txt doesn't curate.