---
url: /learn/is-llms-txt-enough
title: "Is llms.txt enough? Discovery versus delivery — Siere Learn"
description: "llms.txt tells an agent where to look. It does not change what the agent receives when it gets there. This guide separates the two problems and shows how each is checked."
type: article
language: en
author: Tomislav Mladenov
canonical: https://www.siere.ai/learn/is-llms-txt-enough
last_modified: 2026-08-18
aeo_generated: 2026-08-24T15:05:05.772Z
---

# Is llms.txt enough? Discovery versus delivery

llms.txt tells an agent where to look. It does not change what the agent receives when it gets there. This guide separates the two problems and shows how each is checked.

AuthorTomislav Mladenov

Technical reviewerTomislav Mladenov

Last verified2026-08-18

Evidence classDocumented behaviour

Audit dimensionsExperimental surfaces, Discovery

Licence[CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)

Scope · The llms.txt proposal as published at llmstxt.org; crawler adoption as documented by vendors on the verification date.

Reading is optional. Running the check is the point.
[See what an AI agent receives from your URL](/tools/agent-response-check)

## What llms.txt is

[The `/llms.txt` proposal](https://llmstxt.org/) describes a Markdown file at the root of a site — `/llms.txt` — that gives a language model a short summary of the site and a curated list of links, each with a one-line description, grouped under headings. The stated purpose is to help a model decide *which* pages to read within a limited context window. Many sites also publish `/llms-full.txt`, a companion convention with the full text of the linked pages, and per-page `.md` variants.

It is a discovery convention, in the same family as `sitemap.xml` (which lists URLs for crawlers, per the [sitemaps protocol](https://www.sitemaps.org/protocol.html)) and `robots.txt` (which states crawl permissions, per [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309)). It tells a client where to look and what it will find. It does not change the response the client receives when it follows a link.

## Discovery and delivery are separate problems

It helps to name the two problems precisely.

Discovery

Delivery

Question

Which URLs exist and which matter?

What does a request to a URL return?

Files and signals

`robots.txt`, `sitemap.xml`, `llms.txt`, internal links

HTTP status, redirects, response body, headers

Typical failure

Important pages unlisted or disallowed

Empty application shell, access gate, crawler-specific redirect, mismatched canonical

Fix

Publish and maintain the index files and links

Server-render, prerender, or serve a complete representation at request time; correct access policy

How the free audit reports it

Discovery, and Experimental surfaces for `llms.txt` / `llms-full.txt`

Access, Source completeness, Fidelity

A site can be perfect on the left and broken on the right. The most common version of that: a client-rendered application publishes an `llms.txt` that lists forty routes with good descriptions, and each of those routes returns the same near-empty shell to a non-rendering crawler. The index promises content that the responses do not deliver.

The reverse also happens: a well-served site with complete HTML and no `llms.txt` at all. That site is fully readable; it has simply not published a curated map. Which situation is worse depends on what you assume the client does — but only one of them can be verified from the outside, and it is the delivery one.

## Who reads llms.txt

On the verification date of this guide, none of the vendors whose crawlers this site's audit simulates — OpenAI, Anthropic, Perplexity — documented that their crawlers request or use `llms.txt`. Adoption is visible mainly among developer-tool documentation sites, coding assistants and agent frameworks that fetch it explicitly, and in the audit reports of tools that check for its presence.

That is not an argument against publishing it. It is small, cheap, harmless, and useful to any client that does look for it. It is an argument against treating its presence as evidence that agents can read your site. The audit therefore reports it under **Experimental surfaces**, and states in the observation that this surface "does not affect access, source completeness, or fidelity".

## What llms.txt cannot do

- **It cannot make an empty shell readable.** The linked page still returns whatever it returns.

- **It cannot override `robots.txt`.** A crawler disallowed for a path will not fetch it because a Markdown index mentions it.

- **It cannot verify anything.** Its presence says nothing about whether the descriptions match the pages, or whether the pages match themselves across request types.

- **It cannot be a substitute for a sitemap.** Search crawlers document `sitemap.xml`; treat `llms.txt` as an addition, not a replacement.

## A sensible way to publish it

- Generate it from the same catalogue that drives your sitemap and navigation, so the three cannot drift. This site does that: `/llms.txt`, `/sitemap.xml` and the `/learn` index come from one list.

- Keep entries to pages whose *response* you have verified is complete. Listing a route that returns a shell advertises the failure.

- Include a one-line description per URL that a model could quote.

- If you also publish `llms-full.txt` or per-page Markdown, make sure it is generated from current content, not a snapshot that ages.

- Re-check after deployments — a route that was server-rendered last month may not be this month.

## How to check both halves

Discovery:

`curl -s https://example.com/robots.txt
curl -s https://example.com/sitemap.xml | grep -c "<loc>"
curl -s https://example.com/llms.txt
`
Delivery, for a URL that the index lists:

`curl -sL https://example.com/pricing | sed 's/<[^>]*>//g' | tr -s ' \n' | wc -w
curl -sL -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot" \
  https://example.com/pricing | sed 's/<[^>]*>//g' | tr -s ' \n' | wc -w
`
If the second pair of numbers is small, or the two differ materially, `llms.txt` was never the problem. The free audit runs the same checks and reports each side separately, so you can see which one you actually have.

## What this proves

- llms.txt is a discovery convention: a Markdown index of URLs and descriptions at a well-known path.
- A page can publish llms.txt and still return an empty application shell to a crawler; the two are independent and are checked in separate audit dimensions.

## What this does not prove

- That any specific AI vendor reads llms.txt; no major vendor documented support for it on the verification date.
- That publishing llms.txt harms or helps ranking, citation or recommendation.

## Reproduce it yourself

- curl -s https://example.com/llms.txt — confirm it exists and lists your important URLs.
- Pick a URL from that list and fetch it with a documented crawler User-Agent; compare visible text with a browser view.
- Run the free audit: llms.txt appears under Experimental surfaces; the URL itself is measured under Source completeness and Fidelity.

## Sources

- [The /llms.txt file — proposal](https://llmstxt.org/) — llmstxt.org (Answer.AI), accessed 2026-08-18
- [sitemaps.org — Sitemaps XML format](https://www.sitemaps.org/protocol.html) — sitemaps.org, accessed 2026-08-18
- [RFC 9309 — Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309) — IETF, accessed 2026-08-18

## Change history

- 2026-08-18First published.

## Related guides

- [What AI agents actually receive from JavaScript websites Most AI crawlers fetch the initial HTML and move on. If your page is a client-rendered application, that response can be an empty shell. Here is what is in the response, how to inspect it, and how the free audit measures it.](/learn/what-ai-agents-receive-from-javascript-websites)
- [Markdown endpoints versus SSR, prerendering, and edge middleware Four ways to give an agent a complete, structured response — compared on what they change, what they cost, what they can break, and how you prove each one worked.](/learn/markdown-endpoints-vs-ssr-prerendering-edge-middleware)
- [Recognized crawler versus verified crawler identity A User-Agent string is a claim, not proof. This guide explains the difference between recognizing a crawler and verifying it, which vendors publish verification methods, and why the audit labels every request with an evidence level.](/learn/recognized-vs-verified-crawler-identity)

[← What AI agents actually receive from JavaScript websites](/learn/what-ai-agents-receive-from-javascript-websites)[React, Vue, Angular: making client-rendered content agent-readable →](/learn/react-vue-angular-agent-readable)

See what a recognized agent receives from your own URL.
[See what an AI agent receives from your URL](/tools/agent-response-check)

## Structured Data

```json
{
  "@context": "https://schema.org",
  "@type": "Organization",
  "@id": "https://www.siere.ai/#organization",
  "name": "Siere",
  "url": "https://www.siere.ai/",
  "logo": "https://www.siere.ai/social-square.png",
  "sameAs": [],
  "contactPoint": {
    "@type": "ContactPoint",
    "contactType": "customer support",
    "email": "contact@siere.ai",
    "url": "https://www.siere.ai/contact"
  }
}
```

```json
{
  "@context": "https://schema.org",
  "@type": "WebSite",
  "@id": "https://www.siere.ai/#website",
  "name": "Siere",
  "url": "https://www.siere.ai/",
  "description": "Your website is invisible to AI agents. Siere fixes that. We make your site readable, structured, and optimized for the next generation of AI-powered search.",
  "publisher": {
    "@type": "Organization",
    "name": "Siere",
    "@id": "https://www.siere.ai/#organization"
  }
}
```

```json
{
  "@context": "https://schema.org",
  "@type": "SoftwareApplication",
  "@id": "https://www.siere.ai/#software",
  "name": "Agent Experience Optimization",
  "url": "https://www.siere.ai/",
  "description": "Your website is invisible to AI agents. Siere fixes that. We make your site readable, structured, and optimized for the next generation of AI-powered search.",
  "applicationCategory": "BusinessApplication",
  "operatingSystem": "Web",
  "offers": [
    {
      "@type": "Offer",
      "price": "0",
      "priceCurrency": "USD",
      "name": "Starter"
    },
    {
      "@type": "Offer",
      "price": "99",
      "priceCurrency": "USD",
      "name": "Pro"
    }
  ]
}
```

```json
{
  "@context": "https://schema.org",
  "@type": "FAQPage",
  "@id": "https://www.siere.ai/#faq",
  "url": "https://www.siere.ai/",
  "mainEntity": [
    {
      "@type": "Question",
      "name": "What is Agent Experience Optimization?",
      "acceptedAnswer": {
        "@type": "Answer",
        "text": "AEO makes your website readable, structured, and reliably citable by AI agents like ChatGPT, Claude, Perplexity, and Gemini. Most modern websites are JavaScript-rendered and look like blank pages to AI crawlers — Siere fixes that without changing your visitor experience."
      }
    },
    {
      "@type": "Question",
      "name": "How does Siere work?",
      "acceptedAnswer": {
        "@type": "Answer",
        "text": "Siere sits between your server and incoming requests. When an AI agent visits, Siere serves an optimized markdown version of your content. When a human visits, your normal page is served unchanged."
      }
    },
    {
      "@type": "Question",
      "name": "Do I need to change my site?",
      "acceptedAnswer": {
        "@type": "Answer",
        "text": "No. Siere installs as middleware. There are no content changes or rewrites required."
      }
    }
  ]
}
```

```json
{
  "@context": "https://schema.org",
  "@type": "TechArticle",
  "headline": "Is llms.txt enough? Discovery versus delivery",
  "description": "llms.txt tells an agent where to look. It does not change what the agent receives when it gets there. This guide separates the two problems and shows how each is checked.",
  "url": "https://www.siere.ai/learn/is-llms-txt-enough",
  "mainEntityOfPage": "https://www.siere.ai/learn/is-llms-txt-enough",
  "dateModified": "2026-08-18",
  "author": {
    "@type": "Person",
    "name": "Tomislav Mladenov"
  },
  "publisher": {
    "@type": "Organization",
    "name": "Siere",
    "url": "https://www.siere.ai/"
  },
  "license": "https://creativecommons.org/licenses/by/4.0/",
  "isAccessibleForFree": true,
  "keywords": "llms.txt, llms-full.txt, discovery, sitemap, robots.txt, delivery",
  "citation": [
    "https://llmstxt.org/",
    "https://www.sitemaps.org/protocol.html",
    "https://www.rfc-editor.org/rfc/rfc9309"
  ]
}
```