The /llms.txt standard is a proposed web convention (similar to robots.txt) that gives Large Language Models a curated, Markdown-formatted index of your site's most valuable content. Unlike sitemap.xml, which lists URLs for crawling, llms.txt is designed to guide AI agents to clean, token-efficient text files, stripping away the HTML boilerplate that confuses ingestion pipelines.
The Problem: The High Cost of HTML Noise
For the past 20 years, we've optimized websites for the Document Object Model (DOM). We built complex hierarchies of divs, scripts, and styles to render visual interfaces for humans. For an AI crawler, this visual layer is toxic waste.
When an LLM scrapes your landing page, it has to burn valuable context window tokens processing your navigation bar, your footer links, your tracking pixels, and your CSS classes.
Rely solely on standard crawling and you're asking the AI to dig for gold in a landfill. The /llms.txt standard solves this by handing the AI a map directly to the gold vault. The underlying economics of that token waste are detailed in The Context Window Economy.
The Solution: The Markdown Sitemap Strategy
The core philosophy of the /llms.txt standard is text-first delivery. Instead of forcing the AI to scrape your HTML pages, you provide a text file at the root of your domain (yourdomain.com/llms.txt). The file lists your core entities and documentation, and crucially, points to Markdown (.md) or plain-text versions of those pages where they exist.
Markdown is the native tongue of LLMs. It represents structure (# H1, **Bold**, - List) without the overhead of HTML tags. By serving Markdown, you maximize information density: every token the AI reads adds semantic value. The token math behind this is in The Token Tax.
Technical Implementation: Building Your File
A robust /llms.txt file follows a specific hierarchy. It isn't just a list of links. It's a semantic table of contents.
Place the file at the root: https://example.com/llms.txt. Ensure your robots.txt allows access to it.
Note: this doesn't replace sitemap.xml (which is for Google Search Console). It's an additive layer for Answer Engine Optimization.
Divide the file into sections that mirror your Entity Home structure. Use H2 headers within the text file to group content.
Recommended sections: Core Identity (who you are), Product Documentation (technical specs in .md files), Pricing & Policies (the "truth" data to prevent brand hallucinations).
The standard also supports an optional section for a concise summary: a single file that compresses your entire site into less than 10,000 tokens for rapid ingestion.
sitemap.xml vs. llms.txt
Feature | sitemap.xml (Traditional) | llms.txt (The New Standard) |
Target Audience | Googlebot, Bingbot | ChatGPT, Claude, Perplexity |
File Format | XML | Markdown (human/AI readable) |
Content Goal | Indexing (find the URL) | Ingestion (read the content) |
Link Destination | HTML pages (heavy) | Markdown/Text files (light) |
Context | URL + last modified date | Title + description + context |
Code Example: A Working /llms.txt File
Here's a boilerplate you can deploy today. Copy the structure, modify the links, upload it to your root directory.
If you don't have a CMS that generates .md files automatically, point these links to your standard HTML pages. Just make sure those HTML pages are strictly formatted with semantic tags (<article>, <table>) per our HTML formatting guide so they parse correctly.
Generate your llms.txt, robots.txt, and schema in one pass.
Free GEO Asset Generator. Builds a compliant, curated llms.txt from your site structure, no hand-coding.
Generate your llms.txt free →Key Takeaways
- Reduce token cost. llms.txt creates a friction-free path for AI crawlers, stripping away HTML noise.
- Control the context. Curating this list lets you decide which pages the AI prioritizes, preventing it from indexing low-value tag or category pages.
- Markdown is king. Whenever possible, serve content in Markdown to maximize ingestion speed and accuracy.
- Parallel infrastructure. This doesn't replace sitemap.xml. It runs parallel to it specifically for the Search Everywhere ecosystem.
- Future-proofing. Agents like Perplexity already prioritize sites that make data retrieval easy. Low-effort, high-reward signal of technical competence.
References & Further Reading
- llms.txt Proposal: The Unofficial Standard for AI Crawlers. Documentation on the emerging convention for text-first delivery.
- Link: https://llmstxt.org/
- OpenAI Documentation: Optimizing Context Windows. Best practices for feeding data to GPT models, emphasizing token efficiency.
- Google Search Central: Robots.txt Specifications. The foundational standard for crawler directives that inspired llms.txt.
