How to Create an llms.txt File: A Practical Guide

An llms.txt file is a markdown document hosted at your site root that tells LLM crawlers what your site is about and which pages matter most. Think of it as a sitemap for AI — not a list of every URL, but a curated guide to your most important content, written in a format that large language models can parse directly.

The format was proposed in 2024 and has seen meaningful adoption since. Here's the full picture: what it is, who reads it, what to put in it, and how to host it.

What llms.txt Is

Traditional robots.txt controls crawler access — which parts of your site bots are allowed to visit. sitemap.xml lists all your URLs for search engine indexers.

llms.txt is different. It's not a permissions file or a URL inventory. It's a curated summary of your site, written in markdown, that LLM crawlers can read to understand your site's scope and identify your most valuable content.

The format is intentionally simple: a site name as an H1, a summary paragraph describing what the site covers, and one or more sections of linked content with brief descriptions. The result is a document that an LLM can read in a few seconds and extract a complete picture of what your site covers and which pages are authoritative.

Why markdown? Because LLMs are trained on markdown-heavy text. JSON and XML are machine-parseable but less natural for LLMs than markdown. The llms.txt spec leans into LLM-native formats rather than traditional crawler-native formats.

Which AI Systems Actually Read llms.txt

Confirmed adoption as of 2026:

Anthropic — Claude's crawlers read llms.txt files when building knowledge bases. Anthropic was an early adopter and their documentation explicitly mentions llms.txt support.

Perplexity — Perplexity's crawlers read llms.txt and use it to prioritize which pages to retrieve for indexing. Pages listed in llms.txt are more likely to be included in the candidate retrieval pool for relevant queries.

Mistral — Mistral AI reads llms.txt files in their training and retrieval crawls.

Not confirmed by Google or Bing — Google and Bing have not officially confirmed llms.txt support. Google AI Overviews appear to rely on Google's existing search index, not a separate llms.txt signal. That may change as the standard matures.

The practical implication: llms.txt is most clearly valuable for Perplexity and Anthropic's products right now. It's a low-effort setup with a concrete upside for those platforms and potential upside for others as adoption grows.

The Format Specification

The llms.txt spec (documented at llmstxt.org) defines the structure:

# Site Name

> One-paragraph summary of what this site is about, who it's for, and what topics it covers. Be specific and entity-clear — treat this as the paragraph you'd want an LLM to read if it could only read one thing about your site.

## Section Name

- [Page Title](https://yourdomain.com/path/): Brief description of what this page covers.
- [Page Title](https://yourdomain.com/path/): Brief description of what this page covers.

## Another Section

- [Page Title](https://yourdomain.com/path/): Brief description.

Key format rules:

• H1 is your site name — one line, no decoration.
• Blockquote (>) is your site summary — one paragraph describing your site's scope.
• H2 sections organize your content by topic or category.
• List items are your page links — each one with a title, URL, and brief description.
• Everything is markdown. No JSON. No XML.

A Complete Example

Here's an example llms.txt for a content marketing blog:

# ContentPro Blog

> ContentPro is a practical resource for content marketers, SEO professionals, and agency teams. We cover content strategy, SEO optimization, AI search visibility (GEO), and content performance measurement. Our audience ranges from solo content creators to boutique agencies managing 10–50 clients.

## Content Strategy

- [How to Build a Content Calendar That Actually Works](https://contentpro.com/blog/content-calendar/): A step-by-step guide to planning and scheduling content across a year, with template download.
- [Content Audit Framework for SEO](https://contentpro.com/blog/content-audit/): How to systematically audit existing content, identify gaps, and prioritize updates.

## SEO Guides

- [On-Page SEO Checklist](https://contentpro.com/blog/on-page-seo-checklist/): 30-item checklist covering title tags, meta descriptions, heading structure, and internal linking.
- [Schema Markup for Content Sites](https://contentpro.com/blog/schema-markup/): Which schema types matter most for blogs and how to implement them correctly.

## AI Search (GEO)

- [What Is Generative Engine Optimization](https://contentpro.com/blog/what-is-geo/): Plain-English explainer of GEO — how AI Overview citations work and the 5 core practices.
- [How to Get Cited in Perplexity AI](https://contentpro.com/blog/perplexity-citations/): The 6-step playbook for becoming a frequently-cited Perplexity source.
- [AI Overview Citation Tracking](https://contentpro.com/blog/ai-overview-tracking/): Methods for tracking whether your content is cited in Google AI Overviews.

Notes on this example:

• The blockquote summary is specific — it names the audience, the topics, and the scope range. Vague summaries don't help LLMs much.
• Each page description is one sentence that answers "what does this page cover?" — not "this is a great guide to..." filler.
• Sections mirror the site's content architecture. If your site has a guides section and a tools section, those should be the sections in llms.txt.

What to Include (and What to Leave Out)

Include:

• Your highest-traffic, highest-authority pages
• Pages that cover your core topics most comprehensively
• Hub pages or pillar content that links to your full topic clusters
• Any pages you most want AI systems to associate with your brand and topic

Leave out:

• Tag pages, category archives, and pagination pages
• Pages behind login or paywalls (LLM crawlers can't access them)
• Thin content pages (stub posts, placeholder pages)
• Privacy policy, terms of service, and legal pages
• Pages you've blocked in robots.txt

There's no maximum page count in the spec, but long lists degrade the document's usefulness. Aim for 15–40 pages — enough to represent your full content scope without burying the most important items.

How to Host It

llms.txt is a static text file hosted at https://yourdomain.com/llms.txt. That's it.

For static sites (Gatsby, Next.js, Astro, Hugo): Add llms.txt to your public/static folder. It will be served at the root URL automatically.

For WordPress: Upload llms.txt via FTP to your site's root directory (same level as wp-config.php). Verify it's accessible at yourdomain.com/llms.txt.

For other CMS platforms: Check whether the platform lets you add files to the root directory. Most do — look for a "file manager" or static assets section. Alternatively, if you use Cloudflare, you can add the file as a Worker that returns the content for requests to /llms.txt.

Content-type: The file should be served as text/plain or text/markdown. Most servers handle this automatically for .txt files.

Verify your file is accessible and correctly formatted by visiting yourdomain.com/llms.txt in your browser. You should see the raw markdown text.

Maintaining Your llms.txt File

Update your llms.txt when:

• You publish a major new piece of content or a hub page
• You significantly revise an existing page that's already listed
• You restructure your site's content architecture
• You remove or redirect pages that are listed

You don't need to update it every time you publish a minor post. The goal is for it to accurately represent your site's best and most important content — not to be an exhaustive index.

The llms.txt and robots.txt Relationship

llms.txt is not a permissions file. You can't use it to block AI crawlers — that's what robots.txt is for. If you want to allow AI crawlers to access your site (which you generally do, for GEO purposes), your robots.txt should not block the major LLM crawlers:

• Anthropic's crawler: ClaudeBot
• Perplexity's crawler: PerplexityBot
• OpenAI's crawler: GPTBot

Check your robots.txt to confirm none of these are blocked. If they are and you want them to crawl your site, remove those rules.

For the broader framework of how llms.txt fits into AEO strategy, see answer engine optimization for bloggers. For the full AI search optimization checklist, see content optimization for AI search.

The AI search hub covers all the GEO principles that work alongside llms.txt.