llms.txt vs robots.txt vs sitemap.xml: what each one does

Three small files, three very different jobs. People often mix them up, or assume one replaces another. It doesn't. Here's what each one does and how they work together.

The short answer

• robots.txt is about permission: which crawlers may visit which parts of your site. Search engines and many AI crawlers read it.
• sitemap.xml is about discovery: a complete list of the pages you want found. Search engines read it.
• llms.txt is a summary: a short, curated guide to what matters most. AI tools and agents read it if they choose to.

robots.txt: the rules of the road

robots.txt tells automated visitors where they may and may not go. It's how you'd ask a crawler to stay out of your admin area, or decide whether a particular AI company's crawler can read your site at all.

It's a request, not a lock: well-behaved crawlers follow it, but it doesn't secure anything. And one wrong line (for example a leftover Disallow: / from a staging site) can hide your whole site from search.

sitemap.xml: the full inventory

A sitemap lists the pages you want search engines to find, often with dates they last changed. On WordPress, SEO plugins create it automatically. It's meant to be complete: every product, post and page that should be indexed.

llms.txt: the highlights

llms.txt is the opposite of a sitemap. Instead of everything, it's the handful of pages that best explain who you are and what you do, with a sentence of context for each. It's written in Markdown so it's easy for both people and AI tools to read. You can learn more in our plain-English guide to llms.txt.

Common mix-ups

• "llms.txt lets me block AI." No. Blocking or allowing crawlers is robots.txt's job. llms.txt doesn't grant or remove permission.
• "My sitemap covers it." A sitemap tells crawlers what exists, not what matters. llms.txt adds the "why" for your most important pages.
• "I'll put every page in llms.txt." Keep it short. If everything is highlighted, nothing is.
• "robots.txt blocks AI from my llms.txt." It can. If you disallow a crawler entirely, it won't read llms.txt either. Decide your AI crawler policy first.

How they work together

1. robots.txt decides who gets in.
2. sitemap.xml shows search engines everything there is.
3. llms.txt points AI tools at what matters most.

All three are small, and all three can break silently. Check your llms.txt with our free validator, and if you list it on LLMs Ledger we'll also tell you when your robots.txt stops our checker from reading it.