llms.txt Guides · · 6 min read · By Mobilize Cloud
Open yoursite.com/llms.txt in a browser. A working file looks plain: black text on a white background, a # title at the top, lines of links. If you see your home page, your theme's "page not found" design, a "checking your browser" screen or a login form instead, then whatever AI tools are getting at that address isn't your llms.txt. It's a web page.
Our validator flags this as Web page instead of a file and scores it 0, because nothing at that address can be read as llms.txt. It's an llms.txt 404 in disguise, and the cause is almost always one of a handful of things: the file doesn't exist and the site shows a page instead (a "soft 404"), a catch-all route, a plugin, or a CDN or firewall. Here's how to spot each one and fix it.
First, find out what's actually being served
Three quick checks tell you most of what you need to know:
- Look at it in a private or incognito window. If you're logged in to your site, you may see something different from everyone else. Maintenance-mode and coming-soon plugins, for example, often show the real site to logged-in admins and a holding page to everyone else.
- View the page source (right-click, View page source). If it starts with
<!DOCTYPE html>or<html, it's a web page, however plain it looks. - Run it through the validator. It fetches the address the way a tool would, without your browser's cookies, and reports what came back, including the content type. A real llms.txt should be served as
text/plainortext/markdown.
Soft 404s: an llms.txt 404 that looks like success
When a browser or crawler asks for an address that doesn't exist, the server should answer with a 404 status code. A soft 404 is when it doesn't. Google's documentation defines it as a URL that returns a page telling the user the page doesn't exist, along with a 200 (success) status code. See Google's crawl error guide for the full description.
For llms.txt, soft 404s usually look like one of these:
- Your home page appears at /llms.txt. Some sites, and some "redirect all 404s to the home page" plugins, send every unknown address to the home page. It looks like the site works, but there's no file there.
- Your theme's "not found" page appears. This is the normal behavior when no file exists. On a well-configured site it comes with a proper 404 status; on some setups it doesn't.
The fix: in both cases, the real problem is that there's no llms.txt at the root of your site. Create one (our generator takes about two minutes) and publish it at yoursite.com/llms.txt. If you use a plugin that redirects missing pages to the home page, it's worth reviewing that setting anyway: it hides broken links from you and from search engines.
Catch-all routes in website builders and JavaScript apps
Many modern sites are built as JavaScript applications (React, Vue, Angular and the like). They're often set up with a catch-all rule: any address the server doesn't recognize gets the app's main index.html, and the app decides what to show. That's convenient for the app, but it means /llms.txt returns the app's HTML with a success code unless a real file exists.
How to spot it: the page source at /llms.txt is a short HTML document with a <div id="root"> or similar, and lots of script tags.
The fix: put llms.txt in the folder your build copies as-is to the live site. In many frameworks that's a folder called public or static. On most hosts, a real file takes priority over the catch-all rule. If yours doesn't, ask your developer to add an exception for /llms.txt (and /llms-full.txt if you have one) before the catch-all.
The file is in the wrong place
llms.txt has to sit at the root of the domain: yoursite.com/llms.txt, not in a subfolder. We see a few common mix-ups:
- Uploaded through the WordPress media library. That puts it somewhere like
/wp-content/uploads/2026/11/llms.txt. The root address still shows a 404 page. Upload it with your host's File Manager or SFTP into the folder that containswp-config.phpinstead. - WordPress installed in a subfolder. If WordPress lives at
/blog/or/wp/, the file still belongs at the top level of the domain, not inside the WordPress folder. - A hidden double extension. On Windows, a file named
llms.txtin some editors is saved asllms.txt.txtwith the extension hidden. Turn on file extensions in File Explorer and check the name.
Our guide to adding llms.txt to WordPress covers the upload step by step.
Plugins that intercept the address
On WordPress, the plugin list is the next place to look. Things that can catch /llms.txt before your file is served:
- SEO plugins with an llms.txt feature. If one plugin generates the file and you've also uploaded your own, or the feature was switched off during an update, you can end up with the 404 page. Pick one source and make sure it's on.
- Redirect plugins with a broad rule (for example, redirecting everything ending in
.txt, or old URL patterns from a site migration). - Maintenance and coming-soon plugins, which show a holding page to logged-out visitors, crawlers included.
- Security plugins and membership plugins, which can show a block page or login form to visitors they don't recognize, or to anything that looks like a bot.
The fix: check each of these plugins' settings for exceptions or "allowed paths", and add /llms.txt. If you can't tell which plugin is responsible, your host's staging site is a safe place to deactivate them one at a time.
CDNs, firewalls and bot protection
If your site sits behind a CDN or web firewall (Cloudflare, Sucuri, your host's built-in protection and so on), two things can go wrong.
Challenge pages. Bot protection often answers suspected bots with a challenge page, the "checking your browser" or "verify you are human" screen. That page is HTML. Your browser passes the check, so you see the file, but an AI tool fetching it gets the challenge. As an example, Cloudflare's documentation says its Bot Fight Mode issues challenges to detected bots and can't be bypassed or skipped with WAF custom rules. Settings and plan features vary, so check your provider's documentation for how to let well-behaved crawlers through, and look for any separate "block AI bots" setting while you're there. Our post on AI crawlers in robots.txt covers the crawler names.
A cached 404. If the CDN cached the "not found" page before you uploaded the file, it may keep serving that page for a while. Purge the cache for /llms.txt (or the whole site), along with any caching plugin on WordPress, then check again in a private window.
Quick reference
| What you see at /llms.txt | Likely cause | Fix |
|---|---|---|
| Your home page | No file, plus a catch-all redirect or route | Publish the file at the root; review redirect rules |
| Your theme's "not found" page | No file at the root, or it's in a subfolder | Upload to the root folder; check the file name |
| A short page full of scripts | JavaScript app catch-all | Put the file in the public or static folder |
| "Coming soon" or a login form | Maintenance, membership or security plugin | Add an exception for /llms.txt |
| "Checking your browser" | CDN or firewall bot protection | Adjust bot settings to allow crawlers |
| An old version or 404 after uploading | Cache | Purge the CDN and plugin cache |
Make sure it stays fixed
This problem has a habit of coming back. A plugin update, a new firewall rule, a site migration or a host change can all quietly put a web page back where your file was, and nothing on the site itself looks broken. So after any of those changes, open /llms.txt in a private window again, or rerun the validator.
When it's working, list your site on LLMs Ledger. Every plan rechecks the file on a schedule, and paid plans email you if it breaks, so a page sneaking back in at /llms.txt gets caught.
Bottom line: if /llms.txt shows a web page, AI tools aren't getting your file. Check in a private window, view the source, then work through the usual suspects: no file at the root, a catch-all route, a plugin, or a CDN challenge.
Is your llms.txt working?
Check any site for free and get plain-English fixes in seconds.
Keep reading
llms.txt Guides
How to check if your llms.txt is workingllms.txt Guides
llms-full.txt explained: do you need one?llms.txt Guides
10 common llms.txt mistakes (and how to fix each one)