llms.txt is a markdown text file at the root of a website that lists your most useful pages for AI tools, with a short description of each. Jeremy Howard proposed it on September 3, 2024. No major AI company has said it reads the file when choosing which sites to cite, so for most sites it works as a tidy index with no known ranking effect.
That answer disappoints people who were told the file is the new robots.txt. It also leaves out one place where llms.txt does earn its keep. Both points matter before you spend an afternoon on it.
What the file looks like
The llmstxt.org proposal sets out a short structure. The file is named llms.txt and sits at your site root (yourdomain.com/llms.txt), though the proposal also allows it at a subpath such as /docs/llms.txt, where it covers the URLs under that path.
Inside, the order is fixed. An H1 with the name of your site or project comes first, and it’s the only required element. Next comes a blockquote with a short summary.
After that you can add markdown paragraphs or lists with no headings, and then any number of H2 sections that hold file lists. Each entry in a list is a markdown link, optionally followed by a colon and a note.
By convention, an H2 section called “Optional” holds secondary material that an agent can skip when it needs a shorter context.
Here is a made-up example for a small plumbing company:
# Acme Plumbing Pune
> Residential plumbing repair and installation in Pune. Fixed-price quotes, same-day callouts for leaks.
## Services
- [Leak repair](https://example.com/leak-repair): what we fix, response times, price ranges
- [Water heater installation](https://example.com/water-heater): brands we fit and warranty terms
## Company
- [Service areas](https://example.com/areas): neighbourhoods we cover
- [Contact](https://example.com/contact): phone, hours, booking form
## Optional
- [Blog archive](https://example.com/blog): older guides and seasonal tips
That’s the whole format. Ten lines can be enough, and a file for a small site rarely needs more than thirty. The restraint is deliberate: the proposal wants a shortlist an agent can read quickly, not a second copy of your sitemap.
What it was designed to do
Howard’s page says the file is for on-demand use. An AI agent that needs information while helping someone consults it at that moment. The proposal expects it to help with inference, meaning the time a model answers a question, rather than with training.
It sets llms.txt against two older files. Robots.txt tells automated visitors what access is acceptable. The proposal says llms.txt complements that by giving context about content that is allowed. A sitemap lists all the indexable human-readable content, and the proposal argues llms.txt can’t replace it, because sitemaps often omit LLM-readable versions of pages, exclude external URLs and cover more content than fits in a context window.
So the pitch is a curated shortlist. You tell a hurried reader, human or machine, which ten pages matter.
Who reads it today
This is the section the vendors skip, so I’ll slow down.
In April 2025, Google’s John Mueller responded to a question about the file. Search Engine Journal reported his answer: as far as he knew, none of the AI services had said they use it, and he called it comparable to the keywords meta tag. His reasoning was that the file is the site owner’s own claim about the site, so a bot that uses it would still have to check the real pages.
He added that server logs show AI services don’t even check for it.
Mueller hedged with “as far as I know”, and the quote is now more than a year old. Treat it as a dated statement from one Google employee, not a permanent ruling.
Google’s own documentation points the same way for its products. The AI features page says you don’t need new machine-readable or AI text files to appear in AI Overviews or AI Mode.
Then there is the outside evidence. SE Ranking, a vendor with its own interest in AI visibility tooling, analysed nearly 300,000 domains. Only 10.13% had an llms.txt file. The company found no correlation between having the file and being cited by AI models.
Its prediction model also got slightly more accurate when the file was removed as a factor. SE Ranking still calls adding the file a low-effort way to prepare for later.
A correlation study can’t prove that a file has no effect, because the sites that publish one may differ from other sites in many ways. Still, it’s the largest public test I found, and it shows no sign of a lift.
The proposal’s own site paints a rosier picture.
It says thousands of sites publish the file, that documentation platforms generate it automatically, that Chrome’s Lighthouse audits for it as part of agentic browsing checks, and that OpenAI, Anthropic and Google’s Gemini publish llms.txt files for their developer documentation. I could not confirm the Lighthouse claim independently. The last claim also needs reading carefully. A company publishing its own file is not evidence that its product reads yours.
The one place it makes sense
Documentation and developer sites. The proposal came out of that world, where a coding assistant fetches a library’s docs to answer a programmer’s question, and a clean index of markdown pages saves the tool from parsing a heavy website. If you run an API, a software library or a technical product with extensive docs, publishing the file costs little and has a plausible use.
For a service business, a local shop, a blog or an ecommerce store, that logic is weaker. Nobody has shown that a plumbing company’s llms.txt changes what ChatGPT says about plumbers in Pune. Your time buys more elsewhere: indexable pages, clear answers, and mentions on other sites.
If you decide to publish one
Write it by hand. A generated file tends to list every URL on the site, which defeats the purpose of a curated shortlist.
Pick 10 to 25 pages. Choose the ones you’d send a new customer who asked, “where do I start?” Describe each in a sentence that says what the page contains, not what you hope it does for you. Skip marketing language, because context is the entire point of the file.
Upload it to the root so it loads at yourdomain.com/llms.txt, and open it in a browser to check that it returns the raw text. Link only to pages that are public and return a 200 status. A file that points to redirected or deleted URLs is worse than no file, because it hands an agent bad directions with your name on them.
Then put a reminder in your calendar. Pages move, products retire, and an outdated llms.txt is easy to forget because nobody ever sees it.
Keep your sitemap and robots rules as they are, since the new file adds nothing to either. If you’re deciding which AI crawlers to allow, that happens in robots.txt, and my robots.txt checker will show you what your current file actually permits.

Check your own server logs before believing anyone
You can test the claim on your own site in about ten minutes. Publish the file, wait a few weeks, then search your server access logs for requests to /llms.txt and see which user agents asked for it.
Mueller’s point was that this is what the logs show. A Reddit commenter in the same thread, who said they host about 20,000 domains, reported that apart from some niche user agents, no bots were grabbing the files. SEJ’s write-up names BuiltWith’s bot as one that does fetch them. That’s one hosting company’s report and SEJ didn’t verify it, so read it as a data point and not a measurement.
Look for the crawlers OpenAI documents on its bots page: OAI-SearchBot, GPTBot and ChatGPT-User. If those names show up in your logs hitting normal pages but never touching /llms.txt, you have your answer for your site. If they do fetch it, write down the dates. That would be worth knowing, and it would be news.
If you can’t reach raw access logs, ask your host how to export them. Without logs, you’re guessing, and so is everyone who sells you the file as a ranking tactic.
Mistakes I’d avoid
Paying for it. If an agency quotes you a separate fee to “implement llms.txt for AI visibility”, ask for the study that shows the file moves citations. Based on what I found, it doesn’t exist yet.
Stuffing it with keywords. The file’s whole premise is clear context. Keyword lists add nothing a model can use.
Hiding important content there. Anything you want a person or a search engine to see belongs on a normal, crawlable page. Google’s documentation is clear that its AI features rely on indexed, snippet-eligible pages.
What to do with the same afternoon
If you were going to spend a few hours on llms.txt, these jobs have documented effects. Google’s AI features page says a page must be indexed and eligible for a snippet to appear as a supporting link, so run your ten most important URLs through the URL Inspection tool in Search Console and fix whatever blocks indexing.
Then check what your robots.txt lets OpenAI’s three crawlers do. A site can allow search while declining training, because OpenAI documents them as separate bots. Finally, pick the single page on your site that answers your buyers’ most common question and make the first paragraph under its heading a complete answer. That is a change you can see on the page and test against search results.
None of that is glamorous. All of it is further along the evidence than a file that major AI companies haven’t said they read.
Quick answers
Is llms.txt the same as robots.txt for AI?
The two files do different jobs. Robots.txt controls which automated visitors may fetch which URLs. The proposal describes llms.txt as a complement that supplies context, and it blocks nothing at all.
Does Google use llms.txt?
Google’s documentation says no additional AI text files are needed for its AI features, and Mueller has compared the file to the keywords meta tag. Neither source describes Google using it as a signal.
Can the file hurt me?
I found no evidence that it harms rankings. The risks are practical: wasted time, stale links, and the false comfort of thinking you’ve done something for AI visibility.
What to do this week
If you run developer documentation, write a 15-link llms.txt and publish it. If you run anything else, skip it until a platform documents that it reads the file, and spend the afternoon checking that your best ten pages are indexed instead. For help deciding where AI visibility work should start on a specific site, see my GEO consulting page.
