Let's face it - most site owners set up their robots.txt file back in 2008 and probably never looked at it again.
Because the internet never stops developing - whether in the right direction or the wrong one - there's a new file on the block that a lot of people are getting very excited about, which is the llms.txt. A file specifically aimed at large language models (AI engines, for you and I).
Here's your five minute guide to what it is, what it isn't, and whether it's worth ten minutes of your morning.
What an llms.txt file is
The short answer is that it's a 'proposed' standard - note, not an actual web standard, yet. Jeremy Howard, over at Answer AI, put forward, in September 2024, the proposal to have a markdown (simply formatted text) file at the root (or top level) of your site with one single purpose - to give LLMs a curated, clean, summary of your site's most important content.
The truth is, no matter how 'intelligent' AI models are, they still struggle to navigate websites somewhat, particularly if they have multiple navigation menus, JavaScript, cookie banners, marketing popups and the general detritus that litters a lot of modern web pages. Having an llms.txt file reduces the key information to a specific location that, in theory, the AI model can look at and get all the same information in one place that it would have to otherwise struggle to find amongst your website.
What an LLMS.TXT file is not
This is the easy bit. It's not :
A. An official standard - it's not been adopted by W3C or IETF.
B. Confirmed as used by the major LLMs - at the time of writing, Anthropic, OpenAI, Google and the other major AI operators haven't confirmed that their models use it as a crawling directive.
C. A ranking factor - if you're thinking (or someone's telling you) that it's a silver bullet to get your site "showing in ChatGPT", then, no. You / they are wrong.
D. A replacement for your actual robots.txt file or your sitemap.xml - both of those, which are official standards, remain essential.
The two flavours
The proposal suggests two different llms.txt files :
llms.txt - a short index - the site's name, a brief description, links to key pages on the site with one-line summaries; and / or
llms-full.txt - the actual content of the key pages concatenated into one markdown file so that an AI model can slurp up your entire site's knowledge base in one go.
Why might it matter
Like it or not, but AI-driven search is increasingly how people find businesses (we looked at how to monitor that in an earlier blog post) - so if an AI model does ever fetch your live site, then presenting it with a clean, concise, easy-to-ingest summary is much easier for it to take on board than wading through the site itself.
Also, it's low effort (as we'll come on to) and low risk. It doesn't impact anything else on your site.
Why might it not matter (yet)
As we said above, no major AI model has confirmed that they're using it yet, although on the flip side, none have confirmed that they not using it. Also, data that feeds into LLMs is often scraped months in advance, so adding an llms.txt file now doesn't mean that your site will magically start appearing retrospectively in older models.
And, let's face it - as long as your site is well-structured already - sensible HTML structure, a sitemap, relevant schema markup - then you're probably already 90% of the way there.
Our honest verdict
It's probably worth doing, on the basis that it's a small outlay of time at low risk, if you run a content-heavy website, something with a lot of documentation, or anything where being cited by an AI model will drive qualified traffic to your site.
It's probably not worth doing if you're a 12 page local service website where your Google Business profile is doing most of the hard work for you.
Either way, it's a 30 minute job, not a three month project, so if someone's coming to you to convince you to implement one, don't let them tell you otherwise.
In conclusion
It's probably one to file under "sensible housekeeping" rather than "absolute game changer".
We've added it to our site (you can read our llms.txt file and our llms-full.txt file via those links - we're not precious about these things). And we've added it to a handful of our clients' sites as a low-cost experiment. We'll report back in six months on whether we (or they) saw a significant increase in AI traffic.