llms.txt in Laravel: giving AI crawlers a map of your shop

More visitors keep arriving from ChatGPT and Perplexity, which raises an awkward question: what does a language model actually see when it reads your shop? We built llms.txt for PadPad.hu. Why it became a route instead of a static file, what we deliberately left out, and the cache trap that catches everyone.

Over the past year I kept seeing in the shop's analytics that a visitor had arrived from ChatGPT or Perplexity. There were not many, but the number grew month over month. That got me thinking: if a language model is recommending our shop, what is it basing that on, and what does it see of our site at all?

The answer is not too flattering. The HTML of a Laravel webshop is full of things that help a human but are noise to a bot: a cookie banner, a nav menu with seventy links, Livewire attributes, tracking scripts, a footer. The actual information, what we sell and to whom, gets lost somewhere in there. And a model's context window is finite. If it reads the wrong pages, it gives the wrong answer, or does not mention us at all.

There is a proposal aimed at this problem, called llms.txt. We built it for PadPad.hu, and here is how.

What it actually is

Jeremy Howard proposed it in 2024, and the specification can be read at llmstxt.org. The idea is simple, put a Markdown file at the root of the domain, named /llms.txt, in which we briefly describe what the site is and list the important subpages with a short explanation.

It is worth separating it from the two other root files, because they are easy to confuse:

• robots.txt is about what may be crawled. Allow and disallow, nothing more.

• sitemap.xml is about what exists. A complete URL list, machine format, no context.

• llms.txt is about what matters and why. Curated, written in human language, deliberately short.

The format is fixed but not complicated. A # level heading, below it a > blockquote with a one sentence summary, then free text, and finally ## level sections with link lists, where every link gets an explanatory sentence next to it.

Let me put one caveat up front, because the credibility of this article depends on it: this is not an official standard. Neither OpenAI nor Anthropic nor Google has promised to take it into account. It may well come to nothing. On the other hand it is half an hour of work, and if it lands, we won cheap.

Static file or generated route

This is where the first decision came, because the obvious solution would have been to drop a public/llms.txt file into the public folder and be done. Nginx serves that, zero runtime cost, finished. Naturally we rejected it, because the content on the site changes and I did not want to keep poking at that file by hand. In a webshop the categories change, blog posts arrive weekly, information pages get reworked. A hand maintained static file will not merely be stale in three months, it will be lying. And in the case of a map handed to an AI, inaccuracy is worse than absence, in my view.

Instead of that solution it became a route, with a controller and a cache. It is generated from the database, so it is always in sync with what is actually on the site. It is important to note that this is not universal advice. If someone has a static documentation site that changes twice a year, public/llms.txt is completely fine, and it would be silly to write a controller for it. The choice depends more on how fast the content moves.

Let's look at the implementation

The route is one line, and the dot in the filename does not bother Laravel's router:

Route::get('llms.txt', LlmsTxtController::class)
    ->name('llms-txt');

The controller is invokable, and follows the same pattern as the Google Merchant feed that already exists in the project. That was a deliberate decision: if a codebase already has a proven solution to a similar problem, there is no point inventing a new one next to it.

public function __invoke(): Response
{
    $content = Cache::remember('llms-txt.v1', now()->addHours(6), function (): string {
        return view('llms-txt', [
            'categories' => Category::active()
                ->orderBy('name')
                ->get(['name', 'slug']),
            'pages' => Page::query()
                ->orderBy('title')
                ->get(['title', 'slug']),
            'blogPosts' => BlogPost::published()
                ->orderByDesc('published_at')
                ->limit(20)
                ->get(['title', 'slug', 'excerpt']),
        ])->render();
    });

    return response($content, 200, [
        'Content-Type' => 'text/plain; charset=utf-8',
    ]);
}

A few details that are not accidental:

get(['name', 'slug']) pulls only the columns we need, we do not load anything unnecessarily. Hydrating full models for a text output is waste, especially when the model carries media files and SEO data.

Category::active() and BlogPost::published() are existing scopes. If somebody changes tomorrow what counts as published, that takes effect here automatically too. If I hand wrote a where('is_published', true) condition, in two months it would drift away from the logic of the blog page. Wherever it makes sense and you can, use scopes (they were not invented by accident).

Six hours of cache is fresh enough for content that changes at most once a day.

The template lives in a Blade view, because roughly eighty percent of the content is fixed prose, and writing that in a controller with string concatenation would be unreadable. There is one trap with it though: the newlines around Blade directives. In Markdown a stray blank line results in a broken list, so the placement of @if and @foreach needs attention.

What we put in and what we left out

This was a more interesting question than the code. I read up on it, and of course ChatGPT could not be left out either, I asked for its "opinion".

In went a concrete summary of what we sell and to whom. In went the business facts a bot would otherwise have to assemble from five different pages: language, currency, shipping area, payment methods. In went the main functional pages, in our case the mousepad designer and bulk ordering, because these are what set us apart. Then the categories, the legal and information pages, and the twenty most recent blog posts with excerpts.

The full product list was left out, that would be hundreds of lines, it would go stale fast, and this is exactly what the sitemap and the product feed are for, which we do point to at the end of the file. The cart, the checkout, the profile and the orders were left out. These are user specific and contain zero public information. In fact we went a step further and stated explicitly in the last sentence of the file that these are not relevant. If it is reading it anyway, let's tell it where not to go. Keyword stuffing was left out too. The recipient is a language model, which treats that as noise, or in the worse case as manipulation.

Testing

A generated file that nobody looks at will break silently at the first model refactor. So six tests were written for it: HTTP 200 and the correct content type, the presence of the main routes, an active category shows and an inactive one does not, a published blog post shows and a draft does not, and finally the sitemap and feed links.

There is one thing worth watching out for when testing a cached endpoint:

beforeEach(function (): void {
    Cache::forget('llms-txt.v1');
});

Without this, the output saved by the first test bleeds into the next one, and we get a false green. This is the most common mistake in this area, and it goes unnoticed for a long time, because the tests appear to work.

What else was needed

An Allow: /llms.txt line went into robots.txt. Technically redundant, since nothing blocks it by default, but it documents the intent for whoever looks at it later. Beyond that it is worth filtering the server logs for AI crawler user agents such as GPTBot, ClaudeBot or PerplexityBot. Without that we have no idea whether anyone ever fetches the file, and in three months we will have nothing to say to the question of whether it was worth it.

To sum up

Half an hour of work, negligible runtime cost, six tests behind it. It is not an SEO silver bullet, and it may well never become a standard, though I hope it does.

The real lesson, though, is not llms.txt. It is that every output meant for machine consumption, whether a sitemap, a product feed or this, should be generated from the code and covered by tests. Hand maintained files do not die with an error message, they go stale quietly, and nobody notices.

Would you like to build something similar for your website? We are happy to do it for you, get in touch: janes.hu/kapcsolat