What is the LLMS.txt file, and what does it mean for SEO

Much like how robots.txt has helped search engines understand and get direction on how to crawl a website, llms.txt aims to create a standardized way for AI systems to access and interpret web content.

What is llms.txt?

LLMs.txt is a proposed standard by Jeremy Howard that allows website owners to provide LLM-friendly content in a structured format. The standard addresses a critical limitation of large language models: their context windows are often too small to handle entire websites, and converting complex HTML pages with navigation, ads, and JavaScript into LLM-friendly plain text can be challenging and imprecise.

At its core, llms.txt is a markdown file placed in a website's root directory that provides AI models with the following:

• A flattened version of website content
• Guidance on how content should be accessed and used
• Clear, structured text that's easier for LLMs to process

How llms.txt Works

Unlike robots.txt, which directs crawlers on how to crawl/interact with a website, llms.txt provides AIs with content and the content is presented. We can:

1. Provide complete flattened content: Create a single file containing the entire website's textual content, stripped of HTML and other non-essential elements.
2. Include URL lists: Point to specific sections of a website that are relevant for AI processing.
3. Add summaries: Offer concise website content descriptions to help LLMs better understand the context.

The file uses simple markdown language, making it both human and LLM-readable. You can view it kind of like a detailed sitemap.

Tools for Generating llms.txt

I did create a tool that would flatten pages/websites into an llms.txt file, but there are several tools already to help website owners make their llms.txt files:

1. Markdowner: A free, open-source tool that converts website content into structured Markdown files.
2. Appify: Jacob Kopecky's llms.txt generator.
3. Website LLMs: A WordPress plugin that automatically creates llms.txt files from posts and pages.
4. FireCrawl: One of the first tools developed specifically for llms.txt file creation.

Current Adoption

The llms.txt standard is gaining traction, with several organizations already implementing llms.txt files on their documentation sites:

1. Anthropic: https://docs.anthropic.com/llms-full.txt
2. Hugging Face: https://huggingface-projects-docs-llms-txt.hf.space/accelerate/llms.txt
3. Perplexity: https://docs.perplexity.ai/llms-full.txt
4. Zapier: https://docs.zapier.com/llms-full.txt

Current Concerns

Exposure of content to competitors for analysis Technically, a file like this of all your content laid out in markdown format, especially in full, would make it a lot easier for competitors to get a full export of all your content. This has always been possible, but not to the level it would be with a dedicated markdown file.

Potential security risks that require careful management

• If your website uses server-side includes or similar technology, these might be inadvertently included in the llms.txt file, potentially exposing implementation details.
• The file might contain metadata about your content that wasn't intended for public viewing.
• This could include author information, unpublished dates, internal categorization, etc.
• This might provide insights into your site architecture that could be useful for potential attackers.
• Any user data accidentally included in public-facing content would be concentrated in the llms.txt file. This could consist of comments, user contributions, or other data.

Mixed Reception

Some experts remain sceptical about the utility of llms.txt. Brett Tabke, CEO of Pubcon and WebmasterWorld, has argued that existing tools like XML sitemaps and robots.txt already serve similar purposes and that the distinction between search engines and LLMs is increasingly blurred. However, in my opinion, implementing it has no harm unless the costs outweigh the potential gain.