Frequently Asked Questions
Everything you need to know about LLMs.txt, our generator, and the implementation hub
General #
What is LLMs.txt?
LLMs.txt is a proposed web standard file that provides concise, LLM-friendly information about a website. Similar to robots.txt, it's placed at the root of a website (e.g., example.com/llms.txt) to help language models better understand and interact with the site's content.
What does this platform offer?
Our platform provides two main services: 1) A Generator tool that helps you create and validate LLMs.txt files for your website, and 2) A Hub that showcases websites that have implemented LLMs.txt, making it easy to discover and learn from real-world examples.
Why do we need LLMs.txt?
Large language models often struggle with processing entire websites due to context window limitations and complex HTML structures. LLMs.txt provides a curated, simplified version of essential information that helps LLMs better understand and utilize website content during inference time.
How is LLMs.txt different from robots.txt or sitemap.xml?
While robots.txt controls bot access and sitemap.xml lists all pages, LLMs.txt serves a different purpose. It provides curated, LLM-optimized content summaries and context that fits within model context windows. Unlike sitemaps, it can include external references and focuses on what's necessary for understanding rather than comprehensive indexing.
Using the Generator #
How do I generate an LLMs.txt file for my website?
Our generator tool makes it easy to create an LLMs.txt file. Simply enter your website's URL or key information, and the AI-powered generator will create a properly formatted LLMs.txt file following the protocol specification.
What information should I include in my LLMs.txt file?
Your LLMs.txt file should include essential information about your website such as its purpose, key features, important URLs, API documentation references, and any other content that would help LLMs better understand and interact with your site. The generator will help you identify and format this information appropriately.
Can I customize the generated LLMs.txt file?
Yes! While our generator creates a well-structured initial file, you can always edit and customize the content to better match your website's needs. The generator ensures your customizations still follow the protocol specification.
The Hub #
What is the LLMs.txt Hub?
The Hub is a curated collection of websites that have implemented LLMs.txt files. It serves as a discovery platform where you can explore real-world examples, learn from different implementations, and understand how various sites are utilizing the standard.
How can I add my website to the Hub?
Once you've implemented an LLMs.txt file on your website, you can submit it to our Hub. We'll validate your implementation against the protocol specification and add it to our directory, making it discoverable by others.
What can I learn from the Hub?
The Hub provides valuable insights into how different websites implement LLMs.txt. You can explore various implementation patterns, see how different types of websites structure their LLM-friendly content, and learn best practices from real-world examples.
How do you ensure quality in the Hub?
All websites in the Hub go through a validation process to ensure their LLMs.txt files follow the protocol specification. We regularly check for updates and maintain the quality of listings to provide reliable examples for the community.
Technical Details #
Where should I place the LLMs.txt file?
The LLMs.txt file should be placed at the root of your website (e.g., https://example.com/llms.txt), similar to robots.txt. This standardized location makes it easy for LLMs to find and access the file.
What format should the LLMs.txt file follow?
LLMs.txt files use Markdown formatting that's both human-readable and LLM-friendly. The file must start with an H1 header containing your project name, followed by a blockquote summary. Additional sections can include regular Markdown content and H2-delimited file lists with hyperlinked URLs. Our generator automatically ensures your file follows this structured Markdown format defined in the protocol specification at llmstxt.org.
Is LLMs.txt required for my website?
No, LLMs.txt is optional. However, providing one can help LLMs better understand and interact with your website's content, potentially improving the accuracy and relevance of AI-powered tools and services that interact with your site.
Best Practices #
How often should I update my LLMs.txt file?
It's recommended to update your LLMs.txt file whenever significant changes are made to your website's content, structure, or purpose. Regular reviews ensure the information remains accurate and useful for LLMs.
What are common mistakes to avoid?
Common mistakes include providing too much information (exceeding typical LLM context windows), not keeping the file updated, including sensitive information, or using overly complex formatting. Our generator helps you avoid these issues.
How can I verify my LLMs.txt file is working correctly?
You can use our Hub to validate your LLMs.txt file against the protocol specification. Additionally, you can test how well LLMs understand your site's content through various AI-powered tools that support the LLMs.txt standard.