New Docusaurus plugin: `docusaurus-plugin-llms-txt`

[jharrell]

Have you read the Contributing Guidelines on issues?

• I have read the Contributing Guidelines on issues.

Description

Hello!

I'd like to propose the creation of a new docusaurus plugin for the creation of a llms.txt file.

As background, an llms.txt file is a way of co-locating needed information such that an LLM processing web pages is able to do so effectively and quickly. More information can be found on the llms.txt proposal website

The goal of this feature is to create a plugin that will generate this file for your docs.

Has this been requested on Canny?

No

Motivation

Having an llms.txt is a business need for our company, but I'm certain it would be helpful for others. I have already created a proof-of-concept plugin that writes the file, but I wanted to bring this to the community as a whole to see how the design could be improved and/or match others' use cases.

Additionally, many docs sites that I use and admire, such as dub.co and Cloudflare have already implemented llms.txt files. The former via Mintlify, the latter via a script.

Adding a new, optional plugin would give Docusaurus users the ability to opt-in to this new standard and allow their documentation to be accessed easily by LLM agents.

API design

The design is very similar to the sitemap plugin.

Configuration

• ignorePatterns: ability to ignore specific paths from being included in the llms.txt.
• filename: ability to change the name and output path of the llms.txt file.

I also considered configuration modifying the output of the file, but the llms.txt proposal is fairly set on the file structure so I declined to add that as a configuration option.

Have you tried building it?

Yes! I made a simple plugin that works for our use case. I would like to move it into this repo in order to make it available to other folks and to get feedback on the design. I feel it could be made much simpler and cleaner 😄

Self-service

• I'd be willing to contribute this feature to Docusaurus myself.

[slorber]

We usually add to the Docusaurus core repo what we use on our own website. In this case it's not really a need we have, and adding this to core would put the burden on us to maintain over the long term. Many people want to add their plugin to core but we can't do that otherwise we'd end up with too many packages to maintain. I'd prefer to keep a lean core and keep this plugin in the community.

---

Also, I'm really not sure what this plugin adds to the regular sitemap.xml file.

I've checked your POC (prisma/docs#6645), and as far as I understand it basically creates a list of links in Markdown format, but using a .txt extension instead of Markdown (why not use llmt.md? 🤷‍♂ )

Result: https://jharrell-llms-plugin.docs-51g.pages.dev/llms.txt

If we look at the Cloudflare example, it's also a list of links in Markdown format.
https://developers.cloudflare.com/llms.txt

I'm not sure how the h2 headings and occasional labels bring a lot of value to a regular sitemap.

Also your site recommends to link to html.md files, which you don't do here so the LLM would have to read an HTML page anyway. And we use MDX which means that this format is already less "LLM-friendly" that regular Markdown files which cannot contain React components that must be evaluated.

This site is linking to Markdown files, for example: https://docs.fastht.ml/llms.txt

---

This field is not my expertise but I don't find this proposal super convincing, it looks more like an early draft not covering edge cases than a spec. I don't think it should be added to Docusaurus core until the proposal gets more mainstream adoption and gets better documented. I'd also prefer if there was first a community implementation that satisfies early adopters. We can consider adding this to our repo later, but not as a first step. Until then I'm happy to support you developing a community plugin, so let me know if you are unable to achieve your needs.

I also wonder why it has to be a Docusaurus plugin. You could build a generic CLI tool that handles a full generic static deployment, looking for sitemap.xml or HTML files and creating LLM-friendly content.

[jharrell]

Thank you @slorber for the context. I don't disagree with your assessment, so I'll be closing this proposal.

For further information about why this over a sitemap etc there's this section of the proposed standard: https://llmstxt.org/#existing-standards

Thank you for the feedback, though. I'll continue to use my simplified solution.

[DenhamPreen]

@jharrell could you share your "simplified solution", this does sound useful :)

[jharrell]

@DenhamPreen sure. We're still iterating but here's the latest: https://github.com/prisma/docs/blob/22208d52e4168028dbbe8b020b10682e6b526e50/docusaurus.config.ts#L95

It generates an llms-full.txt and llms.txt.

Happy to chat further about it if I can help, my bsky account is in my GitHub profile 😊