Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
WalkthroughExpands Docusaurus config into a multi-doc, multi-plugin setup: adds broad redirects, multiple content-docs instances (including LLM variants), a new plugin to generate LLMS artifacts, new dependencies, and formatting fixes to a HyperIndex guide. Exports remain Changes
Sequence Diagram(s)sequenceDiagram
autonumber
participant Build as Docusaurus Build
participant Docs as plugin-content-docs (multiple)
participant Gen as plugin-generate-llms
participant FS as File System
Build->>Docs: Initialize docs instances (paths, routeBasePath, sidebars)
Build->>Gen: Invoke postBuild(siteConfig)
note right of Gen #ADD8E6: Gen reads options.filesConfigs and redirectsList
Gen->>Docs: Collect metadata (slug, title, description, routeBasePath)
Gen->>Gen: Normalize paths (to POSIX) and apply includeOrder (minimatch)
rect rgba(200,230,255,0.25)
Gen->>FS: Write llms.txt and llms-<name>.txt
alt main config
Gen->>FS: Write stripped `.md` copies for ordered docs
end
end
Gen-->>Build: LLMS artifacts created
Build-->>FS: Finalize build output (assets, redirects)
Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes Possibly related PRs
Suggested reviewers
Poem
Pre-merge checks and finishing touches✅ Passed checks (3 passed)
✨ Finishing touches
🧪 Generate unit tests (beta)
📜 Recent review detailsConfiguration used: CodeRabbit UI Review profile: CHILL Plan: Pro ⛔ Files ignored due to path filters (1)
📒 Files selected for processing (2)
✅ Files skipped from review due to trivial changes (1)
🧰 Additional context used🧬 Code graph analysis (1)docusaurus.config.js (2)
🪛 Gitleaks (8.28.0)docusaurus.config.js[high] 349-349: Detected a Generic API Key, potentially exposing access to various services and sensitive operations. (generic-api-key) 🔇 Additional comments (1)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
the repo doesn’t have a Prettier config, so my local formatter ended up reformatting docusaurus.config.js. That’s why the diff looks bigger than it really is. |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
⛔ Files ignored due to path filters (1)
yarn.lockis excluded by!**/yarn.lock,!**/*.lock
📒 Files selected for processing (5)
docs/HyperSync/HyperRPC/hyperrpc-supported-networks.md(0 hunks)docs/HyperSync/hypersync-supported-networks.md(0 hunks)docusaurus.config.js(1 hunks)package.json(1 hunks)supported-networks.json(1 hunks)
💤 Files with no reviewable changes (2)
- docs/HyperSync/HyperRPC/hyperrpc-supported-networks.md
- docs/HyperSync/hypersync-supported-networks.md
🧰 Additional context used
🧬 Code graph analysis (1)
docusaurus.config.js (1)
docusaurus.config.llm.js (4)
config(6-159)require(1-1)lightCodeTheme(2-2)darkCodeTheme(3-3)
🪛 Gitleaks (8.28.0)
docusaurus.config.js
[high] 344-344: Detected a Generic API Key, potentially exposing access to various services and sensitive operations.
(generic-api-key)
🔇 Additional comments (1)
package.json (1)
44-46: docusaurus-plugin-llms@^0.2.2 is correct and Docusaurus v3–compatible. peerDependencies include@docusaurus/core@^3.0.0; no changes needed.
|
Can you explain where to access this.txt file and how I can check it's working on the Vercel preview link? Also, I know we talk to the other LLM documentations a lot, so could just be worth in future thinking about having multiple LLM documentations. I'm not sure if it would be confusing for users. |
|
@moose-code you can access it Similar to Google crawlers, LLM crawlers need data in one place. For search engine crawlers it was sitemap.xml and for llm crawlers it is llms.txt |
There was a problem hiding this comment.
Actionable comments posted: 2
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
package.json (1)
28-47: Declare theglobdependency explicitly.
plugins/plugin-generate-llms.jsrequiresglob, but it isn’t listed in package.json. In clean installs (or Yarn PnP), the build will crash with “Cannot find module 'glob'”. Add it as a direct dependency so the plugin remains portable.Apply this diff:
"dependencies": { "@cookbookdev/docsbot": "^4.21.23", @@ - "docusaurus-json-schema-plugin": "^1.12.2", + "docusaurus-json-schema-plugin": "^1.12.2", + "glob": "^10.3.10", "minimatch": "^10.0.3",
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
⛔ Files ignored due to path filters (1)
yarn.lockis excluded by!**/yarn.lock,!**/*.lock
📒 Files selected for processing (3)
docusaurus.config.js(1 hunks)package.json(1 hunks)plugins/plugin-generate-llms.js(1 hunks)
🧰 Additional context used
🧬 Code graph analysis (2)
plugins/plugin-generate-llms.js (1)
docusaurus.config.js (2)
require(1-1)config(233-565)
docusaurus.config.js (2)
docusaurus.config.llm.js (4)
config(6-159)require(1-1)lightCodeTheme(2-2)darkCodeTheme(3-3)plugins/plugin-generate-llms.js (1)
require(5-5)
🪛 Gitleaks (8.28.0)
docusaurus.config.js
[high] 344-344: Detected a Generic API Key, potentially exposing access to various services and sensitive operations.
(generic-api-key)
| for (const file of allFiles) { | ||
| const fullPath = path.join(docsPath, file); | ||
| const raw = fs.readFileSync(fullPath, "utf-8"); | ||
| const parsed = matter(raw); | ||
|
|
||
| const slug = parsed.data.slug; | ||
| const title = parsed.data.title; | ||
| const description = parsed.data.description || ""; | ||
|
|
||
| if (!slug || !title) continue; | ||
|
|
||
| const pageUrl = `${url.replace( | ||
| /\/$/, | ||
| "" | ||
| )}/${routeBasePath.replace(/^\//, "")}/${slug.replace( | ||
| /^\//, | ||
| "" | ||
| )}`; | ||
|
|
||
| collectedDocs.push({ | ||
| filePath: path.join(config.path, file), | ||
| title, | ||
| description, | ||
| pageUrl, | ||
| }); | ||
| } |
There was a problem hiding this comment.
Don’t drop docs without an explicit slug.
Most Docusaurus docs rely on the default slug derived from the file path. Because this loop skips any doc lacking frontmatter.slug, large portions of the docs (including plenty of HyperIndex/HyperSync pages) will vanish from llms.txt, breaking the build’s primary goal. Derive a fallback slug from the file path (matching Docusaurus’ default) instead of skipping those entries.
Apply this diff:
- const slug = parsed.data.slug;
+ const slug =
+ parsed.data.slug ??
+ `/${file
+ .replace(/(\/)?index\.(mdx?|md)$/i, "$1")
+ .replace(/\.(mdx?|md)$/i, "")}`;📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| for (const file of allFiles) { | |
| const fullPath = path.join(docsPath, file); | |
| const raw = fs.readFileSync(fullPath, "utf-8"); | |
| const parsed = matter(raw); | |
| const slug = parsed.data.slug; | |
| const title = parsed.data.title; | |
| const description = parsed.data.description || ""; | |
| if (!slug || !title) continue; | |
| const pageUrl = `${url.replace( | |
| /\/$/, | |
| "" | |
| )}/${routeBasePath.replace(/^\//, "")}/${slug.replace( | |
| /^\//, | |
| "" | |
| )}`; | |
| collectedDocs.push({ | |
| filePath: path.join(config.path, file), | |
| title, | |
| description, | |
| pageUrl, | |
| }); | |
| } | |
| for (const file of allFiles) { | |
| const fullPath = path.join(docsPath, file); | |
| const raw = fs.readFileSync(fullPath, "utf-8"); | |
| const parsed = matter(raw); | |
| const slug = | |
| parsed.data.slug ?? | |
| `/${file | |
| .replace(/(\/)?index\.(mdx?|md)$/i, "$1") | |
| .replace(/\.(mdx?|md)$/i, "")}`; | |
| const title = parsed.data.title; | |
| const description = parsed.data.description || ""; | |
| if (!slug || !title) continue; | |
| const pageUrl = `${url.replace( | |
| /\/$/, | |
| "" | |
| )}/${routeBasePath.replace(/^\//, "")}/${slug.replace( | |
| /^\//, | |
| "" | |
| )}`; | |
| collectedDocs.push({ | |
| filePath: path.join(config.path, file), | |
| title, | |
| description, | |
| pageUrl, | |
| }); | |
| } |
🤖 Prompt for AI Agents
In plugins/plugin-generate-llms.js around lines 88 to 113, the loop currently
skips docs that lack frontmatter.slug which drops files that rely on Docusaurus'
default slug; instead, compute a fallback slug from the file path (derive it by
taking the file path relative to the docs directory, remove the file extension,
normalize path separators, and strip any leading/trailing slashes) and use that
when parsed.data.slug is missing, ensuring the resulting slug has no leading
slash before constructing pageUrl and push the entry to collectedDocs as before.
|
@moose-code last plugin used was not useful due to it using folder structure for links. modified it to use our url structure and follow llmstxt.org standards. When possible can you review it? |
|
Sorry @nikbhintade but some merge conflicts again here if you can resolve 🙏 |
|
@moose-code resolved them |
2 participants
Summary
Added support for generating
llms.txtfiles for the docs during build process.Changes
docusaurus-plugin-llmsplugindocusaurus.config.jsSummary by CodeRabbit
New Features
Improvements
Bug Fixes
Chores
Documentation