Very long build times with large docusaurus site #11140
Replies: 2 comments 15 replies
-
|
We have also discussed splitting the site out into multiple sites but wanted to avoid this as possible to keep the sites in sync with packages and styling. We also had the thought about caching sections of our docs. If this is possible, we could cache versions of our API docs that rarely change so that they don't have to be rebuilt which may be a way to reduce build times as well. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for the report, it's great for me because it gives me a relatively large (but IMHOP still an acceptable size we should support well) to benchmark against. I'll investigate later but IMHO the upcoming v3.8 release should improve things. I tried running your site on v3.0 (current version) locally on MacOS M3, and encountered this Was able to fix it by turning these features off: docs: {
showLastUpdateAuthor: false,
showLastUpdateTime: false,
}Running your build with the
By chance, the SSG should greatly improve thanks to v3.8 SSG worker threads (#10826), and building should be faster on second runs (in both start/build) thanks to Rspack persistent cache (#10931) DOCUSAURUS_PERF_LOGGER=true npm run build
> sailpoint-developer-portal@0.0.0 build
> docusaurus build
[PERF] Get locales to build - 144.37 ms - (56mb -> 75mb)
[INFO] [en] Creating an optimized production build...
[PERF] Build > en > Load site > Load context - 10.03 ms - (75mb -> 72mb)
[PERF] Build > en > Load site > Load plugins > Init plugins - 305.14 ms - (72mb -> 115mb)
[PERF] Build > en > Load site > Load plugins > Load plugins content > Load classic@default > translatePluginContent() - 5.88 ms - (116mb -> 117mb)
[PERF] Build > en > Load site > Load plugins > Load plugins content > Load google-tag-manager@default - 6.31 ms - (115mb -> 117mb)
[PERF] Build > en > Load site > Load plugins > Load plugins content > Load pages@default > loadContent() - 10.57 ms - (115mb -> 119mb)
[PERF] Build > en > Load site > Load plugins > Load plugins content > Load blog@default > translatePluginContent() - 10.12 ms - (116mb -> 121mb)
[PERF] Build > en > Load site > Load plugins > Load plugins content > Load search-algolia@default - 15.14 ms - (115mb -> 110mb)
[PERF] Build > en > Load site > Load plugins > Load plugins content > Load classic@default > getDefaultCodeTranslationMessages() - 9.83 ms - (117mb -> 110mb)
[PERF] Build > en > Load site > Load plugins > Load plugins content > Load classic@default - 16.15 ms - (115mb -> 110mb)
[PERF] Build > en > Load site > Load plugins > Load plugins content > Load pages@default - 17.18 ms - (115mb -> 111mb)
[PERF] Build > en > Load site > Load plugins > Load plugins content > Load blog@default > contentLoaded() - 270.54 ms - (111mb -> 211mb)
[PERF] Build > en > Load site > Load plugins > Load plugins content > Load blog@default - 288.22 ms - (115mb -> 211mb)
[PERF] Build > en > Load site > Load plugins > Load plugins content > Load docs@default > loadContent() - 2.57 seconds! - (115mb -> 331mb)
[PERF] Build > en > Load site > Load plugins > Load plugins content > Load docs@default > translatePluginContent() - 21.00 ms - (331mb -> 329mb)
[PERF] Build > en > Load site > Load plugins > Load plugins content > Load docs@default > contentLoaded() - 1.32 seconds! - (329mb -> 459mb)
[PERF] Build > en > Load site > Load plugins > Load plugins content > Load docs@default - 3.92 seconds! - (115mb -> 459mb)
[PERF] Build > en > Load site > Load plugins > Load plugins content - 3.92 seconds! - (115mb -> 459mb)
[PERF] Build > en > Load site > Load plugins - 4.23 seconds! - (72mb -> 460mb)
[PERF] Build > en > Load site > Create site files - 515.15 ms - (469mb -> 462mb)
[PERF] Build > en > Load site - 4.76 seconds! - (75mb -> 462mb)
[PERF] Build > en > Creating rspack bundler configs - 33.98 ms - (462mb -> 471mb)
● Client ██████████████████████████████████████████████████ (100%) emitting after emit
● Server ██████████████████████████████████████████████████ (100%) emitting after emit [PERF] Build > en > Bundling with rspack - 762.87 seconds! - (471mb -> 491mb)
[PERF] Build > en > SSG > Read client manifest - 25.67 ms - (491mb -> 503mb)
[PERF] Build > en > SSG > Generate static files > Load HTML minifier - 7.22 ms - (511mb -> 511mb)
[PERF] Build > en > SSG > Generate static files > Load App renderer > Load server bundle - 30.07 ms - (511mb -> 508mb)
[PERF] Build > en > SSG > Generate static files > Load App renderer > Server bundle size = 16.125 MB
[PERF] Build > en > SSG > Generate static files > Load App renderer > Evaluate server bundle - 326.01 ms - (508mb -> 608mb)
[PERF] Build > en > SSG > Generate static files > Load App renderer - 356.47 ms - (511mb -> 608mb)
[WARNING] Docusaurus static site generation process emitted warnings for 4 paths
This is non-critical and can be disabled with DOCUSAURUS_IGNORE_SSG_WARNINGS=true
Troubleshooting guide: https://github.com/facebook/docusaurus/discussions/10580
- "/docs/tools/sdk/go":
- [HTML minifier diagnostic - error] No "p" element in scope but a "p" end tag seen - {"primary_spans":[{"end":37289,"start":37285}],"span_labels":[]}
- "/docs/tools/sdk/powershell":
- [HTML minifier diagnostic - error] No "p" element in scope but a "p" end tag seen - {"primary_spans":[{"end":39548,"start":39544}],"span_labels":[]}
- "/docs/tools/sdk/python":
- [HTML minifier diagnostic - error] No "p" element in scope but a "p" end tag seen - {"primary_spans":[{"end":36986,"start":36982}],"span_labels":[]}
- "/docs/tools/sdk/typescript":
- [HTML minifier diagnostic - error] No "p" element in scope but a "p" end tag seen - {"primary_spans":[{"end":50030,"start":50026}],"span_labels":[]}
[PERF] Build > en > SSG > Generate static files - 1632.71 seconds! - (511mb -> 13267mb)
[PERF] Build > en > SSG - 1632.78 seconds! - (491mb -> 13267mb)
[WARNING] Will NOT delete server bundle because DOCUSAURUS_KEEP_SERVER_BUNDLE is set to 'true'
[PERF] Build > en > postBuild() - 2.89 seconds! - (13267mb -> 13489mb)
[PERF] Build > en > Broken links checker - 14.22 seconds! - (13489mb -> 13773mb)
[SUCCESS] Generated static files in "build".
[PERF] Build > en - 2417.55 seconds! - (75mb -> 13773mb)
[PERF] Build - 2417.55 seconds! - (75mb -> 13773mb)
[INFO] Use `npm run serve` command to test your build locally. |
Beta Was this translation helpful? Give feedback.
-
|
Good news is I can build your site localle (Mac M3 pro) in:
Bad news:
Mild news:
The Rspack team is likely to improve these 2 things in the future |
Beta Was this translation helpful? Give feedback.
-
|
Hey @slorber this is awesome I tried pulling in these changes to test and when I add the following in our plugins.ts and try and generate our open-api docs, it says it can't find the plugin-ids anymore. Any idea what could cause that? Steps to reproduce.
function disableExpensiveBundlerOptimizationPlugin() {
return {
name: 'disable-expensive-bundler-optimizations',
configureWebpack() {
return {
optimization: {
concatenateModules: false,
mergeDuplicateChunks: false,
removeAvailableModules: true,
},
experiments: {
parallelCodeSplitting: false,
} as any,
};
},
};
},I can build the site with old generated api files and see that it builds in ~20 minutes on my M1 mac |
Beta Was this translation helpful? Give feedback.
-
|
hmm I don't know, this is weird, how is this different from my PR? I'd need to see the code to understand what is going on Docusaurus v3.8 will be released soon, I'd like to ship just one thing before, so we'll upgrade the PRs soon to a stable version |
Beta Was this translation helpful? Give feedback.
-
|
It should be near identical to your PR, I even copied the plugins.ts file from your branch. Do you generate the api docs with That is the step that fails when the function is present. |
Beta Was this translation helpful? Give feedback.
-
|
Hmmm I probably did not regenerate the docs after adding that extra plugin |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
Hello everyone,
My post is very similar to #10895, we have a large site, that displays mostly documentation, and API specifications using https://github.com/PaloAltoNetworks/docusaurus-openapi-docs/.
We use the Docusaurus faster plugin with all options set to true. The site does build, on a Linux machine (16 cores & 64GB of RAM) after about an hour and 15 minutes.
We have approximately 11k markdown files, which didn't seem like a ton compared to other sites we've seen. We have some
.mdsome.mdxI saw one post that mentioned that.mdxfiles can take longer to build. We have a couple custom components and things we've created over the years and I am wondering if we are at the limit of what we can do with Docusaurus. Which is sad for us because we LOVE Docusaurus.Our site is open source here: https://github.com/sailpoint-oss/developer.sailpoint.com, and I would love some feedback about our configuration. I am hoping we have something misconfigured or there are optimizations that we can do to speed up the build.
When running
npm run buildthe site appears to build but then crashes with out of memory on my local windows and mac machines.I am able to run
npm run startand it builds in a minute or two locally then I can edit what I need but the deploy times once we merge PRs in take a long time.@slorber I see you have done a lot of work on performance as of late. I would love your feedback as well.
Beta Was this translation helpful? Give feedback.
All reactions