Docusaurus leaks memory on i18n site builds

[slorber]

When building a Docusaurus site with multiple locales, the memory keeps increasing.

What happens, in pseudo-code:

async function buildSite() {
  await buildLocale("en"); // +20mb
  await buildLocale("fr"); // +20mb
  await buildLocale("ja"); // +20mb
  await buildLocale("es"); // +20mb
}

This can be seen on our website, but also our init template (much smaller leak though).

In #10599 we solved an important leak, but we still leak memory and should investigate so that it's possible to build a Docusaurus site in thousands of locales without having to increase the heap size.

---

Repro

yarn install

yarn clear:website

NODE_OPTIONS="--max-old-space-size=250 --expose-gc" DOCUSAURUS_PERF_LOGGER=true yarn build:website:fast --locale en --locale fr --locale ja --locale es

The logs show memory increasing after each locale despite calling globalThis.gc?.() before measures.

[PERF] Build > en - 20.90 seconds! - (58mb -> 144mb)
[PERF] Build > fr - 30.94 seconds! - (144mb -> 162mb)
[PERF] Build > ja - 32.49 seconds! - (162mb -> 180mb)
[PERF] Build > es - 33.00 seconds! - (180mb -> 197mb)
[PERF] Build - 117.36 seconds! - (58mb -> 197mb)

---

Heap dumps

Heap dumps can be taken after each locale build with little edits in packages/docusaurus/src/commands/build/build.ts

async function runBuildLocaleTask(params: BuildLocaleParams) {
  await buildLocale(params);

  globalThis.gc?.();
  require('v8').writeHeapSnapshot(
    `docusaurus-heap-${Date.now()}-${params.locale}.heapsnapshot`,
  );
}

Access upon request: https://drive.google.com/drive/folders/11JrI_sgw_uwtGBAza52AEyVpiH33Qyq8?usp=sharing

---

Edit: from this discussion, we can assume that the memory leak grows with the number of MDX docs:
#9211 (reply in thread)

---

Edit: these articles presents interesting ways to run memory leak tests:

• https://joyeecheung.github.io/blog/2024/03/17/memory-leak-testing-v8-node-js-1/
• https://joyeecheung.github.io/blog/2024/03/17/memory-leak-testing-v8-node-js-2/
• https://joyeecheung.github.io/blog/2024/03/17/memory-leak-testing-v8-node-js-3/

[slorber]

Some investigations I did recently:

---

gray-matter cache

While investigating build times on a large site (discussion), I noticed that the legacy lib we use to parse front matter has a built-in cache:

https://github.com/jonschlinkert/gray-matter/blob/master/index.js#L225

For that large 11k docs site, it looks like the cache retains approximately 200mb of memory, which could become problematic for i18n sites.

We should probably call require("gray-matter").clearCache() somewhere in our process.

Note: we can disabe the gray-matter cache by providing an empty option {}, but from my benchmarks, this decreases performance a bit and it's faster with the cache (AB=false):

Benchmark 1: DOCUSAURUS_AB_BENCHMARK=true yarn build:website:fast
  Time (mean ± σ):     11.094 s ±  0.462 s    [User: 30.461 s, System: 5.889 s]
  Range (min … max):   10.593 s … 11.503 s    3 runs
 
Benchmark 2: DOCUSAURUS_AB_BENCHMARK=false yarn build:website:fast
  Time (mean ± σ):     11.026 s ±  0.643 s    [User: 30.620 s, System: 5.993 s]
  Range (min … max):   10.337 s … 11.608 s    3 runs
 
Summary
  DOCUSAURUS_AB_BENCHMARK=false yarn build:website:fast ran
    1.01 ± 0.07 times faster than DOCUSAURUS_AB_BENCHMARK=true yarn build:website:fast

This makes sense because versioned docs sites have many docs that have the exact same front matter input. We want to keep the cache, but ideally we should implement our own on top of gray-matter using WeakMap instead of {}.

---

MDX processor cache

Similarly, we have a cache of MDX processors that we never clear:

const ProcessorsCache = new Map<string | Options, SimpleProcessors>();

We should probably clear it after building each locale, but I'm not sure it's what's causing the most important leaks we have. However this might affect some third-party plugins because things like remark plugins can be stateful.

[slorber]

Memory leaks increase with Rspack

On a newly initialized Docusaurus v3.7 site with Docusaurus faster, resolving to the latest Rspack (1.3.9 currently)

const config = {
  future: {
    experimental_faster: {
      mdxCrossCompilerCache: false,
      rspackBundler: true, // Flag to toggle
      swcJsLoader: true,
      swcJsMinimizer: true,
      lightningCssMinimizer: true,
      swcHtmlMinimizer: true,
    },
  },
}

When building with:

yarn clear && NODE_OPTIONS="--max-old-space-size=400 --expose-gc" DOCUSAURUS_PERF_LOGGER=true yarn build --locale en --locale fr --locale ja --locale es

When Webpack, we have a high memory spike while bundling, but the memory leaks are tiny:

[PERF] Build > en - 10.97 seconds! - (50mb -> 136mb)
[PERF] Build > fr - 11.62 seconds! - (136mb -> 136mb)
[PERF] Build > ja - 10.96 seconds! - (136mb -> 138mb)
[PERF] Build > es - 11.50 seconds! - (138mb -> 139mb)

When Rspack is enabled, the memory leaks become larger:

[PERF] Build > en - 3.22 seconds! - (60mb -> 147mb)
[PERF] Build > fr - 3.49 seconds! - (147mb -> 188mb)
[PERF] Build > ja - 3.88 seconds! - (188mb -> 227mb)
[PERF] Build > es - 7.77 seconds! - (227mb -> 265mb)

---

When compiling the very same app (English) twice in a row, we see a memory increase.

From comparing heap dumps before/after the additional compilation, we see a delta causing by strings used by Rspack NormalModules.

I would expect that after closing the Rspack compiler, all its things such as NormalModule can be garbage collected, but that doesn't look to be the case. I think the Rspack Compiler and everything it references stay in memory. Not sure why and if it's related to Rspack or Docusaurus.

(note: I tested that on a very minimal Docusaurus site, with only a page plugin, and without persistent cache or Rspack incremental mode)

---

This retained memory seems to be related to Rspack COMPILATION_WEAK_MAP

https://github.com/web-infra-dev/rspack/blob/v1.3.9/packages/rspack/src/Compiler.ts#L80

Unfortunately, I'm not sure my explicit globalThis.gc?.(); calls clears it properly, and V8 may choose to retain the memory. But maybe it would clean under more memory pressure 🤷‍♂

Anyway, disabling this feature manually by tweaking node modules, we can see that Rpsack doesn't produce a memory delta anymore on new compilation, even though it seems the last compiler remains in memory.

[guillaume-kotulski]

We are currently unable to upgrade from Docusaurus 3.6.3 due to this issue.
On our website each locale compilation with rspack takes around 300mb more. We have around 6500 files per language.

[slorber]

@guillaume-kotulski, you should be able to run multiple times the docusaurus build --locale <my-locale> in a row to mitigate this issue (it might require some little config adjustments)

This only happens if you use docusaurus build, which just loops over all your locales to build them sequentially.