Skip to content

Conversation

@SgtPooki
Copy link

@SgtPooki SgtPooki commented Jun 16, 2025

Fixes #34
Built on top of #35 -- I couldn't get project to compile without this

You can run the test with npm run unittest -- -g 'does not run out of memory'

Note that there is also a lot of memory being consumed by fsExtra in CodeBlockExtractor when reading an entire file into memory. We could speed things up and use much less memory if we used some sort of readline approach to reading the markdown code blocks, and emitted them with an async iterator as they pop up.

I went ahead with the CodeBlockExtractor change.. it speeds things up a bit and should allow for much larger files to be read a little more safely, but can remove from this PR if desired. check the stats from hyperfine run below.

NOTE that in the below hyperfine tests, the memory test removed so test speeds are a little more accurate.

> hyperfine --parameter-list branch fix/dont-use-chai-object-prototype-modifications,test/show-OOM-error --setup "git switch {branch} && npm run build" "npm run unittest -- -g 'TypeScriptDocsVerifier'" --runs 50
Benchmark 1: npm run unittest -- -g 'TypeScriptDocsVerifier' (branch = fix/dont-use-chai-object-prototype-modifications)
  Time (mean ± σ):     13.722 s ±  0.228 s    [User: 20.098 s, System: 2.675 s]
  Range (min … max):   13.451 s … 14.373 s    50 runs
 
Benchmark 2: npm run unittest -- -g 'TypeScriptDocsVerifier' (branch = test/show-OOM-error)
  Time (mean ± σ):     10.674 s ±  0.213 s    [User: 14.952 s, System: 1.856 s]
  Range (min … max):   10.411 s … 11.318 s    50 runs
 
Summary
  npm run unittest -- -g 'TypeScriptDocsVerifier' (branch = test/show-OOM-error) ran
    1.29 ± 0.03 times faster than npm run unittest -- -g 'TypeScriptDocsVerifier' (branch = fix/dont-use-chai-object-prototype-m [20m22.831s]

How I changed the solution to #30 (previously fixed by #33):

Wrap all functions that should be able to be wrapped in an IIFE without error in a named IIFE, and to ensure line number accuracy for errors, recognize that if the code block is wrapped.

It's possible that someone may want to completely prevent this behavior for some reason, so I would propose a <!-- ts-docs-verifier:no-wrap --> or // ts-docs-verifier:no-wrap pragma in order to allow for escaping this new behavior if desired.

@SgtPooki
Copy link
Author

Will update and remove changes from #35

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bug: Out-of-Memory error when parsing many snippets with compileSnippets

1 participant