Looking for feedback: Pointer compression in Node.js

[addaleax]

The V8 JavaScript engine, which is used by Node.js and Google Chrome, has started to provide a feature called “pointer compression”. The following aims to explain the impact of this feature on Node.js, and the questions and challenges around implementing it which the Node.js collaborator group is seeking feedback from the community on.

What is pointer compression?

On 64-bit hardware, which includes most modern end-user devices, references such as those from one JavaScript object to another take up 8 bytes each. A large percentage (about 70 %) of the memory used by a JavaScript application consists of such references.

Pointer compression is a way to reduce the size of these references to 4 bytes each. This reduces memory usage significantly and improves performance, at the cost of limiting the size of the JavaScript memory (the “heap”) to 4 GB, which is equivalent to about 6–8 GB of uncompressed memory.

Future changes could increase this limit to e.g. 8, 16, or 32 GB at some performance cost. This is not being worked on in V8 at this point.

Note that currently, the V8 engine limits the heap size to 4 GB as well unless this is explicitly overridden through command line flags (--max-old-space-size).

ArrayBuffers and some JavaScript strings are stored separately and do not count towards this limit.

For a more in-depth explanation, watch Benedikt Meurer’s NodeConf EU talk “What’s happening in V8? – Benedikt Meurer”:

How does pointer compression affect Node.js?

Unlike Chrome, Node.js currently does not enforce a hard limit on the size of the JavaScript heap, and so an application can generally use much more than 4 GB of memory on a 64-bit platform if configured to do so.

However, the memory footprint and performance improvements brought by pointer compression have a high potential of benefiting many Node.js users, and the need for large heap sizes is partially reduced through the memory footprint improvements which pointer compression yields.

Currently, enabling pointer compression breaks native addon compatibility for Node.js applications (unless N-API is being used), but this is likely going to change in the near future, making builds of Node.js with and without pointer compression almost fully interchangeable.

What questions do we need to answer?

Do we provide two different modes for Node.js?

Do we officially support and test Node.js builds both with pointer compression enabled and without? How would a higher heap limit at some performance cost affect the answer to this question?

Supporting two different build configurations allows Node.js users to make a choice that matches their needs best, although it is unclear what a typical maximum heap size would be and how important it is to provide Node.js builds supporting unusually large heap sizes.

Additionally, providing two different modes means that the non-default one will receive less usage and thus less overall test coverage.
The V8 team does not expect this to be an issue.

In the past, Node.js has also provided 32-bit binaries that would be usable on 64-bit systems. This option is still available when building Node.js from source, but release builds are not provided anymore. 32-bit binaries are incompatible with 64-bit native addons, but provide similar benefits as pointer compression does.

If we do support two different modes, how do we deliver them to users?

The pointer compression mode is hardcoded into the V8 build used by Node.js, bringing up the question of how to give users a choice if we decide to do so out of the box (as opposed to requiring users who pick the non-default option to build Node.js from source).

There are at least two main options for answering this:

• We could provide two separate Node.js binaries for download from https://nodejs.org. This requires the Node.js build working group to perform additional work around maintaining additional release builds.
• We could integrate two different V8 builds into a single Node.js binary. This would increase the Node.js binary size significantly (at least 20 MB).

Both options would increase the load on the Node.js release and testing infrastructure non-trivially, increasing the amount of necessary testing and lengthening the release process.

Which one would be the default mode?

If we do provide the two different modes out of the box, which one should be the default? Are we assuming that the heap limit imposed by pointer compression would be sufficient? If there is an extension to the heap limit as suggested above, what heap limit should we pick as the one that Node.js builds and provides to users?

[Joe8Bit]

Is there any aggregate data on heap-size in the community? Do we have any way to quantify the impact of restricting heaps to 4GB?

It's also worth considering the added complexity in the downstream ecosystem if there are two official binaries, for example:

• How would this effect providers of 'managed' node services? Would they provide both? Only one? How much additional burden would this place on them?
• Would downstream packagers be OK taking on ~2x the burden? e.g. unofficial docker image maintainers

Whichever approach is taken I'd also be interested in the impact on library authors, would they face a similar burden to the node release and test team? Would two binaries or a flag ~2x the testing burden on library authors? Would they only test on the default and therefore leave a big area of untested entropy in the node eco-system when running on the non-default option?

[addaleax]

Is there any aggregate data on heap-size in the community? Do we have any way to quantify the impact of restricting heaps to 4GB?

Not really, no. If you have suggestions for how to gather such data, please feel free to share!

Whichever approach is taken I'd also be interested in the impact on library authors, would they face a similar burden to the node release and test team? Would two binaries or a flag ~2x the testing burden on library authors? Would they only test on the default and therefore leave a big area of untested entropy in the node eco-system when running on the non-default option?

I think it’s safe to answer all these questions with “no” in 99 % of cases – this would only become relevant when libraries would use large heap sizes themselves, I think. Typical npm packages, including native addons, would be unaffected.

[zbjornson]

raises hand interested. Is there a way to try this yet? We run with 4-8 GB heaps and I'm currently trying to resolve an issue that I think is due to v8's GC not performing well when the heap usage gets that high. My guess is this wouldn't remedy the issue (doesn't change object count) but it shrinking the heap would be nice regardless.

For internal strings, does this still reduce memory usage by ~30%? I thought pointer compression only reduces the object handle size. (When we use more than 4GB it's usually lots of internal strings.)

As far as this breaking native add-ons, is that just a matter of rebuilding?

We use a custom build of Node.js, so the default wouldn't matter to us. I'm guessing most users would benefit from PC and don't set the max old space size, so that seems like a reasonable default.

[Joe8Bit]

Is there any aggregate data on heap-size in the community? Do we have any way to quantify the impact of restricting heaps to 4GB?

Not really, no. If you have suggestions for how to gather such data, please feel free to share!

Could platform providers provide anonymised, aggregate heap sizes for Node workloads on their platforms? It would be a limited, point in time perspective, but it might give us some idea. Some of the Microsoft/Google/Amazon/IBM folks in the community might be able to help broker this from their public cloud colleagues!

[addaleax]

raises hand interested. Is there a way to try this yet?

@zbjornson Well, at least there’s going to be a way soon: nodejs/node#30463. For now it breaks ABI compatibility for non-N-API native addons, as you noted.

We run with 4-8 GB heaps and I'm currently trying to resolve an issue that I think is due to v8's GC not performing well when the heap usage gets that high. My guess is this wouldn't remedy the issue (doesn't change object count) but it shrinking the heap would be nice regardless.

I would assume that too, yes.

For internal strings, does this still reduce memory usage by ~30%? I thought pointer compression only reduces the object handle size. (When we use more than 4GB it's usually lots of internal strings.)

The memory reduction estimates are based on statistics gathered in Chrome for typical websites. String sizes would remain the same. However, one possible strategy to reduce the impact of the lower heap limit that was mentioned by the V8 team would be adding something along the lines of an “externalize all strings” step that would move strings off the heap if one would otherwise be running out of heap, as I understand it.

As far as this breaking native add-ons, is that just a matter of rebuilding?

You’d have to set the two gyp variables changed in nodejs/node#30463 manually in the binding.gyp, I think. We could maybe figure out something to make this easier, however, as the ABI incompatibility is intended to be temporary, that’s probably not worth the effort.

Is there any aggregate data on heap-size in the community? Do we have any way to quantify the impact of restricting heaps to 4GB?

Not really, no. If you have suggestions for how to gather such data, please feel free to share!

Could platform providers provide anonymised, aggregate heap sizes for Node workloads on their platforms? It would be a limited, point in time perspective, but it might give us some idea. Some of the Microsoft/Google/Amazon/IBM folks in the community might be able to help broker this from their public cloud colleagues!

At our last Node.js collaborator summit, the idea was floated that we could gather some kinds of usage statistics for Node.js, e.g. feature usage counters etc., that would typically be enabled in CI systems or similar situations. However, no work in that direction has been done so far, I think.

As for cloud platform providers gathering this data themselves, I guess that would be possible with their buy-in, yes?

[Fishrock123]

Some observations:

A good number of people seem to have trouble getting --max-old-space-size and related options to work regardless (a quick search of the node issue tracker should turn several up).

Few user applications that I've seen in past support experiences exceed the default limit. Those that do usually have systemic memory leaks from misuse of language features, particularly Promises. (Or simply unbounded caches. Heh.)

As is likely obvious, retaining a massive heap in a single node application is quite an anti-pattern when examining Node's strengths. Most of the things you'd want to store that would take up a lot of memory are better stored some other way than in Node heap itself.
(You'll likely also be causing some perf degradation, which is all the more reason to look to other ways to scale.) (GC was mentioned above but I suspect other degradations also.)

Additionally, a sizeable about of memory in Node applications is in Buffers, which are unaffected by this.

---

I think this will benefit a vast majority of users. It's possible that some people may run into issues, but I think the largest percentile of those cases will run into issues already regardless.

(Consider: many deployment options these days have low-ish memory limits per cost. Saving people space there would actually be a very large win. Particularly in FaaS e.g. lambda.)

[ljharb]

Often inside enterprise CI systems, builds will suddenly start to fail as the codebase scales to be "too big". While the "best" fix is indeed to address the root cause, often the only practical escape hatch to unblock the entire company's workflow, without disabling critical steps (linting, the build process, tests) is to use --max-old-space-size to temporarily kick the can down the road, and buy time in which to do proper investigation.

It would be unfortunate, I think, if the removal of this ability meant that when an organization scales to a certain size, all work has to stop while the appropriate team tries to discover the root cause.

[addaleax]

@ljharb Thank you for your input! Just for a bit of context, until Node.js 12 the default --max-old-space-size value would be 1400MB on 64-bit platforms, and even now it’s capped at 2GB unless flags are set iiuc. (Thanks @joyeecheung for having pointed that out when drafting the issue text.) So if you do start out with defaults, you could still bump the limit up to at least twice that, and those 4GB correspond to ~6GB of uncompressed heap. This doesn’t make your argument any less valid, I just feel like this should be taken into account there.

[ljharb]

@addaleax totally fair! I believe Airbnb's jest tests were taking 8GB the last time I checked prior to leaving, as an example.

[newtack]

We use WebAssembly extensively and that is where most of our memory usage is. So as long as this does not impact WebAssembly memory (which could go into 10s of GB) this would probably be OK.

[configurator]

I once wrote a node server that would handle JSOM messages from a queue, each one about 1 MB in size. The operation it did benefited a lot from batching, and doubling the batch size increased the clock run time per batch by only about 10%, so it would handle 10000 messages at a time. If I weren't able to increase my heap size to 16 GB, I'd lose a great deal of performance by limiting to 2500 messages at a time.

[puzpuzpuz]

A potentially useless, off-topic feedback from a developer with Java background.

If we do provide the two different modes out of the box, which one should be the default?

This is probably a question for V8 team, but what prevents from having heaps larger that 4GB with enabled compressed pointers? A similar optimization is present in JVMs for many years and in Java 7+ it's enabled by default when the maximum heap size is less than 32GB. If V8 also adds paddings to objects stored on the heap, a similar trick could be used.

However, one possible strategy to reduce the impact of the lower heap limit that was mentioned by the V8 team would be adding something along the lines of an “externalize all strings” step that would move strings off the heap if one would otherwise be running out of heap, as I understand it.

In mainstream JVMs there are several approaches that decrease memory footprint of strings. I'd like to name a couple of them.

The first one is called Compact Strings, which reduces footprint of internal representation of each char from 2 bytes to potentially 1 byte. As many applications store huge amount of strings that consist of ASCII characters, this optimization may be valuable for such apps.

The second one is string deduplication (the name is quite straight-forward), but it's bound to concrete GC algorithm.

In general, 4GB max heap size sounds like a critical restriction to me. I think that such restriction would prevent the compressed pointers mode from being used in many web applications. But I may be wrong, especially considering V8's default heap size restriction.

[addaleax]

If we do provide the two different modes out of the box, which one should be the default?

This is probably a question for V8 team, but what prevents from having heaps larger that 4GB with enabled compressed pointers?

Yes, if padding is being added, the heap limit could be increased – that won’t be worked on until some time next year, but it’s possible. The extra shifts are likely to introduce some performance overhead, though.

The first one is called Compact Strings, which reduces footprint of internal representation of each char from 2 bytes to potentially 1 byte. As many applications store huge amount of strings that consist of ASCII characters, this optimization may be valuable for such apps.

I’m not an expert on this topic, but there are definitely a number of cases where V8 already applies this optimization.

The second one is string deduplication (the name is quite straight-forward), but it's bound to concrete GC algorithm.

This also already happens in V8, afaik.

[puzpuzpuz]

Thanks for these insights @addaleax!

Personally, I'd vote for a single (larger) binary with compressed pointers mode enabled by default. And if max heap size is explicitly set to >4GB value, 64-bit pointers mode would kick in.

To me, memory footprint (and, potentially, performance) benefit is much more valuable than a larger binary size.

[JMTK]

I currently work on a Discord bot using the hydrabolt/discord.js library. This library is built upon "caching everything", so there are large Maps built out of references to other objects and sometimes circularly. I frequently run into issues due to many references to the same object across the application(8GB on the server).

I'd be very interested in trying out this functionality to see what the differences would be. I would also be happy to provide some heapdumps that I have to see if I would actually benefit from this.