Streams, Buffers, and Worker Threads

[jasnell]

There is a bit of undefined grey area when passing a Buffer instance into a stream.Writable: who has ownership responsibility for the Buffer?

See this example as reference: https://github.com/jasnell/piscina/pull/34/files#diff-123d6328fe4f2579686ee49111e38427

In the example, I create a stream.Duplex instance that wraps a MessagePort. On _write, I post the given Buffer to the MessagePort and include it in the transfer list to avoid copying the data:

  _write (chunk, encoding, callback) {
    if (typeof chunk === 'string') {
      chunk = Buffer.from(chunk, encoding);
    }
    this.#port.postMessage(chunk, [chunk.buffer]);
    callback();
  }

This is where it gets fun. In my example, from within a worker thread, I open a pipeline that reads from a file, passes that into a gzip compressor, then out to my custom duplex. What happens is a fun little Abort...

james@ubuntu:~/nearform/piscina/examples/stream$ node index
node[30456]: ../src/node_zlib.cc:323:static void node::{anonymous}::CompressionStream<CompressionContext>::Write(const v8::FunctionCallbackInfo<v8::Value>&) [with bool async = true; CompressionContext = node::{anonymous}::ZlibContext]: Assertion `Buffer::IsWithinBounds(out_off, out_len, Buffer::Length(out_buf))' failed.
 1: 0xa295b0 node::Abort() [node]
 2: 0xa2962e  [node]
 3: 0xae6b1a  [node]
 4: 0xc0251b  [node]
 5: 0xc03ac6  [node]
 6: 0xc04146 v8::internal::Builtin_HandleApiCall(int, unsigned long*, v8::internal::Isolate*) [node]
2
 7: 0x13a5919  [node]
Aborted (core dumped)

The reason for the Abort is that my custom Duplex just transferred ownership of the Buffer away to the main thread while the gzip compressor was still using it.

Replace the gzip compressor with the brotli compressor, and things work! Change the postMessage() call to remove the transferList and things work!

Obviously the Abort itself is problematic and we need to figure out how to avoid that in the zlib, so that's issue #1.... issue #2 is that we really should try to specify some ownership expectations on Buffer instances passed into streams instances.

/cc @nodejs/streams @ronag @addaleax @nodejs/zlib

[ronag]

Oh, move semantics in javascript. How exciting.

I would prefer if the ownership was transferred down the pipe. However, I find it unlikely that we will be able to enforce this in user land...

It's not the most performant way but I think in this case you should not transfer the Buffer, but copy it.

[ronag]

Maybe stream implementators can set a standardised flag on the Buffer to indicate it is safe to transfer?

[jasnell]

Yep, for my case, I'm going to allow an option that does both, with copy as the default. We should, however, deal with the Abort case here more elegantly to avoid the crash as it's just a matter of time before other end users hit it.

[addaleax]

I think the question you posed in the linked issue is the relevant one here for the second issue:

when using a Writable, and passing a Buffer, who takes ownership responsibility for that Buffer? The caller or the receiver? I don't think we have a clear answer for that but we definitely need a fix that does not trigger an Abort.

We kind of do have an answer for that: Modifications to the Buffer that happen between write() and _write() are reflected in the written data, which implies that the caller is responsible for maintaining ownership of the Buffer. As @ronag said, that’s not really preferable, but it’s unlikely that we can do something about it.

There kind of is a good and performand solution here, which would be having copy-on-write semantics for ArrayBuffers (where detaching an ArrayBuffer would qualify as a “write”). I’m afraid that that’s a language-level feature that we can’t necessarily implement, though.

And for issue 1, yeah, it might make sense to audit the codebase for calls that read from buffers and check how they react to detaching ArrayBuffers.

(None of this is really Worker-thread specific, btw.)

[jasnell]

Nope, definitely not worker-thread specific but makes it easy to spot the issue :-)

[mafintosh]

@jasnell I think your issue is due to your stream "breaking" the pooling contract here https://github.com/jasnell/piscina/pull/34/files#r419929366

[jasnell]

@mafintosh yeah, that's precisely the point here tho. The streams contract is underdefined here with an implicit, undocumented rule that the writer owns the buffer most of the time. That doesn't hold true all of the time, however. Such as when the Buffer is sliced off the pool, or when the stream instance itself creates the Buffer because it was given a string instead. Meanwhile, there aren't protections downstream to prevent the Buffer from being misused (such as being put into a transfer list a triggering an abort).

There are two actions here:

1. Make the implicit rule explicit through better documentation
2. As @addaleax suggested, audit our uses of Buffer to ensure we are catching and handling better violations of that rule.

[mafintosh]

@jasnell I don't think this is a stream issue tho. Node pooling makes this impossible to solve, in general.

const a = Buffer.from('hello')
const b = Buffer.from('world')

console.log(a.buffer === b.buffer)

[mafintosh]

To expand on o/, is there ever a case where transferring an unsafe buffer using transferlist won't cause undefined behaviour in your Node program, since it'll "free" all kinds of unrelated buffers?

[ronag]

Given that, is there any case where we can allow buffer in the transferlist? Shouldn't we always make a copy (event if the user puts the buffer into the transferlist), i.e. we support the API but with a slow implementation. Until we have a better solution?

[addaleax]

@mafintosh Just fyi, you are right, but in the near future, that is no longer going to be a concern here: #32759