regression: Response does not accept a async-iterator anymore

[mcollina]

#48395 which shipped in Node v20.6.0 broke part of the fetch() spec and was caught by CITGM with some of the undici test failing everywhere.

This stopped working:

  const asyncIterable = {
    async * [Symbol.asyncIterator] () {
      yield 'a'
      yield 'b'
      yield 'c'
    }
  }

  const response = new Response(asyncIterable)
  await response.text()

I'm shipping a fix in undici in nodejs/undici#2251.

Ref. nodejs/citgm#959

(FWIW we should have caught this before release)

[mcollina]

cc @Trott

[nicolo-ribaudo]

Q (coming here from the citgm PR with zero background 😅): what is the expected behavior of that code? In browser, await response.text() returns [object Object] because Response does not have special handling for async iterables.

[lucacasonato]

@mcollina maybe your example is wrong, but you can't pass AsyncIterables as response bodies directly. The only valid streaming BodyInit is ReadableStream.

[mcollina]

maybe your example is wrong, but you can't pass AsyncIterables as response bodies directly. The only valid streaming BodyInit is ReadableStream.

Have you got a spec reference @lucacasonato?
That used to work before, and it broke.
I think it's part of our backward-compat with node-fetch (we lifted some of the tests there to minimize disruption).

@KhafraDev wdyt?

[lucacasonato]

• Response interface first argument is BodyInit
• BodyInit is either ReadableStream or XMLHttpRequestBodyInit
• XMLHttpRequestBodyInit is either Blob, BufferSource (Uint8Array, ArrayBuffer etc), FormData, URLSearchParams, or USVString

None of those types are IDL types that can be created directly from an async iterable / async iterator.

Your example should be new Response(ReadableStream.from(asyncIterator)); to be spec compliant.

[mcollina]

Thanks for the explanation!

@KhafraDev should we change this or leave as-is for backward compat with node-fetch?

[KhafraDev]

This was added so fetch could support node streams. We should definitely keep it.

[dnalborczyk]

doesn't seem to be great for interoperable code between runtimes and browsers. that said, how would users know about this? the docs are just mentioning Response as a browser compatible global: https://nodejs.org/api/globals.html#response

This was added so fetch could support node streams.

I haven't looked into this, but doesn't node provide a node stream/web api stream converter?

[lucacasonato]

This was added so fetch could support node streams. We should definitely keep it.

@KhafraDev Fetch can support node streams through new Response(ReadableStream.from(nodeStream)).

It is obviously not my call to make, but I want to advise you that keeping this behaviour is a dangerous hazard for future spec compatibility for Node.js. This is because if fetch gains support for other object types in the BodyInit position in the future, and these objects happen to already implement the async iterator protocol, and a developer is already passing these objects in a BodyInit position, then the behaviour that the developer experiences may breakingly change when Node is updated with that spec change.

If Node did not accept arbitrary async iterables, but rather only branded node streams in the BodyInit position, your usecase would be resolved without exposing Node to this dangerous future compatibility hazard. I think this would be a much safer alternative you could consider.

[KhafraDev]

that said, how would users know about this?

It's mentioned in the undici docs, there probably should be more extensive docs for fetch in node core.
https://github.com/nodejs/undici#requestbody

@lucacasonato

I agree with you and it wasn't my idea to support arbitrary async iterables, but I don't think it's as harmful as it's being made out to be. We've already implemented breaking changes that were made in the spec that broke users, so it would be the same here. For example, when a change was introduced that removes Authorization headers from cross-origin redirects or when a duplex: 'half' option started being required when passing a ReadableStream as a body. All environments choose to go against the spec in some capacity, and for node's use case it makes sense to support async iterables.

I couldn't find past discussions but @ronag added it so he might be able to give a better answer.

[ronag]

It was added so that node-fetch could be more easily be replaced with native fetch. The value was deemed high and any possible negative side effects were considered negligible.

Not supporting it is an unnecessary breaking change.

[ronag]

Ref: whatwg/fetch#1291
Ref: whatwg/fetch#809

[lucacasonato]

Folks now think this is standard behavior and are opening issues on the Deno repo about us not supporting this. This is why you don't do random willy nilly spec extensions folks 🙄

denoland/deno#21063

[KhafraDev]

Deno doesn't support forbidden headers, is that not also a "willy nilly spec extension"? Node supported them at one point, and removed it to match Deno's behavior (and we've had multiple issues regarding it).