Generalize ScriptTransform constructor to allow main-thread processing

[alvestrand]

The RTCRtpScriptTransform constructor takes a Worker argument, limiting the usage of this form of the transform to Workers.

The older createEncodedStreams() function was agnostic as to where the processing was going to take place; a number of existing demos and apps have been written that do processing on the main thread; some have even prototyped both worker-based and main-thread-based processing and deliberately chosen main-thread-based processing.

The normal use case should be worker - but other use cases should be possible.

Proposal: Change the argument type of the constructor from Worker to (Worker or MessagePort). Dispatch the event (which could then be a message) on either the worker's implicit port or the explicit MessagePort.

This allows all the use cases that the older API allowed, but ensures that the simplest code will be the one invoking a Worker.

[alvestrand]

Adopting this change would remove my objection to landing #64.

[youennf]

This allows all the use cases that the older API allowed

Can you give details on the applications/JavaScript code that cannot be migrated to RTCRtpScriptTransform?

[youennf]

Also, MessagePort can potentially be used for out-of-process message exchanges, for instance between window and service worker, or opener/opened-window.
DedicatedWorker was used here to limit the complexity of the implementation while retaining most of the functionality.

[alvestrand]

I first wanted to suggest Window as the alternate target, but Window's PostMessage() seems to have a different signature than workers' PostMessage (which is derived from MessagePort); I wanted to have a single API to relate to.

[alvestrand]

Given that streams are transferable, once we let them out, we don't really have any way to limit how much they spread by PostMessage(), so I don't see much implementation simplification by this restriction.

It seems useful to have only one mechanism for interprocess communication (PostMessage), not two (PostMessage and the special-case internal processing that creates an event).

This variant was also proposed as an API in the January interim (slide 20, https://docs.google.com/presentation/d/1crumgYj4eHkjo04faLktPTg0QoYJhTFoosEBudfJBuw/edit#slide=id.g7957cd038b_9_116), but wasn't further explored in the February interim (slide 34, https://docs.google.com/presentation/d/1-ZB5pjq0hTl4iNxGhIc3polH9wjCps2Uzqg7D4v1n14/edit#slide=id.gbdce051084_0_3)

The discussion in the linked issue ( #48 ) does not give any pro/con discussion of event versus message.

[youennf]

so I don't see much implementation simplification by this restriction.

By restricting to Worker, we do not open the door to out-of-process transforms, this simplify implementations a great deal.
This also makes using the API straightforward to understand and use (in-process, out-of-main-thread).

[alvestrand]

Still don't understand: How, with transferable streams, does restricting to Worker close the door on out-of-process transforms?

I am not necessarily saying that closing the door on out-of-process transforms is a Bad Thing, but I don't understand how the stated tool ("restricting to Worker") accomplishes the task ("do not open the door to out-of-process transforms").

[alvestrand]

Consider reimplementing https://webrtc.github.io/samples/src/content/insertable-streams/video-analyzer/ (the obvious example of an app where background processing doesn't matter) under the event-sending interface:

One can write a worker that receives the stream on an event, and then transfers the stream right back to the main thread, using an API that is part of the Streams API and not at all part of the webrtc API.
There is no simplification of implementation compared to a more generic, more common IPC API using PostMessage with a MessagePort target.

[youennf]

The MessagePort code path is much more complex than Worker.postMessage code path.
We keep things simple for WebRTC encoded transform implementation by staying in process and we support all cases we actually care.

Additional API (transferable streams) can be used for more complex cases on top of webrtc encoded transform.
That is great, this is composability a la Unix.

This does not mean we should bake into the simple WebRTC encoded transform API everything that WebRTC encoded transform+transferable streams allow.

[youennf]

If the goal is to be able to properly shim createEncodedTransform in JS, I think the following can be written:

• Create a ReadableStream R1 and WritableStream W1 in window environment with JS sources SR1 and SW1
• Create a worker and a script transform
• Set the transform on the sender/receiver
• In the worker, when receiving the readable/writable R2/W2, transfer them back to window (async operation).
• When receiving R2 and W2 in window, use them to make SR1 and SW1 functional (async operation)
• Return R1 and W1

[alvestrand]

Re #89 (comment) about complexity: Why is the MessagePort code path more complex than Worker.postMessage? For the user or for the browser implementor?
After all, DedicatedWorker is defined as "act as if it has an internal MessagePort" (https://html.spec.whatwg.org/multipage/workers.html#dedicated-workers-and-the-dedicatedworkerglobalscope-interface), so I assumed that the underlying implementation was OOM the same, since the API is identical.

Re workaround: Yes, that's exactly what I was suggesting in #89 (comment), and it shows that:

1. Requiring posting to a worker does not lead to decreased implementation complexity, since one still has to support processing the stream if it happens to be executing on the main thread (or some other thread)
2. Requiring posting to a worker increases the complexity for the user in the (few) cases where main thread processing makes sense.

[youennf]

Why is the MessagePort code path more complex than Worker.postMessage? For the user or for the browser implementor?

Both actually.
For the implementor, posting to a worker is easy as it is a destination that does not move (the implicit port is not transferred). Posting to a general MessagePort is harder since it might be transferred at will. In Safari, there is usually a central registry that does the routing and message management. Not sure about other implementations, let me know how Chrome is handling this.

As of user impact, using a MessagePort in the API will require the user to create a worker, create a MessageChannel, transfer one of the message port to the worker and use the other one in the script transform constructor.
For the user, it seems simpler and easier to understand to directly pass the Worker as a parameter, which covers all cases we know about.

1. Requiring posting to a worker does not lead to decreased implementation complexity, since one still has to support processing the stream if it happens to be executing on the main thread

Only if the UA supports transferable streams, which can be implemented in a fully orthogonal way. This orthogonality is decreasing complexity.
The only case where it would not decrease complexity would be if UA would want to optimise the script transform+transfer stream code path. I haven't heard of use cases that would make this needed.

If there are such use cases, I would first try to see how we could update the API and resort on specific optimization if API change is not feasible.

2. Requiring posting to a worker increases the complexity for the user in the (few) cases where main thread processing makes sense.

I am not clear about these few cases.

[alvestrand]

Replying in reverse order:

The use case I use as an example is https://webrtc.github.io/samples/src/content/insertable-streams/video-analyzer/, where the whole point of the processing is to get data that is presented on the page by code running on the main thread.

I mentioned this 21 hours ago in #89 (comment) too.

Using a (MessagePort or Worker) will not lead to any user increase in API user complexity for the worker case; that's what I suggested in #89 (comment) - I have not suggested removing Worker from that construct.

[alvestrand]

FWIW - MessagePort in Blink is implemented as a Mojo IPC operation, and postMessage() in DedicatedWorkerGlobalScope is implemetned on top of MessagePort. So for Chrome, the underlying mechanism is identical for the two cases.
If I've read the code correctly.

[youennf]

The use case I use as an example is https://webrtc.github.io/samples/src/content/insertable-streams/video-analyzer/, where the whole point of the processing is to get data that is presented on the page by code running on the main thread.

I have not looked precisely at what this page is doing but I think the usecase can be defined as:

• Do the video rtc pipeline as efficiently as possible (as usual). In addition, analyse the data to gather stats/monitor the pipeline
• Show the stats/analyze results in the web page
  I would implement it as:
• Use script transform to efficiently do the processing out of main thread. Gather the stats in the worker.
• Send the stats back to the page.
  Based on that, the current API supports that use case and would lead to more optimal results than doing the whole processing in main thread.

MessagePort in Blink is implemented as a Mojo IPC operation

In the previous message, I was referring to Worker.postMessage, not WorkerGlobalScope.postMessage (which might be the same for shared worker, service worker and dedicated worker in Chrome).
In any case, allowing a MessagePort would lead to require Mojo IPC operations. Currently, RTCRtpScriptTransform only requires posting a task on the worker thread, which is much simpler to implement.