Unable to stream videos in chat

[baconsalad]

Steps to reproduce

Using roughly this command it was previously possible to stream video into a room

lk join-room \
--room "${ROOM}" \
--identity "videobot" \
${API} \
--publish "${VIDEO_SOCK}" \
--publish "${AUDIO_SOCK}" \
--fps "${FPS}"

but it no longer works any more. Instead now it spits into the browser console [MatrixAudioRenderer] Audio track videobot has no matching matrix call member current members: @aaa:bbb.com:CDEFGHIJK track will not get rendered (this is from src/livekit/MatrixAudioRenderer.tsx)

The reason I think this is a bug is that using livekit cli to "join-room" the user is not considered part of the room despite everything saying otherwise. eg.

LIVEKIT_URL='http://aaa:7880' LIVEKIT_API_KEY='bbb' LIVEKIT_API_SECRET='abc123' lk room participants list '!ABC:def.com'
Using url, api-key, api-secret from environment
@aaa:bbb.com:CDEFGHIJK (ACTIVE)    tracks: 1
videobot (ACTIVE)        tracks: 2

Outcome

Used to work, no longer does.

Operating system

Linux

Browser information

Any

URL for webapp

0.13.1

Will you send logs?

No

[rwjack]

I noticed this for my phone, made a call from EX to EW, turned on the camera on EX - EW did not see any video. Tried sharing my screen from EW, and the screen was properly displayed on EX.

[toger5]

I noticed this for my phone, made a call from EX to EW, turned on the camera on EX - EW did not see any video. Tried sharing my screen from EW, and the screen was properly displayed on EX.

This seems unrelated since when using EW you should create a call member state event.

[fkwp]

@baconsalad what you described is the intended behavior for the following reasons:

• As with Matrix with MatrixRTC we want to encrypt everything by default which requires a secure and trusted channel to exchange the individual sender encryption keys
• Hence, MatrixRTC is build on top of Matrix and combines signaling via Matrix with media exchange via an RTC backend. For now this backend is LiveKit's flavor of WebRTC but it is designed such that we can plug another RTC backend if a better one is available, e.g., Webtransport & web codecs in the future
• In order to be able to trust the participants of a MatrixRTC session the sessions's participants are announced/signaled via the Matrix room state. We only present participants from the livekit side if they do have a proper MatrixRTC membership in the corresponding Matrix room.

[toger5]

@baconsalad This is very much by design.
It is part of the inevitable tradeoff between security and convenience you will find in all technology.

Making it very convenient and easy to inject streams into a conference is an attack vector.

One can easily mimic another participant or send abusive content into a video conference.

MatrixRTC does not allow this attack by design and EC now also supports all those security precautions.
In addition to that EC is mostly designed to be used in an encrypted scenario where we need to know the device via the state event to know whom to distribute keys to.

A short summary of how it works now:

• A MatrixRTC participant is primarily defined by the room state (matrix room state hast access control and a permission + authentication mechanism and hence is what we want for a video call)
• A livekit participant on the other hand is very easy to fake and can easily be injected especially since matrixRTC is designed to work with an SFU you do not necessarily need to trust. We use it mostly as a pool of media streams.
• EC will use the room state to choose what data it actually wants to load from the pool of media streams.

To work around your issue you need to extend you script you used to inject the stream to also join the room/rtc session via a membership state event. See the MatrixRTC MSC's for more details

(sorry for the redundancy with @fkwp post)

[baconsalad]

As you can see this wasn't a script, just a command. So short of writing a lengthy script to login and emulate a user what are my options?

[toger5]

• Write a command that also sends the room state with a bot token.
• Create a pre-existing state event as a placeholder and add the stream with your command and always leave the state in the room sometimes there is just no associated stream.
• Activate the developer option to show "ghosts" (Show tiles for non-member media) streams that are not part of the rtc session but that are in the livekit room (unrelated to matrixRTC)