A camera track made without the processor lets the room see the camera
until the processor is built. So the processor travels in the publish
request: the lobby opens the camera with it, the join publishes with it,
the room's capture defaults follow the requests made since the join, and
a camera turned off before the processor was chosen is replaced on turning
it on rather than unmuted, so that the new track starts with it.
Ported from spec/background-effects/2-pipeline (#4282): the publish half
of a91dab8f. The replacement lives in the mute switch's track supply,
which the call view model shares with the lobby.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Blur runs through one pipeline, shared by the pre-join preview and the
call: attached the first time an effect is wanted and switched in place
after, never detached. The logic is BackgroundEffects in src/state;
BackgroundEffectsContext only builds it, from the app's view model, so
the standalone app and the component hand the same instance to every
camera. Disabled drops the blur radius, so no effect costs no
segmentation. The SDK gains fromLivekitProcessor beside the adapter,
now toLivekitProcessor.
Ported from spec/background-effects/2-pipeline (#4282): a91dab8f with
its review tidy-ups 445e010c, bbc73d07, 51729bcf, f52d2b79, 813f1e6f
and 584bbaa0. Left out as already solved here: OneStepPipeline (the SDK
runs a processor's steps in turn) and the LiveKit-track sync helpers
(the lobby's camera is an SDK track, synced by followProcessor).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A lobby captures the microphone and the camera on the browser's devices
alone, with `createLocalMedia`, so that any backend can take them over:
`LocalMedia` hands out the same `LocalAudioMediaTrack` and
`LocalVideoMediaTrack` a call renders, with mute, device and facing
switches and the host's `VideoProcessor` run on the preview. Given to
`join` as `publish`, the tracks are moved rather than shared: the
publisher takes them over as it starts publishing, their lobby objects
are finished, and the LiveKit backend adopts the device tracks as its
own and the processor with its running output, so the join captures
nothing and initialises nothing again.
Also here, found on the way: a track already on the requested device is
left alone (LiveKit would restart it, always on Firefox, which echoes no
device id in `getConstraints()`), identical captures in flight are
coalesced (React's strict mode mounts twice), and every operation on a
processor takes its turn, since a stopped track tears its processor down
without waiting for the next track's init.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
By keying the connections by the transport's resolved auth flow. This is important because connecting to the same SFU twice with the same participant ID will cause a 'duplicate participant ID' error that stops one of the connections.
The second connection only happens when the first one actually failed
on the WebSocket. connect() rejects after its socket has failed and the
engine is closed, so the retry never overlaps the failing socket.
This removes the notion of an auth flow fallback for now. We will no longer try the CS API when authenticating with transports that include both a `url` and `livekit_service_url`.
Behind a proxy, Firefox fails the first WebSocket it opens to a host, so
the first join fails. The livekit-client in the report turned this into an
insufficient capacity error; current versions report a generic connection
error. Opening a throwaway WebSocket first lets the real connection
through.
Co-authored-by: Marc <10048869+MarcWadai@users.noreply.github.com>
Port of edf1feae from johannes/rtc-remote-publish onto the SDK split,
where
the MatrixRTC logic lives in sdk/ rather than src/state/CallViewModel/.
Where the original lands:
- getLocalTransport's remote fallback: the LiveKit backend's new
findRemotePublishTransport (sdk/src/media-backend/livekit/
RemotePublishTransport.ts), which tries the candidates from the
js-sdk's
getRemoteLivekitPublishCandidates oldest first and keeps the first
whose
homeserver issues a token with the publish grant.
discoverLocalTransport
asks the backend for it after the host's fallback, with a snapshot of
the
session's memberships from RTCParticipation, and throws
MatrixRTCTransportMissingError where that finds nothing too. Transport
discovery stays backend neutral: the MSC4195 token check is the
backend's.
- LocalMember's "no delegation on a remote SFU" and the custom
transport's
server_name: LivekitBackend.prepareLocalTransport locates the
transport on
the homeserver it names (serverNameOfTransport) and skips the
delegation
probe where that is not our own.
- SessionBehaviors' membership.getTransportServerName: the same helper
in
LivekitBackend.mediaFor.
- decodeSFUJWTPayload, SFUJWTPayload.video.canPublish and the exported
isCsApiUnavailable: sdk/src/media-backend/livekit/auth/.
- DeveloperSettingsTab, the widget capabilities, the locale string and
the
js-sdk pin (9405287c, 44.0.0): applied as is. The error boundary
snapshots follow the string.
Tests: the LocalTransport cases go to RemotePublishTransport.test.ts and
LocalTransport.test.ts, the LocalMember remote case and the
SessionBehaviors
server-name case to LivekitBackend.test.ts, plus the memberships
snapshot in
RTCParticipation.test.ts.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Unit tests for the pieces that had none: which transport the local
member advertises (override, homeserver, fallback, none), what the js-sdk
join is called with, how a publish request becomes LiveKit capture and
publish options, and the room options the connection factory builds from
the initial requests and the media quality settings, including what a
resolved connection reports while it is up. The backend's `mediaFor` gains
the cases from Element Call's member matching tests: a member on another
transport, a member changing transport, and a member without one.
The publisher is driven through a real LocalParticipant whose device and
signalling calls are faked, so that LiveKit's own pending-publication
bookkeeping is part of what is tested: the cases ported from Element
Call's Publisher tests (one permission prompt for both devices, nothing
created while muted, the upstream held back until the member publishes,
a mute that lands while the track is still being created) plus the
publish, unpublish and destroy paths.
The local media runs on a fake connection map: publishing before and
after the connection exists, republishing on a new connection, mutes
remembered across connections, the first publish error kept, and the
output device applied to the current room and the ones to come.
The participation runs on a fake media backend: the join options, the
state machine through connecting, connected, reconnecting, failed and
left, publishing only while joined, the delayed leave delegation, the
member ids per mode, the key replay and the data channel's sender check.
The loop that drives `scope.reconcile` was marked as running until the
promise of its last run had settled, which is a few microtasks after the
loop itself had checked for new values. A value arriving in that window
found the loop "running", was left to it, and was never processed: a
connection appearing at that moment would never get its publisher.
The flag is now cleared the moment the loop ends, so such a value starts
a new loop. The regression test drives the window microtask by microtask.
The developer setting that overrode the LiveKit service URL becomes a
JSON transport, so that a `url`-only or combined transport can be tested
against a homeserver that does not advertise one yet. The settings modal
shows each LiveKit room's role instead of a never-populated isLocal flag.
On the SDK the participation options take a whole Transport for the override and the fallback instead of two LiveKit service URLs, which also takes the last LiveKit knowledge out of transport discovery.
(cherry picked from commit d3eb40db0bcf7370418c4948049b3eac176bbdc1)
The OpenID flow moves to src/livekit/auth/openID.ts with the SFU config
types in a sibling file, takes an options object, and replaces the
MatrixRTC mode with the role we take on the transport plus the session's
slot ID instead of a hard-coded m.call#ROOM.
This changes behaviour: publishers on a livekit_service_url transport
always use the legacy /sfu/get endpoint, so that subscribers can predict
their user:device participant identity, and subscribers always use the
default /get_token endpoint. The MatrixRTC mode no longer influences which
endpoint is used.
On the SDK this lands as sdk/src/media-backend/livekit/auth/ with the connection map, the discriminated connection state and the client-server API flow of the following commits of element-hq/element-call#4331, which rewrite the same files:
- Model ConnectionState as a discriminated union
- Show the whole transport in debug UI and logs (SDK part)
- Let the publishing Connection authenticate itself
- Reduce ConnectionManager to a ConnectionMap
- Use the unstable LiveKit transport type
- Identify transports by server name as well as URLs
- Get SFU tokens and delegate delayed leaves through the MSC4195 CS API
- Fall back to get_token as subscriber on widget API errors
- Share a connection between homeservers using the same JWT service
- Make the auth choice mode independent again
Co-authored-by: Johannes Marbach <n0-0ne+github@mailbox.org>
(cherry picked from commit adb380cd9ed108856e42d992fc03ced8c44f1e56)
A member authenticated through the legacy JWT endpoint publishes under a
`user:device` identity, while one authenticated through the default
endpoint publishes under a hashed identity. Both are listed in the
membership's backend identities, so match against the whole list rather
than only the primary one.
On the SDK this is RTCMember.backendIdentities, which the LiveKit backend matches media and data packets against; the harness keys its tiles by the membership id, which the backend identity no longer tells apart.
(cherry picked from commit c9aad2a2b3279782d101c9bc6c80af0417ccbb92)
The js-sdk at this commit also renamed MatrixRTCSession.leaveRoomSession to leave, which the SDK follows.
(cherry picked from commit 405348945febc1df10edf9c00748f1edd6b39e71)
The SDK now takes `publish` requests and audio output control directly
in the main options object, removing the separate `LocalMediaInputs`
type. Audio output device selection moves from a reactive behavior to an
imperative method on `MatrixRTCClient`, giving hosts direct control over
when devices switch.
The session speaks MatrixRTC and sees media through MediaBackend in
sdk/src/backend/api.ts. Everything LiveKit lives in
sdk/src/backend/livekit/,
the only place in the SDK that imports livekit-client; the import
boundary rule
now enforces that outside sdk/src/backend/ the LiveKit packages appear
in type
imports only.
The seam sits above the connection manager: one connection per transport
url
is the LiveKit multi-SFU strategy, not a MatrixRTC concept, so a
cascading SFU
or full mesh replaces the folder, not the session. A backend prepares
the local
transport the session discovered (the token exchange and the delegation
probe,
reported as canDelegateDelayedLeave), hands the delayed leave over per
delay
id, gives the local member one publish switch plus publish/unpublish,
builds a
remote member's media from its membership and registers that
membership's
transport for as long as the member lives, and reports its connections
for the
transport registry, which keys on the raw transport serialised with
sorted
keys. The backend's context carries no Matrix client (a backend picks
what it
calls in its own options), no membership list, and no js-sdk session:
media
keys arrive as a stream replayed on subscribe, and MatrixKeyProvider
subscribes
to it. The MatrixRTC mode no longer reaches LiveKit; it becomes a
tokenEndpoint
choice made where the default backend is built.
Two behaviour changes ride along. Publishing is one boolean, joined and
homeserver reachable, where before a sync hiccup after leave() resumed
the
upstream of tracks that were meant to stay silent. And
ResolvedConnection keeps
one ResolvedTransport object per fetch, so a state change no longer
reads as a
change of what was fetched.
MatrixRTCClientOptions gains backend?: MediaBackendFactory, defaulting
to
createLivekitBackend built from the client options.
authenticateWithTransport
takes tokenEndpoint instead of matrixRTCMode.
membership$ is typed as RTCMembership, the subset of the js-sdk CallMembership that hosts read. UnknownCallError is UnknownRTCError throughout, not only in the export. applicationData reaches the membership whole, since the js-sdk takes it next to the intent. The plan records the three slices as implemented.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
One way to put media on the session: publish(request) resolves with the local track once LiveKit has it, and unpublish(id) removes it. A request names a source with its device, processor and capture settings, so CaptureSettings on the client options goes, as do setMicrophoneEnabled and setCameraEnabled on the client and the four screen share fields on the local member; sharing is tracks$ having a screenShare track, and a capture failure rejects the promise. LocalMediaInputs becomes the list of initial requests, which also become the room defaults so that one permission prompt covers microphone and camera. The publisher keeps the requests across a reconnection and republishes what is not muted. The harness gains a Share screen button.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
