- The effects state's timeline through preparing, settling and attached,
and that a failed trial build withholds the effects for the session while
keeping the choice.
- The footer offers nothing once the pipeline fails to build, and passes
on the wait as the state's single wait field.
- The call leaves the camera out of the join while the pipeline is
prepared, publishes it once that is known, and turns on a camera turned
off before the processor was chosen; the lobby opens no camera meanwhile.
Both view models are driven by one state subject.
- The transformer's trial build answers whether a segmenter can be made.
- End to end, a browser whose pipeline cannot be built sees the effects
withheld.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
- A segmenter that can't be built, with the model unreachable or a GPU
its delegate refuses, took the camera down with the pipeline: the SDK
fails the track it was attaching to, and the lobby muted camera and
microphone, on every load while the choice was kept.
- One is now built on trial when an effect is first wanted, before the
pipeline is attached to anything, on the CPU where the GPU refuses.
Where it can't be built, the options are disabled with the existing
notice for the rest of the session. The choice is kept: a browser
update may fix what failed.
- The view model takes that verdict from the pipeline's state rather
than
asking the SDK, so both say the same, and it no longer builds a WebGL
context of its own to ask.
- Until the trial is over, the lobby opens no camera and the call makes
no
camera track, so neither starts without the effect.
Rebased onto main, where the Publisher is gone and the call view model
publishes the camera through the SDK:
- CallViewModel takes videoProcessorPreparing$. While it is true, the
camera's mute handler waits before making a track, and the join leaves
the camera out of its requests (initialPublishRequests); the view
model
publishes it once it is known whether the pipeline builds. The
Publisher's test became CallViewModel tests.
Cherry-picked from spec/background-effects/12-unbuildable-pipeline onto
toger5/background-effect/camera-preview, where the camera menu's preview
takes the camera from the footer's caller rather than from a track
synced
on the background-effects state. The footer factories take the
pipeline's
state again, as this commit needs its verdict and not only the settling
flag; the synced camera and the preview's wrapper type are not brought
along.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The SDK now builds a processor chosen while the camera was off when the
camera turns on, so the call view model no longer unpublishes the
camera and publishes a new one. TrackSupply.replace, the record of
which camera took which processor, and followProcessor's applied
callback go with it.
Answers robintown's review on #4282 ("what's wrong with attaching
processors to already published camera tracks?").
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A camera track made without the processor lets the room see the camera
until the processor is built. So the processor travels in the publish
request: the lobby opens the camera with it, the join publishes with it,
the room's capture defaults follow the requests made since the join, and
a camera turned off before the processor was chosen is replaced on turning
it on rather than unmuted, so that the new track starts with it.
Ported from spec/background-effects/2-pipeline (#4282): the publish half
of a91dab8f. The replacement lives in the mute switch's track supply,
which the call view model shares with the lobby.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
join
The lobby captured its own LiveKit tracks and the join captured again:
two permission prompts, a grey tile while the camera restarted, and the
background pipeline built twice. The lobby now opens the devices on the
SDK's `LocalMedia`, follows the mute switches as they change (they start
off until the devices are enumerated), the device choice and the
processor through controls shared with the call view model, and the call
joins with that media, so the tile carries the very track the lobby
previewed.
`useBehavior` subscribes without `useSyncExternalStore`, which left a
view without a re-render for an emission React had been told about,
about one time in three, when its store had switched. The LiveKit
preview hook and the processor sync helpers go, with
`@livekit/components-react`.
Background effects run under the dev server: the node polyfill no longer
rewrites MediaPipe's WASM loader. Taken from fkwp's 2fad4e08 on
spec/background-effects/2-pipeline (#4282).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The SDK replaced `src/livekit` and the remote-member modules before that PR
landed on `main`, so its `src/livekit/auth/*` and `remoteMembers/ConnectionMap.ts`
have no consumer here: the same code lives under `sdk/src/media-backend/livekit`.
The imports of them that `main` added to the view model, the developer
settings tab and the test utils go with them.
The bumped js-sdk reports the legacy identity as rtcBackendIdentity for every membership, so a device that rejoins after an improper leave, which has two memberships until the stale one expires, produced two tiles with the same key and crashed the call view. The membership id tells them apart; the audio renderer keys by it too.
A transport may soon carry a LiveKit SFU `url` instead of, or as well as,
a `livekit_service_url`, so a service URL alone no longer identifies it.
Pass the whole transport to the tiles' debug stats and print it as JSON in
log lines instead of picking out the service URL.
Element Call's media view models carry the member's raw transport from the SDK's TransportMetadata to the tiles.
The developer setting that overrode the LiveKit service URL becomes a
JSON transport, so that a `url`-only or combined transport can be tested
against a homeserver that does not advertise one yet. The settings modal
shows each LiveKit room's role instead of a never-populated isLocal flag.
src/livekit/auth now fronts two flows. For a transport with a LiveKit SFU
`url`, tokens come from our own homeserver's MSC4195 get_token endpoint
and delayed leaves are delegated through delegate_delayed_leave, both via
the js-sdk's unstable client methods. Transports with only a
livekit_service_url keep using the OpenID flow against the JWT service.
A transport advertising both shapes lets a subscriber fall back to the
OpenID flow when the homeserver answers get_token with a 404 (Synapse
reports M_UNRECOGNIZED for unknown routes rather than M_NOT_FOUND), so a
client can still join calls hosted on servers that have not deployed
MSC4195 yet. Publishers never fall back: a homeserver that advertises a
`url` but does not implement the endpoints is misconfigured and the
admin should hear about it, so they get NoMatrix2AuthorizationService.
Delegation uses the same flow as the token so that the participant
identity the service computes matches.
The local member only probes a transport's delegate_delayed_leave
endpoint when it has a service URL. The developer settings tab validates
a custom service URL through the same facade.
A transport with only a LiveKit SFU `url` has to be authenticated through
our own homeserver, which needs to know the name of the server owning the
SFU. Carry that name alongside the transport as a TransportLocator,
derived from the member's user ID for remote transports and from the
client's domain for our own, and key the connection map on the SFU URL,
the service URL and the server name together so that `url`, service URL
and combined transport shapes are deduplicated consistently.
UnstableLivekitTransport covers every shape a homeserver may advertise
while MSC4195 evolves: a LiveKit SFU `url`, a `livekit_service_url`, or
both. Switching to it means a service URL is no longer guaranteed, so the
remaining code that relies on one checks for it explicitly: delegation
probing skips transports without one, the connection map keys transports
by whichever URL they carry, and the OpenID flow rejects transports
without a service URL until the client-server API flow lands.
createConnectionMap$ now returns the behavior of connections and their
participants directly and takes an already resolved local transport
instead of an observable of it. The CallViewModel wraps it with mapScoped
so that a new map is built, in a fresh scope, whenever the local transport
changes, and swallows local transport errors there since the local member
already surfaces them in the UI.
getLocalTransport no longer authenticates with the SFU up front and hands
the resulting config to the connection. Instead the ConnectionManager tags
the connection whose transport equals the local one as the publisher, and
every Connection fetches its own SFU config for its role.
The local member now only joins the MatrixRTC session once its connection
is authenticated, so that the LiveKit room exists before remote members
try to get a subscriber token for it. Subscribers authenticating with the
OpenID flow lack permission to create the room and would otherwise fail
without retrying, causing a cascade of leaves and rejoins at the start of
a call.
The OpenID flow moves to src/livekit/auth/openID.ts with the SFU config
types in a sibling file, takes an options object, and replaces the
MatrixRTC mode with the role we take on the transport plus the session's
slot ID instead of a hard-coded m.call#ROOM.
This changes behaviour: publishers on a livekit_service_url transport
always use the legacy /sfu/get endpoint, so that subscribers can predict
their user:device participant identity, and subscribers always use the
default /get_token endpoint. The MatrixRTC mode no longer influences which
endpoint is used.
The old reducer keyed items by service URL and mutated the accumulator's
entries in place, so a transport without a service URL could not be
represented and items were not safe to compare. Group members by their
Connection object instead and carry the whole transport on the item, which
the audio renderer now receives in place of a URL.
A transport may soon carry a LiveKit SFU `url` instead of, or as well as,
a `livekit_service_url`, so a service URL alone no longer identifies it.
Pass the whole transport to the tiles' debug stats and print it as JSON in
log lines instead of picking out the service URL.
The enum mixed our own lifecycle states with a copy of livekit-client's
connection states, and errors were smuggled through the same observable as
a separate union member. A discriminated union keeps the LiveKit state
nested under "authenticated", carries the error as data, and lets the
local member represent the time before a connection exists as an
"initializing" state instead of null.
A member authenticated through the legacy JWT endpoint publishes under a
`user:device` identity, while one authenticated through the default
endpoint publishes under a hashed identity. Both are listed in the
membership's backend identities, so match against the whole list rather
than only the primary one.
I learned that makeKeyDelay has been deprecated in matrix-js-sdk ever since the switch to the new encryption manager, so this config option has no effect.