diff --git a/sdk/MediaBackendPlan.md b/sdk/MediaBackendPlan.md deleted file mode 100644 index 222aa1557..000000000 --- a/sdk/MediaBackendPlan.md +++ /dev/null @@ -1,716 +0,0 @@ -# Media backend: trapping LiveKit behind one seam - -> **Status: implemented, 4 October 2026.** All six steps landed in one change; -> the interface is in `sdk/src/backend/api.ts`, the LiveKit backend in -> `sdk/src/backend/livekit/`, and the architecture document has the section -> "Media backend". The text below is the plan as it was written, kept for the -> reasoning behind the shape. - -The SDK's public API in `sdk/src/api.ts` names LiveKit in three places: the video -processor type in `PublishRequest` and `LocalVideoMediaTrack.setProcessor`, the -video codec in `VideoCaptureSettings`, and the LiveKit-flavoured client options -(`mediaQuality`, `transportUrl`, `fallbackTransportUrl`). The implementation is a -different story. LiveKit participants, rooms, connections, -tokens and data packets run through most of `sdk/src/session/`. Only the two -adapters in `sdk/src/media/` are the clean boundary they were meant to be. - -MatrixRTC will get media backends other than a LiveKit SFU, full mesh and a -cascading SFU among them. Neither fits the model the session is written against -today, one connection per transport URL with a subscribe-only connection to every -remote SFU. That model is the LiveKit multi-SFU strategy, so the seam has to sit -above it. Cascading keeps the membership transport exactly as it is and only -changes which SFUs the client connects to, which makes the connection manager a -LiveKit-internal policy. Full mesh has one peer connection per remote member and -no token service at all. - -This plan introduces one interface, `MediaBackend`, created once per session. It -takes the memberships and the local media inputs and hands back the media the -session consumes, plus the three things the MatrixRTC join needs from the media -side: the transport to advertise, whether the backend can take over the delayed -leave, and the handover of the delayed leave once its id is known. Everything -LiveKit moves into `sdk/src/backend/livekit/` behind it. The public API does not -change, behaviour does not change, and the `playwright/sdk` specs verify every step. - -This is slice 12 in the status table of [`SdkMigration.md`](./SdkMigration.md). It -is independent of slices 6 to 11 there, and slice 7 (a neutral processor type) and -slice 11 (the error renames) get easier once the folder exists. - -## What is LiveKit today - -Everything under `sdk/src/session/` that imports `livekit-client`, -`@livekit/components-core`, or speaks the LiveKit token protocol: - -| Module | What it is | -| ----------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `media/LivekitMediaTrack.ts`, `media/LivekitMemberMedia.ts` | Participant to `MemberMedia`. Clean already. | -| `session/Connection.ts`, `ConnectionFactory.ts`, `ConnectionManager.ts` | One LiveKit room per transport URL, the multi-SFU policy, the token fetch per connection. | -| `session/MatrixLivekitMembers.ts` | Matches a participant identity to a membership, epoch-paired. | -| `session/Publisher.ts`, `publishOptions.ts` | Publish requests onto the local participant, upstream pausing, audio output switching, the capture and publish option builders. | -| `session/KeyProvider.ts`, `MatrixKeyProvider.ts` | LiveKit's key provider on the js-sdk key stream. | -| `session/DataChannel.ts` | LiveKit data packets. | -| `session/openIDSFU.ts`, `livekitOptions.ts` | The JWT service protocol, including delegation, and room options. | -| `session/LocalTransport.ts` | The pre-join token fetch. Discovery itself is a homeserver endpoint and stays; see "Discovery stays in the session". | -| `session/RtcTransportAutoDiscovery.ts` | Generic apart from its `isLivekitTransport` filter and the fallback URL. Stays. | -| `session/LocalMember.ts` | `participant$`, `connection$`, upstream pausing, the delegation probe and handover, and `publish` resolving with a `LocalTrackPublication`. Roughly 120 of 490 lines. | -| `session/Members.ts` | `mediaFor` takes a participant and a room; `publish` maps a `trackSid` back to a track. | -| `session/Transports.ts` | Transport id, the connected check and `resolved$` assume LiveKit. | -| `session/SessionBehaviors.ts` | `membershipsAndTransports$` filters to LiveKit transports. Deleted in step 3, not moved. | -| `session/MatrixRTCClient.ts` | Wires all of the above. | -| `utils/test.ts` | `mockRemoteParticipant`, `exampleTransport`. | - -`errors.ts` has a type-only import for `LivekitConnectionError`; that is slice 11's -business and stays. `index.ts` exports `getSFUConfigWithOpenID` as -`authenticateWithTransport` for the developer panel; it keeps being exported, from -the backend folder. - -## The interface - -All of it lives in `sdk/src/backend/api.ts`. Types not defined here come from -`sdk/src/api.ts` (`MemberMedia`, `LocalMemberMedia`, `LocalMediaInputs`, -`ResolvedTransport`), `sdk/src/config.ts` (`SessionTimings`), -`sdk/src/encryption.ts` and matrix-js-sdk (`Transport`, -`CallMembershipIdentityParts`). Memberships reach a backend as the SDK's -`RTCMembership`, whose pick covers the two things a backend reads, -`rtcBackendIdentity` and `getTransport()`. The js-sdk session itself never reaches a -backend: `MatrixKeyProvider` reads three things from it today, the -`EncryptionKeyChanged` event, one `reemitEncryptionKeys()` call to replay the -keys it already holds, and the room id for a log line, and `mediaKeys$` covers -all three. - -### Connection state, backend neutral - -Replaces `ConnectionState` from `Connection.ts` inside the session. The LiveKit -backend maps its room state onto it; `Reconnecting` and `SignalReconnecting` both -become `Reconnecting`. - -```ts -export enum MediaConnectionState { - /** No connection attempt yet, or no connection at all. */ - Initialized = "initialized", - /** Fetching whatever the connection needs first; the SFU token for LiveKit. */ - Preparing = "preparing", - Connecting = "connecting", - Connected = "connected", - Reconnecting = "reconnecting", - Disconnected = "disconnected", - Stopped = "stopped", -} -``` - -### What the session hands the backend - -```ts -export interface MediaBackendContext { - roomId: string; - ownMembershipIdentity: CallMembershipIdentityParts; - /** What the local member publishes; the exact type is below. */ - localMedia: LocalMediaInputs; - encryptionSystem: EncryptionSystem; - /** - * Every media key the session knows: the current ones replayed on - * subscribe, new ones as they arrive. The backend feeds them into its own - * frame encryption. Empty for a shared key, which is in `encryptionSystem`. - */ - mediaKeys$: Observable; - /** For the delegated leave's timeout, and whatever else a backend times. */ - timings: SessionTimings; - logger: Logger; -} - -/** One per-participant media key, as the js-sdk's `EncryptionKeyChanged` carries it. */ -export interface MediaKey { - /** The `rtcBackendIdentity` of the member the key belongs to. */ - participantId: string; - index: number; - key: Uint8Array; -} - -/** - * The type of the `backend` option: a function the session calls once with - * its scope and the context to get the backend. Only an alias, so the option - * and a second backend's module do not spell the signature out. A backend's - * own factory keeps the codebase's shape, - * `createLivekitBackend(scope, context, options): MediaBackend`, and the host - * closes over its options: `(scope, ctx) => createLivekitBackend(scope, ctx, - * { client, tokenEndpoint: "msc4195" })`. No currying, so `createX` returns - * an X here as everywhere else. - * - * The Matrix client is not in the context: a backend that needs it takes it in - * its own options, as a `Pick` of exactly what it calls, so the type says what - * each backend reaches for. LiveKit picks the OpenID token and the base URL; - * a mesh backend would pick to-device sending. - */ -export type MediaBackendFactory = ( - scope: ObservableScope, - context: MediaBackendContext, -) => MediaBackend; -``` - -`LocalMediaInputs` is unchanged from `sdk/src/api.ts` and is the only input the -backend reads about the local media. Since the `MemberMediaPlan.md` slices, the -devices and the processor are no longer behaviors the SDK watches: they are -fields of the initial requests, and later changes go to the track's own -`setDevice` and `setProcessor`. - -```ts -export interface LocalMediaInputs { - /** Published at the join; `publish` on the local member adds to it from then on. */ - publish: PublishRequest[]; - /** Undefined where the host routes audio itself, or to leave the browser's choice. */ - audioOutputDeviceId$: Behavior; -} - -export type PublishRequest = - | { source: "microphone"; deviceId?: string; capture?: AudioCaptureSettings } - | { - source: "camera"; - deviceId?: string; - processor?: TrackProcessor; - capture?: VideoCaptureSettings; - } - | { source: "screenShare"; audio?: boolean; capture?: VideoCaptureSettings }; -``` - -The processor type in the camera request is the LiveKit type on the public API; -slice 7 of the migration. - -### The backend - -```ts -/** What preparing the local transport established; what the MatrixRTC join needs from the media side. */ -export interface TransportCapabilities { - /** - * Whether the backend can restart the delayed leave on the client's behalf, - * so that a client that vanishes is removed by the backend rather than by - * the timeout. Decides which delayed leave timings the join uses, and - * whether `delegateDelayedLeave` is ever called. - */ - canDelegateDelayedLeave: boolean; -} - -/** One connection a backend holds. */ -export interface BackendConnection { - /** The transport as it appears in memberships; what the registry keys on. */ - transport: Transport; - state: MediaConnectionState | Error; - /** - * What the backend fetched to connect; undefined until it has, and again - * after the connection stops. One stable object per fetch, so that the - * registry's `distinctUntilChanged` holds across state changes. - */ - resolved: ResolvedTransport | undefined; -} - -export interface MediaBackend { - /** The membership transport type this backend serves: "livekit". */ - readonly transportType: string; - - /** - * Makes `transport` usable as the one the local member publishes on. The - * session found it; the backend does whatever has to succeed before it is - * advertised in the membership. Resolves with what the join needs to know, - * rejects with the error the session reports as fatal. Called once per - * session. For LiveKit: the OpenID and JWT exchange, whose result the - * backend keeps for its own connection, and the delegation probe. - */ - prepareLocalTransport(transport: Transport): Promise; - - /** - * Hands the delayed leave event to the backend, which restarts it from - * then on. Called by the session each time the membership manager reports - * a new delay id, only when `canDelegateDelayedLeave` is true. A rejection - * is logged, not fatal: the client keeps restarting the leave itself. - */ - delegateDelayedLeave(delayId: string): Promise; - - /** The local member's side; the exact type is below. */ - readonly local: LocalMediaBackend; - - /** - * The media of one remote member, null while nothing has arrived for it on - * its transport ("waiting for media"). Built in `scope`, which the caller - * ends with the member. The backend matches on `rtcBackendIdentity` and the - * membership's transport, both read from `membership$`. - * - * This is also how the backend learns which remote transports exist: it - * registers the membership's transport for as long as `scope` lives, and - * connects to the union of registered transports plus the local one. No - * membership list reaches the backend; the session decides which members - * exist and asks for their media. - */ - mediaFor$( - scope: ObservableScope, - membership$: Behavior, - ): Behavior; - - /** - * Every connection the backend holds, for diagnostics. The session's - * transport registry derives `connectedTransports$` from the entries in - * `Connected` state, and each `TransportMetadata.resolved$` from the entry - * whose transport has the same canonical id, with `distinctUntilChanged`. - */ - readonly connections$: Behavior; - - /** - * A text channel beside the media. Sent on the local transport, rejected - * while it is not connected; received on every transport. The session maps - * `senderId` to a member and drops what no member sent. - */ - sendData(topic: string, text: string): Promise; - readonly data$: Observable<{ senderId: string; topic: string; text: string }>; -} -``` - -There is no `transportId` on the backend. The session's transport registry keys -on a canonical serialisation of the raw transport object, sorted keys. Transports -are plain JSON out of membership events, so identical transports serialise -identically, and a mesh transport carrying per-member data is correctly distinct. - -### The local side - -This is what `createLocalMembership$` in `session/LocalMember.ts` reads instead of -`participant$`, `connection$`, `connection.livekitRoom` and the publisher. The -join state machine, the Matrix join and leave, the error aggregation, the delay -id signal and the screen share error handling stay in the session; the LiveKit -calls and the per-connection publisher lifecycle move. - -```ts -export interface LocalMediaBackend { - /** - * State of the connection the local member publishes on. `Initialized` - * while there is none. An `Error` is the connection's failure. - */ - readonly connectionState$: Behavior; - - /** The local member's own media. Null until the connection has a local participant. */ - readonly media$: Behavior; - - /** - * Whether the local tracks reach the transport. The session sets it to - * "joined and the homeserver is reachable", so that a member who has left, - * or who may already have been dropped from the session, is not heard. - * Remembered across connections and applied to each new one: the tracks - * are created on the first `true`, and their upstream is paused while - * `false`, so the local preview stays live while nothing is heard. - */ - setPublishing(publish: boolean): void; - /** The first failure to publish, kept until the session ends. Not fatal. */ - readonly publishError$: Behavior; - - /** - * Publishes a source and resolves with its track once it is in `media$`'s - * `tracks$`. Remembered across connections, so a request before the - * transport is up is applied when the connection comes, and republished - * after a reconnection. Rejects where the device could not be used, - * including the user closing the screen picker. Today's `DesiredMedia` - * map lives behind this. - */ - publish( - request: PublishRequest, - ): Promise; - /** Removes one of our tracks by id; a screen share takes its audio with it. */ - unpublish(id: string): Promise; -} -``` - -Muting, device switching and the processor are not on the backend: they are -methods of `LocalAudioMediaTrack` and `LocalVideoMediaTrack`, which the backend -builds in `media$`. The `setEnabled` callback that `createLocalLivekitMemberMedia` -takes today, so that a track's mute keeps the desired map in step, is wired -inside the backend. Screen sharing has no members of its own any more: it is a -`publish` request, and "sharing" is `tracks$` holding a `screenShare` track. - -There is one publish switch, not two. Today `LocalMember.ts` pauses the -upstream from two places: `stopPublishing()` on leave, and a separate -subscription on the homeserver connectivity that pauses every publication while -the sync is down and resumes every publication when it is back. Both are the -same LiveKit operation, and the second does not look at the join intent: after -`leave()` the tracks still exist with their upstream paused, so a sync hiccup -resumes them and a member who has left is heard again. The session computes -the one boolean, joined and reachable, and the backend applies it. The cost is -that the tracks are now created on the first `true`, so a join while the sync is -down waits for the sync before asking for devices. A join cannot send its -membership without the sync either, and pre-join track creation is slice 6's. - -There is no publisher on the interface. Today's `Publisher` wraps one LiveKit -room's local participant and so lives and dies with the local connection, and -`LocalMember.ts` re-applies the join intent to each new one. That lifecycle is a -LiveKit detail: the backend keeps the publish intent, as it keeps the desired -requests for `publish`, and re-applies both to every connection it opens. A -mesh backend has no publisher object at all. - -### Delayed leave delegation - -How it works today, all in `LocalMember.ts` and `openIDSFU.ts`: - -1. The js-sdk membership manager sends the membership and a delayed leave event, - and emits the leave's delay id (`MembershipManagerEvent.DelayIdChanged`), again - whenever it has to send a new one. `MatrixRTCClient.ts` turns that into - `delayId$`. -2. When the local transport is known, the session probes two MSC4195 endpoints - without credentials: the homeserver's - `/_matrix/client/unstable/io.element.msc4195/rtc/livekit/delegate_delayed_leave` - and the transport's `/delegate_delayed_leave`. A 404 means - no support. Either one supporting it is enough. -3. The join uses `timings.delegatedDelayedLeave` if supported, else - `timings.delayedLeave`. The difference is an hour versus eighteen seconds of - delay, because a delegated leave is restarted by the SFU and the client stops - restarting it. -4. On every delay id, if supported, the session calls the token service again with - `delay_id`, `delay_timeout` and the homeserver's URL. The token it returns is - discarded; the request is the handover. Failure is logged and the client keeps - restarting the leave itself. - -Steps 1 and 3 are MatrixRTC and stay in the session. Steps 2 and 4 are the LiveKit -token protocol and move behind the interface, as two members rather than one: - -- The probe goes into `prepareLocalTransport`, reported as `canDelegateDelayedLeave`. - It needs the transport's URL and the homeserver's, both of which the backend has - at that moment, and the join needs its answer before it fires. The backend runs - the probe concurrently with the token fetch, as today. -- The handover is `delegateDelayedLeave(delayId)`, because the delay id arrives - after the join and changes over the session's life. The session owns that signal - and drives the method from it. - -The session side after the change: - -```ts -// discoverLocalTransport is the session's; see "Discovery stays in the session" -const prepared$ = from( - discoverLocalTransport(client, backend.transportType, options, logger).then( - async (transport) => ({ - transport, - ...(await backend.prepareLocalTransport(transport)), - }), - ), -); // errors become fatalTransportError$ - -scope.reconcile( - combineLatest([prepared$, joinRequested$]), - ([prepared, join]) => { - if (prepared === null || !join) return; - joinMatrixRTC( - prepared.transport, - prepared.canDelegateDelayedLeave - ? timings.delegatedDelayedLeave - : timings.delayedLeave, - ); - return leaveOnCleanup; - }, -); - -scope.reconcile( - combineLatest([prepared$, delayId$]), - async ([prepared, delayId]) => { - if (!prepared?.canDelegateDelayedLeave || delayId === null) return; - try { - await backend.delegateDelayedLeave(delayId); - } catch (e) { - logger.error("Failed to delegate the leave", e); - } - }, -); -``` - -A mesh backend answers `canDelegateDelayedLeave: false` and is never asked to -delegate. Should a backend later delegate to something other than a token -service, it does so behind the same method. - -### Lifetimes: what ends what - -`mediaFor$` is bound to the scope it is given, not to its subscribers. -`scope.behavior` subscribes eagerly and tears down with the scope, so the -behavior and the transport registration live exactly as long as the member -does, whether or not a view reads them. Lazy registration on first subscribe -would be wrong: the audio renderer needs the microphone track of a member -nobody has a tile for, and that member's transport would never connect. - -- **A remote member leaves.** `generateItems` ends the member's scope. From - that one scope the `mapScoped` item ends and detaches the `MemberMedia`, the - behavior completes, and the `onEnd` callback releases the transport. If the - count for that transport hits zero the connection manager drops it, and the - connection's own scope ending calls `stop()`, as today. -- **A member changes transport.** Registration follows `membership$` with - `distinctUntilChanged` on the transport id: release the old, register the - new, inside the same scope. -- **The local member calls `leave()`.** Nothing changes on the remote side. - The memberships are still in the room, so the members still exist and the - connections stay open. That is today's behaviour too: the connection - manager never looked at `joinRequested$`. -- **The network drops.** The member still exists, so `mediaFor$` is live. The - connection reports reconnecting, its participant list empties, the - participant behind the member becomes null, the media item ends and - `media$` reads null, "waiting for media". A new `MemberMedia` is built on - reconnect. As today. -- **The session scope ends.** The backend and every member scope end in the - same tick, in subscription order, so the backend's internals tear down - before the members' release callbacks run. Release must therefore do - nothing but update the registry map and `next` a subject, both no-ops with - nobody listening. It must never stop a connection or touch LiveKit itself; - the connection scopes do that. - -### Where the LiveKit backend gets each member from - -| Interface member | Today | Moves from | -| -------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------- | -| `prepareLocalTransport` | OpenID and JWT fetch kept as `existingSFUConfig`, delegation probe | `LocalTransport.ts`, `LocalMember.ts` | -| `delegateDelayedLeave` | `getSFUConfigWithOpenID` with `delayId` | `LocalMember.ts` | -| `local.connectionState$` | `connection$.state$` for the local transport | `LocalMember.ts` | -| `local.media$` | `mediaFor(participant$, connection$)` with `createLocalLivekitMemberMedia` | `Members.ts` | -| `local.setPublishing`, `publishError$` | `scope.reconcile(connection$, createPublisher)`, the `[publisher$, joinRequested$]` reconcile with its `publishError$`, and the homeserver-connectivity loop over `participant.trackPublications` | `LocalMember.ts`, `MatrixRTCClient.ts` | -| `local.publish`, `unpublish` | `publish` and `unpublish` with the `DesiredMedia` map and `publication$`, and the `trackSid` to track lookup | `LocalMember.ts`, `Members.ts`, `Publisher.ts` | -| internal | the `setEnabled(source, enabled)` callback into `createLocalLivekitMemberMedia` | `LocalMember.ts`, `Members.ts` | -| `mediaFor$` | `createRemoteMatrixLivekitMembers$` matching plus `mediaFor` | `Members.ts`, `MatrixLivekitMembers.ts` | -| `connections$` | `connectionManagerData$` joined with each connection's `state$` and `ResolvedConnection.resolved$` | `Transports.ts`, `ConnectionFactory.ts` | -| `sendData`, `data$` | `createDataChannel$` minus the member lookup | `DataChannel.ts` | -| internal | key provider on `mediaKeys$` instead of `setRTCSession`, room options, E2EE worker, connection manager fed by the transports registered through `mediaFor$` | `KeyProvider.ts`, `MatrixKeyProvider.ts`, `livekitOptions.ts`, `ConnectionFactory.ts`, `ConnectionManager.ts` | - -### Discovery stays in the session - -Which transport the local member advertises is answered by the homeserver's -MSC4143 transport list, and that list can name transports of any type. So -discovery is the session's, in `session/LocalTransport.ts`, and a backend is -only asked to prepare what the session picked: - -```ts -/** In session/LocalTransport.ts. The homeserver's list, retried, filtered to what the backend serves. */ -export async function discoverLocalTransport( - client: Pick, - transportType: string, - options: Pick< - MatrixRTCClientOptions, - "transportUrl" | "fallbackTransportUrl" - >, - logger: Logger, -): Promise; // throws MatrixRTCTransportMissingError -``` - -`RtcTransportAutoDiscovery` does the list fetch today and stays where it is, -with its `isLivekitTransport` filter replaced by a filter on `transportType`. - -The two URL options are the awkward part. `transportUrl` and -`fallbackTransportUrl` are LiveKit service URLs, and turning a URL into a -membership transport means building `{ type: "livekit", livekit_service_url }`. -That is LiveKit knowledge, but it is a workaround for hosts that configure a URL -rather than a transport, and it lives in `discoverLocalTransport`, not in -`backend/livekit/`. The backend folder never learns where its transport came -from. It is one `Transport` literal and the js-sdk's `LivekitTransport` type, -no `livekit-client` import, so the lint guard of step 6 is not affected. The -clean follow-up, outside this plan, is `transport?: Transport` and -`fallbackTransport?: Transport` on the client options, with the host building -the literal; then the workaround is deleted. - -### LiveKit's own options - -`mediaQuality` and `capture` on `MatrixRTCClientOptions` configure LiveKit, not -MatrixRTC. They become `LivekitBackendOptions`, the third parameter of -`createLivekitBackend(scope, context, options)`, together with the slice of the -Matrix client the backend calls. They also stay on `MatrixRTCClientOptions`, where they feed the -default backend when the host passes no `backend`, so Element Call's -`InCallView` does not change. A host that supplies its own backend configures -it itself, client included. `transportUrl` and `fallbackTransportUrl` are -discovery inputs and stay on the client options for good, as above. - -```ts -export interface LivekitBackendOptions { - /** - * `getOpenIdToken` for the token exchange, `baseUrl` for the delegation - * probe and handover. Nothing else: discovery is the session's. This is - * today's `OpenIDClientParts` from `openIDSFU.ts` minus its unused - * `getDeviceId`, plus `baseUrl`. - */ - client: Pick; - mediaQuality?: MediaQuality; - /** - * Which token service endpoint to use for the local transport: the MSC4195 - * one, or the legacy one with the old identity format. Remote transports - * always try both. The default backend derives it from the client's - * `matrixRTCMode`; the mode itself never reaches a backend, since what it - * also decides, sticky events and the member id format, is the session's. - */ - tokenEndpoint: "msc4195" | "legacy"; -} -``` - -Today `openIDSFU.ts` takes the `MatrixRTCMode` itself and reads the endpoint -choice off it. Step 5 changes it to take `tokenEndpoint`, with the mapping -`Matrix_2_0` to `msc4195` and `Compatibility` to `legacy` done once where the -default backend is built, in `MatrixRTCClient.ts`, which also hands the -client through. The exported `authenticateWithTransport` keeps its signature by -doing the same mapping. - -## Transition - -Six steps, each green on its own and each a PR of under 400 lines. There is one -connection manager at every point: the backend owns it from step 2 on, and the -consumers not yet switched read it through a transitional field until step 4 -removes it. Running the old wiring and the backend side by side would open a -second connection to every SFU, so the plan never does that. - -### Step 1: carve the folder - -Pure move, no code change beyond import paths. - -- `git mv` into `sdk/src/backend/livekit/`: `media/*`, `Connection`, - `ConnectionFactory`, `ConnectionManager`, `DataChannel`, `KeyProvider`, - `MatrixKeyProvider`, `MatrixLivekitMembers`, `Publisher`, `publishOptions`, - `livekitOptions`, `openIDSFU`, each with its test. `utils/tracks.ts` is - neutral and stays. `RtcTransportAutoDiscovery` and - `LocalTransport` stay in `session/`. -- Split `utils/test.ts`: `mockRemoteParticipant` and `exampleTransport` go to - `backend/livekit/test.ts`. -- `index.ts` re-exports `authenticateWithTransport` from the new path. - -Check: `pnpm lint`, `pnpm test`, the `playwright/sdk` specs. - -### Step 2: the interface, the backend shell, the registry, the data channel - -- Add `sdk/src/backend/api.ts` with the types above. -- Add `sdk/src/backend/livekit/LivekitBackend.ts` exporting - `createLivekitBackend(scope, context, options): MediaBackend`. It owns what - `MatrixRTCClient.ts` builds today: the key provider, the - `LivekitConnectionFactory` and the connection manager. Until step 3 the - connection manager is still fed from `membershipsAndTransports$`, passed in - as a transitional `remoteTransports$` input beside the transitional - `connectionManager` field; both go in step 3. It implements - `transportType`, `connections$`, `sendData` and `data$`. `ResolvedConnection` - keeps one `ResolvedTransport` object per fetch instead of rebuilding it on - every state change. -- `MatrixRTCClient.ts` creates the backend, takes the connection manager from - it, and maps `data$` senders to members itself. It builds `mediaKeys$` from - the js-sdk session: `fromEvent(EncryptionKeyChanged)` with a - `reemitEncryptionKeys()` call on subscribe. `MatrixKeyProvider` subscribes - to that instead of taking the session; its test feeds a `Subject`. -- `Transports.ts` derives both `connected$` and each `TransportMetadata.resolved$` - from `backend.connections$`, keys the registry on the canonical JSON of the - raw transport, and loses `ResolvedConnection`, `isLivekitTransport` and - `ConnectionState`. - -New unit test: `LivekitBackend.test.ts` with a fake LiveKit room through the -existing `livekitRoomFactory` seam, covering `connections$` through a connect, -a reconnect and a stop, and the data channel. - -### Step 3: remote media - -- Implement `mediaFor$` in the LiveKit backend. It registers the membership's - transport, following `membership$` and released when `scope` ends; looks - the transport and identity up in `connectionManagerData$`; - `distinctUntilChanged` on the participant and room pair; then `mapScoped` - into `createLivekitMemberMedia`. The "participant matched or missing" log - line moves here. -- The connection manager's `remoteTransports$` becomes the registered set: - a `Map` of transport id to count, exposed as a behavior of its keys, with - `membershipsAndTransports$`'s LiveKit filter and dedup applied there. The - transitional input from step 2 goes. -- `Members.ts` drops `mediaFor` and the participant and room parameters; - `createRemoteRTCMember` takes `backend.mediaFor$(scope, membership$)`. -- `MatrixRTCClient.ts` generates `remoteMembers$` straight from `memberships$` - minus the local one, keyed by `membershipKeys`, and stops using - `createRemoteMatrixLivekitMembers$`. `membershipsAndTransports$` is deleted - from `SessionBehaviors.ts`. - -The epoch pairing that `createRemoteMatrixLivekitMembers$` does today is not -needed any more. It existed to keep a membership list and the connection data -from different ticks apart; now a member's media depends on its own membership -only, and the connections exist because members asked for them. A connection -that does not exist yet reads as null, which is "waiting for media" and correct. -What has to hold is that `MemberMedia` instances stay stable while the -participant and room do not change, which the `distinctUntilChanged` above -guarantees, and that a transport stays registered while any member names it, -which the count does. `media.spec.ts` is the check that nothing flickers. - -New unit test in `LivekitBackend.test.ts`: two members on one remote transport -open one connection, it survives the first member's scope ending, and closes -with the second's. - -### Step 4: local media - -- Implement `local` in the LiveKit backend: `connectionState$` from the local - transport's connection mapped onto `MediaConnectionState`, `media$` through - `createLocalLivekitMemberMedia`, `setPublishing` and `publishError$` by - reconciling a `Publisher` per connection against the remembered intent, - which is the two reconciles from `LocalMember.ts` moved as they are, and - `publish` and `unpublish` over a backend-owned `DesiredMedia` map and the - current publisher, with the `trackSid` to track lookup that `Members.ts` - does today done inside. The `setEnabled` callback into - `createLocalLivekitMemberMedia` is wired here too. The - homeserver-connectivity loop is not moved; `setPublishing` covers it. -- `LocalMember.ts` takes `backend.local` instead of `connectionManager`, - `createPublisher` and `desired`. Its publisher reconcile and the pause and - resume loop become one subscription, - `joinRequested$ && homeserverConnected.combined$` into - `backend.local.setPublishing`, and `backend.local.publishError$` feeds - `state$` where the local subject did. It loses `participant$`, - `connection$`, the `livekitRoom` access, the pause and resume loop, - `publish`, `unpublish`, `setEnabled` and `publication$`, and compares - against `MediaConnectionState.Connected` in `mediaState$` and - `disconnectReason$`. `LocalMembership` shrinks to the join state machine - and no longer names a LiveKit type. -- `Members.ts`: `createLocalRTCMember` forwards `publish` and `unpublish` to - `backend.local` and drops its own `trackSid` lookup. -- `MatrixRTCClient.ts` stops passing `createPublisher`, `desired` and the - connection manager, and stops building the `DesiredMedia` map. The - transitional field goes. -- `status.test.ts` moves to `MediaConnectionState`. - -New unit test: `LocalMember.test.ts` against a fake `LocalMediaBackend`. This is -the first time the local state machine is testable without LiveKit; cover join -before the connection is up, connection loss while publishing, a sync outage -after `leave()` leaving `setPublishing` at false, and a rejected -local transport. - -### Step 5: the local transport and the delegation - -- `session/LocalTransport.ts` becomes `discoverLocalTransport` as above: the - URL override, else `RtcTransportAutoDiscovery` filtered on the backend's - type, else the fallback URL. The token fetch leaves it, and so does the - `LocalTransport` pair type with its `sfuConfig`. -- `RtcTransportAutoDiscovery` takes a `transportType` and returns a - `Transport`; its `isLivekitTransport` filter goes. -- Implement `prepareLocalTransport` in the LiveKit backend from the token - fetch in `getLocalTransport` and the two `checkDelegationSupport` calls, - the probe running concurrently with the token fetch. The SFU config is - kept for the local connection, keyed by service URL, replacing - `existingSFUConfig` flowing through `LocalTransport`. -- Implement `delegateDelayedLeave` from the handover `reconcile` in - `LocalMember.ts`. -- `MatrixRTCClient.ts` chains discovery and preparation into `prepared$` as - sketched above. -- `LocalMember.ts` takes `prepared$` instead of `localTransport$`, drops - `joinParams$`, `checkDelegationSupport`, `client`, `roomId`, - `ownMembershipIdentity` and `matrixRTCMode` from its props, and drives - `backend.delegateDelayedLeave` from `delayId$`. -- `joinJsSdkSession` takes a `Transport`. `encryptMedia` derives from - `encryptionSystem.kind`, not from whether a key provider exists. - -New unit tests: in `LivekitBackend.test.ts`, the probe outcomes (homeserver -supports it, only the transport does, neither, probe throws) and the handover -request body; in `LocalMember.test.ts`, that the join picks the delegated timings -and that a new delay id triggers a handover. - -### Step 6: the `backend` option and the guard - -- `MatrixRTCClientOptions.backend?: MediaBackendFactory`. When absent, - `MatrixRTCClient.ts` supplies - `(scope, ctx) => createLivekitBackend(scope, ctx, livekitOptions)`, with - `livekitOptions` built from the client and the LiveKit options on the client - options. -- Extend `eslint/SdkImportBoundary.js`: under `sdk/src/session/`, in - `sdk/src/api.ts` and in `sdk/index.ts`, `livekit-client` and `@livekit/*` are - banned except as `import type`. From here on the seam is enforced, not - documented. -- `SdkArchitecture.md` gets a "Media backend" section built from this plan, the - README's description of what imports LiveKit is updated, and `SdkMigration.md` - records the slice as done. - -## After this - -- Full mesh or a cascading SFU is a new folder under `sdk/src/backend/` with - its own `createMeshBackend(scope, context, options)`, handed in as a - `MediaBackendFactory`; nothing in `session/` changes. -- A member whose transport type has no backend gets null media. That is what - happens today through the `isLivekitTransport` filter; now it is explicit. -- Slice 7, the neutral processor type, and slice 11, the error renames, each - touch one folder. -- Several backends at once, grouping memberships by transport type, fits the - interface but is out of scope until a second backend exists. diff --git a/sdk/MemberMediaPlan.md b/sdk/MemberMediaPlan.md deleted file mode 100644 index a5faf231d..000000000 --- a/sdk/MemberMediaPlan.md +++ /dev/null @@ -1,457 +0,0 @@ -# `MemberMedia`: a list of tracks instead of a call's four slots - -`MemberMedia` in `sdk/src/api.ts` is written in the vocabulary of a video call. It -has four named slots, `microphone$`, `camera$`, `screenShare$` and -`screenShareAudio$`, and two member-level flags, `speaking$` and -`screenShareEnabled$`. A MatrixRTC session is not always a call: a whiteboard that -streams one video track, a translation bot with two audio tracks, a member -publishing two cameras, or a future application the SDK has not heard of all have -media, and none of it fits the four slots. The flags are worse than the slots. -`screenShareEnabled$` says only that the `screenShare$` slot is filled, and -`speaking$` names the microphone's activity after what people do with it in a call. - -This plan replaces the slots and the flags with one `tracks$` list, and moves the -activity signal to where it is measured: `isActive$` on the audio track. The public -API changes; behaviour does not; the `playwright/sdk` specs and the dev harness -verify it. - -The SDK's consumers of these fields today are the dev harness (`sdk/dev/main.ts`) -and, through it, the specs. Element Call's media view models on this branch still -read the LiveKit participant directly, so nothing under `src/` changes. This is the -cheapest moment to make the change: once Element Call consumes `MemberMedia`, every -view model in `src/state/media/` will be a call site. - -> Note for the status table of [`SdkMigration.md`](./SdkMigration.md): it records -> slice 4 (view models take `media$`) as done, but on this branch -> `src/state/media/*.ts` still imports `livekit-client` and `CallViewModel.ts` -> does not import the SDK. Either the table is ahead of this branch or the slice -> lives elsewhere; the table should say which before this plan is added to it. - -## What the tracks already know - -The four slot getters were one line each: `participant.getTrackPublication( -livekitSources[source])`. That is a lookup by `source` in the participant's -`trackPublications` map. Everything the slot name told the consumer, the track -already carries: - -| On the LiveKit publication | On `MediaTrack` today | Set by | -| -------------------------- | ---------------------- | ----------------------------------------------------------------------- | -| `source: Track.Source` | `source: MediaSource` | the publisher, at publish time; the SFU forwards it to every subscriber | -| `kind: Track.Kind` | `kind` | the media stream track | -| `trackSid` | `id` | the SFU, stable for the life of the publication | -| `trackName` | not exposed | the publisher; free-form, empty for everything Element Call publishes | -| `mimeType`, `dimensions` | not exposed | the SFU; `stats$` carries the live values, these are what was announced | -| `isMuted`, `isEncrypted` | `muted$`, `encrypted$` | the publisher and the SFU | - -So a flat list is lossless: `tracks$.find((t) => t.source === "camera")` is exactly -what `camera$` returned, and `tracks$.some((t) => t.source === "screenShare")` is -exactly `screenShareEnabled$`, since LiveKit's `isScreenShareEnabled` is defined as -`!!getTrackPublication(Track.Source.ScreenShare)`, presence and not mute state. - -One thing the slots lost that the list keeps: a publication with -`Track.Source.Unknown` never appeared in any slot. The list includes it, so -`MediaSource` grows a fifth value, `"unknown"`. A consumer that wants the call -vocabulary filters for the four names it knows; nothing is dropped on the way. - -What the list does not try to do: the `trackName` and any application-specific -meaning stay out of the interface for now. Whether a non-call application tags -its tracks through the name, through the membership's `applicationData`, or -through a MatrixRTC-level description is a question for when such an application -exists; see "Other possible call-related refactors". - -## The interface - -```ts -export type MediaSource = - | "microphone" - | "camera" - | "screenShare" - | "screenShareAudio" - /** Published without a source; the application knows what it is. */ - | "unknown"; - -export interface AudioMediaTrack extends MediaTrack { - kind: "audio"; - /** - * Whether the track carries sound right now, as the backend measures it. - * False while muted. A call reads this on the microphone track and calls - * it "speaking". - */ - isActive$: Behavior; - setAudioContext(ctx: AudioContext | undefined, plugins?: AudioNode[]): void; - setVolume(volume: number): void; -} - -/** - * The media of one member. One entry per published track, in publication - * order; an entry is the same object for as long as the same publication is - * behind it. There is no identity field: the backend identity is the member's - * `id`. - */ -export interface MemberMedia { - local: boolean; - tracks$: Behavior<(AudioMediaTrack | VideoMediaTrack)[]>; - /** Emits when the SFU reports a key problem for this member. */ - encryptionError$: Observable; -} - -export interface LocalMemberMedia extends MemberMedia { - local: true; - switchCamera(): Promise; -} -``` - -Removed: `speaking$`, `screenShareEnabled$`, `microphone$`, `camera$`, -`screenShare$`, `screenShareAudio$`. `switchCamera` stays in this plan because it -is a different question (see the refactors section). - -### One helper, exported - -Every consumer will want "the member's track for a source, as a behavior". The -harness needs it twice already, and Element Call will need it in every media view -model. Rather than let each caller write the `map` + `distinctUntilChanged` chain, -the SDK exports one function from `sdk/src/utils/tracks.ts`: - -```ts -/** The first track of a source, undefined while there is none. */ -export function trackBySource$( - scope: ObservableScope, - tracks$: Behavior, - source: S, -): Behavior | undefined>; -``` - -`TrackOfSource` narrows `"microphone" | "screenShareAudio"` to `AudioMediaTrack` -and `"camera" | "screenShare"` to `VideoMediaTrack`, so the typed getters the slots -gave are not lost. It is a pure function over `Behavior`s, tested with marbles. -Nothing else is added; `some`, `filter` and `find` on the array cover the rest. - -## `isActive$`: how it replaces `speaking$` - -### What LiveKit measures - -The SFU runs speaker detection from the audio-level RTP header extension on the -audio it forwards, and pushes `SpeakerInfo { sid, level, active }` updates to every -client. `livekit-client` turns them into `participant.audioLevel`, -`participant.isSpeaking` and `ParticipantEvent.IsSpeakingChanged` (see -`Room.handleActiveSpeakersUpdate` and `handleSpeakersChanged`). The `sid` is the -participant's; there is no per-track speaking signal in the client API. `speaking$` -today is `IsSpeakingChanged` mapped to `isSpeaking`, nothing more. - -### What the SDK exposes - -`isActive$` on an audio track is the participant's speaking signal gated by the -track's own mute state: - -``` -isActive = participant.isSpeaking && !publication.isMuted -``` - -For the LiveKit backend this maps one participant-level signal onto every audio -track of the member. In practice a member has one audio track, the microphone, and -the mapping is exact. A member that also shares a screen with audio gets the same -value on both tracks while either carries sound; the mute gate keeps a muted -microphone dark while the screen share plays. This is a documented approximation of -the LiveKit adapter, not of the interface: a full-mesh backend has a peer connection -per remote member and measures each track's level itself, and a stats-based -refinement exists for LiveKit too (below). The track is the right place for the -behavior because that is where every backend other than a single SFU would produce -it. - -### Why a call loses nothing - -"Speaking" is a call's reading of "the member's microphone track is active". The -replacement for `media.speaking$` is: - -```ts -trackBySource$(scope, media.tracks$, "microphone").pipe( - switchMap((track) => track?.isActive$ ?? of(false)), -); -``` - -False when there is no microphone track, which is what `isSpeaking` reported in -that case too. Element Call's 1 s / 60 s hysteresis (`observeSpeaker$`) and the -spotlight speaker selection stay in the call view model, where they are: they are -call UX, and they consume a boolean exactly as before. - -What is gained: screen share audio can show activity ("the presenter's video has -sound"), and an application with several audio tracks per member can tell them -apart once the backend can. What is lost: nothing the SFU knew. The signal was -never per track in LiveKit; the interface now says so instead of pretending the -member speaks. - -### Later refinement, not in this plan - -A genuinely per-track level for LiveKit is available from the RTP statistics the -SDK already polls for `stats$`: `audioLevel` and `totalAudioEnergy` on -`RTCInboundRtpStreamStats`, and on the `media-source` stats for a local track. The -poll is once a second, which the call's hysteresis tolerates but which is too -coarse for an unfiltered indicator, and it costs a stats request per audio track -per second while subscribed. If a second audio track per member ever matters, that -is the first thing to try. - -## What changes, file by file - -### `sdk/src/api.ts` - -The interface above. The comment on `MemberMedia` says the list is in publication -order and that entries are stable per publication; the comment on `isActive$` says -what it measures and names the LiveKit approximation. - -### `sdk/src/media/LivekitMediaTrack.ts` - -- `createLivekitMediaTrack` stops taking `source` as a parameter and derives it - from `publication.source` through the inverse of `livekitSources`, with - `Track.Source.Unknown` as `"unknown"`. -- Audio tracks get `isActive$`: `observeParticipantEvents(participant, -ParticipantEvent.IsSpeakingChanged)` mapped to `isSpeaking`, combined with - `muted$`, as a `scope.behavior` seeded from `participant.isSpeaking && -!publication.isMuted`. -- `setVolume` acts on the track instead of the source. Today it branches on - `source` and calls `RemoteParticipant.setVolume(volume, Track.Source.X)`, which - has nothing to do for an `"unknown"` audio track. `RemoteAudioTrack.setVolume` - exists and is per track; the local no-op stays. `track$` is already a behavior - here, so the volume is reapplied when a remote track arrives on subscription, the - same way the audio context is. - -### `sdk/src/media/LivekitMemberMedia.ts` - -- `speaking$`, `screenShareEnabled$` and the four `track$` calls go; `memberTrack$` - goes with them. -- `tracks$` is built from `observeParticipantMedia(participant)` (it already fires - on publish, unpublish, subscribe and mute) mapped to - `[...participant.trackPublications.values()]`, then through `generateItems` from - `reactive/observable.ts`, keyed by `trackSid`, so each publication gets one - `createLivekitMediaTrack` in a scope that ends when the publication leaves the - map. This is the pattern `MatrixRTCClient.ts` uses for `remoteMembers$`; nothing - new is written for it. -- Publication order is the map's insertion order, which `generateItems` preserves. - -### `sdk/src/utils/tracks.ts` - -`trackBySource$` as above, exported from `sdk/index.ts`. - -### `sdk/dev/main.ts` - -`memberTile` renders from `trackBySource$(scope, tracks$, "camera")` and -`"microphone"`, where `tracks$` is `media$` switched to `media?.tracks$ ?? of([])`. -`render` adds one label, `data-active`, from `isActive$` when the track is audio. -The `member`, `video` and `audio` test ids do not change, so the existing specs -keep passing untouched. - -### Documentation - -- `SdkArchitecture.md`: the `MemberMedia` block in "Media: `MemberMedia`", the - diagram line that lists `speaking$ microphone$ camera$ screenShare$`, and the - harness description that mentions `media$.camera$` and `media$.microphone$`. -- `SdkMigration.md`: a new row in the status table once the branch question above - is settled. -- `MediaBackendPlan.md`: no change; it already treats `media/` as the clean seam and - this plan only makes that seam less call-shaped. - -## Tests - -### Unit - -- `LivekitMediaTrack.test.ts`, existing file: `isActive$` follows - `IsSpeakingChanged`; a muted publication reports `false` while the participant - speaks; `source` is derived from the publication, including `"unknown"`; - `setVolume` reaches the remote track, and again when a track arrives later. -- `LivekitMemberMedia.test.ts`, new: `tracks$` lists every publication in order; - the same object is returned for a publication across unrelated media events (a - mute on another track); a publication replaced under the same source yields a new - object and ends the old one's scope; an unknown-source publication appears with - `source: "unknown"`; screen share presence is derivable with `some`. -- `tracks.test.ts`, new: `trackBySource$` with marbles, including the transition - from undefined to a track and back. - -The fakes come from `sdk/src/utils/test.ts` (`mockRemoteParticipant`) and the -`fakes()` helper already in `LivekitMediaTrack.test.ts`; the latter moves to -`test.ts` if the new file needs it, rather than being copied. - -### End-to-end - -`playwright/sdk/media.spec.ts` gains one test: the peer's `audio` element reaches -`data-active="true"` after a mute and unmute, and `"false"` while muted. It is -marked `fixme`: Chromium's fake microphone never registers as a speaker at the -SFU, and Firefox's steady tone does so in about two runs out of three, with or -without the browser's audio processing. It passed four times in six, which proves -the pipeline, and it stays out of the gate until the dev SFU gets a lower -speaker threshold or the harness a louder fake device. The existing five tests -run as they are. - -## Order of work - -1. `api.ts`, `tracks.ts` and the unit tests for `trackBySource$`. -2. `LivekitMediaTrack.ts`: source from the publication, `isActive$`, per-track - volume, with its tests. -3. `LivekitMemberMedia.ts`: `tracks$`, remove the rest, with its new test file. -4. `dev/main.ts` and the new spec. -5. Documentation. - -Steps 1 to 3 do not compile against the harness until step 4, so they land as one -PR; it is under 400 lines because the media adapter is small. `pnpm lint`, -`pnpm test` and the `playwright/sdk` project are the gates. `knip` will not -complain about `trackBySource$` since the harness uses it. - -## Other possible call-related refactors - -Everything else in `sdk/src/api.ts` and `sdk/src/session/` that only makes sense -for a call. None of it is in this plan; each would be its own slice. The first -three are the ones this plan makes visible. - -| What | Why it is call-specific | What instead | -| --------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `MediaSource` itself | Four call names plus `"unknown"`. LiveKit's `Track.Source` has the same four, so the type is faithful to the wire today, but a whiteboard or a bot has no name for its track. | Keep `source` for the backend's notion; add an optional `name` (LiveKit `trackName`) or a description in `applicationData`, once an application needs it. | -| `LocalRTCMember.sharingScreen$`, `toggleScreenSharing`, `screenShareError$`, `dismissScreenShareError` | A screen share is one publication among others. `sharingScreen$` is `participant.isScreenShareEnabled`, now `tracks$.some(...)` on the local media. | A generic `publish(source, capture)` / `unpublish(id)` on the local member, with its error on the returned promise; `sharingScreen$` derived by the host from `tracks$`. | -| `LocalMemberMedia.switchCamera` | "Camera" is the call's name for the local video track with a `facingMode$`. | `switchFacingMode()` on the local `VideoMediaTrack`, next to `facingMode$`. | -| `MatrixRTCClient.setMicrophoneEnabled` / `setCameraEnabled` | Two fixed publications the client knows by name. | `setEnabled(enabled)` on the local track, or `publish`/`unpublish`; `LocalMember.setEnabled` and `Publisher.setEnabled` already take a `Track.Source` and would take a track id. | -| `LocalMediaInputs` (`microphoneEnabled`, `cameraEnabled`, `audioInputDeviceId$`, `videoInputDeviceId$`, `videoProcessor$`) | Describes exactly one microphone and one camera. `audioOutputDeviceId$` is about playback and is not call-specific. | A list of desired local publications, each with its device, processor and enabled state; `Publisher.desired.microphone$` / `camera$` become entries in it. | -| `CaptureSettings { audio, camera, screenShare }` and `MediaQuality { video, screen_share }` | Keyed by the four sources. | Per-publication capture and encoding options, passed with `publish`. `config.ts` keeps the host-facing `MediaQuality` shape for `config.json`, mapped at the edge. | -| `AudioMediaTrack.setVolume` branching on source | Covered by this plan (per-track volume); listed so the row above it is complete. | Done here. | -| `Track.Source` plumbing in `Publisher.ts` (`pauseUpstreams`, `resumeUpstreams`, the processor on the camera) and `LocalMember.ts` | Internal, but the three sources are enumerated by hand in five places. | Iterate the local publications; attach the processor to every local video track whose publication asks for one. | -| `applicationData` carrying only `m.call.intent`; `application` defaulting to `m.call` | The js-sdk's membership format carries one key. The default is a convenience for the one application that exists. | Pass the record through once the js-sdk can; keep the default but say so in the SDK README. Already an open item in `SdkArchitecture.md`. | -| `membership$: Behavior` on `RTCMember` | The js-sdk's type name. The SDK does not get to rename it, but every host sees the word "call" in a session API. | Re-export under an SDK alias (`RTCMembership`) and narrow to what hosts read: `userId`, `deviceId`, `memberId`, `rtcBackendIdentity`, `getTransport()`, `applicationData`. | -| `UnknownCallError` in `errors.ts` | Name only. | Rename with the two `Livekit*` errors in slice 11 of `SdkMigration.md`. | -| `observeSharingScreen$` in `LocalMember.ts` | Reads `participant.isScreenShareEnabled`. | Goes with `sharingScreen$`, or reads `tracks$` from the local media in the meantime. | -| `sendNotificationType` | Kept on purpose; see "Call-specific things kept out of the client" in `SdkArchitecture.md`. | Nothing. | - -## Plan for the refactors above - -Four slices, in dependency order. Each is one PR, green on its own, with the -`playwright/sdk` specs and the dev harness as the proof that behaviour is kept. -Element Call follows on its own branch after each. - -### 1. Local track controls - -The local member's tracks become objects a host can act on, instead of methods -named after a call's devices. - -- **Add** `LocalAudioMediaTrack` and `LocalVideoMediaTrack` (`local: true`) in - `api.ts`, with `setEnabled(enabled)`, `setDevice(deviceId)` and, on video, - `switchFacingMode()` and `setProcessor(processor)`. `tracks$` on - `LocalMemberMedia` is typed to them. -- **Remove** `LocalMemberMedia.switchCamera`, and `audioInputDeviceId$`, - `videoInputDeviceId$`, `videoProcessor$` from `LocalMediaInputs`. The host - subscribes to its settings and calls the track; the SDK stops watching - behaviors it does not own. `audioOutputDeviceId$` stays: playback is not a - track. -- **Keep** `setMicrophoneEnabled` / `setCameraEnabled` for now: they also cover - "no track yet", which slice 2 solves. -- **Touches** `LivekitMediaTrack.ts`, `Publisher.ts` (the processor and device - code moves onto the track), `MatrixRTCClient.ts`, the harness buttons. - -### 2. `publish` and `unpublish` - -One way to put media on the session, whatever it is. - -- **Add** to `LocalRTCMember`: `publish(request): Promise` and - `unpublish(id): Promise`. A request is `{ source, deviceId?, capture? }` - for `"microphone"` and `"camera"`, `{ source: "screenShare", audio?, capture? -}` for a screen. Failures reject the promise with a `MatrixRTCError`. -- **Remove** `sharingScreen$`, `toggleScreenSharing`, `screenShareError$`, - `dismissScreenShareError`, `observeSharingScreen$`, and the two - `set*Enabled` methods on the client. "Sharing" is `tracks$.some(...)` on the - local media; "enable" is `publish` or `track.setEnabled`. -- **Replace** `LocalMediaInputs.microphoneEnabled` / `cameraEnabled` with - `publish: PublishRequest[]`, applied once the transport is up, as today. - `CaptureSettings` becomes the `capture` of a request; `MediaQuality` keeps its - `config.json` shape and is mapped to per-request capture at the edge. -- **Touches** `LocalMember.ts` (the screen share block and `setEnabled`), - `Publisher.ts` (iterates requests instead of enumerating three sources), - `livekitOptions.ts`, `lifecycle.spec.ts` and `media.spec.ts` for the new - buttons. - -### 3. Names that say "call" - -No behaviour change; one PR of renames and type narrowing. - -- `RTCMember.membership$` is typed as an SDK `RTCMembership`: the subset of - `CallMembership` hosts read (`userId`, `deviceId`, `memberId`, - `rtcBackendIdentity`, `getTransport()`, `applicationData`). The js-sdk object - is still what is behind it. -- `UnknownCallError` is renamed with the two `Livekit*` errors, as slice 11 of - `SdkMigration.md` already plans; `index.ts` drops its alias. -- `applicationData` is passed through whole once the js-sdk accepts more than - `m.call.intent`; until then the README says which key survives. - -### Not planned - -- `MediaSource` keeps its five values. A `name` on `MediaTrack` (LiveKit's - `trackName`) is a one-line addition when the first non-call application asks - for it; adding it now would be a guess at that application's needs. -- `sendNotificationType` stays, as `SdkArchitecture.md` already decided. - -## Changed behaviour - -What a host sees differently, per slice. All four blocks are implemented. - -### `tracks$` and `isActive$` (this plan, implemented) - -| Before | After | -| ---------------------------------------------------------------------- | -------------------------------------------------------------------- | -| `media.camera$`, `.microphone$`, `.screenShare$`, `.screenShareAudio$` | `trackBySource$(scope, media.tracks$, "camera")`, and so on | -| `media.screenShareEnabled$` | `media.tracks$` mapped to `some((t) => t.source === "screenShare")` | -| `media.speaking$` | `isActive$` on the `"microphone"` track; `false` while there is none | -| `createLivekitMediaTrack(…, source)` | the source is read from the publication (internal) | - -Differences in what the values mean: - -- **Tracks published without a source now appear**, with `source: "unknown"`. - Before, they were invisible. A host that renders every track must be ready - for a track it has no name for. -- **`isActive$` is `false` while the track is muted**, even if LiveKit still - reports the member as speaking. `speaking$` was not gated by mute. -- **`isActive$` is the same on every audio track of a member**, because LiveKit - detects speakers per participant. A member sharing a screen with audio lights - both tracks while either carries sound. -- **`setVolume` applies to the track, not the source.** It is remembered and - reapplied when the remote track arrives on subscription, so a volume set - before the track exists is no longer lost. It still does nothing on our own - tracks. -- **One `MediaTrack` object per publication**, stable across unrelated media - events. Before, a slot's object also changed only with its publication, so - nothing changes for the four known sources. -- **The list is in publication order** (the participant's map order). Nothing - before had an order. -- The harness labels audio elements with `data-active`. Test ids are unchanged. - -### Slice 1, local track controls - -- `switchCamera()` on the local media becomes `switchFacingMode()` on the local - camera track, so there is always a track to call it on; it still resolves - `undefined` where the facing mode is unknown. -- Input device and processor changes are calls on the track. The SDK no longer - reacts to `audioInputDeviceId$`, `videoInputDeviceId$` or `videoProcessor$`; - a host that changed a setting and expected the SDK to follow now calls - `setDevice` or `setProcessor` itself. The initial device and processor travel - with the publish request instead. -- `setEnabled(false)` on a local track mutes it and keeps the publication, as - `setMicrophoneEnabled(false)` does today. - -### Slice 2, `publish` and `unpublish` - -- Screen sharing is `publish({ source: "screenShare" })` and `unpublish(id)`. - A capture failure rejects the promise; there is no `screenShareError$` to - dismiss. -- "Am I sharing" is derived from `tracks$` by the host. The SDK no longer - exposes `sharingScreen$`. -- `setMicrophoneEnabled` / `setCameraEnabled` are gone. Before the transport - is connected, `publish` is remembered and applied once it is, which is what - the two methods did. -- The initial microphone and camera are `LocalMediaInputs.publish`, a list; - an empty list joins without publishing, which needed `microphoneEnabled: -false, cameraEnabled: false` before. -- Capture settings travel with each request. A host that set - `capture.screenShare` once sets it on each screen share publish instead. - -### Slice 3, names - -- `membership$` is typed as `RTCMembership`. A host that reached for a - `CallMembership` member outside the narrowed set must import the js-sdk type - itself; the object is unchanged. -- `UnknownCallError` is `UnknownRTCError` in the SDK too, not only in the - export. Error codes do not change. The two `Livekit*` errors are left to - slice 11 of `SdkMigration.md`, which renames them on both sides at once. -- `applicationData` is forwarded whole: the js-sdk takes it next to - `m.call.intent`, which the membership format carries as a field of its own. diff --git a/sdk/README.md b/sdk/README.md index a592c740d..b1e8252ed 100644 --- a/sdk/README.md +++ b/sdk/README.md @@ -1,55 +1,95 @@ # MatrixRTC SDK (EXPERIMENTAL) `@element-hq/matrixrtc-sdk` is the call model under Element Call, on its own: -MatrixRTC memberships and transports, media connections, publishing, E2EE keys and -the media of every member, as observables. It has no UI. Element Call's own -`CallViewModel` is meant to become one consumer of it. The design is in -[`SdkArchitecture.md`](./SdkArchitecture.md), the migration from the view model -in [`SdkMigration.md`](./SdkMigration.md). Media goes through one media backend -behind `MediaBackend`; LiveKit, under `src/media-backend/livekit/`, is the only one -today and the only place that imports `livekit-client`. +it joins a MatrixRTC slot in a Matrix room, writes and watches the memberships, +connects to the media transport (LiveKit today), publishes the local media and +exposes every member and their tracks as observables. It has no UI. Element +Call's own `CallViewModel` is one consumer of it; any host can build a different +view on top. The design is in [`SdkArchitecture.md`](./SdkArchitecture.md), what +moved out of Element Call's `src/` in [`SdkMigration.md`](./SdkMigration.md). -**Status:** first implementation. `createMatrixRTCClient` joins the session, connects to -the transport, publishes the local media and exposes every member's media; the -development harness and its e2e tests in `playwright/sdk` drive it, and Element -Call's own `CallViewModel` is built on it. It depends on nothing in Element Call's -`src/`. +**Status:** first implementation. `createRTCSlot` and `slot.join()` join the +session, connect to the transport, publish the local media and expose every +member's media; the development harness and its e2e tests in `playwright/sdk` +drive it, and Element Call's own `CallViewModel` is built on it. It depends on +nothing in Element Call's `src/`. ## Using it ```ts import { - createMatrixRTCClient, + createRTCSlot, E2eeType, MatrixRTCMode, ObservableScope, + trackBySource$, } from "@element-hq/matrixrtc-sdk"; const scope = new ObservableScope(); -const rtcClient = createMatrixRTCClient( +const slot = createRTCSlot( scope, client, // a matrix-js-sdk MatrixClient, logged in and syncing room, // the matrix-js-sdk Room, from client.getRoom() once the join has synced { encryptionSystem: { kind: E2eeType.PER_PARTICIPANT }, matrixRTCMode: MatrixRTCMode.Compatibility, - publish: [{ source: "microphone" }, { source: "camera" }], }, ); -rtcClient.join(); +const participation = slot.join({ + publish: [{ source: "microphone" }, { source: "camera" }], +}); -rtcClient.remoteMembers$.subscribe((members) => { - // each member has displayName$, tracks$ and more; a media track is rendered - // by handing it a