mirror of
https://github.com/vector-im/element-call.git
synced 2026-10-16 23:45:50 +00:00
[SDK] Drop the landed plan documents and cut the migration document down
MediaBackendPlan.md and MemberMediaPlan.md described changes that have landed. SdkMigration.md keeps only what moved from src/ into sdk/, with the paths as they are today. README.md gains the onboarding notes on the three API files, and its example uses createRTCSlot, which replaced createMatrixRTCClient.
This commit is contained in:
Timo K.
@@ -1,716 +0,0 @@
|
||||
# Media backend: trapping LiveKit behind one seam
|
||||
|
||||
> **Status: implemented, 4 October 2026.** All six steps landed in one change;
|
||||
> the interface is in `sdk/src/backend/api.ts`, the LiveKit backend in
|
||||
> `sdk/src/backend/livekit/`, and the architecture document has the section
|
||||
> "Media backend". The text below is the plan as it was written, kept for the
|
||||
> reasoning behind the shape.
|
||||
|
||||
The SDK's public API in `sdk/src/api.ts` names LiveKit in three places: the video
|
||||
processor type in `PublishRequest` and `LocalVideoMediaTrack.setProcessor`, the
|
||||
video codec in `VideoCaptureSettings`, and the LiveKit-flavoured client options
|
||||
(`mediaQuality`, `transportUrl`, `fallbackTransportUrl`). The implementation is a
|
||||
different story. LiveKit participants, rooms, connections,
|
||||
tokens and data packets run through most of `sdk/src/session/`. Only the two
|
||||
adapters in `sdk/src/media/` are the clean boundary they were meant to be.
|
||||
|
||||
MatrixRTC will get media backends other than a LiveKit SFU, full mesh and a
|
||||
cascading SFU among them. Neither fits the model the session is written against
|
||||
today, one connection per transport URL with a subscribe-only connection to every
|
||||
remote SFU. That model is the LiveKit multi-SFU strategy, so the seam has to sit
|
||||
above it. Cascading keeps the membership transport exactly as it is and only
|
||||
changes which SFUs the client connects to, which makes the connection manager a
|
||||
LiveKit-internal policy. Full mesh has one peer connection per remote member and
|
||||
no token service at all.
|
||||
|
||||
This plan introduces one interface, `MediaBackend`, created once per session. It
|
||||
takes the memberships and the local media inputs and hands back the media the
|
||||
session consumes, plus the three things the MatrixRTC join needs from the media
|
||||
side: the transport to advertise, whether the backend can take over the delayed
|
||||
leave, and the handover of the delayed leave once its id is known. Everything
|
||||
LiveKit moves into `sdk/src/backend/livekit/` behind it. The public API does not
|
||||
change, behaviour does not change, and the `playwright/sdk` specs verify every step.
|
||||
|
||||
This is slice 12 in the status table of [`SdkMigration.md`](./SdkMigration.md). It
|
||||
is independent of slices 6 to 11 there, and slice 7 (a neutral processor type) and
|
||||
slice 11 (the error renames) get easier once the folder exists.
|
||||
|
||||
## What is LiveKit today
|
||||
|
||||
Everything under `sdk/src/session/` that imports `livekit-client`,
|
||||
`@livekit/components-core`, or speaks the LiveKit token protocol:
|
||||
|
||||
| Module | What it is |
|
||||
| ----------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `media/LivekitMediaTrack.ts`, `media/LivekitMemberMedia.ts` | Participant to `MemberMedia`. Clean already. |
|
||||
| `session/Connection.ts`, `ConnectionFactory.ts`, `ConnectionManager.ts` | One LiveKit room per transport URL, the multi-SFU policy, the token fetch per connection. |
|
||||
| `session/MatrixLivekitMembers.ts` | Matches a participant identity to a membership, epoch-paired. |
|
||||
| `session/Publisher.ts`, `publishOptions.ts` | Publish requests onto the local participant, upstream pausing, audio output switching, the capture and publish option builders. |
|
||||
| `session/KeyProvider.ts`, `MatrixKeyProvider.ts` | LiveKit's key provider on the js-sdk key stream. |
|
||||
| `session/DataChannel.ts` | LiveKit data packets. |
|
||||
| `session/openIDSFU.ts`, `livekitOptions.ts` | The JWT service protocol, including delegation, and room options. |
|
||||
| `session/LocalTransport.ts` | The pre-join token fetch. Discovery itself is a homeserver endpoint and stays; see "Discovery stays in the session". |
|
||||
| `session/RtcTransportAutoDiscovery.ts` | Generic apart from its `isLivekitTransport` filter and the fallback URL. Stays. |
|
||||
| `session/LocalMember.ts` | `participant$`, `connection$`, upstream pausing, the delegation probe and handover, and `publish` resolving with a `LocalTrackPublication`. Roughly 120 of 490 lines. |
|
||||
| `session/Members.ts` | `mediaFor` takes a participant and a room; `publish` maps a `trackSid` back to a track. |
|
||||
| `session/Transports.ts` | Transport id, the connected check and `resolved$` assume LiveKit. |
|
||||
| `session/SessionBehaviors.ts` | `membershipsAndTransports$` filters to LiveKit transports. Deleted in step 3, not moved. |
|
||||
| `session/MatrixRTCClient.ts` | Wires all of the above. |
|
||||
| `utils/test.ts` | `mockRemoteParticipant`, `exampleTransport`. |
|
||||
|
||||
`errors.ts` has a type-only import for `LivekitConnectionError`; that is slice 11's
|
||||
business and stays. `index.ts` exports `getSFUConfigWithOpenID` as
|
||||
`authenticateWithTransport` for the developer panel; it keeps being exported, from
|
||||
the backend folder.
|
||||
|
||||
## The interface
|
||||
|
||||
All of it lives in `sdk/src/backend/api.ts`. Types not defined here come from
|
||||
`sdk/src/api.ts` (`MemberMedia`, `LocalMemberMedia`, `LocalMediaInputs`,
|
||||
`ResolvedTransport`), `sdk/src/config.ts` (`SessionTimings`),
|
||||
`sdk/src/encryption.ts` and matrix-js-sdk (`Transport`,
|
||||
`CallMembershipIdentityParts`). Memberships reach a backend as the SDK's
|
||||
`RTCMembership`, whose pick covers the two things a backend reads,
|
||||
`rtcBackendIdentity` and `getTransport()`. The js-sdk session itself never reaches a
|
||||
backend: `MatrixKeyProvider` reads three things from it today, the
|
||||
`EncryptionKeyChanged` event, one `reemitEncryptionKeys()` call to replay the
|
||||
keys it already holds, and the room id for a log line, and `mediaKeys$` covers
|
||||
all three.
|
||||
|
||||
### Connection state, backend neutral
|
||||
|
||||
Replaces `ConnectionState` from `Connection.ts` inside the session. The LiveKit
|
||||
backend maps its room state onto it; `Reconnecting` and `SignalReconnecting` both
|
||||
become `Reconnecting`.
|
||||
|
||||
```ts
|
||||
export enum MediaConnectionState {
|
||||
/** No connection attempt yet, or no connection at all. */
|
||||
Initialized = "initialized",
|
||||
/** Fetching whatever the connection needs first; the SFU token for LiveKit. */
|
||||
Preparing = "preparing",
|
||||
Connecting = "connecting",
|
||||
Connected = "connected",
|
||||
Reconnecting = "reconnecting",
|
||||
Disconnected = "disconnected",
|
||||
Stopped = "stopped",
|
||||
}
|
||||
```
|
||||
|
||||
### What the session hands the backend
|
||||
|
||||
```ts
|
||||
export interface MediaBackendContext {
|
||||
roomId: string;
|
||||
ownMembershipIdentity: CallMembershipIdentityParts;
|
||||
/** What the local member publishes; the exact type is below. */
|
||||
localMedia: LocalMediaInputs;
|
||||
encryptionSystem: EncryptionSystem;
|
||||
/**
|
||||
* Every media key the session knows: the current ones replayed on
|
||||
* subscribe, new ones as they arrive. The backend feeds them into its own
|
||||
* frame encryption. Empty for a shared key, which is in `encryptionSystem`.
|
||||
*/
|
||||
mediaKeys$: Observable<MediaKey>;
|
||||
/** For the delegated leave's timeout, and whatever else a backend times. */
|
||||
timings: SessionTimings;
|
||||
logger: Logger;
|
||||
}
|
||||
|
||||
/** One per-participant media key, as the js-sdk's `EncryptionKeyChanged` carries it. */
|
||||
export interface MediaKey {
|
||||
/** The `rtcBackendIdentity` of the member the key belongs to. */
|
||||
participantId: string;
|
||||
index: number;
|
||||
key: Uint8Array;
|
||||
}
|
||||
|
||||
/**
|
||||
* The type of the `backend` option: a function the session calls once with
|
||||
* its scope and the context to get the backend. Only an alias, so the option
|
||||
* and a second backend's module do not spell the signature out. A backend's
|
||||
* own factory keeps the codebase's shape,
|
||||
* `createLivekitBackend(scope, context, options): MediaBackend`, and the host
|
||||
* closes over its options: `(scope, ctx) => createLivekitBackend(scope, ctx,
|
||||
* { client, tokenEndpoint: "msc4195" })`. No currying, so `createX` returns
|
||||
* an X here as everywhere else.
|
||||
*
|
||||
* The Matrix client is not in the context: a backend that needs it takes it in
|
||||
* its own options, as a `Pick` of exactly what it calls, so the type says what
|
||||
* each backend reaches for. LiveKit picks the OpenID token and the base URL;
|
||||
* a mesh backend would pick to-device sending.
|
||||
*/
|
||||
export type MediaBackendFactory = (
|
||||
scope: ObservableScope,
|
||||
context: MediaBackendContext,
|
||||
) => MediaBackend;
|
||||
```
|
||||
|
||||
`LocalMediaInputs` is unchanged from `sdk/src/api.ts` and is the only input the
|
||||
backend reads about the local media. Since the `MemberMediaPlan.md` slices, the
|
||||
devices and the processor are no longer behaviors the SDK watches: they are
|
||||
fields of the initial requests, and later changes go to the track's own
|
||||
`setDevice` and `setProcessor`.
|
||||
|
||||
```ts
|
||||
export interface LocalMediaInputs {
|
||||
/** Published at the join; `publish` on the local member adds to it from then on. */
|
||||
publish: PublishRequest[];
|
||||
/** Undefined where the host routes audio itself, or to leave the browser's choice. */
|
||||
audioOutputDeviceId$: Behavior<string | undefined>;
|
||||
}
|
||||
|
||||
export type PublishRequest =
|
||||
| { source: "microphone"; deviceId?: string; capture?: AudioCaptureSettings }
|
||||
| {
|
||||
source: "camera";
|
||||
deviceId?: string;
|
||||
processor?: TrackProcessor<Track.Kind.Video>;
|
||||
capture?: VideoCaptureSettings;
|
||||
}
|
||||
| { source: "screenShare"; audio?: boolean; capture?: VideoCaptureSettings };
|
||||
```
|
||||
|
||||
The processor type in the camera request is the LiveKit type on the public API;
|
||||
slice 7 of the migration.
|
||||
|
||||
### The backend
|
||||
|
||||
```ts
|
||||
/** What preparing the local transport established; what the MatrixRTC join needs from the media side. */
|
||||
export interface TransportCapabilities {
|
||||
/**
|
||||
* Whether the backend can restart the delayed leave on the client's behalf,
|
||||
* so that a client that vanishes is removed by the backend rather than by
|
||||
* the timeout. Decides which delayed leave timings the join uses, and
|
||||
* whether `delegateDelayedLeave` is ever called.
|
||||
*/
|
||||
canDelegateDelayedLeave: boolean;
|
||||
}
|
||||
|
||||
/** One connection a backend holds. */
|
||||
export interface BackendConnection {
|
||||
/** The transport as it appears in memberships; what the registry keys on. */
|
||||
transport: Transport;
|
||||
state: MediaConnectionState | Error;
|
||||
/**
|
||||
* What the backend fetched to connect; undefined until it has, and again
|
||||
* after the connection stops. One stable object per fetch, so that the
|
||||
* registry's `distinctUntilChanged` holds across state changes.
|
||||
*/
|
||||
resolved: ResolvedTransport | undefined;
|
||||
}
|
||||
|
||||
export interface MediaBackend {
|
||||
/** The membership transport type this backend serves: "livekit". */
|
||||
readonly transportType: string;
|
||||
|
||||
/**
|
||||
* Makes `transport` usable as the one the local member publishes on. The
|
||||
* session found it; the backend does whatever has to succeed before it is
|
||||
* advertised in the membership. Resolves with what the join needs to know,
|
||||
* rejects with the error the session reports as fatal. Called once per
|
||||
* session. For LiveKit: the OpenID and JWT exchange, whose result the
|
||||
* backend keeps for its own connection, and the delegation probe.
|
||||
*/
|
||||
prepareLocalTransport(transport: Transport): Promise<TransportCapabilities>;
|
||||
|
||||
/**
|
||||
* Hands the delayed leave event to the backend, which restarts it from
|
||||
* then on. Called by the session each time the membership manager reports
|
||||
* a new delay id, only when `canDelegateDelayedLeave` is true. A rejection
|
||||
* is logged, not fatal: the client keeps restarting the leave itself.
|
||||
*/
|
||||
delegateDelayedLeave(delayId: string): Promise<void>;
|
||||
|
||||
/** The local member's side; the exact type is below. */
|
||||
readonly local: LocalMediaBackend;
|
||||
|
||||
/**
|
||||
* The media of one remote member, null while nothing has arrived for it on
|
||||
* its transport ("waiting for media"). Built in `scope`, which the caller
|
||||
* ends with the member. The backend matches on `rtcBackendIdentity` and the
|
||||
* membership's transport, both read from `membership$`.
|
||||
*
|
||||
* This is also how the backend learns which remote transports exist: it
|
||||
* registers the membership's transport for as long as `scope` lives, and
|
||||
* connects to the union of registered transports plus the local one. No
|
||||
* membership list reaches the backend; the session decides which members
|
||||
* exist and asks for their media.
|
||||
*/
|
||||
mediaFor$(
|
||||
scope: ObservableScope,
|
||||
membership$: Behavior<RTCMembership>,
|
||||
): Behavior<MemberMedia | null>;
|
||||
|
||||
/**
|
||||
* Every connection the backend holds, for diagnostics. The session's
|
||||
* transport registry derives `connectedTransports$` from the entries in
|
||||
* `Connected` state, and each `TransportMetadata.resolved$` from the entry
|
||||
* whose transport has the same canonical id, with `distinctUntilChanged`.
|
||||
*/
|
||||
readonly connections$: Behavior<BackendConnection[]>;
|
||||
|
||||
/**
|
||||
* A text channel beside the media. Sent on the local transport, rejected
|
||||
* while it is not connected; received on every transport. The session maps
|
||||
* `senderId` to a member and drops what no member sent.
|
||||
*/
|
||||
sendData(topic: string, text: string): Promise<void>;
|
||||
readonly data$: Observable<{ senderId: string; topic: string; text: string }>;
|
||||
}
|
||||
```
|
||||
|
||||
There is no `transportId` on the backend. The session's transport registry keys
|
||||
on a canonical serialisation of the raw transport object, sorted keys. Transports
|
||||
are plain JSON out of membership events, so identical transports serialise
|
||||
identically, and a mesh transport carrying per-member data is correctly distinct.
|
||||
|
||||
### The local side
|
||||
|
||||
This is what `createLocalMembership$` in `session/LocalMember.ts` reads instead of
|
||||
`participant$`, `connection$`, `connection.livekitRoom` and the publisher. The
|
||||
join state machine, the Matrix join and leave, the error aggregation, the delay
|
||||
id signal and the screen share error handling stay in the session; the LiveKit
|
||||
calls and the per-connection publisher lifecycle move.
|
||||
|
||||
```ts
|
||||
export interface LocalMediaBackend {
|
||||
/**
|
||||
* State of the connection the local member publishes on. `Initialized`
|
||||
* while there is none. An `Error` is the connection's failure.
|
||||
*/
|
||||
readonly connectionState$: Behavior<MediaConnectionState | Error>;
|
||||
|
||||
/** The local member's own media. Null until the connection has a local participant. */
|
||||
readonly media$: Behavior<LocalMemberMedia | null>;
|
||||
|
||||
/**
|
||||
* Whether the local tracks reach the transport. The session sets it to
|
||||
* "joined and the homeserver is reachable", so that a member who has left,
|
||||
* or who may already have been dropped from the session, is not heard.
|
||||
* Remembered across connections and applied to each new one: the tracks
|
||||
* are created on the first `true`, and their upstream is paused while
|
||||
* `false`, so the local preview stays live while nothing is heard.
|
||||
*/
|
||||
setPublishing(publish: boolean): void;
|
||||
/** The first failure to publish, kept until the session ends. Not fatal. */
|
||||
readonly publishError$: Behavior<Error | null>;
|
||||
|
||||
/**
|
||||
* Publishes a source and resolves with its track once it is in `media$`'s
|
||||
* `tracks$`. Remembered across connections, so a request before the
|
||||
* transport is up is applied when the connection comes, and republished
|
||||
* after a reconnection. Rejects where the device could not be used,
|
||||
* including the user closing the screen picker. Today's `DesiredMedia`
|
||||
* map lives behind this.
|
||||
*/
|
||||
publish(
|
||||
request: PublishRequest,
|
||||
): Promise<LocalAudioMediaTrack | LocalVideoMediaTrack>;
|
||||
/** Removes one of our tracks by id; a screen share takes its audio with it. */
|
||||
unpublish(id: string): Promise<void>;
|
||||
}
|
||||
```
|
||||
|
||||
Muting, device switching and the processor are not on the backend: they are
|
||||
methods of `LocalAudioMediaTrack` and `LocalVideoMediaTrack`, which the backend
|
||||
builds in `media$`. The `setEnabled` callback that `createLocalLivekitMemberMedia`
|
||||
takes today, so that a track's mute keeps the desired map in step, is wired
|
||||
inside the backend. Screen sharing has no members of its own any more: it is a
|
||||
`publish` request, and "sharing" is `tracks$` holding a `screenShare` track.
|
||||
|
||||
There is one publish switch, not two. Today `LocalMember.ts` pauses the
|
||||
upstream from two places: `stopPublishing()` on leave, and a separate
|
||||
subscription on the homeserver connectivity that pauses every publication while
|
||||
the sync is down and resumes every publication when it is back. Both are the
|
||||
same LiveKit operation, and the second does not look at the join intent: after
|
||||
`leave()` the tracks still exist with their upstream paused, so a sync hiccup
|
||||
resumes them and a member who has left is heard again. The session computes
|
||||
the one boolean, joined and reachable, and the backend applies it. The cost is
|
||||
that the tracks are now created on the first `true`, so a join while the sync is
|
||||
down waits for the sync before asking for devices. A join cannot send its
|
||||
membership without the sync either, and pre-join track creation is slice 6's.
|
||||
|
||||
There is no publisher on the interface. Today's `Publisher` wraps one LiveKit
|
||||
room's local participant and so lives and dies with the local connection, and
|
||||
`LocalMember.ts` re-applies the join intent to each new one. That lifecycle is a
|
||||
LiveKit detail: the backend keeps the publish intent, as it keeps the desired
|
||||
requests for `publish`, and re-applies both to every connection it opens. A
|
||||
mesh backend has no publisher object at all.
|
||||
|
||||
### Delayed leave delegation
|
||||
|
||||
How it works today, all in `LocalMember.ts` and `openIDSFU.ts`:
|
||||
|
||||
1. The js-sdk membership manager sends the membership and a delayed leave event,
|
||||
and emits the leave's delay id (`MembershipManagerEvent.DelayIdChanged`), again
|
||||
whenever it has to send a new one. `MatrixRTCClient.ts` turns that into
|
||||
`delayId$`.
|
||||
2. When the local transport is known, the session probes two MSC4195 endpoints
|
||||
without credentials: the homeserver's
|
||||
`/_matrix/client/unstable/io.element.msc4195/rtc/livekit/delegate_delayed_leave`
|
||||
and the transport's `<livekit_service_url>/delegate_delayed_leave`. A 404 means
|
||||
no support. Either one supporting it is enough.
|
||||
3. The join uses `timings.delegatedDelayedLeave` if supported, else
|
||||
`timings.delayedLeave`. The difference is an hour versus eighteen seconds of
|
||||
delay, because a delegated leave is restarted by the SFU and the client stops
|
||||
restarting it.
|
||||
4. On every delay id, if supported, the session calls the token service again with
|
||||
`delay_id`, `delay_timeout` and the homeserver's URL. The token it returns is
|
||||
discarded; the request is the handover. Failure is logged and the client keeps
|
||||
restarting the leave itself.
|
||||
|
||||
Steps 1 and 3 are MatrixRTC and stay in the session. Steps 2 and 4 are the LiveKit
|
||||
token protocol and move behind the interface, as two members rather than one:
|
||||
|
||||
- The probe goes into `prepareLocalTransport`, reported as `canDelegateDelayedLeave`.
|
||||
It needs the transport's URL and the homeserver's, both of which the backend has
|
||||
at that moment, and the join needs its answer before it fires. The backend runs
|
||||
the probe concurrently with the token fetch, as today.
|
||||
- The handover is `delegateDelayedLeave(delayId)`, because the delay id arrives
|
||||
after the join and changes over the session's life. The session owns that signal
|
||||
and drives the method from it.
|
||||
|
||||
The session side after the change:
|
||||
|
||||
```ts
|
||||
// discoverLocalTransport is the session's; see "Discovery stays in the session"
|
||||
const prepared$ = from(
|
||||
discoverLocalTransport(client, backend.transportType, options, logger).then(
|
||||
async (transport) => ({
|
||||
transport,
|
||||
...(await backend.prepareLocalTransport(transport)),
|
||||
}),
|
||||
),
|
||||
); // errors become fatalTransportError$
|
||||
|
||||
scope.reconcile(
|
||||
combineLatest([prepared$, joinRequested$]),
|
||||
([prepared, join]) => {
|
||||
if (prepared === null || !join) return;
|
||||
joinMatrixRTC(
|
||||
prepared.transport,
|
||||
prepared.canDelegateDelayedLeave
|
||||
? timings.delegatedDelayedLeave
|
||||
: timings.delayedLeave,
|
||||
);
|
||||
return leaveOnCleanup;
|
||||
},
|
||||
);
|
||||
|
||||
scope.reconcile(
|
||||
combineLatest([prepared$, delayId$]),
|
||||
async ([prepared, delayId]) => {
|
||||
if (!prepared?.canDelegateDelayedLeave || delayId === null) return;
|
||||
try {
|
||||
await backend.delegateDelayedLeave(delayId);
|
||||
} catch (e) {
|
||||
logger.error("Failed to delegate the leave", e);
|
||||
}
|
||||
},
|
||||
);
|
||||
```
|
||||
|
||||
A mesh backend answers `canDelegateDelayedLeave: false` and is never asked to
|
||||
delegate. Should a backend later delegate to something other than a token
|
||||
service, it does so behind the same method.
|
||||
|
||||
### Lifetimes: what ends what
|
||||
|
||||
`mediaFor$` is bound to the scope it is given, not to its subscribers.
|
||||
`scope.behavior` subscribes eagerly and tears down with the scope, so the
|
||||
behavior and the transport registration live exactly as long as the member
|
||||
does, whether or not a view reads them. Lazy registration on first subscribe
|
||||
would be wrong: the audio renderer needs the microphone track of a member
|
||||
nobody has a tile for, and that member's transport would never connect.
|
||||
|
||||
- **A remote member leaves.** `generateItems` ends the member's scope. From
|
||||
that one scope the `mapScoped` item ends and detaches the `MemberMedia`, the
|
||||
behavior completes, and the `onEnd` callback releases the transport. If the
|
||||
count for that transport hits zero the connection manager drops it, and the
|
||||
connection's own scope ending calls `stop()`, as today.
|
||||
- **A member changes transport.** Registration follows `membership$` with
|
||||
`distinctUntilChanged` on the transport id: release the old, register the
|
||||
new, inside the same scope.
|
||||
- **The local member calls `leave()`.** Nothing changes on the remote side.
|
||||
The memberships are still in the room, so the members still exist and the
|
||||
connections stay open. That is today's behaviour too: the connection
|
||||
manager never looked at `joinRequested$`.
|
||||
- **The network drops.** The member still exists, so `mediaFor$` is live. The
|
||||
connection reports reconnecting, its participant list empties, the
|
||||
participant behind the member becomes null, the media item ends and
|
||||
`media$` reads null, "waiting for media". A new `MemberMedia` is built on
|
||||
reconnect. As today.
|
||||
- **The session scope ends.** The backend and every member scope end in the
|
||||
same tick, in subscription order, so the backend's internals tear down
|
||||
before the members' release callbacks run. Release must therefore do
|
||||
nothing but update the registry map and `next` a subject, both no-ops with
|
||||
nobody listening. It must never stop a connection or touch LiveKit itself;
|
||||
the connection scopes do that.
|
||||
|
||||
### Where the LiveKit backend gets each member from
|
||||
|
||||
| Interface member | Today | Moves from |
|
||||
| -------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------- |
|
||||
| `prepareLocalTransport` | OpenID and JWT fetch kept as `existingSFUConfig`, delegation probe | `LocalTransport.ts`, `LocalMember.ts` |
|
||||
| `delegateDelayedLeave` | `getSFUConfigWithOpenID` with `delayId` | `LocalMember.ts` |
|
||||
| `local.connectionState$` | `connection$.state$` for the local transport | `LocalMember.ts` |
|
||||
| `local.media$` | `mediaFor(participant$, connection$)` with `createLocalLivekitMemberMedia` | `Members.ts` |
|
||||
| `local.setPublishing`, `publishError$` | `scope.reconcile(connection$, createPublisher)`, the `[publisher$, joinRequested$]` reconcile with its `publishError$`, and the homeserver-connectivity loop over `participant.trackPublications` | `LocalMember.ts`, `MatrixRTCClient.ts` |
|
||||
| `local.publish`, `unpublish` | `publish` and `unpublish` with the `DesiredMedia` map and `publication$`, and the `trackSid` to track lookup | `LocalMember.ts`, `Members.ts`, `Publisher.ts` |
|
||||
| internal | the `setEnabled(source, enabled)` callback into `createLocalLivekitMemberMedia` | `LocalMember.ts`, `Members.ts` |
|
||||
| `mediaFor$` | `createRemoteMatrixLivekitMembers$` matching plus `mediaFor` | `Members.ts`, `MatrixLivekitMembers.ts` |
|
||||
| `connections$` | `connectionManagerData$` joined with each connection's `state$` and `ResolvedConnection.resolved$` | `Transports.ts`, `ConnectionFactory.ts` |
|
||||
| `sendData`, `data$` | `createDataChannel$` minus the member lookup | `DataChannel.ts` |
|
||||
| internal | key provider on `mediaKeys$` instead of `setRTCSession`, room options, E2EE worker, connection manager fed by the transports registered through `mediaFor$` | `KeyProvider.ts`, `MatrixKeyProvider.ts`, `livekitOptions.ts`, `ConnectionFactory.ts`, `ConnectionManager.ts` |
|
||||
|
||||
### Discovery stays in the session
|
||||
|
||||
Which transport the local member advertises is answered by the homeserver's
|
||||
MSC4143 transport list, and that list can name transports of any type. So
|
||||
discovery is the session's, in `session/LocalTransport.ts`, and a backend is
|
||||
only asked to prepare what the session picked:
|
||||
|
||||
```ts
|
||||
/** In session/LocalTransport.ts. The homeserver's list, retried, filtered to what the backend serves. */
|
||||
export async function discoverLocalTransport(
|
||||
client: Pick<MatrixClient, "getDomain" | "_unstable_getRTCTransports">,
|
||||
transportType: string,
|
||||
options: Pick<
|
||||
MatrixRTCClientOptions,
|
||||
"transportUrl" | "fallbackTransportUrl"
|
||||
>,
|
||||
logger: Logger,
|
||||
): Promise<Transport>; // throws MatrixRTCTransportMissingError
|
||||
```
|
||||
|
||||
`RtcTransportAutoDiscovery` does the list fetch today and stays where it is,
|
||||
with its `isLivekitTransport` filter replaced by a filter on `transportType`.
|
||||
|
||||
The two URL options are the awkward part. `transportUrl` and
|
||||
`fallbackTransportUrl` are LiveKit service URLs, and turning a URL into a
|
||||
membership transport means building `{ type: "livekit", livekit_service_url }`.
|
||||
That is LiveKit knowledge, but it is a workaround for hosts that configure a URL
|
||||
rather than a transport, and it lives in `discoverLocalTransport`, not in
|
||||
`backend/livekit/`. The backend folder never learns where its transport came
|
||||
from. It is one `Transport` literal and the js-sdk's `LivekitTransport` type,
|
||||
no `livekit-client` import, so the lint guard of step 6 is not affected. The
|
||||
clean follow-up, outside this plan, is `transport?: Transport` and
|
||||
`fallbackTransport?: Transport` on the client options, with the host building
|
||||
the literal; then the workaround is deleted.
|
||||
|
||||
### LiveKit's own options
|
||||
|
||||
`mediaQuality` and `capture` on `MatrixRTCClientOptions` configure LiveKit, not
|
||||
MatrixRTC. They become `LivekitBackendOptions`, the third parameter of
|
||||
`createLivekitBackend(scope, context, options)`, together with the slice of the
|
||||
Matrix client the backend calls. They also stay on `MatrixRTCClientOptions`, where they feed the
|
||||
default backend when the host passes no `backend`, so Element Call's
|
||||
`InCallView` does not change. A host that supplies its own backend configures
|
||||
it itself, client included. `transportUrl` and `fallbackTransportUrl` are
|
||||
discovery inputs and stay on the client options for good, as above.
|
||||
|
||||
```ts
|
||||
export interface LivekitBackendOptions {
|
||||
/**
|
||||
* `getOpenIdToken` for the token exchange, `baseUrl` for the delegation
|
||||
* probe and handover. Nothing else: discovery is the session's. This is
|
||||
* today's `OpenIDClientParts` from `openIDSFU.ts` minus its unused
|
||||
* `getDeviceId`, plus `baseUrl`.
|
||||
*/
|
||||
client: Pick<MatrixClient, "getOpenIdToken" | "baseUrl">;
|
||||
mediaQuality?: MediaQuality;
|
||||
/**
|
||||
* Which token service endpoint to use for the local transport: the MSC4195
|
||||
* one, or the legacy one with the old identity format. Remote transports
|
||||
* always try both. The default backend derives it from the client's
|
||||
* `matrixRTCMode`; the mode itself never reaches a backend, since what it
|
||||
* also decides, sticky events and the member id format, is the session's.
|
||||
*/
|
||||
tokenEndpoint: "msc4195" | "legacy";
|
||||
}
|
||||
```
|
||||
|
||||
Today `openIDSFU.ts` takes the `MatrixRTCMode` itself and reads the endpoint
|
||||
choice off it. Step 5 changes it to take `tokenEndpoint`, with the mapping
|
||||
`Matrix_2_0` to `msc4195` and `Compatibility` to `legacy` done once where the
|
||||
default backend is built, in `MatrixRTCClient.ts`, which also hands the
|
||||
client through. The exported `authenticateWithTransport` keeps its signature by
|
||||
doing the same mapping.
|
||||
|
||||
## Transition
|
||||
|
||||
Six steps, each green on its own and each a PR of under 400 lines. There is one
|
||||
connection manager at every point: the backend owns it from step 2 on, and the
|
||||
consumers not yet switched read it through a transitional field until step 4
|
||||
removes it. Running the old wiring and the backend side by side would open a
|
||||
second connection to every SFU, so the plan never does that.
|
||||
|
||||
### Step 1: carve the folder
|
||||
|
||||
Pure move, no code change beyond import paths.
|
||||
|
||||
- `git mv` into `sdk/src/backend/livekit/`: `media/*`, `Connection`,
|
||||
`ConnectionFactory`, `ConnectionManager`, `DataChannel`, `KeyProvider`,
|
||||
`MatrixKeyProvider`, `MatrixLivekitMembers`, `Publisher`, `publishOptions`,
|
||||
`livekitOptions`, `openIDSFU`, each with its test. `utils/tracks.ts` is
|
||||
neutral and stays. `RtcTransportAutoDiscovery` and
|
||||
`LocalTransport` stay in `session/`.
|
||||
- Split `utils/test.ts`: `mockRemoteParticipant` and `exampleTransport` go to
|
||||
`backend/livekit/test.ts`.
|
||||
- `index.ts` re-exports `authenticateWithTransport` from the new path.
|
||||
|
||||
Check: `pnpm lint`, `pnpm test`, the `playwright/sdk` specs.
|
||||
|
||||
### Step 2: the interface, the backend shell, the registry, the data channel
|
||||
|
||||
- Add `sdk/src/backend/api.ts` with the types above.
|
||||
- Add `sdk/src/backend/livekit/LivekitBackend.ts` exporting
|
||||
`createLivekitBackend(scope, context, options): MediaBackend`. It owns what
|
||||
`MatrixRTCClient.ts` builds today: the key provider, the
|
||||
`LivekitConnectionFactory` and the connection manager. Until step 3 the
|
||||
connection manager is still fed from `membershipsAndTransports$`, passed in
|
||||
as a transitional `remoteTransports$` input beside the transitional
|
||||
`connectionManager` field; both go in step 3. It implements
|
||||
`transportType`, `connections$`, `sendData` and `data$`. `ResolvedConnection`
|
||||
keeps one `ResolvedTransport` object per fetch instead of rebuilding it on
|
||||
every state change.
|
||||
- `MatrixRTCClient.ts` creates the backend, takes the connection manager from
|
||||
it, and maps `data$` senders to members itself. It builds `mediaKeys$` from
|
||||
the js-sdk session: `fromEvent(EncryptionKeyChanged)` with a
|
||||
`reemitEncryptionKeys()` call on subscribe. `MatrixKeyProvider` subscribes
|
||||
to that instead of taking the session; its test feeds a `Subject`.
|
||||
- `Transports.ts` derives both `connected$` and each `TransportMetadata.resolved$`
|
||||
from `backend.connections$`, keys the registry on the canonical JSON of the
|
||||
raw transport, and loses `ResolvedConnection`, `isLivekitTransport` and
|
||||
`ConnectionState`.
|
||||
|
||||
New unit test: `LivekitBackend.test.ts` with a fake LiveKit room through the
|
||||
existing `livekitRoomFactory` seam, covering `connections$` through a connect,
|
||||
a reconnect and a stop, and the data channel.
|
||||
|
||||
### Step 3: remote media
|
||||
|
||||
- Implement `mediaFor$` in the LiveKit backend. It registers the membership's
|
||||
transport, following `membership$` and released when `scope` ends; looks
|
||||
the transport and identity up in `connectionManagerData$`;
|
||||
`distinctUntilChanged` on the participant and room pair; then `mapScoped`
|
||||
into `createLivekitMemberMedia`. The "participant matched or missing" log
|
||||
line moves here.
|
||||
- The connection manager's `remoteTransports$` becomes the registered set:
|
||||
a `Map` of transport id to count, exposed as a behavior of its keys, with
|
||||
`membershipsAndTransports$`'s LiveKit filter and dedup applied there. The
|
||||
transitional input from step 2 goes.
|
||||
- `Members.ts` drops `mediaFor` and the participant and room parameters;
|
||||
`createRemoteRTCMember` takes `backend.mediaFor$(scope, membership$)`.
|
||||
- `MatrixRTCClient.ts` generates `remoteMembers$` straight from `memberships$`
|
||||
minus the local one, keyed by `membershipKeys`, and stops using
|
||||
`createRemoteMatrixLivekitMembers$`. `membershipsAndTransports$` is deleted
|
||||
from `SessionBehaviors.ts`.
|
||||
|
||||
The epoch pairing that `createRemoteMatrixLivekitMembers$` does today is not
|
||||
needed any more. It existed to keep a membership list and the connection data
|
||||
from different ticks apart; now a member's media depends on its own membership
|
||||
only, and the connections exist because members asked for them. A connection
|
||||
that does not exist yet reads as null, which is "waiting for media" and correct.
|
||||
What has to hold is that `MemberMedia` instances stay stable while the
|
||||
participant and room do not change, which the `distinctUntilChanged` above
|
||||
guarantees, and that a transport stays registered while any member names it,
|
||||
which the count does. `media.spec.ts` is the check that nothing flickers.
|
||||
|
||||
New unit test in `LivekitBackend.test.ts`: two members on one remote transport
|
||||
open one connection, it survives the first member's scope ending, and closes
|
||||
with the second's.
|
||||
|
||||
### Step 4: local media
|
||||
|
||||
- Implement `local` in the LiveKit backend: `connectionState$` from the local
|
||||
transport's connection mapped onto `MediaConnectionState`, `media$` through
|
||||
`createLocalLivekitMemberMedia`, `setPublishing` and `publishError$` by
|
||||
reconciling a `Publisher` per connection against the remembered intent,
|
||||
which is the two reconciles from `LocalMember.ts` moved as they are, and
|
||||
`publish` and `unpublish` over a backend-owned `DesiredMedia` map and the
|
||||
current publisher, with the `trackSid` to track lookup that `Members.ts`
|
||||
does today done inside. The `setEnabled` callback into
|
||||
`createLocalLivekitMemberMedia` is wired here too. The
|
||||
homeserver-connectivity loop is not moved; `setPublishing` covers it.
|
||||
- `LocalMember.ts` takes `backend.local` instead of `connectionManager`,
|
||||
`createPublisher` and `desired`. Its publisher reconcile and the pause and
|
||||
resume loop become one subscription,
|
||||
`joinRequested$ && homeserverConnected.combined$` into
|
||||
`backend.local.setPublishing`, and `backend.local.publishError$` feeds
|
||||
`state$` where the local subject did. It loses `participant$`,
|
||||
`connection$`, the `livekitRoom` access, the pause and resume loop,
|
||||
`publish`, `unpublish`, `setEnabled` and `publication$`, and compares
|
||||
against `MediaConnectionState.Connected` in `mediaState$` and
|
||||
`disconnectReason$`. `LocalMembership` shrinks to the join state machine
|
||||
and no longer names a LiveKit type.
|
||||
- `Members.ts`: `createLocalRTCMember` forwards `publish` and `unpublish` to
|
||||
`backend.local` and drops its own `trackSid` lookup.
|
||||
- `MatrixRTCClient.ts` stops passing `createPublisher`, `desired` and the
|
||||
connection manager, and stops building the `DesiredMedia` map. The
|
||||
transitional field goes.
|
||||
- `status.test.ts` moves to `MediaConnectionState`.
|
||||
|
||||
New unit test: `LocalMember.test.ts` against a fake `LocalMediaBackend`. This is
|
||||
the first time the local state machine is testable without LiveKit; cover join
|
||||
before the connection is up, connection loss while publishing, a sync outage
|
||||
after `leave()` leaving `setPublishing` at false, and a rejected
|
||||
local transport.
|
||||
|
||||
### Step 5: the local transport and the delegation
|
||||
|
||||
- `session/LocalTransport.ts` becomes `discoverLocalTransport` as above: the
|
||||
URL override, else `RtcTransportAutoDiscovery` filtered on the backend's
|
||||
type, else the fallback URL. The token fetch leaves it, and so does the
|
||||
`LocalTransport` pair type with its `sfuConfig`.
|
||||
- `RtcTransportAutoDiscovery` takes a `transportType` and returns a
|
||||
`Transport`; its `isLivekitTransport` filter goes.
|
||||
- Implement `prepareLocalTransport` in the LiveKit backend from the token
|
||||
fetch in `getLocalTransport` and the two `checkDelegationSupport` calls,
|
||||
the probe running concurrently with the token fetch. The SFU config is
|
||||
kept for the local connection, keyed by service URL, replacing
|
||||
`existingSFUConfig` flowing through `LocalTransport`.
|
||||
- Implement `delegateDelayedLeave` from the handover `reconcile` in
|
||||
`LocalMember.ts`.
|
||||
- `MatrixRTCClient.ts` chains discovery and preparation into `prepared$` as
|
||||
sketched above.
|
||||
- `LocalMember.ts` takes `prepared$` instead of `localTransport$`, drops
|
||||
`joinParams$`, `checkDelegationSupport`, `client`, `roomId`,
|
||||
`ownMembershipIdentity` and `matrixRTCMode` from its props, and drives
|
||||
`backend.delegateDelayedLeave` from `delayId$`.
|
||||
- `joinJsSdkSession` takes a `Transport`. `encryptMedia` derives from
|
||||
`encryptionSystem.kind`, not from whether a key provider exists.
|
||||
|
||||
New unit tests: in `LivekitBackend.test.ts`, the probe outcomes (homeserver
|
||||
supports it, only the transport does, neither, probe throws) and the handover
|
||||
request body; in `LocalMember.test.ts`, that the join picks the delegated timings
|
||||
and that a new delay id triggers a handover.
|
||||
|
||||
### Step 6: the `backend` option and the guard
|
||||
|
||||
- `MatrixRTCClientOptions.backend?: MediaBackendFactory`. When absent,
|
||||
`MatrixRTCClient.ts` supplies
|
||||
`(scope, ctx) => createLivekitBackend(scope, ctx, livekitOptions)`, with
|
||||
`livekitOptions` built from the client and the LiveKit options on the client
|
||||
options.
|
||||
- Extend `eslint/SdkImportBoundary.js`: under `sdk/src/session/`, in
|
||||
`sdk/src/api.ts` and in `sdk/index.ts`, `livekit-client` and `@livekit/*` are
|
||||
banned except as `import type`. From here on the seam is enforced, not
|
||||
documented.
|
||||
- `SdkArchitecture.md` gets a "Media backend" section built from this plan, the
|
||||
README's description of what imports LiveKit is updated, and `SdkMigration.md`
|
||||
records the slice as done.
|
||||
|
||||
## After this
|
||||
|
||||
- Full mesh or a cascading SFU is a new folder under `sdk/src/backend/` with
|
||||
its own `createMeshBackend(scope, context, options)`, handed in as a
|
||||
`MediaBackendFactory`; nothing in `session/` changes.
|
||||
- A member whose transport type has no backend gets null media. That is what
|
||||
happens today through the `isLivekitTransport` filter; now it is explicit.
|
||||
- Slice 7, the neutral processor type, and slice 11, the error renames, each
|
||||
touch one folder.
|
||||
- Several backends at once, grouping memberships by transport type, fits the
|
||||
interface but is out of scope until a second backend exists.
|
||||
@@ -1,457 +0,0 @@
|
||||
# `MemberMedia`: a list of tracks instead of a call's four slots
|
||||
|
||||
`MemberMedia` in `sdk/src/api.ts` is written in the vocabulary of a video call. It
|
||||
has four named slots, `microphone$`, `camera$`, `screenShare$` and
|
||||
`screenShareAudio$`, and two member-level flags, `speaking$` and
|
||||
`screenShareEnabled$`. A MatrixRTC session is not always a call: a whiteboard that
|
||||
streams one video track, a translation bot with two audio tracks, a member
|
||||
publishing two cameras, or a future application the SDK has not heard of all have
|
||||
media, and none of it fits the four slots. The flags are worse than the slots.
|
||||
`screenShareEnabled$` says only that the `screenShare$` slot is filled, and
|
||||
`speaking$` names the microphone's activity after what people do with it in a call.
|
||||
|
||||
This plan replaces the slots and the flags with one `tracks$` list, and moves the
|
||||
activity signal to where it is measured: `isActive$` on the audio track. The public
|
||||
API changes; behaviour does not; the `playwright/sdk` specs and the dev harness
|
||||
verify it.
|
||||
|
||||
The SDK's consumers of these fields today are the dev harness (`sdk/dev/main.ts`)
|
||||
and, through it, the specs. Element Call's media view models on this branch still
|
||||
read the LiveKit participant directly, so nothing under `src/` changes. This is the
|
||||
cheapest moment to make the change: once Element Call consumes `MemberMedia`, every
|
||||
view model in `src/state/media/` will be a call site.
|
||||
|
||||
> Note for the status table of [`SdkMigration.md`](./SdkMigration.md): it records
|
||||
> slice 4 (view models take `media$`) as done, but on this branch
|
||||
> `src/state/media/*.ts` still imports `livekit-client` and `CallViewModel.ts`
|
||||
> does not import the SDK. Either the table is ahead of this branch or the slice
|
||||
> lives elsewhere; the table should say which before this plan is added to it.
|
||||
|
||||
## What the tracks already know
|
||||
|
||||
The four slot getters were one line each: `participant.getTrackPublication(
|
||||
livekitSources[source])`. That is a lookup by `source` in the participant's
|
||||
`trackPublications` map. Everything the slot name told the consumer, the track
|
||||
already carries:
|
||||
|
||||
| On the LiveKit publication | On `MediaTrack` today | Set by |
|
||||
| -------------------------- | ---------------------- | ----------------------------------------------------------------------- |
|
||||
| `source: Track.Source` | `source: MediaSource` | the publisher, at publish time; the SFU forwards it to every subscriber |
|
||||
| `kind: Track.Kind` | `kind` | the media stream track |
|
||||
| `trackSid` | `id` | the SFU, stable for the life of the publication |
|
||||
| `trackName` | not exposed | the publisher; free-form, empty for everything Element Call publishes |
|
||||
| `mimeType`, `dimensions` | not exposed | the SFU; `stats$` carries the live values, these are what was announced |
|
||||
| `isMuted`, `isEncrypted` | `muted$`, `encrypted$` | the publisher and the SFU |
|
||||
|
||||
So a flat list is lossless: `tracks$.find((t) => t.source === "camera")` is exactly
|
||||
what `camera$` returned, and `tracks$.some((t) => t.source === "screenShare")` is
|
||||
exactly `screenShareEnabled$`, since LiveKit's `isScreenShareEnabled` is defined as
|
||||
`!!getTrackPublication(Track.Source.ScreenShare)`, presence and not mute state.
|
||||
|
||||
One thing the slots lost that the list keeps: a publication with
|
||||
`Track.Source.Unknown` never appeared in any slot. The list includes it, so
|
||||
`MediaSource` grows a fifth value, `"unknown"`. A consumer that wants the call
|
||||
vocabulary filters for the four names it knows; nothing is dropped on the way.
|
||||
|
||||
What the list does not try to do: the `trackName` and any application-specific
|
||||
meaning stay out of the interface for now. Whether a non-call application tags
|
||||
its tracks through the name, through the membership's `applicationData`, or
|
||||
through a MatrixRTC-level description is a question for when such an application
|
||||
exists; see "Other possible call-related refactors".
|
||||
|
||||
## The interface
|
||||
|
||||
```ts
|
||||
export type MediaSource =
|
||||
| "microphone"
|
||||
| "camera"
|
||||
| "screenShare"
|
||||
| "screenShareAudio"
|
||||
/** Published without a source; the application knows what it is. */
|
||||
| "unknown";
|
||||
|
||||
export interface AudioMediaTrack extends MediaTrack {
|
||||
kind: "audio";
|
||||
/**
|
||||
* Whether the track carries sound right now, as the backend measures it.
|
||||
* False while muted. A call reads this on the microphone track and calls
|
||||
* it "speaking".
|
||||
*/
|
||||
isActive$: Behavior<boolean>;
|
||||
setAudioContext(ctx: AudioContext | undefined, plugins?: AudioNode[]): void;
|
||||
setVolume(volume: number): void;
|
||||
}
|
||||
|
||||
/**
|
||||
* The media of one member. One entry per published track, in publication
|
||||
* order; an entry is the same object for as long as the same publication is
|
||||
* behind it. There is no identity field: the backend identity is the member's
|
||||
* `id`.
|
||||
*/
|
||||
export interface MemberMedia {
|
||||
local: boolean;
|
||||
tracks$: Behavior<(AudioMediaTrack | VideoMediaTrack)[]>;
|
||||
/** Emits when the SFU reports a key problem for this member. */
|
||||
encryptionError$: Observable<EncryptionError>;
|
||||
}
|
||||
|
||||
export interface LocalMemberMedia extends MemberMedia {
|
||||
local: true;
|
||||
switchCamera(): Promise<string | undefined>;
|
||||
}
|
||||
```
|
||||
|
||||
Removed: `speaking$`, `screenShareEnabled$`, `microphone$`, `camera$`,
|
||||
`screenShare$`, `screenShareAudio$`. `switchCamera` stays in this plan because it
|
||||
is a different question (see the refactors section).
|
||||
|
||||
### One helper, exported
|
||||
|
||||
Every consumer will want "the member's track for a source, as a behavior". The
|
||||
harness needs it twice already, and Element Call will need it in every media view
|
||||
model. Rather than let each caller write the `map` + `distinctUntilChanged` chain,
|
||||
the SDK exports one function from `sdk/src/utils/tracks.ts`:
|
||||
|
||||
```ts
|
||||
/** The first track of a source, undefined while there is none. */
|
||||
export function trackBySource$<S extends MediaSource>(
|
||||
scope: ObservableScope,
|
||||
tracks$: Behavior<MediaTrack[]>,
|
||||
source: S,
|
||||
): Behavior<TrackOfSource<S> | undefined>;
|
||||
```
|
||||
|
||||
`TrackOfSource` narrows `"microphone" | "screenShareAudio"` to `AudioMediaTrack`
|
||||
and `"camera" | "screenShare"` to `VideoMediaTrack`, so the typed getters the slots
|
||||
gave are not lost. It is a pure function over `Behavior`s, tested with marbles.
|
||||
Nothing else is added; `some`, `filter` and `find` on the array cover the rest.
|
||||
|
||||
## `isActive$`: how it replaces `speaking$`
|
||||
|
||||
### What LiveKit measures
|
||||
|
||||
The SFU runs speaker detection from the audio-level RTP header extension on the
|
||||
audio it forwards, and pushes `SpeakerInfo { sid, level, active }` updates to every
|
||||
client. `livekit-client` turns them into `participant.audioLevel`,
|
||||
`participant.isSpeaking` and `ParticipantEvent.IsSpeakingChanged` (see
|
||||
`Room.handleActiveSpeakersUpdate` and `handleSpeakersChanged`). The `sid` is the
|
||||
participant's; there is no per-track speaking signal in the client API. `speaking$`
|
||||
today is `IsSpeakingChanged` mapped to `isSpeaking`, nothing more.
|
||||
|
||||
### What the SDK exposes
|
||||
|
||||
`isActive$` on an audio track is the participant's speaking signal gated by the
|
||||
track's own mute state:
|
||||
|
||||
```
|
||||
isActive = participant.isSpeaking && !publication.isMuted
|
||||
```
|
||||
|
||||
For the LiveKit backend this maps one participant-level signal onto every audio
|
||||
track of the member. In practice a member has one audio track, the microphone, and
|
||||
the mapping is exact. A member that also shares a screen with audio gets the same
|
||||
value on both tracks while either carries sound; the mute gate keeps a muted
|
||||
microphone dark while the screen share plays. This is a documented approximation of
|
||||
the LiveKit adapter, not of the interface: a full-mesh backend has a peer connection
|
||||
per remote member and measures each track's level itself, and a stats-based
|
||||
refinement exists for LiveKit too (below). The track is the right place for the
|
||||
behavior because that is where every backend other than a single SFU would produce
|
||||
it.
|
||||
|
||||
### Why a call loses nothing
|
||||
|
||||
"Speaking" is a call's reading of "the member's microphone track is active". The
|
||||
replacement for `media.speaking$` is:
|
||||
|
||||
```ts
|
||||
trackBySource$(scope, media.tracks$, "microphone").pipe(
|
||||
switchMap((track) => track?.isActive$ ?? of(false)),
|
||||
);
|
||||
```
|
||||
|
||||
False when there is no microphone track, which is what `isSpeaking` reported in
|
||||
that case too. Element Call's 1 s / 60 s hysteresis (`observeSpeaker$`) and the
|
||||
spotlight speaker selection stay in the call view model, where they are: they are
|
||||
call UX, and they consume a boolean exactly as before.
|
||||
|
||||
What is gained: screen share audio can show activity ("the presenter's video has
|
||||
sound"), and an application with several audio tracks per member can tell them
|
||||
apart once the backend can. What is lost: nothing the SFU knew. The signal was
|
||||
never per track in LiveKit; the interface now says so instead of pretending the
|
||||
member speaks.
|
||||
|
||||
### Later refinement, not in this plan
|
||||
|
||||
A genuinely per-track level for LiveKit is available from the RTP statistics the
|
||||
SDK already polls for `stats$`: `audioLevel` and `totalAudioEnergy` on
|
||||
`RTCInboundRtpStreamStats`, and on the `media-source` stats for a local track. The
|
||||
poll is once a second, which the call's hysteresis tolerates but which is too
|
||||
coarse for an unfiltered indicator, and it costs a stats request per audio track
|
||||
per second while subscribed. If a second audio track per member ever matters, that
|
||||
is the first thing to try.
|
||||
|
||||
## What changes, file by file
|
||||
|
||||
### `sdk/src/api.ts`
|
||||
|
||||
The interface above. The comment on `MemberMedia` says the list is in publication
|
||||
order and that entries are stable per publication; the comment on `isActive$` says
|
||||
what it measures and names the LiveKit approximation.
|
||||
|
||||
### `sdk/src/media/LivekitMediaTrack.ts`
|
||||
|
||||
- `createLivekitMediaTrack` stops taking `source` as a parameter and derives it
|
||||
from `publication.source` through the inverse of `livekitSources`, with
|
||||
`Track.Source.Unknown` as `"unknown"`.
|
||||
- Audio tracks get `isActive$`: `observeParticipantEvents(participant,
|
||||
ParticipantEvent.IsSpeakingChanged)` mapped to `isSpeaking`, combined with
|
||||
`muted$`, as a `scope.behavior` seeded from `participant.isSpeaking &&
|
||||
!publication.isMuted`.
|
||||
- `setVolume` acts on the track instead of the source. Today it branches on
|
||||
`source` and calls `RemoteParticipant.setVolume(volume, Track.Source.X)`, which
|
||||
has nothing to do for an `"unknown"` audio track. `RemoteAudioTrack.setVolume`
|
||||
exists and is per track; the local no-op stays. `track$` is already a behavior
|
||||
here, so the volume is reapplied when a remote track arrives on subscription, the
|
||||
same way the audio context is.
|
||||
|
||||
### `sdk/src/media/LivekitMemberMedia.ts`
|
||||
|
||||
- `speaking$`, `screenShareEnabled$` and the four `track$` calls go; `memberTrack$`
|
||||
goes with them.
|
||||
- `tracks$` is built from `observeParticipantMedia(participant)` (it already fires
|
||||
on publish, unpublish, subscribe and mute) mapped to
|
||||
`[...participant.trackPublications.values()]`, then through `generateItems` from
|
||||
`reactive/observable.ts`, keyed by `trackSid`, so each publication gets one
|
||||
`createLivekitMediaTrack` in a scope that ends when the publication leaves the
|
||||
map. This is the pattern `MatrixRTCClient.ts` uses for `remoteMembers$`; nothing
|
||||
new is written for it.
|
||||
- Publication order is the map's insertion order, which `generateItems` preserves.
|
||||
|
||||
### `sdk/src/utils/tracks.ts`
|
||||
|
||||
`trackBySource$` as above, exported from `sdk/index.ts`.
|
||||
|
||||
### `sdk/dev/main.ts`
|
||||
|
||||
`memberTile` renders from `trackBySource$(scope, tracks$, "camera")` and
|
||||
`"microphone"`, where `tracks$` is `media$` switched to `media?.tracks$ ?? of([])`.
|
||||
`render` adds one label, `data-active`, from `isActive$` when the track is audio.
|
||||
The `member`, `video` and `audio` test ids do not change, so the existing specs
|
||||
keep passing untouched.
|
||||
|
||||
### Documentation
|
||||
|
||||
- `SdkArchitecture.md`: the `MemberMedia` block in "Media: `MemberMedia`", the
|
||||
diagram line that lists `speaking$ microphone$ camera$ screenShare$`, and the
|
||||
harness description that mentions `media$.camera$` and `media$.microphone$`.
|
||||
- `SdkMigration.md`: a new row in the status table once the branch question above
|
||||
is settled.
|
||||
- `MediaBackendPlan.md`: no change; it already treats `media/` as the clean seam and
|
||||
this plan only makes that seam less call-shaped.
|
||||
|
||||
## Tests
|
||||
|
||||
### Unit
|
||||
|
||||
- `LivekitMediaTrack.test.ts`, existing file: `isActive$` follows
|
||||
`IsSpeakingChanged`; a muted publication reports `false` while the participant
|
||||
speaks; `source` is derived from the publication, including `"unknown"`;
|
||||
`setVolume` reaches the remote track, and again when a track arrives later.
|
||||
- `LivekitMemberMedia.test.ts`, new: `tracks$` lists every publication in order;
|
||||
the same object is returned for a publication across unrelated media events (a
|
||||
mute on another track); a publication replaced under the same source yields a new
|
||||
object and ends the old one's scope; an unknown-source publication appears with
|
||||
`source: "unknown"`; screen share presence is derivable with `some`.
|
||||
- `tracks.test.ts`, new: `trackBySource$` with marbles, including the transition
|
||||
from undefined to a track and back.
|
||||
|
||||
The fakes come from `sdk/src/utils/test.ts` (`mockRemoteParticipant`) and the
|
||||
`fakes()` helper already in `LivekitMediaTrack.test.ts`; the latter moves to
|
||||
`test.ts` if the new file needs it, rather than being copied.
|
||||
|
||||
### End-to-end
|
||||
|
||||
`playwright/sdk/media.spec.ts` gains one test: the peer's `audio` element reaches
|
||||
`data-active="true"` after a mute and unmute, and `"false"` while muted. It is
|
||||
marked `fixme`: Chromium's fake microphone never registers as a speaker at the
|
||||
SFU, and Firefox's steady tone does so in about two runs out of three, with or
|
||||
without the browser's audio processing. It passed four times in six, which proves
|
||||
the pipeline, and it stays out of the gate until the dev SFU gets a lower
|
||||
speaker threshold or the harness a louder fake device. The existing five tests
|
||||
run as they are.
|
||||
|
||||
## Order of work
|
||||
|
||||
1. `api.ts`, `tracks.ts` and the unit tests for `trackBySource$`.
|
||||
2. `LivekitMediaTrack.ts`: source from the publication, `isActive$`, per-track
|
||||
volume, with its tests.
|
||||
3. `LivekitMemberMedia.ts`: `tracks$`, remove the rest, with its new test file.
|
||||
4. `dev/main.ts` and the new spec.
|
||||
5. Documentation.
|
||||
|
||||
Steps 1 to 3 do not compile against the harness until step 4, so they land as one
|
||||
PR; it is under 400 lines because the media adapter is small. `pnpm lint`,
|
||||
`pnpm test` and the `playwright/sdk` project are the gates. `knip` will not
|
||||
complain about `trackBySource$` since the harness uses it.
|
||||
|
||||
## Other possible call-related refactors
|
||||
|
||||
Everything else in `sdk/src/api.ts` and `sdk/src/session/` that only makes sense
|
||||
for a call. None of it is in this plan; each would be its own slice. The first
|
||||
three are the ones this plan makes visible.
|
||||
|
||||
| What | Why it is call-specific | What instead |
|
||||
| --------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `MediaSource` itself | Four call names plus `"unknown"`. LiveKit's `Track.Source` has the same four, so the type is faithful to the wire today, but a whiteboard or a bot has no name for its track. | Keep `source` for the backend's notion; add an optional `name` (LiveKit `trackName`) or a description in `applicationData`, once an application needs it. |
|
||||
| `LocalRTCMember.sharingScreen$`, `toggleScreenSharing`, `screenShareError$`, `dismissScreenShareError` | A screen share is one publication among others. `sharingScreen$` is `participant.isScreenShareEnabled`, now `tracks$.some(...)` on the local media. | A generic `publish(source, capture)` / `unpublish(id)` on the local member, with its error on the returned promise; `sharingScreen$` derived by the host from `tracks$`. |
|
||||
| `LocalMemberMedia.switchCamera` | "Camera" is the call's name for the local video track with a `facingMode$`. | `switchFacingMode()` on the local `VideoMediaTrack`, next to `facingMode$`. |
|
||||
| `MatrixRTCClient.setMicrophoneEnabled` / `setCameraEnabled` | Two fixed publications the client knows by name. | `setEnabled(enabled)` on the local track, or `publish`/`unpublish`; `LocalMember.setEnabled` and `Publisher.setEnabled` already take a `Track.Source` and would take a track id. |
|
||||
| `LocalMediaInputs` (`microphoneEnabled`, `cameraEnabled`, `audioInputDeviceId$`, `videoInputDeviceId$`, `videoProcessor$`) | Describes exactly one microphone and one camera. `audioOutputDeviceId$` is about playback and is not call-specific. | A list of desired local publications, each with its device, processor and enabled state; `Publisher.desired.microphone$` / `camera$` become entries in it. |
|
||||
| `CaptureSettings { audio, camera, screenShare }` and `MediaQuality { video, screen_share }` | Keyed by the four sources. | Per-publication capture and encoding options, passed with `publish`. `config.ts` keeps the host-facing `MediaQuality` shape for `config.json`, mapped at the edge. |
|
||||
| `AudioMediaTrack.setVolume` branching on source | Covered by this plan (per-track volume); listed so the row above it is complete. | Done here. |
|
||||
| `Track.Source` plumbing in `Publisher.ts` (`pauseUpstreams`, `resumeUpstreams`, the processor on the camera) and `LocalMember.ts` | Internal, but the three sources are enumerated by hand in five places. | Iterate the local publications; attach the processor to every local video track whose publication asks for one. |
|
||||
| `applicationData` carrying only `m.call.intent`; `application` defaulting to `m.call` | The js-sdk's membership format carries one key. The default is a convenience for the one application that exists. | Pass the record through once the js-sdk can; keep the default but say so in the SDK README. Already an open item in `SdkArchitecture.md`. |
|
||||
| `membership$: Behavior<CallMembership>` on `RTCMember` | The js-sdk's type name. The SDK does not get to rename it, but every host sees the word "call" in a session API. | Re-export under an SDK alias (`RTCMembership`) and narrow to what hosts read: `userId`, `deviceId`, `memberId`, `rtcBackendIdentity`, `getTransport()`, `applicationData`. |
|
||||
| `UnknownCallError` in `errors.ts` | Name only. | Rename with the two `Livekit*` errors in slice 11 of `SdkMigration.md`. |
|
||||
| `observeSharingScreen$` in `LocalMember.ts` | Reads `participant.isScreenShareEnabled`. | Goes with `sharingScreen$`, or reads `tracks$` from the local media in the meantime. |
|
||||
| `sendNotificationType` | Kept on purpose; see "Call-specific things kept out of the client" in `SdkArchitecture.md`. | Nothing. |
|
||||
|
||||
## Plan for the refactors above
|
||||
|
||||
Four slices, in dependency order. Each is one PR, green on its own, with the
|
||||
`playwright/sdk` specs and the dev harness as the proof that behaviour is kept.
|
||||
Element Call follows on its own branch after each.
|
||||
|
||||
### 1. Local track controls
|
||||
|
||||
The local member's tracks become objects a host can act on, instead of methods
|
||||
named after a call's devices.
|
||||
|
||||
- **Add** `LocalAudioMediaTrack` and `LocalVideoMediaTrack` (`local: true`) in
|
||||
`api.ts`, with `setEnabled(enabled)`, `setDevice(deviceId)` and, on video,
|
||||
`switchFacingMode()` and `setProcessor(processor)`. `tracks$` on
|
||||
`LocalMemberMedia` is typed to them.
|
||||
- **Remove** `LocalMemberMedia.switchCamera`, and `audioInputDeviceId$`,
|
||||
`videoInputDeviceId$`, `videoProcessor$` from `LocalMediaInputs`. The host
|
||||
subscribes to its settings and calls the track; the SDK stops watching
|
||||
behaviors it does not own. `audioOutputDeviceId$` stays: playback is not a
|
||||
track.
|
||||
- **Keep** `setMicrophoneEnabled` / `setCameraEnabled` for now: they also cover
|
||||
"no track yet", which slice 2 solves.
|
||||
- **Touches** `LivekitMediaTrack.ts`, `Publisher.ts` (the processor and device
|
||||
code moves onto the track), `MatrixRTCClient.ts`, the harness buttons.
|
||||
|
||||
### 2. `publish` and `unpublish`
|
||||
|
||||
One way to put media on the session, whatever it is.
|
||||
|
||||
- **Add** to `LocalRTCMember`: `publish(request): Promise<LocalMediaTrack>` and
|
||||
`unpublish(id): Promise<void>`. A request is `{ source, deviceId?, capture? }`
|
||||
for `"microphone"` and `"camera"`, `{ source: "screenShare", audio?, capture?
|
||||
}` for a screen. Failures reject the promise with a `MatrixRTCError`.
|
||||
- **Remove** `sharingScreen$`, `toggleScreenSharing`, `screenShareError$`,
|
||||
`dismissScreenShareError`, `observeSharingScreen$`, and the two
|
||||
`set*Enabled` methods on the client. "Sharing" is `tracks$.some(...)` on the
|
||||
local media; "enable" is `publish` or `track.setEnabled`.
|
||||
- **Replace** `LocalMediaInputs.microphoneEnabled` / `cameraEnabled` with
|
||||
`publish: PublishRequest[]`, applied once the transport is up, as today.
|
||||
`CaptureSettings` becomes the `capture` of a request; `MediaQuality` keeps its
|
||||
`config.json` shape and is mapped to per-request capture at the edge.
|
||||
- **Touches** `LocalMember.ts` (the screen share block and `setEnabled`),
|
||||
`Publisher.ts` (iterates requests instead of enumerating three sources),
|
||||
`livekitOptions.ts`, `lifecycle.spec.ts` and `media.spec.ts` for the new
|
||||
buttons.
|
||||
|
||||
### 3. Names that say "call"
|
||||
|
||||
No behaviour change; one PR of renames and type narrowing.
|
||||
|
||||
- `RTCMember.membership$` is typed as an SDK `RTCMembership`: the subset of
|
||||
`CallMembership` hosts read (`userId`, `deviceId`, `memberId`,
|
||||
`rtcBackendIdentity`, `getTransport()`, `applicationData`). The js-sdk object
|
||||
is still what is behind it.
|
||||
- `UnknownCallError` is renamed with the two `Livekit*` errors, as slice 11 of
|
||||
`SdkMigration.md` already plans; `index.ts` drops its alias.
|
||||
- `applicationData` is passed through whole once the js-sdk accepts more than
|
||||
`m.call.intent`; until then the README says which key survives.
|
||||
|
||||
### Not planned
|
||||
|
||||
- `MediaSource` keeps its five values. A `name` on `MediaTrack` (LiveKit's
|
||||
`trackName`) is a one-line addition when the first non-call application asks
|
||||
for it; adding it now would be a guess at that application's needs.
|
||||
- `sendNotificationType` stays, as `SdkArchitecture.md` already decided.
|
||||
|
||||
## Changed behaviour
|
||||
|
||||
What a host sees differently, per slice. All four blocks are implemented.
|
||||
|
||||
### `tracks$` and `isActive$` (this plan, implemented)
|
||||
|
||||
| Before | After |
|
||||
| ---------------------------------------------------------------------- | -------------------------------------------------------------------- |
|
||||
| `media.camera$`, `.microphone$`, `.screenShare$`, `.screenShareAudio$` | `trackBySource$(scope, media.tracks$, "camera")`, and so on |
|
||||
| `media.screenShareEnabled$` | `media.tracks$` mapped to `some((t) => t.source === "screenShare")` |
|
||||
| `media.speaking$` | `isActive$` on the `"microphone"` track; `false` while there is none |
|
||||
| `createLivekitMediaTrack(…, source)` | the source is read from the publication (internal) |
|
||||
|
||||
Differences in what the values mean:
|
||||
|
||||
- **Tracks published without a source now appear**, with `source: "unknown"`.
|
||||
Before, they were invisible. A host that renders every track must be ready
|
||||
for a track it has no name for.
|
||||
- **`isActive$` is `false` while the track is muted**, even if LiveKit still
|
||||
reports the member as speaking. `speaking$` was not gated by mute.
|
||||
- **`isActive$` is the same on every audio track of a member**, because LiveKit
|
||||
detects speakers per participant. A member sharing a screen with audio lights
|
||||
both tracks while either carries sound.
|
||||
- **`setVolume` applies to the track, not the source.** It is remembered and
|
||||
reapplied when the remote track arrives on subscription, so a volume set
|
||||
before the track exists is no longer lost. It still does nothing on our own
|
||||
tracks.
|
||||
- **One `MediaTrack` object per publication**, stable across unrelated media
|
||||
events. Before, a slot's object also changed only with its publication, so
|
||||
nothing changes for the four known sources.
|
||||
- **The list is in publication order** (the participant's map order). Nothing
|
||||
before had an order.
|
||||
- The harness labels audio elements with `data-active`. Test ids are unchanged.
|
||||
|
||||
### Slice 1, local track controls
|
||||
|
||||
- `switchCamera()` on the local media becomes `switchFacingMode()` on the local
|
||||
camera track, so there is always a track to call it on; it still resolves
|
||||
`undefined` where the facing mode is unknown.
|
||||
- Input device and processor changes are calls on the track. The SDK no longer
|
||||
reacts to `audioInputDeviceId$`, `videoInputDeviceId$` or `videoProcessor$`;
|
||||
a host that changed a setting and expected the SDK to follow now calls
|
||||
`setDevice` or `setProcessor` itself. The initial device and processor travel
|
||||
with the publish request instead.
|
||||
- `setEnabled(false)` on a local track mutes it and keeps the publication, as
|
||||
`setMicrophoneEnabled(false)` does today.
|
||||
|
||||
### Slice 2, `publish` and `unpublish`
|
||||
|
||||
- Screen sharing is `publish({ source: "screenShare" })` and `unpublish(id)`.
|
||||
A capture failure rejects the promise; there is no `screenShareError$` to
|
||||
dismiss.
|
||||
- "Am I sharing" is derived from `tracks$` by the host. The SDK no longer
|
||||
exposes `sharingScreen$`.
|
||||
- `setMicrophoneEnabled` / `setCameraEnabled` are gone. Before the transport
|
||||
is connected, `publish` is remembered and applied once it is, which is what
|
||||
the two methods did.
|
||||
- The initial microphone and camera are `LocalMediaInputs.publish`, a list;
|
||||
an empty list joins without publishing, which needed `microphoneEnabled:
|
||||
false, cameraEnabled: false` before.
|
||||
- Capture settings travel with each request. A host that set
|
||||
`capture.screenShare` once sets it on each screen share publish instead.
|
||||
|
||||
### Slice 3, names
|
||||
|
||||
- `membership$` is typed as `RTCMembership`. A host that reached for a
|
||||
`CallMembership` member outside the narrowed set must import the js-sdk type
|
||||
itself; the object is unchanged.
|
||||
- `UnknownCallError` is `UnknownRTCError` in the SDK too, not only in the
|
||||
export. Error codes do not change. The two `Livekit*` errors are left to
|
||||
slice 11 of `SdkMigration.md`, which renames them on both sides at once.
|
||||
- `applicationData` is forwarded whole: the js-sdk takes it next to
|
||||
`m.call.intent`, which the membership format carries as a field of its own.
|
||||
+60
-20
@@ -1,55 +1,95 @@
|
||||
# MatrixRTC SDK (EXPERIMENTAL)
|
||||
|
||||
`@element-hq/matrixrtc-sdk` is the call model under Element Call, on its own:
|
||||
MatrixRTC memberships and transports, media connections, publishing, E2EE keys and
|
||||
the media of every member, as observables. It has no UI. Element Call's own
|
||||
`CallViewModel` is meant to become one consumer of it. The design is in
|
||||
[`SdkArchitecture.md`](./SdkArchitecture.md), the migration from the view model
|
||||
in [`SdkMigration.md`](./SdkMigration.md). Media goes through one media backend
|
||||
behind `MediaBackend`; LiveKit, under `src/media-backend/livekit/`, is the only one
|
||||
today and the only place that imports `livekit-client`.
|
||||
it joins a MatrixRTC slot in a Matrix room, writes and watches the memberships,
|
||||
connects to the media transport (LiveKit today), publishes the local media and
|
||||
exposes every member and their tracks as observables. It has no UI. Element
|
||||
Call's own `CallViewModel` is one consumer of it; any host can build a different
|
||||
view on top. The design is in [`SdkArchitecture.md`](./SdkArchitecture.md), what
|
||||
moved out of Element Call's `src/` in [`SdkMigration.md`](./SdkMigration.md).
|
||||
|
||||
**Status:** first implementation. `createMatrixRTCClient` joins the session, connects to
|
||||
the transport, publishes the local media and exposes every member's media; the
|
||||
development harness and its e2e tests in `playwright/sdk` drive it, and Element
|
||||
Call's own `CallViewModel` is built on it. It depends on nothing in Element Call's
|
||||
`src/`.
|
||||
**Status:** first implementation. `createRTCSlot` and `slot.join()` join the
|
||||
session, connect to the transport, publish the local media and expose every
|
||||
member's media; the development harness and its e2e tests in `playwright/sdk`
|
||||
drive it, and Element Call's own `CallViewModel` is built on it. It depends on
|
||||
nothing in Element Call's `src/`.
|
||||
|
||||
## Using it
|
||||
|
||||
```ts
|
||||
import {
|
||||
createMatrixRTCClient,
|
||||
createRTCSlot,
|
||||
E2eeType,
|
||||
MatrixRTCMode,
|
||||
ObservableScope,
|
||||
trackBySource$,
|
||||
} from "@element-hq/matrixrtc-sdk";
|
||||
|
||||
const scope = new ObservableScope();
|
||||
const rtcClient = createMatrixRTCClient(
|
||||
const slot = createRTCSlot(
|
||||
scope,
|
||||
client, // a matrix-js-sdk MatrixClient, logged in and syncing
|
||||
room, // the matrix-js-sdk Room, from client.getRoom() once the join has synced
|
||||
{
|
||||
encryptionSystem: { kind: E2eeType.PER_PARTICIPANT },
|
||||
matrixRTCMode: MatrixRTCMode.Compatibility,
|
||||
publish: [{ source: "microphone" }, { source: "camera" }],
|
||||
},
|
||||
);
|
||||
rtcClient.join();
|
||||
const participation = slot.join({
|
||||
publish: [{ source: "microphone" }, { source: "camera" }],
|
||||
});
|
||||
|
||||
rtcClient.remoteMembers$.subscribe((members) => {
|
||||
// each member has displayName$, tracks$ and more; a media track is rendered
|
||||
// by handing it a <video> or <audio> element: track.attach(element)
|
||||
// A <video> for each remote member's camera and an <audio> for its microphone.
|
||||
// A track plays on an element once attached to it; detach before the element
|
||||
// leaves the DOM. (`tileFor`, `setName` and `removeTilesExcept` are the host's.)
|
||||
participation.remoteMembers$.subscribe((members) => {
|
||||
for (const member of members) {
|
||||
const { video, audio } = tileFor(member.memberId);
|
||||
member.displayName$.subscribe((name) => setName(member.memberId, name));
|
||||
trackBySource$(scope, member.tracks$, "camera").subscribe((track) =>
|
||||
track?.attach(video),
|
||||
);
|
||||
trackBySource$(scope, member.tracks$, "microphone").subscribe((track) =>
|
||||
track?.attach(audio),
|
||||
);
|
||||
}
|
||||
removeTilesExcept(members.map((m) => m.memberId));
|
||||
});
|
||||
|
||||
// later
|
||||
rtcClient.leave();
|
||||
participation.leave();
|
||||
scope.end();
|
||||
```
|
||||
|
||||
`participation.localMember$` is one member of the same shape, null until our own
|
||||
membership is in the room; its microphone would only echo.
|
||||
|
||||
[`dev/main.ts`](./dev/main.ts) is this example as a working page.
|
||||
|
||||
## Reading the code
|
||||
|
||||
Three files define the surface. Read them in this order.
|
||||
|
||||
- [`src/api.ts`](./src/api.ts): the MatrixRTC side. An `RTCSlot` is the
|
||||
namespace in a room that memberships join into; `join()` returns an
|
||||
`RTCParticipation`, which lasts until its `leave()` and exposes `state$`,
|
||||
`localMember$` and `remoteMembers$`. Members, transports and the participation
|
||||
state machine live here. `createRTCSlot` in `RTCSlot.ts` is the only way to get
|
||||
an implementation.
|
||||
- [`src/media-api.ts`](./src/media-api.ts): media as a host sees it, on
|
||||
`remoteMembers$` and `localMember$`. `PublishRequest` says what to publish
|
||||
(microphone, camera, screen share); `MediaTrack` is how it arrives. The
|
||||
underlying tracks stay hidden: the host hands an element to `attach` and the
|
||||
SDK plays into it. The local variants carry the controls (`setEnabled`,
|
||||
`setDevice`, `setProcessor`).
|
||||
- [`src/media-backend/api.ts`](./src/media-backend/api.ts): internal, the seam
|
||||
between the session and whatever carries the media. The session speaks
|
||||
MatrixRTC; a `MediaBackend` speaks one media protocol and hands back the
|
||||
`MemberMedia` of each member through `mediaFor()`. A host rarely touches it;
|
||||
it matters when working on the LiveKit backend under
|
||||
`src/media-backend/livekit/`, the only place that imports `livekit-client`, or
|
||||
when writing a new backend.
|
||||
|
||||
## Development
|
||||
|
||||
Everything runs from the repository root. This directory is a pnpm project of its
|
||||
|
||||
@@ -10,8 +10,8 @@ interface.
|
||||
Element Call becomes one consumer of it: `CallViewModel` keeps what makes a session
|
||||
a _call_ (ringing, auto-leave, layout, sounds, reactions, settings, the host
|
||||
bridge) and takes everything else from the client. The dependency direction is
|
||||
therefore `Element Call → sdk`, never the reverse. The migration from today's
|
||||
`CallViewModel` to that shape is in [`SdkMigration.md`](./SdkMigration.md).
|
||||
therefore `Element Call → sdk`, never the reverse. What moved out of Element
|
||||
Call's `src/` to get there is listed in [`SdkMigration.md`](./SdkMigration.md).
|
||||
|
||||
## The architecture
|
||||
|
||||
@@ -570,9 +570,7 @@ and learns of remote members only through `mediaFor$`, registering each
|
||||
membership's transport for as long as the member's scope lives. Media keys reach
|
||||
it as a stream, `mediaKeys$`, replayed on subscribe; the js-sdk session never
|
||||
does. A host supplies its own backend through `MatrixRTCClientOptions.backend`;
|
||||
without one, LiveKit is built from the LiveKit-flavoured client options. The
|
||||
reasoning behind each of these choices is in
|
||||
[`MediaBackendPlan.md`](./MediaBackendPlan.md).
|
||||
without one, LiveKit is built from the LiveKit-flavoured client options.
|
||||
|
||||
### Authenticating with a LiveKit transport
|
||||
|
||||
@@ -769,7 +767,7 @@ sdk/
|
||||
src/utils/ mapScoped, network retry, display names, test helpers
|
||||
dev/ the harness (below)
|
||||
SdkArchitecture.md this document
|
||||
SdkMigration.md how Element Call gets from CallViewModel to the client
|
||||
SdkMigration.md what moved from Element Call's src/ into the SDK
|
||||
```
|
||||
|
||||
The lint rule `element-call/sdk-import-boundary` keeps it that way: nothing under
|
||||
|
||||
+47
-360
@@ -1,366 +1,53 @@
|
||||
# From `CallViewModel` to `MatrixRTCClient`
|
||||
# What moved from `src/` into the SDK
|
||||
|
||||
How Element Call gets from today's `CallViewModel`, which does everything, to a
|
||||
view model that consumes the SDK. The target shape is in
|
||||
[`SdkArchitecture.md`](./SdkArchitecture.md); this document is the inventory of
|
||||
what moves, what stays, and how far along it is. It refers to source files and
|
||||
symbols, so it ages with the code; the architecture document should not.
|
||||
Element Call's `CallViewModel` used to do everything; now it consumes the SDK and
|
||||
keeps only what makes a session a call. This is the inventory of what moved out
|
||||
of `src/` on the way, so that a reader of either side knows where to look. It
|
||||
names source files, so it ages with the code; the shape they make is in
|
||||
[`SdkArchitecture.md`](./SdkArchitecture.md).
|
||||
|
||||
## Status
|
||||
Everything in the table is gone from `src/`. Each module came with its tests;
|
||||
`sdk/src/utils/test.ts` and `test-fixtures.ts` are the subset of Element Call's
|
||||
test helpers they need. `Behavior`, `ObservableScope`, the observable operators,
|
||||
`E2eeType`, `EncryptionSystem` and `MatrixRTCMode` are re-exported from the SDK
|
||||
by their old `src/` paths, so the rest of Element Call did not have to change.
|
||||
|
||||
| Slice | State |
|
||||
| -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------- |
|
||||
| 1. Scaffold: `sdk/` packaging, the interface, the harness, the smoke test | done, October 2026 |
|
||||
| 2. First implementation of `createMatrixRTCClient` under `sdk/src`, the media adapter, the e2e suite | done, 1 October 2026 |
|
||||
| 3. Copy the shared modules into `sdk/src/` with their tests, drop every `src/` import, turn on the import boundary | done, 2 October 2026 |
|
||||
| 4. `createCallViewModel$` consumes the client; media view models take `media$`; Element Call imports the primitives and enums from the SDK and deletes its own copies | done, 2 October 2026 |
|
||||
| 5. `sendData`/`data$` on the client; the legacy bundle `sdk-target-based-on-call-view-model/` is rebuilt on the SDK with its interface unchanged, since it is in production use | done, 2 October 2026 |
|
||||
| 12. The media backend seam: everything LiveKit behind `MediaBackend` in `sdk/src/media-backend/livekit/`, the `backend` option, the lint guard; see [`MediaBackendPlan.md`](./MediaBackendPlan.md) | done, 4 October 2026 |
|
||||
| Was in `src/` | Is in `sdk/src/` | What changed on the way |
|
||||
| ----------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `state/Behavior.ts`, `state/ObservableScope.ts`, `utils/observable.ts` | `reactive/` | as is |
|
||||
| `state/SessionBehaviors.ts` | `matrixrtc/JsRtcSessionBehaviors.ts` | the memberships and key rotation behaviors only |
|
||||
| `state/CallViewModel/localMember/HomeserverConnected.ts` | `matrixrtc/HomeserverConnected.ts` | the grace period defaults to `defaultSessionTimings` instead of `Config.get()` |
|
||||
| `state/CallViewModel/localMember/LocalMember.ts` | `matrixrtc/LocalMembership.ts`, `media-backend/livekit/LocalMedia.ts`, `participationState.ts` | split into the Matrix side, the media side and the state machine that combines them |
|
||||
| `state/CallViewModel/localMember/LocalTransport.ts`, `RtcTransportAutoDiscovery.ts` | `matrixrtc/LocalTransport.ts`, `TransportAutoDiscovery.ts` | the override and the fallback are transports from the options, not a setting and a resolved config |
|
||||
| `state/CallViewModel/remoteMembers/MatrixLivekitMembers.ts` | `media-backend/livekit/LivekitBackend.ts` (`mediaFor`) | matches participants to members inside the backend; the SDK hands out `RTCMember`s |
|
||||
| `state/CallViewModel/remoteMembers/MatrixMemberMetadata.ts`, `utils/displayname.ts` | `matrixrtc/MatrixMemberMetadata.ts`, `utils/displayname.ts` | as is |
|
||||
| `state/CallViewModel/remoteMembers/Connection.ts` | `media-backend/livekit/Connection.ts` | throws `MatrixRTCError`s; `ConnectionState` is a discriminated union; fetches its own token for its role |
|
||||
| `state/CallViewModel/remoteMembers/ConnectionFactory.ts` | `media-backend/livekit/ConnectionFactory.ts` | `LivekitConnectionFactory` builds the room options from the publish requests and `mediaQuality`, not from settings |
|
||||
| `state/CallViewModel/remoteMembers/ConnectionManager.ts` | `media-backend/livekit/ConnectionMap.ts` | keyed by `TransportLocator`; the publisher role is the local transport's |
|
||||
| `state/CallViewModel/localMember/Publisher.ts` | `media-backend/livekit/Publisher.ts` | rewritten on publish requests instead of `MuteStates` |
|
||||
| `livekit/openIDSFU.ts`, later `livekit/auth/` | `media-backend/livekit/auth/` | the flow is chosen by the transport's shape and our role; the delegated leave timeout is a parameter |
|
||||
| `livekit/options.ts` | `media-backend/livekit/livekitOptions.ts` | `buildLiveKitOptions(mediaQuality?)` only; the `Config` singleton is gone |
|
||||
| `e2ee/matrixKeyProvider.ts`, the key provider selection | `media-backend/livekit/MatrixKeyProvider.ts`, `KeyProvider.ts` | as is; selection keyed off `encryptionSystem` |
|
||||
| `e2ee/e2eeType.ts`, `EncryptionSystem` from `e2ee/sharedKeyManagement.ts` | `encryption.ts` | the React hooks stayed behind |
|
||||
| `MatrixRTCMode`, the timings and media quality types from `config/ConfigOptions.ts` | `config.ts` | `defaultSessionTimings` and `defaultMediaQuality` replace `DEFAULT_CONFIG` |
|
||||
| `utils/errors.ts` | `errors.ts` | `MatrixRTCError` with `code` and `category` replaces `ElementCallError`; messages are plain English, the host translates by code |
|
||||
| `doNetworkOperationWithRetry` from `utils/matrix.ts` | `utils/network.ts` | takes a `shouldRetry` predicate |
|
||||
| the join in `CallViewModel.ts` (`enterRTCSession`) | `matrixrtc/joinJsSdkSession.ts` | the timings and the delayed leave variant are parameters |
|
||||
| `livekit/MatrixAudioRenderer.tsx`, the participant reads in `state/media/` | `media-backend/livekit/LivekitMediaTrack.ts`, `LivekitMemberMedia.ts` | the view models get `MediaTrack`s with `attach`; nothing in `src/` imports a LiveKit participant any more |
|
||||
|
||||
Element Call now consumes the SDK. `InCallView` creates the client from the
|
||||
configuration, settings and URL parameters, and hands it to `createCallViewModel$`,
|
||||
which keeps what makes the session a call. The modules below were deleted from
|
||||
`src/` once the SDK had its copies, and `Behavior`, `ObservableScope`, the
|
||||
observable operators, `E2eeType`, `EncryptionSystem` and `MatrixRTCMode` are
|
||||
re-exported from the SDK by their old paths so the rest of Element Call did not
|
||||
have to change in the same step. The view model tests drive a fake
|
||||
`MatrixRTCClient` (`mockMatrixRTCClient` in `src/utils/test.ts`) instead of mocked
|
||||
LiveKit rooms; the tests of the deleted modules live on in `sdk/src`.
|
||||
## What still binds Element Call to LiveKit
|
||||
|
||||
## What still binds Element Call to LiveKit, as slices
|
||||
Each is a gap in the SDK's API rather than something to rename; the import goes
|
||||
away when the SDK grows the piece. The names Element Call owes to the outside
|
||||
stay as they are: the `livekit.livekit_service_url` key in `config.json`, the
|
||||
persisted setting keys `extended-livekit-logs` and `custom-transport`, the
|
||||
Posthog fields `callBackend` and `callReconnectingCountLivekit`, the rageshake
|
||||
field `call_backend`, and the translated error strings.
|
||||
|
||||
Element Call still imports the LiveKit packages in a few places. Each is a gap in
|
||||
the SDK's API rather than something to rename: the import goes away when the SDK
|
||||
grows the piece. The names Element Call owes to the outside stay as they are: the
|
||||
`livekit.livekit_service_url` key in `config.json`, the persisted setting keys
|
||||
`extended-livekit-logs` and `custom-transport`, the Posthog fields
|
||||
`callBackend` and `callReconnectingCountLivekit`, the rageshake field
|
||||
`call_backend`, and the translated error strings.
|
||||
|
||||
| Slice | Where it shows today |
|
||||
| --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------ |
|
||||
| 6. Preview before `join()`: an SDK API that creates the local tracks for the lobby and hands them out as `MediaTrack`s, so the preview renders like `MediaView` | `LobbyView` (`usePreviewTracks`), `VideoPreview` (`LocalVideoTrack`, `facingModeFromLocalTrack`) |
|
||||
| 7. A transport-neutral processor type on `LocalMediaInputs.videoProcessor$`, so the blur feature in `src/tracks/` no longer types against LiveKit | `TrackProcessorContext`, `BlurBackgroundTransformer`, the `TrackProcessor` type in `api.ts` |
|
||||
| 8. Device enumeration without LiveKit's helpers, on the browser API or an SDK utility | `MediaDevices` (`createMediaDeviceObserver`), `CallView` (`Room.getLocalDevices`) |
|
||||
| 9. A `logging` option on the client: log level and a sink for the transport's logs, so the rageshake wiring leaves `initializer.tsx` | `initializer.tsx`, the `extended-livekit-logs` setting |
|
||||
| 10. A `capabilities` field: output selection and background processors, asked of the SDK rather than of LiveKit | `CallFooterViewModel` (`supportsAudioOutputSelection`, `supportsBackgroundProcessors`) |
|
||||
| 11. Element Call's error and analytics types on the SDK's: `DisconnectReason` and the error classes, and a rename of the two `Livekit*` error classes on both sides at once | `PosthogEvents` (`DisconnectReason`), `errors.ts` (`ConnectionError`, `LivekitConnectionError`) |
|
||||
|
||||
## What was copied from `src/`, and what changed on the way
|
||||
|
||||
All of these are gone from `src/` as of slice 4; the table records where they went.
|
||||
|
||||
Each module came with its tests (`sdk/src/utils/test.ts` and `test-fixtures.ts` are
|
||||
the subset of `src/utils/test.ts` and `test-fixtures.ts` they need). In slice 4
|
||||
Element Call switches to the SDK's copy and deletes its own.
|
||||
|
||||
| In `src/` | In `sdk/src/` | What changed |
|
||||
| ----------------------------------------------------------------------------------- | --------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `state/Behavior.ts`, `state/ObservableScope.ts`, `utils/observable.ts` | `reactive/` | as is |
|
||||
| `state/SessionBehaviors.ts` | `session/SessionBehaviors.ts` | as is |
|
||||
| `state/CallViewModel/remoteMembers/Connection.ts` | `session/Connection.ts` | throws `MatrixRTCError`s; the SDK still subclasses it as `ResolvedConnection` to keep the JWT answer, to be folded in |
|
||||
| `state/CallViewModel/remoteMembers/ConnectionFactory.ts` | `session/ConnectionFactory.ts` | the interface only; `ECConnectionFactory` reads settings and `MediaDevices`, the SDK has `LivekitConnectionFactory` |
|
||||
| `state/CallViewModel/remoteMembers/ConnectionManager.ts` | `session/ConnectionManager.ts` | as is |
|
||||
| `state/CallViewModel/remoteMembers/MatrixLivekitMembers.ts` | `session/MatrixLivekitMembers.ts` | as is; the SDK maps its result to `RTCMember`, `TaggedParticipant` is internal |
|
||||
| `state/CallViewModel/remoteMembers/MatrixMemberMetadata.ts`, `utils/displayname.ts` | `session/MatrixMemberMetadata.ts`, `utils/displayname.ts` | as is |
|
||||
| `state/CallViewModel/localMember/HomeserverConnected.ts` | `session/HomeserverConnected.ts` | the grace period defaults to `defaultSessionTimings` instead of `Config.get()` |
|
||||
| `state/CallViewModel/localMember/LocalTransport.ts` | `session/LocalTransport.ts` | the type and `isLocalTransport`; discovery without the custom-url setting |
|
||||
| `state/CallViewModel/localMember/RtcTransportAutoDiscovery.ts` | `session/RtcTransportAutoDiscovery.ts` | takes `fallbackTransportUrl` instead of a resolved config |
|
||||
| `state/CallViewModel/localMember/LocalMember.ts` | `session/LocalMember.ts` | rewritten; only `observeSharingScreen$` was copied |
|
||||
| `livekit/openIDSFU.ts` | `session/openIDSFU.ts` | the delegated leave timeout is a parameter (`delayTimeoutMs`) instead of `Config.get()` |
|
||||
| `livekit/options.ts` | `session/livekitOptions.ts` | `buildLiveKitOptions(mediaQuality?)` only; `getLiveKitOptions` and the `Config` singleton are gone |
|
||||
| `e2ee/matrixKeyProvider.ts` | `session/MatrixKeyProvider.ts` | as is |
|
||||
| `e2ee/e2eeType.ts`, the `EncryptionSystem` type from `e2ee/sharedKeyManagement.ts` | `encryption.ts` | the React hooks stayed behind |
|
||||
| `MatrixRTCMode`, the timings and media quality types from `config/ConfigOptions.ts` | `config.ts` | `defaultSessionTimings` and `defaultMediaQuality` replace `DEFAULT_CONFIG` |
|
||||
| `utils/errors.ts` | `errors.ts` | `MatrixRTCError` with `code` and `category` replaces `ElementCallError`; messages are plain English, the host translates by code |
|
||||
| `doNetworkOperationWithRetry` from `utils/matrix.ts` | `utils/network.ts` | as is |
|
||||
|
||||
Rewritten in the SDK rather than copied, because the originals read settings,
|
||||
config, analytics or the host bridge: the local member (`session/LocalMember.ts`),
|
||||
the publisher (`session/Publisher.ts`), the connection factory and LiveKit room
|
||||
options (`session/ConnectionFactory.ts`), transport discovery
|
||||
(`session/LocalTransport.ts`), the join (`session/joinJsSdkSession.ts`) and the
|
||||
key provider selection (`session/KeyProvider.ts`).
|
||||
|
||||
## What becomes what in `createCallViewModel$`
|
||||
|
||||
Everything below `createCallViewModel$` that is not presentation and not in the
|
||||
call-specific table of the architecture document. Grouped by what it is today in
|
||||
`src/state/CallViewModel/CallViewModel.ts`.
|
||||
|
||||
| Binding today | In the SDK |
|
||||
| ---------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------- |
|
||||
| `livekitKeyProvider` via `getE2eeKeyProvider` | internal, keyed off `options.encryptionSystem` |
|
||||
| `matrixRTCMode` resolution from `Config.get()` | an input; resolution stays in the caller |
|
||||
| `matrixRTCSession` parameter | internal: the session manager for the slot, `getRoomSession(room)` |
|
||||
| `memberships$`, `membershipsAndTransports` | internal; surfaces as `RTCMember.transport$` |
|
||||
| `ownMembershipIdentity` | internal; surfaces as `RTCMember.id` |
|
||||
| `localTransport$` via `getLocalTransport` | internal |
|
||||
| `connectionFactory`, `connectionManager` | internal; surfaces as `connectedTransports$` |
|
||||
| `remoteMatrixLivekitMembers$` | `remoteMembers$` |
|
||||
| `localMembership` via `createLocalMembership$`, `enterRTCSession`, `Publisher`, `delayId$` | internal; surfaces as `join`, `leave`, `status$`, `connected$`, `reconnecting$`, the screen share fields |
|
||||
| `localRtcMembership$`, `localMatrixLivekitMember$` | `localMember$` |
|
||||
| `matrixLivekitMembers$`, `participantCount$` | `memberCount$` |
|
||||
| `matrixRoomMembers$`, `matrixMemberMetadataStore` | `RTCMember.displayName$` / `avatarUrl$` |
|
||||
| `allConnections$` | `connectedTransports$` |
|
||||
| `livekitRoomItems$` | dropped; audio renders from `remoteMembers$[].media$` |
|
||||
| `keyRotationSuppressed$` | same |
|
||||
| `sharingScreen$`, `toggleScreenSharing` | `localMember$.sharingScreen$`, `.toggleScreenSharing` |
|
||||
| `errors$` and the `fatalError$` mapping | `fatalError$`, as `MatrixRTCError` |
|
||||
| `join`, `leave`, `connected$`, `reconnecting$`, `screenShareError$`, `dismissScreenShareError` | same names on `MatrixRTCClient` / `LocalRTCMember` |
|
||||
| `callIntent` option, `updateCallIntent` from the camera | `applicationData["m.call.intent"]`; the camera sync is still in the SDK, see the open decisions |
|
||||
|
||||
Changes to the call sites that the rename brings: `localMember$` and
|
||||
`remoteMembers$` replace `localMatrixLivekitMember$` and
|
||||
`remoteMatrixLivekitMembers$`; `connection$` on a member goes away, since the view
|
||||
model only ever used `livekitRoom` (for `waitingForMedia$` and encryption errors,
|
||||
both on `MemberMedia` now) and `transport.livekit_service_url` (now
|
||||
`transport$.id`); `connectedTransports$` replaces `allConnections$`, since the
|
||||
developer panel only needed which transports exist and whether they are up.
|
||||
|
||||
## What stays in `CallViewModel`
|
||||
|
||||
Everything in the call-specific table, plus everything that takes `userMedia$` or a
|
||||
UI input and turns it into something to draw:
|
||||
|
||||
- Media view models (`createWrappedUserMedia` and below), built from
|
||||
`remoteMembers$` / `localMember$` instead of participants.
|
||||
- `CallNotificationLifecycle.ts` in full: `ringAttempts$`, `autoLeave$`,
|
||||
`ringingMedia$`, `ringingStatusLocation`.
|
||||
- `leave$` (= `autoLeave$` + `hangup()` + `hostBridge.hangUp$`), `hangup()`.
|
||||
- `screenShares$`.
|
||||
- Sounds: `joinSoundEffect$`, `leaveSoundEffect$`, `newHandRaised$`,
|
||||
`newScreenShare$`, `audibleReactions$`, `visibleReactions$`.
|
||||
- `handsRaised$`, `reactions$`, until they move onto `RTCMember`.
|
||||
- All layout: `spotlightSpeaker$`, `grid$`, `spotlight$`, `pip*`, `windowMode$`,
|
||||
`layoutSwitchVm`, every `*LayoutMedia$`, `layout$`, `visibleTiles$`,
|
||||
`tileStoreGeneration$`, `overflowing$`, `spotlightExpanded$`.
|
||||
- Chrome: `showHeader$`, `showFooter$`, `edgeToEdge$`, `showModals$`,
|
||||
`settingsOpen$`, `showSpotlightIndicators$`, `showSpeakingIndicators$`,
|
||||
`showNameTags$`, tap and hover subjects.
|
||||
- Audio routing: `earpieceMode$`, `audioOutputSwitcher$`, which read
|
||||
`MediaDevices`.
|
||||
- `CallViewModelOptions` keeps `hostBridge`, `controlledAudioDevices`, `header`,
|
||||
`showControls`, `hideScreensharing`, `waitForCallPickup`,
|
||||
`autoLeaveWhenOthersLeft`, `windowSize$`, `toggleScreensharing` (test override),
|
||||
and forwards `encryptionSystem`, `matrixRTCMode`, `sendNotificationType` and
|
||||
`applicationData` to `MatrixRTCClientOptions`.
|
||||
|
||||
## The media view models and the participant
|
||||
|
||||
The goal of slice 4 on the media side: nothing in Element Call imports
|
||||
`Participant`, `RemoteParticipant`, `LocalParticipant`, `TrackPublication`,
|
||||
`Track` or `TrackReference`. This is where the participant flows today, and what
|
||||
each use becomes.
|
||||
|
||||
### Where the participant flows today
|
||||
|
||||
Start: `remoteMatrixLivekitMembers$` in `CallViewModel.ts`, built by
|
||||
`createRemoteMatrixLivekitMembers$` in
|
||||
`src/state/CallViewModel/remoteMembers/MatrixLivekitMembers.ts`, as
|
||||
`participant: { type: "remote"; value$: Behavior<RemoteParticipant | null> }`.
|
||||
Consumers, in order of depth:
|
||||
|
||||
1. `MatrixLivekitMembers.ts` matches `participant.identity` against
|
||||
`membership.rtcBackendIdentity`; logs `participant.sid`.
|
||||
2. `CallViewModel.ts` `livekitRoomItems$` reads `isLocal`, `identity`, hands
|
||||
`connection.livekitRoom` plus identities to `LivekitRoomAudioRenderer`.
|
||||
3. `CallViewModel.ts` `userMedia$` passes the whole `TaggedParticipant` into
|
||||
`createWrappedUserMedia`.
|
||||
4. `src/state/media/WrappedUserMediaViewModel.ts`: `observeParticipantEvents(p,
|
||||
TrackPublished, TrackUnpublished, LocalTrackPublished, LocalTrackUnpublished)`
|
||||
then `p.isScreenShareEnabled` to spawn screen share view models.
|
||||
5. `src/state/media/UserMediaViewModel.ts`: `observeParticipantMedia(p)` for
|
||||
`microphoneTrack?.isMuted` / `cameraTrack?.isMuted`; `IsSpeakingChanged` →
|
||||
`p.isSpeaking`; `observeRtpStreamStats$(p, source, type)`.
|
||||
6. `src/state/media/MemberMediaViewModel.ts`: `observeTrackReference$(p, source)`
|
||||
for audio and video; `publication.isEncrypted`; `p.isLocal`; `p.identity`
|
||||
matched inside `RoomEvent.EncryptionError` messages; inbound RTP frame counters.
|
||||
7. `src/state/media/RemoteUserMediaViewModel.ts`, `RemoteScreenShareViewModel.ts`:
|
||||
`p.setVolume(v)` / `p.setVolume(v, ScreenShareAudio)`; `participant === null`
|
||||
means "waiting for media".
|
||||
8. `src/state/observeTrackReference.ts`: `p.getTrackPublication(source)` wrapped as
|
||||
`{ participant, publication, source }`.
|
||||
9. `src/state/media/observeRtpStreamStats.ts`: `publication.track` must be
|
||||
`RemoteTrack | LocalTrack`, then `track.getRTCStatsReport()`.
|
||||
10. Views: `src/tile/MediaView.tsx` renders `<VideoTrack trackRef={video}>` from
|
||||
`@livekit/components-react`; `src/livekit/MatrixAudioRenderer.tsx` uses
|
||||
`useTracks` on the LiveKit room, filters on `ref.participant.isLocal` /
|
||||
`.identity`, and calls `RemoteAudioTrack.setAudioContext` /
|
||||
`setWebAudioPlugins`.
|
||||
|
||||
Local-only extras, same shape but on `LocalParticipant`
|
||||
(`localMember/LocalMember.ts`, `Publisher.ts`,
|
||||
`src/state/media/LocalUserMediaViewModel.ts`): `trackPublications`,
|
||||
`audioTrackPublications`, `setScreenShareEnabled`, `setMicrophoneEnabled`,
|
||||
`setCameraEnabled`, `enableCameraAndMicrophone`, `isMicrophoneEnabled`,
|
||||
`isCameraEnabled`, `unpublishTrack`, and on the camera `LocalVideoTrack`:
|
||||
`facingModeFromLocalTrack`, `restartTrack`, `mediaStreamTrack`,
|
||||
`TrackEvent.Restarted` / `Muted`.
|
||||
|
||||
### Participant API actually used, and what covers it
|
||||
|
||||
| LiveKit surface | Used for | Used in | Now |
|
||||
| --------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------- | ----------- | ------------------------------------------------ |
|
||||
| `identity` | match to `rtcBackendIdentity`; audio renderer allow-list; E2EE error matching; logs | 1, 2, 6, 10 | matched inside the SDK; `RTCMember.id` |
|
||||
| `sid` | logs only | 1 | SDK logs |
|
||||
| `isLocal` | skip local audio; encryption status short-circuit | 2, 6, 10 | `MemberMedia.local` |
|
||||
| `isSpeaking` + `IsSpeakingChanged` | `speaking$` | 5 | `MemberMedia.speaking$` |
|
||||
| `isScreenShareEnabled` + publish events | spawn screen share VM | 4 | `MemberMedia.screenShareEnabled$` |
|
||||
| `observeParticipantMedia(p)` → `microphoneTrack?.isMuted`, `cameraTrack?.isMuted` | `audioEnabled$`, `videoEnabled$` | 5 | `microphone$` / `camera$` → `muted$` |
|
||||
| `getTrackPublication(source)` for the four sources | `TrackReference` for video, encryption, stats, screen share audio presence | 6, 7, 8 | the four track behaviors |
|
||||
| `publication.isEncrypted` | `unencryptedWarning$` | 6 | `MediaTrack.encrypted$` |
|
||||
| `publication.track.getRTCStatsReport()` | `audioStreamStats$`, `videoStreamStats$`, "receiving okay" heuristic | 5, 6, 9 | `MediaTrack.stats$` |
|
||||
| `setVolume(v)`, `setVolume(v, ScreenShareAudio)` | `VolumeControls` sink | 7 | `AudioMediaTrack.setVolume` |
|
||||
| `TrackReference` → `<VideoTrack>` | render video | 10 | `MediaTrack.attach` on a `<video>` |
|
||||
| room `useTracks` + `RemoteAudioTrack.setAudioContext/setWebAudioPlugins` | render audio with earpiece pan/gain | 10 | one `<audio>` per audio track, `setAudioContext` |
|
||||
| room `RoomEvent.EncryptionError` with `identity` + `"MissingKey"`/`"InvalidKey"` | `encryptionStatus$` | 6 | `MemberMedia.encryptionError$` |
|
||||
|
||||
Everything else in the media view models (`observeSpeaker$`, `VolumeControls`,
|
||||
`SortingBin`, hand raise, reactions, display name, avatar, aspect ratio) already
|
||||
works on plain booleans and observables and does not change.
|
||||
|
||||
### How the call sites map onto `MemberMedia`
|
||||
|
||||
- `audioEnabled$` = `microphone$ → track?.muted$ === false`; `videoEnabled$`
|
||||
likewise from `camera$`.
|
||||
- `unencryptedWarning$` = any of the four tracks with `encrypted$ === false`.
|
||||
- `encryptionStatus$` = `encryptionError$` plus a "frames decoded" check on
|
||||
`camera$.stats$` / `microphone$.stats$`; same logic, no participant.
|
||||
- `screenShares$` in `WrappedUserMediaViewModel` keys off `screenShareEnabled$`.
|
||||
- `VolumeControls` sink = `microphone$.value?.setVolume` /
|
||||
`screenShareAudio$.value?.setVolume`.
|
||||
- `waitingForMedia$` = transport present and `media === null`, unchanged
|
||||
semantics; `TaggedParticipant` goes away.
|
||||
- `video$: Behavior<TrackReference | undefined>` on `BaseMemberMediaViewModel`
|
||||
becomes `Behavior<VideoMediaTrack | undefined>`; `MediaView` replaces
|
||||
`<VideoTrack trackRef>` with a `<video>` plus `attach` / `detach` in an effect,
|
||||
keeping the `onResize` callback for the aspect ratio. `VideoPreview.tsx` already
|
||||
follows this pattern.
|
||||
- `LivekitRoomAudioRenderer` stops using `useTracks` on the room and renders one
|
||||
`<audio>` per `microphone$` / `screenShareAudio$` across `remoteMembers$`. That
|
||||
removes the "no matching matrix call member" warning path, since an unmatched
|
||||
track can no longer be seen.
|
||||
- The `showConnectionStats` setting gates the subscription to `stats$` in the view
|
||||
model; the SDK polls only while subscribed.
|
||||
- Stories and tests mock `attach` by setting `srcObject` from
|
||||
`canvas.captureStream()` on the element, or by doing nothing and asserting the
|
||||
call.
|
||||
|
||||
## Packaging, as it was planned
|
||||
|
||||
`sdk/` mirrors `component/` file for file. The table that drove the scaffold, kept
|
||||
for the record:
|
||||
|
||||
| `component/` | `sdk/` | Differences |
|
||||
| ------------------------------------------------------------------------------------ | ----------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `package.json` (`@element-hq/element-call-component`) | `package.json` (`@element-hq/matrixrtc-sdk`) | peers: `matrix-js-sdk`, `livekit-client`, `rxjs`. No React, no Compound. `exports`: `"."` only. No `./style.css`, no `sideEffects`. Same `prepare`. |
|
||||
| `pnpm-workspace.yaml` (marker, `autoInstallPeers: false`) | same | none |
|
||||
| `tsconfig.build.json` | same | `include: ["./index.ts"]` |
|
||||
| `index.tsx` + `api.ts` | `index.ts` + `src/api.ts` | one entry; no separate types entry until a host asks |
|
||||
| `host.ts`, `localization.ts`, `ElementCall.module.css`, `build/scopeStylesToRoot.ts` | none | no host bridge, no i18n, no styles |
|
||||
| `dev/` on port 3001 | `dev/` on port 3002 via `vite-sdk-dev.config.ts` | plain DOM |
|
||||
| `vite-component.config.ts` | `vite-sdk.config.ts` | `vitePluginsConfig({ mode, html: false })`, `outDir: "sdk/dist"`, externals = js-sdk and LiveKit entries plus `rxjs` |
|
||||
| root scripts `build:component*` | `build:sdk*` | `build:sdk:types` = `tsc -p sdk/tsconfig.build.json` |
|
||||
| root `tsconfig.json` `include` | adds `./sdk/**/*.ts` | |
|
||||
| `scripts/check-component-externals.mjs` | parameterised by config; `sdk` added to the sources, `sdk/dev` excluded | |
|
||||
| `knip.ts` | `vite-sdk.config.ts`, `vite-sdk-dev.config.ts`, `sdk/index.ts` as entry | |
|
||||
| `vitest.config.ts` unit project | adds `sdk/**/*.test.ts` | |
|
||||
| `package.json` `lint:oxlint` paths | adds `sdk` | |
|
||||
|
||||
Why not a pnpm workspace package yet: the root has no `packages:` in
|
||||
`pnpm-workspace.yaml`, so a workspace split means restructuring the app, the
|
||||
component's `prepare` trick, CI and the lockfile at the same time as moving the
|
||||
code, and dev-time HMR across workspace packages needs the same source alias
|
||||
anyway. It is the right end state if the SDK gets a release cadence of its own.
|
||||
|
||||
## Reference: the `CallViewModel` public API before the migration
|
||||
|
||||
Source: `src/state/CallViewModel/CallViewModel.ts`. Only consumer outside tests:
|
||||
`src/room/InCallView.tsx`. A snapshot, so that the split can be checked against
|
||||
what was there.
|
||||
|
||||
### Factory
|
||||
|
||||
`createCallViewModel$(scope, matrixRTCSession, matrixRoom, mediaDevices, muteStates, options, handsRaisedSubject$, reactionsSubject$, trackProcessorState$): CallViewModel`
|
||||
|
||||
- `scope: ObservableScope`: ending the scope is one of the two ways to leave.
|
||||
- `matrixRTCSession: MatrixRTCSession`, `matrixRoom: MatrixRoom`: one room per view model.
|
||||
- `mediaDevices: MediaDevices`, `muteStates: MuteStates`
|
||||
- `options: CallViewModelOptions` (below)
|
||||
- `handsRaisedSubject$: Observable<Record<string, RaisedHandInfo>>`
|
||||
- `reactionsSubject$: Observable<Record<string, ReactionInfo>>`
|
||||
- `trackProcessorState$: Behavior<ProcessorState>`
|
||||
|
||||
Helper: `callViewModelOptionsFromParams(params: UrlParams)` picks
|
||||
`controlledAudioDevices`, `header`, `showControls`, `hideScreensharing`,
|
||||
`sendNotificationType`, `callIntent` out of the URL params. `autoLeaveWhenOthersLeft`
|
||||
and `waitForCallPickup` are left to the caller.
|
||||
|
||||
### `CallViewModelOptions`
|
||||
|
||||
| Field | Type | Note |
|
||||
| ----------------------------------------------------------------------------------------------------------- | --------------------------- | --------------------------------------------------- |
|
||||
| `encryptionSystem` | `EncryptionSystem` | required |
|
||||
| `hostBridge?` | `HostBridge` | host can hang up / observe join+leave; default none |
|
||||
| `controlledAudioDevices?` | `boolean` | host controls audio output devices |
|
||||
| `header?` | `HeaderStyle` | default `Standard` |
|
||||
| `showControls?` | `boolean` | default `true` |
|
||||
| `hideScreensharing?` | `boolean` | default `false` |
|
||||
| `sendNotificationType?` | `RTCNotificationType` | notification sent on join |
|
||||
| `callIntent?` | `RTCCallIntent` | |
|
||||
| `autoLeaveWhenOthersLeft?` | `boolean` | |
|
||||
| `waitForCallPickup?` | `boolean` | telephone-style ringing UI |
|
||||
| `windowSize$` | `Behavior<{width; height}>` | required; drives `WindowMode` |
|
||||
| `matrixRTCMode?` | `MatrixRTCMode` | MatrixRTC version / compat mode |
|
||||
| `livekitRoomFactory?`, `connectionState$?`, `localTransport?`, `connectionFactory?`, `toggleScreensharing?` | | test overrides |
|
||||
|
||||
### The interface, grouped
|
||||
|
||||
**Lifecycle / connection** (to the client): `join()`, `leave()`, `hangup()`,
|
||||
`leave$: Observable<"user" | AutoLeaveReason>`, `autoLeave$`, `connected$`,
|
||||
`reconnecting$`, `allConnections$: Behavior<ConnectionManagerData>`,
|
||||
`fatalError$: Behavior<ElementCallError | null>`, `keyRotationSuppressed$`.
|
||||
|
||||
**Members / participants** (to the client): `participantCount$`,
|
||||
`localMatrixLivekitMember$`, `remoteMatrixLivekitMembers$`, `livekitRoomItems$`,
|
||||
`handsRaised$`, `reactions$` (keyed by `${userId}:${deviceId}`).
|
||||
|
||||
**Screen sharing** (capability to the client, error UI stays): `toggleScreenSharing`,
|
||||
`sharingScreen$`, `screenShareError$`, `dismissScreenShareError()`.
|
||||
|
||||
**Ringing** (stays): `ringingVm$`, `ringingStatusLocation`.
|
||||
|
||||
**Audio routing** (stays): `earpieceMode$`, `audioOutputSwitcher$`.
|
||||
|
||||
**Sounds and transient events** (stay): `joinSoundEffect$`, `leaveSoundEffect$`,
|
||||
`newHandRaised$`, `newScreenShare$`, `audibleReactions$`, `visibleReactions$`,
|
||||
`MAX_PARTICIPANT_COUNT_FOR_SOUND = 8`, `THROTTLE_SOUND_EFFECT_MS = 500`.
|
||||
|
||||
**Layout and tiles** (stay): `layout$`, `tileStoreGeneration$`,
|
||||
`showSpotlightIndicators$`, `showSpeakingIndicators$`, `showNameTags$`,
|
||||
`spotlightExpanded$`, `toggleSpotlightExpanded$`, `layoutSwitchVm$`.
|
||||
|
||||
**Chrome visibility and UI interaction** (stay): `showHeader$`, `showFooter$`,
|
||||
`edgeToEdge$`, `overflowing$`, `showModals$`, `settingsOpen$`, `setSettingsOpen$`,
|
||||
`tapScreen()`, `tapControls()`, `hoverScreen()`, `unhoverScreen()`.
|
||||
|
||||
### Pending rename noted in the source
|
||||
|
||||
A TODO above `CallViewModelOptions` proposes: member/membership → `rtcMember`,
|
||||
participant → `livekitParticipant`, matrixLivekitItem → `callMember`; and in the
|
||||
js-sdk `callMembership` → `rtcMembership`. The SDK's vocabulary (architecture
|
||||
document, "Vocabulary") settles the first three; the js-sdk rename is still open.
|
||||
|
||||
### Build targets before the migration
|
||||
|
||||
`vite-sdk.config.ts` (`pnpm build:sdk`), `vite-component.config.ts`,
|
||||
`vite-embedded.config.ts`, `vite.config.ts`. The old SDK target
|
||||
(`sdk-target-based-on-call-view-model/`) is the one whose dependency direction the
|
||||
migration flips. It is in production use, so slice 5 rebuilt it on the SDK instead of
|
||||
removing it: `createMatrixRTCSdk` keeps its signature, `join`/`leave`/`stop`,
|
||||
`connected$`, `data$` (`{ rtcBackendIdentity, data }` on the `matrixRTC` topic),
|
||||
`sendData`, `sendRoomMessage`, `localMember$` and `remoteMembers$` keep their shapes.
|
||||
The one break is inside the member objects: `connection` and `participant` are
|
||||
always `null` now, because the SDK does not hand out LiveKit objects; `membership`
|
||||
is unchanged. The bundle is removed once its consumers use the package directly.
|
||||
| Gap | Where it shows |
|
||||
| ----------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------ |
|
||||
| Preview before `join()`: local tracks for the lobby, handed out as `MediaTrack`s | `LobbyView` (`usePreviewTracks`), `VideoPreview` (`LocalVideoTrack`, `facingModeFromLocalTrack`) |
|
||||
| Device enumeration without LiveKit's helpers | `MediaDevices` (`createMediaDeviceObserver`), `CallView` (`Room.getLocalDevices`) |
|
||||
| A `logging` option: log level and a sink for the transport's logs | `initializer.tsx`, the `extended-livekit-logs` setting |
|
||||
| A `capabilities` field: output selection and background processors, asked of the SDK rather than of LiveKit | `CallFooterViewModel` (`supportsAudioOutputSelection`, `supportsBackgroundProcessors`) |
|
||||
| Element Call's error and analytics types on the SDK's | `PosthogEvents` (`DisconnectReason`), `errors.ts` (`ConnectionError`, `LivekitConnectionError`) |
|
||||
|
||||
Reference in New Issue
Block a user