mirror of
https://github.com/vector-im/element-call.git
synced 2026-10-16 23:45:50 +00:00
Put the media behind a backend seam and move LiveKit into it
The session speaks MatrixRTC and sees media through MediaBackend in sdk/src/backend/api.ts. Everything LiveKit lives in sdk/src/backend/livekit/, the only place in the SDK that imports livekit-client; the import boundary rule now enforces that outside sdk/src/backend/ the LiveKit packages appear in type imports only. The seam sits above the connection manager: one connection per transport url is the LiveKit multi-SFU strategy, not a MatrixRTC concept, so a cascading SFU or full mesh replaces the folder, not the session. A backend prepares the local transport the session discovered (the token exchange and the delegation probe, reported as canDelegateDelayedLeave), hands the delayed leave over per delay id, gives the local member one publish switch plus publish/unpublish, builds a remote member's media from its membership and registers that membership's transport for as long as the member lives, and reports its connections for the transport registry, which keys on the raw transport serialised with sorted keys. The backend's context carries no Matrix client (a backend picks what it calls in its own options), no membership list, and no js-sdk session: media keys arrive as a stream replayed on subscribe, and MatrixKeyProvider subscribes to it. The MatrixRTC mode no longer reaches LiveKit; it becomes a tokenEndpoint choice made where the default backend is built. Two behaviour changes ride along. Publishing is one boolean, joined and homeserver reachable, where before a sync hiccup after leave() resumed the upstream of tracks that were meant to stay silent. And ResolvedConnection keeps one ResolvedTransport object per fetch, so a state change no longer reads as a change of what was fetched. MatrixRTCClientOptions gains backend?: MediaBackendFactory, defaulting to createLivekitBackend built from the client options. authenticateWithTransport takes tokenEndpoint instead of matrixRTCMode.
This commit is contained in:
Timo K.
+132
-95
@@ -1,9 +1,16 @@
|
||||
# Media backend: trapping LiveKit behind one seam
|
||||
|
||||
> **Status: implemented, 4 October 2026.** All six steps landed in one change;
|
||||
> the interface is in `sdk/src/backend/api.ts`, the LiveKit backend in
|
||||
> `sdk/src/backend/livekit/`, and the architecture document has the section
|
||||
> "Media backend". The text below is the plan as it was written, kept for the
|
||||
> reasoning behind the shape.
|
||||
|
||||
The SDK's public API in `sdk/src/api.ts` names LiveKit in three places: the video
|
||||
processor type, the video codec in the capture settings, and the LiveKit-flavoured
|
||||
client options (`mediaQuality`, `capture`, `transportUrl`, `fallbackTransportUrl`).
|
||||
The implementation is a different story. LiveKit participants, rooms, connections,
|
||||
processor type in `PublishRequest` and `LocalVideoMediaTrack.setProcessor`, the
|
||||
video codec in `VideoCaptureSettings`, and the LiveKit-flavoured client options
|
||||
(`mediaQuality`, `transportUrl`, `fallbackTransportUrl`). The implementation is a
|
||||
different story. LiveKit participants, rooms, connections,
|
||||
tokens and data packets run through most of `sdk/src/session/`. Only the two
|
||||
adapters in `sdk/src/media/` are the clean boundary they were meant to be.
|
||||
|
||||
@@ -33,23 +40,23 @@ slice 11 (the error renames) get easier once the folder exists.
|
||||
Everything under `sdk/src/session/` that imports `livekit-client`,
|
||||
`@livekit/components-core`, or speaks the LiveKit token protocol:
|
||||
|
||||
| Module | What it is |
|
||||
| --- | --- |
|
||||
| `media/LivekitMediaTrack.ts`, `media/LivekitMemberMedia.ts` | Participant to `MemberMedia`. Clean already. |
|
||||
| `session/Connection.ts`, `ConnectionFactory.ts`, `ConnectionManager.ts` | One LiveKit room per transport URL, the multi-SFU policy, the token fetch per connection. |
|
||||
| `session/MatrixLivekitMembers.ts` | Matches a participant identity to a membership, epoch-paired. |
|
||||
| `session/Publisher.ts` | Local tracks, upstream pausing, device switching, the processor. |
|
||||
| `session/KeyProvider.ts`, `MatrixKeyProvider.ts` | LiveKit's key provider on the js-sdk key stream. |
|
||||
| `session/DataChannel.ts` | LiveKit data packets. |
|
||||
| `session/openIDSFU.ts`, `livekitOptions.ts` | The JWT service protocol, including delegation, and room options. |
|
||||
| `session/LocalTransport.ts` | The pre-join token fetch. Discovery itself is a homeserver endpoint and stays; see "Discovery stays in the session". |
|
||||
| `session/RtcTransportAutoDiscovery.ts` | Generic apart from its `isLivekitTransport` filter and the fallback URL. Stays. |
|
||||
| `session/LocalMember.ts` | `participant$`, `connection$`, upstream pausing, screen share, the delegation probe and handover, `Track.Source` in `setEnabled`. Roughly 150 of 530 lines. |
|
||||
| `session/Members.ts` | `mediaFor` takes a participant and a room. |
|
||||
| `session/Transports.ts` | Transport id, the connected check and `resolved$` assume LiveKit. |
|
||||
| `session/SessionBehaviors.ts` | `membershipsAndTransports$` filters to LiveKit transports. Deleted in step 3, not moved. |
|
||||
| `session/MatrixRTCClient.ts` | Wires all of the above. |
|
||||
| `utils/test.ts` | `mockRemoteParticipant`, `exampleTransport`. |
|
||||
| Module | What it is |
|
||||
| ----------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `media/LivekitMediaTrack.ts`, `media/LivekitMemberMedia.ts` | Participant to `MemberMedia`. Clean already. |
|
||||
| `session/Connection.ts`, `ConnectionFactory.ts`, `ConnectionManager.ts` | One LiveKit room per transport URL, the multi-SFU policy, the token fetch per connection. |
|
||||
| `session/MatrixLivekitMembers.ts` | Matches a participant identity to a membership, epoch-paired. |
|
||||
| `session/Publisher.ts`, `publishOptions.ts` | Publish requests onto the local participant, upstream pausing, audio output switching, the capture and publish option builders. |
|
||||
| `session/KeyProvider.ts`, `MatrixKeyProvider.ts` | LiveKit's key provider on the js-sdk key stream. |
|
||||
| `session/DataChannel.ts` | LiveKit data packets. |
|
||||
| `session/openIDSFU.ts`, `livekitOptions.ts` | The JWT service protocol, including delegation, and room options. |
|
||||
| `session/LocalTransport.ts` | The pre-join token fetch. Discovery itself is a homeserver endpoint and stays; see "Discovery stays in the session". |
|
||||
| `session/RtcTransportAutoDiscovery.ts` | Generic apart from its `isLivekitTransport` filter and the fallback URL. Stays. |
|
||||
| `session/LocalMember.ts` | `participant$`, `connection$`, upstream pausing, the delegation probe and handover, and `publish` resolving with a `LocalTrackPublication`. Roughly 120 of 490 lines. |
|
||||
| `session/Members.ts` | `mediaFor` takes a participant and a room; `publish` maps a `trackSid` back to a track. |
|
||||
| `session/Transports.ts` | Transport id, the connected check and `resolved$` assume LiveKit. |
|
||||
| `session/SessionBehaviors.ts` | `membershipsAndTransports$` filters to LiveKit transports. Deleted in step 3, not moved. |
|
||||
| `session/MatrixRTCClient.ts` | Wires all of the above. |
|
||||
| `utils/test.ts` | `mockRemoteParticipant`, `exampleTransport`. |
|
||||
|
||||
`errors.ts` has a type-only import for `LivekitConnectionError`; that is slice 11's
|
||||
business and stays. `index.ts` exports `getSFUConfigWithOpenID` as
|
||||
@@ -61,8 +68,10 @@ the backend folder.
|
||||
All of it lives in `sdk/src/backend/api.ts`. Types not defined here come from
|
||||
`sdk/src/api.ts` (`MemberMedia`, `LocalMemberMedia`, `LocalMediaInputs`,
|
||||
`ResolvedTransport`), `sdk/src/config.ts` (`SessionTimings`),
|
||||
`sdk/src/encryption.ts` and matrix-js-sdk (`Transport`, `CallMembership`,
|
||||
`CallMembershipIdentityParts`). The js-sdk session itself never reaches a
|
||||
`sdk/src/encryption.ts` and matrix-js-sdk (`Transport`,
|
||||
`CallMembershipIdentityParts`). Memberships reach a backend as the SDK's
|
||||
`RTCMembership`, whose pick covers the two things a backend reads,
|
||||
`rtcBackendIdentity` and `getTransport()`. The js-sdk session itself never reaches a
|
||||
backend: `MatrixKeyProvider` reads three things from it today, the
|
||||
`EncryptionKeyChanged` event, one `reemitEncryptionKeys()` call to replay the
|
||||
keys it already holds, and the room id for a log line, and `mediaKeys$` covers
|
||||
@@ -138,22 +147,33 @@ export type MediaBackendFactory = (
|
||||
```
|
||||
|
||||
`LocalMediaInputs` is unchanged from `sdk/src/api.ts` and is the only input the
|
||||
backend reads about the local devices:
|
||||
backend reads about the local media. Since the `MemberMediaPlan.md` slices, the
|
||||
devices and the processor are no longer behaviors the SDK watches: they are
|
||||
fields of the initial requests, and later changes go to the track's own
|
||||
`setDevice` and `setProcessor`.
|
||||
|
||||
```ts
|
||||
export interface LocalMediaInputs {
|
||||
/** Initial state; `setMicrophoneEnabled` and `setCameraEnabled` change it from then on. */
|
||||
microphoneEnabled: boolean;
|
||||
cameraEnabled: boolean;
|
||||
audioInputDeviceId$: Behavior<string | undefined>;
|
||||
videoInputDeviceId$: Behavior<string | undefined>;
|
||||
/** Published at the join; `publish` on the local member adds to it from then on. */
|
||||
publish: PublishRequest[];
|
||||
/** Undefined where the host routes audio itself, or to leave the browser's choice. */
|
||||
audioOutputDeviceId$: Behavior<string | undefined>;
|
||||
/** Background blur and the like. The LiveKit type on the public API; slice 7. */
|
||||
videoProcessor$: Behavior<TrackProcessor<Track.Kind.Video> | undefined>;
|
||||
}
|
||||
|
||||
export type PublishRequest =
|
||||
| { source: "microphone"; deviceId?: string; capture?: AudioCaptureSettings }
|
||||
| {
|
||||
source: "camera";
|
||||
deviceId?: string;
|
||||
processor?: TrackProcessor<Track.Kind.Video>;
|
||||
capture?: VideoCaptureSettings;
|
||||
}
|
||||
| { source: "screenShare"; audio?: boolean; capture?: VideoCaptureSettings };
|
||||
```
|
||||
|
||||
The processor type in the camera request is the LiveKit type on the public API;
|
||||
slice 7 of the migration.
|
||||
|
||||
### The backend
|
||||
|
||||
```ts
|
||||
@@ -220,7 +240,7 @@ export interface MediaBackend {
|
||||
*/
|
||||
mediaFor$(
|
||||
scope: ObservableScope,
|
||||
membership$: Behavior<CallMembership>,
|
||||
membership$: Behavior<RTCMembership>,
|
||||
): Behavior<MemberMedia | null>;
|
||||
|
||||
/**
|
||||
@@ -278,22 +298,28 @@ export interface LocalMediaBackend {
|
||||
readonly publishError$: Behavior<Error | null>;
|
||||
|
||||
/**
|
||||
* Enables or disables a source and resolves with what resulted, which
|
||||
* differs from the request where the device could not be used. Remembered
|
||||
* across connections, so a request before the transport is up is applied
|
||||
* when the tracks are created. Today's `DesiredMedia` lives behind this.
|
||||
* Publishes a source and resolves with its track once it is in `media$`'s
|
||||
* `tracks$`. Remembered across connections, so a request before the
|
||||
* transport is up is applied when the connection comes, and republished
|
||||
* after a reconnection. Rejects where the device could not be used,
|
||||
* including the user closing the screen picker. Today's `DesiredMedia`
|
||||
* map lives behind this.
|
||||
*/
|
||||
setEnabled(source: "microphone" | "camera", enabled: boolean): Promise<boolean>;
|
||||
|
||||
readonly sharingScreen$: Behavior<boolean>;
|
||||
/**
|
||||
* Rejects with the browser's `DOMException` when the user closes the
|
||||
* picker; the session filters that out before showing an error.
|
||||
*/
|
||||
setScreenShareEnabled(enabled: boolean): Promise<void>;
|
||||
publish(
|
||||
request: PublishRequest,
|
||||
): Promise<LocalAudioMediaTrack | LocalVideoMediaTrack>;
|
||||
/** Removes one of our tracks by id; a screen share takes its audio with it. */
|
||||
unpublish(id: string): Promise<void>;
|
||||
}
|
||||
```
|
||||
|
||||
Muting, device switching and the processor are not on the backend: they are
|
||||
methods of `LocalAudioMediaTrack` and `LocalVideoMediaTrack`, which the backend
|
||||
builds in `media$`. The `setEnabled` callback that `createLocalLivekitMemberMedia`
|
||||
takes today, so that a track's mute keeps the desired map in step, is wired
|
||||
inside the backend. Screen sharing has no members of its own any more: it is a
|
||||
`publish` request, and "sharing" is `tracks$` holding a `screenShare` track.
|
||||
|
||||
There is one publish switch, not two. Today `LocalMember.ts` pauses the
|
||||
upstream from two places: `stopPublishing()` on leave, and a separate
|
||||
subscription on the homeserver connectivity that pauses every publication while
|
||||
@@ -310,12 +336,9 @@ There is no publisher on the interface. Today's `Publisher` wraps one LiveKit
|
||||
room's local participant and so lives and dies with the local connection, and
|
||||
`LocalMember.ts` re-applies the join intent to each new one. That lifecycle is a
|
||||
LiveKit detail: the backend keeps the publish intent, as it keeps the desired
|
||||
media for `setEnabled`, and re-applies both to every connection it opens. A
|
||||
requests for `publish`, and re-applies both to every connection it opens. A
|
||||
mesh backend has no publisher object at all.
|
||||
|
||||
Whether the platform can share a screen at all stays in the session: it is a
|
||||
`getDisplayMedia` check, the same for every backend.
|
||||
|
||||
### Delayed leave delegation
|
||||
|
||||
How it works today, all in `LocalMember.ts` and `openIDSFU.ts`:
|
||||
@@ -362,23 +385,31 @@ const prepared$ = from(
|
||||
),
|
||||
); // errors become fatalTransportError$
|
||||
|
||||
scope.reconcile(combineLatest([prepared$, joinRequested$]), ([prepared, join]) => {
|
||||
if (prepared === null || !join) return;
|
||||
joinMatrixRTC(
|
||||
prepared.transport,
|
||||
prepared.canDelegateDelayedLeave ? timings.delegatedDelayedLeave : timings.delayedLeave,
|
||||
);
|
||||
return leaveOnCleanup;
|
||||
});
|
||||
scope.reconcile(
|
||||
combineLatest([prepared$, joinRequested$]),
|
||||
([prepared, join]) => {
|
||||
if (prepared === null || !join) return;
|
||||
joinMatrixRTC(
|
||||
prepared.transport,
|
||||
prepared.canDelegateDelayedLeave
|
||||
? timings.delegatedDelayedLeave
|
||||
: timings.delayedLeave,
|
||||
);
|
||||
return leaveOnCleanup;
|
||||
},
|
||||
);
|
||||
|
||||
scope.reconcile(combineLatest([prepared$, delayId$]), async ([prepared, delayId]) => {
|
||||
if (!prepared?.canDelegateDelayedLeave || delayId === null) return;
|
||||
try {
|
||||
await backend.delegateDelayedLeave(delayId);
|
||||
} catch (e) {
|
||||
logger.error("Failed to delegate the leave", e);
|
||||
}
|
||||
});
|
||||
scope.reconcile(
|
||||
combineLatest([prepared$, delayId$]),
|
||||
async ([prepared, delayId]) => {
|
||||
if (!prepared?.canDelegateDelayedLeave || delayId === null) return;
|
||||
try {
|
||||
await backend.delegateDelayedLeave(delayId);
|
||||
} catch (e) {
|
||||
logger.error("Failed to delegate the leave", e);
|
||||
}
|
||||
},
|
||||
);
|
||||
```
|
||||
|
||||
A mesh backend answers `canDelegateDelayedLeave: false` and is never asked to
|
||||
@@ -420,19 +451,19 @@ nobody has a tile for, and that member's transport would never connect.
|
||||
|
||||
### Where the LiveKit backend gets each member from
|
||||
|
||||
| Interface member | Today | Moves from |
|
||||
| --- | --- | --- |
|
||||
| `prepareLocalTransport` | OpenID and JWT fetch kept as `existingSFUConfig`, delegation probe | `LocalTransport.ts`, `LocalMember.ts` |
|
||||
| `delegateDelayedLeave` | `getSFUConfigWithOpenID` with `delayId` | `LocalMember.ts` |
|
||||
| `local.connectionState$` | `connection$.state$` for the local transport | `LocalMember.ts` |
|
||||
| `local.media$` | `mediaFor(participant$, connection$)` with `createLocalLivekitMemberMedia` | `Members.ts` |
|
||||
| `local.setPublishing`, `publishError$` | `scope.reconcile(connection$, createPublisher)`, the `[publisher$, joinRequested$]` reconcile with its `publishError$`, and the homeserver-connectivity loop over `participant.trackPublications` | `LocalMember.ts`, `MatrixRTCClient.ts` |
|
||||
| `local.setEnabled` | `DesiredMedia` plus `Publisher.setEnabled(Track.Source, …)` | `LocalMember.ts`, `Publisher.ts` |
|
||||
| `local.sharingScreen$`, `setScreenShareEnabled` | `observeSharingScreen$`, `setScreenShareEnabled` with capture and publish options | `LocalMember.ts` |
|
||||
| `mediaFor$` | `createRemoteMatrixLivekitMembers$` matching plus `mediaFor` | `Members.ts`, `MatrixLivekitMembers.ts` |
|
||||
| `connections$` | `connectionManagerData$` joined with each connection's `state$` and `ResolvedConnection.resolved$` | `Transports.ts`, `ConnectionFactory.ts` |
|
||||
| `sendData`, `data$` | `createDataChannel$` minus the member lookup | `DataChannel.ts` |
|
||||
| internal | key provider on `mediaKeys$` instead of `setRTCSession`, room options, E2EE worker, connection manager fed by the transports registered through `mediaFor$` | `KeyProvider.ts`, `MatrixKeyProvider.ts`, `livekitOptions.ts`, `ConnectionFactory.ts`, `ConnectionManager.ts` |
|
||||
| Interface member | Today | Moves from |
|
||||
| -------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------- |
|
||||
| `prepareLocalTransport` | OpenID and JWT fetch kept as `existingSFUConfig`, delegation probe | `LocalTransport.ts`, `LocalMember.ts` |
|
||||
| `delegateDelayedLeave` | `getSFUConfigWithOpenID` with `delayId` | `LocalMember.ts` |
|
||||
| `local.connectionState$` | `connection$.state$` for the local transport | `LocalMember.ts` |
|
||||
| `local.media$` | `mediaFor(participant$, connection$)` with `createLocalLivekitMemberMedia` | `Members.ts` |
|
||||
| `local.setPublishing`, `publishError$` | `scope.reconcile(connection$, createPublisher)`, the `[publisher$, joinRequested$]` reconcile with its `publishError$`, and the homeserver-connectivity loop over `participant.trackPublications` | `LocalMember.ts`, `MatrixRTCClient.ts` |
|
||||
| `local.publish`, `unpublish` | `publish` and `unpublish` with the `DesiredMedia` map and `publication$`, and the `trackSid` to track lookup | `LocalMember.ts`, `Members.ts`, `Publisher.ts` |
|
||||
| internal | the `setEnabled(source, enabled)` callback into `createLocalLivekitMemberMedia` | `LocalMember.ts`, `Members.ts` |
|
||||
| `mediaFor$` | `createRemoteMatrixLivekitMembers$` matching plus `mediaFor` | `Members.ts`, `MatrixLivekitMembers.ts` |
|
||||
| `connections$` | `connectionManagerData$` joined with each connection's `state$` and `ResolvedConnection.resolved$` | `Transports.ts`, `ConnectionFactory.ts` |
|
||||
| `sendData`, `data$` | `createDataChannel$` minus the member lookup | `DataChannel.ts` |
|
||||
| internal | key provider on `mediaKeys$` instead of `setRTCSession`, room options, E2EE worker, connection manager fed by the transports registered through `mediaFor$` | `KeyProvider.ts`, `MatrixKeyProvider.ts`, `livekitOptions.ts`, `ConnectionFactory.ts`, `ConnectionManager.ts` |
|
||||
|
||||
### Discovery stays in the session
|
||||
|
||||
@@ -446,9 +477,12 @@ only asked to prepare what the session picked:
|
||||
export async function discoverLocalTransport(
|
||||
client: Pick<MatrixClient, "getDomain" | "_unstable_getRTCTransports">,
|
||||
transportType: string,
|
||||
options: Pick<MatrixRTCClientOptions, "transportUrl" | "fallbackTransportUrl">,
|
||||
options: Pick<
|
||||
MatrixRTCClientOptions,
|
||||
"transportUrl" | "fallbackTransportUrl"
|
||||
>,
|
||||
logger: Logger,
|
||||
): Promise<Transport>; // throws MatrixRTCTransportMissingError
|
||||
): Promise<Transport>; // throws MatrixRTCTransportMissingError
|
||||
```
|
||||
|
||||
`RtcTransportAutoDiscovery` does the list fetch today and stays where it is,
|
||||
@@ -487,7 +521,6 @@ export interface LivekitBackendOptions {
|
||||
*/
|
||||
client: Pick<MatrixClient, "getOpenIdToken" | "baseUrl">;
|
||||
mediaQuality?: MediaQuality;
|
||||
capture?: CaptureSettings;
|
||||
/**
|
||||
* Which token service endpoint to use for the local transport: the MSC4195
|
||||
* one, or the legacy one with the old identity format. Remote transports
|
||||
@@ -520,8 +553,9 @@ Pure move, no code change beyond import paths.
|
||||
|
||||
- `git mv` into `sdk/src/backend/livekit/`: `media/*`, `Connection`,
|
||||
`ConnectionFactory`, `ConnectionManager`, `DataChannel`, `KeyProvider`,
|
||||
`MatrixKeyProvider`, `MatrixLivekitMembers`, `Publisher`, `livekitOptions`,
|
||||
`openIDSFU`, each with its test. `RtcTransportAutoDiscovery` and
|
||||
`MatrixKeyProvider`, `MatrixLivekitMembers`, `Publisher`, `publishOptions`,
|
||||
`livekitOptions`, `openIDSFU`, each with its test. `utils/tracks.ts` is
|
||||
neutral and stays. `RtcTransportAutoDiscovery` and
|
||||
`LocalTransport` stay in `session/`.
|
||||
- Split `utils/test.ts`: `mockRemoteParticipant` and `exampleTransport` go to
|
||||
`backend/livekit/test.ts`.
|
||||
@@ -595,24 +629,28 @@ with the second's.
|
||||
transport's connection mapped onto `MediaConnectionState`, `media$` through
|
||||
`createLocalLivekitMemberMedia`, `setPublishing` and `publishError$` by
|
||||
reconciling a `Publisher` per connection against the remembered intent,
|
||||
which is the two reconciles from `LocalMember.ts` moved as they are,
|
||||
`setEnabled` over a backend-owned `DesiredMedia` and the current publisher,
|
||||
`sharingScreen$` and `setScreenShareEnabled` with today's capture and publish
|
||||
options. The homeserver-connectivity loop is not moved; `setPublishing`
|
||||
covers it.
|
||||
which is the two reconciles from `LocalMember.ts` moved as they are, and
|
||||
`publish` and `unpublish` over a backend-owned `DesiredMedia` map and the
|
||||
current publisher, with the `trackSid` to track lookup that `Members.ts`
|
||||
does today done inside. The `setEnabled` callback into
|
||||
`createLocalLivekitMemberMedia` is wired here too. The
|
||||
homeserver-connectivity loop is not moved; `setPublishing` covers it.
|
||||
- `LocalMember.ts` takes `backend.local` instead of `connectionManager`,
|
||||
`createPublisher` and `desired`. Its publisher reconcile and the pause and
|
||||
resume loop become one subscription,
|
||||
`joinRequested$ && homeserverConnected.combined$` into
|
||||
`backend.local.setPublishing`, and `backend.local.publishError$` feeds
|
||||
`state$` where the local subject did. It
|
||||
loses `participant$`, `connection$`, the `livekitRoom` access, the pause and
|
||||
resume loop, `Track.Source`, `ScreenShareCaptureOptions` and
|
||||
`TrackPublishOptions`, and compares against `MediaConnectionState.Connected`
|
||||
in `mediaState$` and `disconnectReason$`. `LocalMembership` no longer exposes
|
||||
`participant$` or `connection$`.
|
||||
`state$` where the local subject did. It loses `participant$`,
|
||||
`connection$`, the `livekitRoom` access, the pause and resume loop,
|
||||
`publish`, `unpublish`, `setEnabled` and `publication$`, and compares
|
||||
against `MediaConnectionState.Connected` in `mediaState$` and
|
||||
`disconnectReason$`. `LocalMembership` shrinks to the join state machine
|
||||
and no longer names a LiveKit type.
|
||||
- `Members.ts`: `createLocalRTCMember` forwards `publish` and `unpublish` to
|
||||
`backend.local` and drops its own `trackSid` lookup.
|
||||
- `MatrixRTCClient.ts` stops passing `createPublisher`, `desired` and the
|
||||
connection manager. The transitional field goes.
|
||||
connection manager, and stops building the `DesiredMedia` map. The
|
||||
transitional field goes.
|
||||
- `status.test.ts` moves to `MediaConnectionState`.
|
||||
|
||||
New unit test: `LocalMember.test.ts` against a fake `LocalMediaBackend`. This is
|
||||
@@ -660,8 +698,7 @@ and that a new delay id triggers a handover.
|
||||
- Extend `eslint/SdkImportBoundary.js`: under `sdk/src/session/`, in
|
||||
`sdk/src/api.ts` and in `sdk/index.ts`, `livekit-client` and `@livekit/*` are
|
||||
banned except as `import type`. From here on the seam is enforced, not
|
||||
documented. `oxlint` currently reports no files under `sdk/`, so that has to
|
||||
be fixed first for the rule to bite at all.
|
||||
documented.
|
||||
- `SdkArchitecture.md` gets a "Media backend" section built from this plan, the
|
||||
README's description of what imports LiveKit is updated, and `SdkMigration.md`
|
||||
records the slice as done.
|
||||
|
||||
+4
-2
@@ -1,11 +1,13 @@
|
||||
# MatrixRTC SDK (EXPERIMENTAL)
|
||||
|
||||
`@element-hq/matrixrtc-sdk` is the call model under Element Call, on its own:
|
||||
MatrixRTC memberships and transports, LiveKit connections, publishing, E2EE keys and
|
||||
MatrixRTC memberships and transports, media connections, publishing, E2EE keys and
|
||||
the media of every member, as observables. It has no UI. Element Call's own
|
||||
`CallViewModel` is meant to become one consumer of it. The design is in
|
||||
[`SdkArchitecture.md`](./SdkArchitecture.md), the migration from the view model
|
||||
in [`SdkMigration.md`](./SdkMigration.md).
|
||||
in [`SdkMigration.md`](./SdkMigration.md). Media goes through one media backend
|
||||
behind `MediaBackend`; LiveKit, under `src/backend/livekit/`, is the only one
|
||||
today and the only place that imports `livekit-client`.
|
||||
|
||||
**Status:** first implementation. `createMatrixRTCClient` joins the session, connects to
|
||||
the transport, publishes the local media and exposes every member's media; the
|
||||
|
||||
+114
-41
@@ -25,7 +25,7 @@ therefore `Element Call → sdk`, never the reverse. The migration from today's
|
||||
│ │
|
||||
│ public API, all Behaviors: │
|
||||
│ join() leave() status$ connected$ reconnecting$ fatalError$ │
|
||||
│ localMember$ remoteMembers$ memberCount$ │
|
||||
│ localMember$ remoteMembers$ memberCount$ sendData() data$ │
|
||||
│ keyRotationSuppressed$ connectedTransports$ │
|
||||
│ │
|
||||
│ per member RTCMember { id userId deviceId membership$ displayName$ │
|
||||
@@ -33,33 +33,42 @@ therefore `Element Call → sdk`, never the reverse. The migration from today's
|
||||
│ per media MemberMedia { tracks$ encryptionError$ } │
|
||||
│ per track MediaTrack { source kind muted$ encrypted$ stats$ attach() }│
|
||||
│ │
|
||||
│ ┌──────────────────────────┐ memberships ┌───────────────────────────┐ │
|
||||
│ │ MatrixRTC core │ ───────────────▶ │ transports │ │
|
||||
│ │ (js-sdk MatrixRTCSession,│ transports │ │ │
|
||||
│ │ later the rust-rtc │ │ ConnectionManager │ │
|
||||
│ │ crate behind a bridge) │ │ one Connection per │ │
|
||||
│ │ │ │ transport url │ │
|
||||
│ │ memberships from room │ │ Connection │ │
|
||||
│ │ state and sticky events│ │ OpenID → JWT → LiveKit │ │
|
||||
│ │ own membership, delayed │ │ room, participants, │ │
|
||||
│ │ leave, retries │ media keys │ tracks │ │
|
||||
│ │ media keys over │ ───────────────▶ │ Publisher │ │
|
||||
│ │ to-device messages │ (key provider) │ local tracks, upstream │ │
|
||||
│ └──────────────────────────┘ └───────────────────────────┘ │
|
||||
│ │ │ │
|
||||
│ └──── RTCMember = membership + participant ─┘ │
|
||||
│ matched on rtcBackendIdentity = LiveKit identity │
|
||||
│ ┌───────────────────────────┐ MediaBackend ┌───────────────────────────┐ │
|
||||
│ │ session: MatrixRTC │ backend/api.ts │ media backend │ │
|
||||
│ │ │ │ backend/livekit/ today, │ │
|
||||
│ │ js-sdk MatrixRTCSession, │ │ or options.backend │ │
|
||||
│ │ later the rust-rtc │ prepareLocal │ │ │
|
||||
│ │ crate behind a bridge │ Transport ───▶│ token service: │ │
|
||||
│ │ memberships from room │ delegateDelayed │ OpenID → JWT, │ │
|
||||
│ │ state and sticky events │ Leave ───────▶│ delegation probe │ │
|
||||
│ │ own membership, delayed │ │ ConnectionManager: │ │
|
||||
│ │ leave, retries │ mediaKeys$ ────▶│ one LiveKit room per │ │
|
||||
│ │ discovery of the local │ │ transport url, local │ │
|
||||
│ │ transport (homeserver) │ mediaFor$ ─────▶│ published, remote │ │
|
||||
│ │ join state machine │ ◀────── media$ │ subscribed │ │
|
||||
│ │ (LocalMember) │ local.publish, │ Publisher: local tracks, │ │
|
||||
│ │ transport registry │ setPublishing▶│ upstream paused until │ │
|
||||
│ │ (TransportMetadata) │ ◀─ connections$ │ joined │ │
|
||||
│ │ data packet → member │ ◀─ data$ │ LivekitMemberMedia: │ │
|
||||
│ │ │ │ participant → tracks │ │
|
||||
│ └───────────────────────────┘ └───────────────────────────┘ │
|
||||
│ │
|
||||
│ RTCMember = membership + mediaFor$(membership): the backend matches the │
|
||||
│ member's rtcBackendIdentity to the identity it sees on the transport │
|
||||
└──────────────────────────────────────────────────────────────────────────────┘
|
||||
│ │
|
||||
▼ ▼
|
||||
MatrixClient (sync, room state, to-device, OpenID) LiveKit SFU
|
||||
```
|
||||
|
||||
The left box is MatrixRTC: who is in the session, with which transport, and the
|
||||
keys. The right box is media: a connection to every transport any member
|
||||
advertises, and our own publication on the one we advertise. The client is the
|
||||
join of the two: a member exists once its membership does, and gets its media once
|
||||
a participant with the matching identity shows up on the member's transport.
|
||||
The left box is MatrixRTC: who is in the session, with which transport, the
|
||||
keys, the delayed leave, and the join state machine. The right box is a media
|
||||
backend: whatever carries the media, behind the `MediaBackend` interface in
|
||||
`sdk/src/backend/api.ts`. The client is the join of the two: a member exists
|
||||
once its membership does, and gets its media once the backend has something
|
||||
for that membership on its transport. LiveKit is the one backend today; a
|
||||
cascading SFU or full mesh is another folder under `sdk/src/backend/`, handed
|
||||
in through `options.backend`, and nothing in the left box changes.
|
||||
|
||||
### How Element Call uses it
|
||||
|
||||
@@ -88,26 +97,33 @@ LiveKit participants.
|
||||
### Joining, in order
|
||||
|
||||
```
|
||||
host MatrixRTCClient homeserver / JWT service LiveKit SFU
|
||||
│ create ───▶│ │ │
|
||||
│ │ GET rtc/transports ──────────────▶│ │
|
||||
│ │◀── preferred transport ───────────│ │
|
||||
│ │ OpenID token, then /sfu/get ─────▶│ │
|
||||
│ │◀── sfu url, jwt, alias, identity ─│ = resolved$ of the │
|
||||
│ │ │ local transport │
|
||||
│ │ connect(url, jwt) ───────────────────────────────────────▶│
|
||||
│ join() ───▶│ │ │
|
||||
│ │ membership state event + delayed leave ──▶│ │
|
||||
│ │ create tracks, resume upstream ──────────────────────────▶│
|
||||
│ │ media key to every other member (to-device) ──▶│ │
|
||||
│ │◀── other memberships (sync) ──────│◀── participants ────────│
|
||||
│◀─ status$ "connected", localMember$, remoteMembers$ with media$ ──────────│
|
||||
host MatrixRTCClient / backend homeserver / JWT service LiveKit SFU
|
||||
│ create ───▶│ │ │
|
||||
│ │ GET rtc/transports (session) ────────▶│ │
|
||||
│ │◀── first transport of the backend's type │
|
||||
│ │ prepareLocalTransport (backend): │ │
|
||||
│ │ OpenID token, then /sfu/get ───────▶│ │
|
||||
│ │◀── sfu url, jwt, alias, identity ─────│ = resolved$ of the │
|
||||
│ │ probe delegate_delayed_leave ──────▶│ local transport │
|
||||
│ │ connect(url, jwt) ───────────────────────────────────────────▶│
|
||||
│ join() ───▶│ │ │
|
||||
│ │ membership state event + delayed leave ──▶│ │
|
||||
│ │ delegateDelayedLeave(delayId) (backend) ─▶│ │
|
||||
│ │ create tracks, resume upstream ──────────────────────────────▶│
|
||||
│ │ media key to every other member (to-device) ──▶│ │
|
||||
│ │◀── other memberships (sync) ──────────│◀── participants ────────│
|
||||
│◀─ status$ "connected", localMember$, remoteMembers$ with media$ ──────────────│
|
||||
```
|
||||
|
||||
The transport is resolved and connected before `join()`, so that the membership
|
||||
can name it and the first frames go out as soon as the state event is sent. Until
|
||||
`join()` the tracks exist but their upstream is paused: a host can show a preview
|
||||
without anyone hearing it. `leave()` pauses the upstream again and sends the leave.
|
||||
The session discovers the transport from the homeserver and the backend prepares
|
||||
it, both before `join()`, so that the membership can name it and the first frames
|
||||
go out as soon as the state event is sent. Preparing is the backend's own
|
||||
business: for LiveKit the token exchange, and the probe that decides whether the
|
||||
SFU can take over the delayed leave, which picks the delayed leave timings the
|
||||
join uses. Until `join()` the tracks exist but their upstream is paused: a host
|
||||
can show a preview without anyone hearing it. The upstream flows exactly while the
|
||||
member is joined and the homeserver is reachable; `leave()` or a sync outage pauses
|
||||
it, and `leave()` also sends the leave.
|
||||
|
||||
## Vocabulary
|
||||
|
||||
@@ -123,6 +139,7 @@ without anyone hearing it. `leave()` pauses the upstream again and sends the lea
|
||||
| active | `isActive$` on an audio track: the track carries sound, as the backend measures it. "Speaking" is a call's reading of it on the microphone track | LiveKit "speaking", `isSpeaking` |
|
||||
| attach, detach | handing a `<video>` or `<audio>` element to a track and taking it back. The SDK owns the stream and the observers on the element | LiveKit `Track.attach` |
|
||||
| transport | where media is exchanged: a LiveKit service url today, described by `TransportMetadata` | MSC4143 "focus" |
|
||||
| media backend | a `MediaBackend`: what carries the media, behind `sdk/src/backend/api.ts`. One per client, chosen with `options.backend`; LiveKit under `sdk/src/backend/livekit/` is the one that exists | "SFU", a LiveKit room, a peer connection |
|
||||
| application, slot | what the session is for (`m.call`) and which one of them in the room (`ROOM`) | MSC4143 |
|
||||
|
||||
Names use `RTC` in capitals, as the js-sdk does: `MatrixRTCClient`, `RTCMember`,
|
||||
@@ -421,7 +438,7 @@ import { type Transport } from "matrix-js-sdk/lib/matrixrtc";
|
||||
export interface TransportMetadata {
|
||||
/** `"livekit"` today. */
|
||||
type: string;
|
||||
/** Stable key, unique per transport in the session. For LiveKit, the service url. */
|
||||
/** Stable key, unique per transport in the session: the raw transport, serialised with sorted keys. */
|
||||
id: string;
|
||||
/** The transport object as it appears in the membership, e.g. `{ type, livekit_service_url }`. */
|
||||
raw: Transport;
|
||||
@@ -468,6 +485,62 @@ Kept despite the name: `sendNotificationType`, because the js-sdk join sends the
|
||||
notification as part of entering the session. Only _reacting_ to it is
|
||||
call-specific.
|
||||
|
||||
## Media backend: `MediaBackend`
|
||||
|
||||
The client speaks MatrixRTC: memberships, the transports named in them, the
|
||||
delayed leave, the keys. What carries the media is a backend behind one
|
||||
interface, `MediaBackend` in `sdk/src/backend/api.ts`, created once per client.
|
||||
LiveKit under `sdk/src/backend/livekit/` is the only backend today, and the only
|
||||
place in the SDK that imports `livekit-client`; the import boundary lint rule
|
||||
enforces that outside `sdk/src/backend/` the LiveKit packages appear in type
|
||||
imports only.
|
||||
|
||||
The seam sits above the connection manager on purpose. One connection per
|
||||
transport url, with a subscribe-only connection to every remote SFU, is the
|
||||
LiveKit multi-SFU strategy, not a MatrixRTC concept: a cascading SFU keeps the
|
||||
membership transport as it is and only changes which SFUs the client connects to,
|
||||
and full mesh has one peer connection per member and no token service. So a
|
||||
backend is handed the whole session's view and gives back media:
|
||||
|
||||
```ts
|
||||
export interface MediaBackend {
|
||||
readonly transportType: string; // "livekit"
|
||||
/** Authenticate with the transport the client found; the probe for delegation runs here. */
|
||||
prepareLocalTransport(
|
||||
transport: Transport,
|
||||
): Promise<{ canDelegateDelayedLeave: boolean }>;
|
||||
/** Hand over the delayed leave, each time the membership manager has a new delay id. */
|
||||
delegateDelayedLeave(delayId: string): Promise<void>;
|
||||
readonly local: LocalMediaBackend; // connectionState$ media$ setPublishing publishError$ publish unpublish
|
||||
/** A remote member's media; also how the backend learns which transports to connect to. */
|
||||
mediaFor$(
|
||||
scope: ObservableScope,
|
||||
membership$: Behavior<RTCMembership>,
|
||||
): Behavior<MemberMedia | null>;
|
||||
readonly connections$: Behavior<BackendConnection[]>; // transport, state, resolved; diagnostics
|
||||
sendData(topic: string, text: string): Promise<void>;
|
||||
readonly data$: Observable<DataPacket>;
|
||||
}
|
||||
```
|
||||
|
||||
What stays on the client's side: discovery of the local transport from the
|
||||
homeserver's list (a homeserver endpoint, so not a backend's), the join state
|
||||
machine in `LocalMember.ts`, the choice of delayed leave timings, mapping a data
|
||||
packet's sender to a member, and the transport registry, which keys on the raw
|
||||
transport serialised with sorted keys and derives `connectedTransports$` and each
|
||||
`TransportMetadata.resolved$` from `connections$`.
|
||||
|
||||
The backend's context carries no Matrix client and no membership list. A backend
|
||||
that needs the client takes a `Pick` of exactly what it calls in its own options
|
||||
(`createLivekitBackend(scope, context, { client, mediaQuality, tokenEndpoint })`),
|
||||
and learns of remote members only through `mediaFor$`, registering each
|
||||
membership's transport for as long as the member's scope lives. Media keys reach
|
||||
it as a stream, `mediaKeys$`, replayed on subscribe; the js-sdk session never
|
||||
does. A host supplies its own backend through `MatrixRTCClientOptions.backend`;
|
||||
without one, LiveKit is built from the LiveKit-flavoured client options. The
|
||||
reasoning behind each of these choices is in
|
||||
[`MediaBackendPlan.md`](./MediaBackendPlan.md).
|
||||
|
||||
## Media: `MemberMedia`
|
||||
|
||||
Every member has media. Today, in Element Call, that media _is_ a `livekit-client`
|
||||
|
||||
+8
-7
@@ -8,13 +8,14 @@ symbols, so it ages with the code; the architecture document should not.
|
||||
|
||||
## Status
|
||||
|
||||
| Slice | State |
|
||||
| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------- |
|
||||
| 1. Scaffold: `sdk/` packaging, the interface, the harness, the smoke test | done, October 2026 |
|
||||
| 2. First implementation of `createMatrixRTCClient` under `sdk/src`, the media adapter, the e2e suite | done, 1 October 2026 |
|
||||
| 3. Copy the shared modules into `sdk/src/` with their tests, drop every `src/` import, turn on the import boundary | done, 2 October 2026 |
|
||||
| 4. `createCallViewModel$` consumes the client; media view models take `media$`; Element Call imports the primitives and enums from the SDK and deletes its own copies | done, 2 October 2026 |
|
||||
| 5. `sendData`/`data$` on the client; the legacy bundle `sdk-target-based-on-call-view-model/` is rebuilt on the SDK with its interface unchanged, since it is in production use | done, 2 October 2026 |
|
||||
| Slice | State |
|
||||
| -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------- |
|
||||
| 1. Scaffold: `sdk/` packaging, the interface, the harness, the smoke test | done, October 2026 |
|
||||
| 2. First implementation of `createMatrixRTCClient` under `sdk/src`, the media adapter, the e2e suite | done, 1 October 2026 |
|
||||
| 3. Copy the shared modules into `sdk/src/` with their tests, drop every `src/` import, turn on the import boundary | done, 2 October 2026 |
|
||||
| 4. `createCallViewModel$` consumes the client; media view models take `media$`; Element Call imports the primitives and enums from the SDK and deletes its own copies | done, 2 October 2026 |
|
||||
| 5. `sendData`/`data$` on the client; the legacy bundle `sdk-target-based-on-call-view-model/` is rebuilt on the SDK with its interface unchanged, since it is in production use | done, 2 October 2026 |
|
||||
| 12. The media backend seam: everything LiveKit behind `MediaBackend` in `sdk/src/backend/livekit/`, the `backend` option, the lint guard; see [`MediaBackendPlan.md`](./MediaBackendPlan.md) | done, 4 October 2026 |
|
||||
|
||||
Element Call now consumes the SDK. `InCallView` creates the client from the
|
||||
configuration, settings and URL parameters, and hands it to `createCallViewModel$`,
|
||||
|
||||
+9
-1
@@ -53,7 +53,15 @@ export {
|
||||
} from "./src/errors";
|
||||
|
||||
export * from "./src/api";
|
||||
export * from "./src/backend/api";
|
||||
export { trackBySource$, type TrackOfSource } from "./src/utils/tracks";
|
||||
export { createMatrixRTCClient } from "./src/session/MatrixRTCClient";
|
||||
export {
|
||||
createLivekitBackend,
|
||||
type LivekitBackendOptions,
|
||||
} from "./src/backend/livekit/LivekitBackend";
|
||||
// For a developer panel to check a transport url before using it
|
||||
export { getSFUConfigWithOpenID as authenticateWithTransport } from "./src/session/openIDSFU";
|
||||
export {
|
||||
getSFUConfigWithOpenID as authenticateWithTransport,
|
||||
type TokenEndpoint,
|
||||
} from "./src/backend/livekit/openIDSFU";
|
||||
|
||||
+7
-1
@@ -23,6 +23,7 @@ import {
|
||||
import { type Observable } from "rxjs";
|
||||
|
||||
import { type Behavior } from "./reactive/Behavior";
|
||||
import { type MediaBackendFactory } from "./backend/api";
|
||||
import { type EncryptionSystem } from "./encryption";
|
||||
import {
|
||||
type MatrixRTCMode,
|
||||
@@ -62,6 +63,11 @@ export interface MatrixRTCClientOptions {
|
||||
transportUrl?: string;
|
||||
/** Use this transport when the homeserver advertises none. */
|
||||
fallbackTransportUrl?: string;
|
||||
/**
|
||||
* What carries the media. LiveKit, configured from the options above, when
|
||||
* left out; see `createLivekitBackend` for building one with other options.
|
||||
*/
|
||||
backend?: MediaBackendFactory;
|
||||
}
|
||||
|
||||
export interface AudioCaptureSettings {
|
||||
@@ -186,7 +192,7 @@ export interface DataMessage {
|
||||
export interface TransportMetadata {
|
||||
/** `"livekit"` today. */
|
||||
type: string;
|
||||
/** Stable key, unique per transport in the session. For LiveKit, the service url. */
|
||||
/** Stable key, unique per transport in the session: the raw transport, serialised with sorted keys. */
|
||||
id: string;
|
||||
/** The transport object as it appears in the membership. */
|
||||
raw: Transport;
|
||||
|
||||
@@ -0,0 +1,190 @@
|
||||
/*
|
||||
Copyright 2026 Element Creations Ltd.
|
||||
|
||||
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
|
||||
Please see LICENSE in the repository root for full details.
|
||||
*/
|
||||
|
||||
/**
|
||||
* The seam between the session and whatever carries its media. The session
|
||||
* speaks MatrixRTC: memberships, transports as they appear in them, the
|
||||
* delayed leave. A backend speaks one media protocol and hands back the
|
||||
* media of each member. `backend/livekit/` is the one backend today.
|
||||
*/
|
||||
|
||||
import { type Logger } from "matrix-js-sdk/lib/logger";
|
||||
import { type Transport } from "matrix-js-sdk/lib/matrixrtc";
|
||||
import { type CallMembershipIdentityParts } from "matrix-js-sdk/lib/matrixrtc/EncryptionManager";
|
||||
import { type Observable } from "rxjs";
|
||||
|
||||
import {
|
||||
type LocalAudioMediaTrack,
|
||||
type LocalMediaInputs,
|
||||
type LocalMemberMedia,
|
||||
type LocalVideoMediaTrack,
|
||||
type MemberMedia,
|
||||
type PublishRequest,
|
||||
type ResolvedTransport,
|
||||
type RTCMembership,
|
||||
} from "../api";
|
||||
import { type SessionTimings } from "../config";
|
||||
import { type EncryptionSystem } from "../encryption";
|
||||
import { type Behavior } from "../reactive/Behavior";
|
||||
import { type ObservableScope } from "../reactive/ObservableScope";
|
||||
|
||||
export enum MediaConnectionState {
|
||||
/** No connection attempt yet, or no connection at all. */
|
||||
Initialized = "initialized",
|
||||
/** Fetching whatever the connection needs first; the SFU token for LiveKit. */
|
||||
Preparing = "preparing",
|
||||
Connecting = "connecting",
|
||||
Connected = "connected",
|
||||
Reconnecting = "reconnecting",
|
||||
Disconnected = "disconnected",
|
||||
Stopped = "stopped",
|
||||
}
|
||||
|
||||
/** One per-participant media key, as the session's key exchange delivers it. */
|
||||
export interface MediaKey {
|
||||
/** The `rtcBackendIdentity` of the member the key belongs to. */
|
||||
participantId: string;
|
||||
index: number;
|
||||
key: Uint8Array<ArrayBuffer>;
|
||||
}
|
||||
|
||||
export interface MediaBackendContext {
|
||||
roomId: string;
|
||||
ownMembershipIdentity: CallMembershipIdentityParts;
|
||||
localMedia: LocalMediaInputs;
|
||||
encryptionSystem: EncryptionSystem;
|
||||
/**
|
||||
* Every media key the session knows: the current ones replayed on
|
||||
* subscribe, new ones as they arrive. Empty for a shared key, which is in
|
||||
* `encryptionSystem`.
|
||||
*/
|
||||
mediaKeys$: Observable<MediaKey>;
|
||||
timings: SessionTimings;
|
||||
logger: Logger;
|
||||
}
|
||||
|
||||
/**
|
||||
* The type of the `backend` option: the session calls it once with its scope
|
||||
* and the context. A backend's own factory keeps the shape of every other
|
||||
* factory here, `createLivekitBackend(scope, context, options)`, and a host
|
||||
* closes over its options. The Matrix client is not in the context: a backend
|
||||
* that needs it takes a `Pick` of exactly what it calls in its own options.
|
||||
*/
|
||||
export type MediaBackendFactory = (
|
||||
scope: ObservableScope,
|
||||
context: MediaBackendContext,
|
||||
) => MediaBackend;
|
||||
|
||||
/** What preparing the local transport established; what the MatrixRTC join needs from the media side. */
|
||||
export interface TransportCapabilities {
|
||||
/**
|
||||
* Whether the backend can restart the delayed leave on the client's behalf,
|
||||
* so that a client that vanishes is removed by the backend rather than by
|
||||
* the timeout. Decides which delayed leave timings the join uses, and
|
||||
* whether `delegateDelayedLeave` is ever called.
|
||||
*/
|
||||
canDelegateDelayedLeave: boolean;
|
||||
}
|
||||
|
||||
/** One connection a backend holds, for diagnostics. */
|
||||
export interface BackendConnection {
|
||||
/** The transport as it appears in memberships; what the registry keys on. */
|
||||
transport: Transport;
|
||||
state: MediaConnectionState | Error;
|
||||
/**
|
||||
* What the backend fetched to connect; undefined until it has, and again
|
||||
* after the connection stops. One object per fetch, so that a change of
|
||||
* state does not read as a change of this.
|
||||
*/
|
||||
resolved: ResolvedTransport | undefined;
|
||||
}
|
||||
|
||||
export interface DataPacket {
|
||||
/** The backend identity of the sender. */
|
||||
senderId: string;
|
||||
topic: string;
|
||||
text: string;
|
||||
}
|
||||
|
||||
export interface MediaBackend {
|
||||
/** The membership transport type this backend serves: "livekit". */
|
||||
readonly transportType: string;
|
||||
|
||||
/**
|
||||
* Makes `transport` usable as the one the local member publishes on. The
|
||||
* session found it; the backend does whatever has to succeed before it is
|
||||
* advertised in the membership. Rejects with the error the session reports
|
||||
* as fatal. Called once per session.
|
||||
*/
|
||||
prepareLocalTransport(transport: Transport): Promise<TransportCapabilities>;
|
||||
|
||||
/**
|
||||
* Hands the delayed leave event to the backend, which restarts it from
|
||||
* then on. Called each time the membership manager reports a new delay id,
|
||||
* only when `canDelegateDelayedLeave` is true. A rejection is logged by
|
||||
* the session, not fatal.
|
||||
*/
|
||||
delegateDelayedLeave(delayId: string): Promise<void>;
|
||||
|
||||
readonly local: LocalMediaBackend;
|
||||
|
||||
/**
|
||||
* The media of one remote member, null while nothing has arrived for it on
|
||||
* its transport. Built in `scope`, which the caller ends with the member.
|
||||
* This is also how the backend learns which remote transports exist: it
|
||||
* follows the membership's transport for as long as `scope` lives, and
|
||||
* connects to every transport some member is on.
|
||||
*/
|
||||
mediaFor$(
|
||||
scope: ObservableScope,
|
||||
membership$: Behavior<RTCMembership>,
|
||||
): Behavior<MemberMedia | null>;
|
||||
|
||||
/** Every connection the backend holds. For debugging and devtool purposes.*/
|
||||
readonly connections$: Behavior<BackendConnection[]>;
|
||||
|
||||
/**
|
||||
* A text channel beside the media: sent on the local transport, received on every one.
|
||||
* This could well be just another track kind. Not needed for element call. just for the sdk target.
|
||||
* Keeping it on the top level for visibility. Should be tackled in a future refactor.
|
||||
*/
|
||||
sendData(topic: string, text: string): Promise<void>;
|
||||
readonly data$: Observable<DataPacket>;
|
||||
}
|
||||
|
||||
export interface LocalMediaBackend {
|
||||
/**
|
||||
* State of the connection the local member publishes on. `Initialized`
|
||||
* while there is none; an `Error` is the connection's failure.
|
||||
*/
|
||||
readonly connectionState$: Behavior<MediaConnectionState | Error>;
|
||||
|
||||
/** The local member's own media; null until the connection carries a local participant. */
|
||||
readonly media$: Behavior<LocalMemberMedia | null>;
|
||||
|
||||
/**
|
||||
* Whether the local tracks reach the transport. The session sets it to
|
||||
* "joined and the homeserver is reachable". Remembered across connections:
|
||||
* the tracks are created on the first `true`, and their upstream is paused
|
||||
* while `false`, so the local preview stays live while nothing is heard.
|
||||
*/
|
||||
setPublishing(publish: boolean): void;
|
||||
/** The first failure to publish, kept until the session ends. Not fatal. */
|
||||
readonly publishError$: Behavior<Error | null>;
|
||||
|
||||
/**
|
||||
* Publishes a source and resolves with its track once it is in `media$`'s
|
||||
* `tracks$`. Remembered across connections, so a request before the
|
||||
* transport is up is applied once it is, and republished after a
|
||||
* reconnection. Rejects where the device could not be used.
|
||||
*/
|
||||
publish(
|
||||
request: PublishRequest,
|
||||
): Promise<LocalAudioMediaTrack | LocalVideoMediaTrack>;
|
||||
/** Removes one of our tracks by id; a screen share takes its audio with it. */
|
||||
unpublish(id: string): Promise<void>;
|
||||
}
|
||||
@@ -37,11 +37,12 @@ import {
|
||||
ConnectionState,
|
||||
type ConnectionOpts,
|
||||
} from "./Connection.ts";
|
||||
import { ObservableScope } from "../reactive/ObservableScope";
|
||||
import { ObservableScope } from "../../reactive/ObservableScope";
|
||||
import { type OpenIDClientParts } from "./openIDSFU";
|
||||
import { MatrixRTCError, FailToGetOpenIdToken } from "../errors";
|
||||
import { testJWTToken } from "../utils/test-fixtures";
|
||||
import { mockRemoteParticipant, ownMemberMock } from "../utils/test";
|
||||
import { MatrixRTCError, FailToGetOpenIdToken } from "../../errors";
|
||||
import { testJWTToken } from "../../utils/test-fixtures";
|
||||
import { ownMemberMock } from "../../utils/test";
|
||||
import { mockRemoteParticipant } from "./test";
|
||||
|
||||
let testScope: ObservableScope;
|
||||
|
||||
@@ -32,8 +32,8 @@ import {
|
||||
type OpenIDClientParts,
|
||||
type SFUConfig,
|
||||
} from "./openIDSFU";
|
||||
import { type Behavior } from "../reactive/Behavior";
|
||||
import { type ObservableScope } from "../reactive/ObservableScope";
|
||||
import { type Behavior } from "../../reactive/Behavior";
|
||||
import { type ObservableScope } from "../../reactive/ObservableScope";
|
||||
import {
|
||||
MatrixRTCError,
|
||||
InsufficientCapacityError,
|
||||
@@ -41,7 +41,7 @@ import {
|
||||
PeerConnectionTimeoutError,
|
||||
SFURoomCreationRestrictedError,
|
||||
UnknownRTCError,
|
||||
} from "../errors";
|
||||
} from "../../errors";
|
||||
|
||||
export interface ConnectionOpts {
|
||||
/**
|
||||
+13
-10
@@ -20,11 +20,11 @@ import { BehaviorSubject, combineLatest, map } from "rxjs";
|
||||
|
||||
import { buildLiveKitOptions } from "./livekitOptions";
|
||||
import { type OpenIDClientParts, type SFUConfig } from "./openIDSFU";
|
||||
import { type Behavior } from "../reactive/Behavior";
|
||||
import { type Behavior } from "../../reactive/Behavior";
|
||||
import { Connection, type ConnectionOpts, ConnectionState } from "./Connection";
|
||||
import { type ObservableScope } from "../reactive/ObservableScope";
|
||||
import { type LocalMediaInputs, type ResolvedTransport } from "../api";
|
||||
import { type MediaQuality } from "../config";
|
||||
import { type ObservableScope } from "../../reactive/ObservableScope";
|
||||
import { type LocalMediaInputs, type ResolvedTransport } from "../../api";
|
||||
import { type MediaQuality } from "../../config";
|
||||
import {
|
||||
audioCaptureOptions,
|
||||
type PublishRequestFor,
|
||||
@@ -54,14 +54,17 @@ export class ResolvedConnection extends Connection {
|
||||
public constructor(opts: ConnectionOpts, logger: Logger) {
|
||||
super(opts, logger);
|
||||
this.sfuConfig$ = new BehaviorSubject(opts.existingSFUConfig);
|
||||
// One object per fetch, so that a change of state does not read as a
|
||||
// change of what was fetched
|
||||
const resolved$ = this.sfuConfig$.pipe(
|
||||
map((config) => config && resolvedTransport(config)),
|
||||
);
|
||||
this.resolved$ = opts.scope.behavior(
|
||||
combineLatest([this.sfuConfig$, this.state$]).pipe(
|
||||
map(([config, state]) =>
|
||||
config === undefined ||
|
||||
state === ConnectionState.Stopped ||
|
||||
state instanceof Error
|
||||
combineLatest([resolved$, this.state$]).pipe(
|
||||
map(([resolved, state]) =>
|
||||
state === ConnectionState.Stopped || state instanceof Error
|
||||
? undefined
|
||||
: resolvedTransport(config),
|
||||
: resolved,
|
||||
),
|
||||
),
|
||||
);
|
||||
+8
-4
@@ -11,16 +11,20 @@ import { type LivekitTransport } from "matrix-js-sdk/lib/matrixrtc";
|
||||
import { type RemoteParticipant } from "livekit-client";
|
||||
import { logger } from "matrix-js-sdk/lib/logger";
|
||||
|
||||
import { Epoch, mapEpoch, ObservableScope } from "../reactive/ObservableScope";
|
||||
import {
|
||||
Epoch,
|
||||
mapEpoch,
|
||||
ObservableScope,
|
||||
} from "../../reactive/ObservableScope";
|
||||
import {
|
||||
createConnectionManager$,
|
||||
ConnectionManagerData,
|
||||
areLivekitTransportsEqual,
|
||||
} from "./ConnectionManager.ts";
|
||||
import { type ConnectionFactory } from "./ConnectionFactory.ts";
|
||||
import { type Connection } from "./Connection.ts";
|
||||
import { ownMemberMock, withTestScheduler } from "../utils/test";
|
||||
import { areLivekitTransportsEqual } from "./MatrixLivekitMembers.ts";
|
||||
import { type Behavior } from "../reactive/Behavior";
|
||||
import { ownMemberMock, withTestScheduler } from "../../utils/test";
|
||||
import { type Behavior } from "../../reactive/Behavior";
|
||||
|
||||
// Some test constants
|
||||
|
||||
+23
-5
@@ -21,13 +21,11 @@ import { type Logger } from "matrix-js-sdk/lib/logger";
|
||||
import { type RemoteParticipant } from "livekit-client";
|
||||
import { type CallMembershipIdentityParts } from "matrix-js-sdk/lib/matrixrtc/EncryptionManager";
|
||||
|
||||
import { type Behavior } from "../reactive/Behavior";
|
||||
import { type Behavior } from "../../reactive/Behavior";
|
||||
import { type Connection } from "./Connection.ts";
|
||||
import { Epoch, type ObservableScope } from "../reactive/ObservableScope";
|
||||
import { generateItemsWithEpoch } from "../reactive/observable";
|
||||
import { areLivekitTransportsEqual } from "./MatrixLivekitMembers.ts";
|
||||
import { Epoch, type ObservableScope } from "../../reactive/ObservableScope";
|
||||
import { generateItemsWithEpoch } from "../../reactive/observable";
|
||||
import { type ConnectionFactory } from "./ConnectionFactory.ts";
|
||||
import { isLocalTransport, type LocalTransport } from "./LocalTransport";
|
||||
import { type SFUConfig } from "./openIDSFU";
|
||||
|
||||
export class ConnectionManagerData {
|
||||
@@ -277,3 +275,23 @@ function removeDuplicateTransports<T extends LivekitTransport>(
|
||||
return acc;
|
||||
}, [] as T[]);
|
||||
}
|
||||
|
||||
/** The transport the local member publishes on, authenticated with. */
|
||||
export interface LocalTransport {
|
||||
transport: LivekitTransport;
|
||||
sfuConfig: SFUConfig;
|
||||
}
|
||||
|
||||
export function isLocalTransport(
|
||||
obj: LivekitTransport | LocalTransport,
|
||||
): obj is LocalTransport {
|
||||
return "transport" in obj && "sfuConfig" in obj;
|
||||
}
|
||||
|
||||
export function areLivekitTransportsEqual<T extends LivekitTransport>(
|
||||
t1: T | null,
|
||||
t2: T | null,
|
||||
): boolean {
|
||||
if (t1 && t2) return t1.livekit_service_url === t2.livekit_service_url;
|
||||
return !t1 && !t2;
|
||||
}
|
||||
@@ -7,17 +7,17 @@ Please see LICENSE in the repository root for full details.
|
||||
|
||||
import { EventEmitter } from "events";
|
||||
import { type RemoteParticipant, RoomEvent } from "livekit-client";
|
||||
import { logger } from "matrix-js-sdk/lib/logger";
|
||||
import { BehaviorSubject } from "rxjs";
|
||||
import { describe, expect, it, vi } from "vitest";
|
||||
|
||||
import { type DataMessage, type RemoteRTCMember } from "../api";
|
||||
import { Epoch } from "../reactive/ObservableScope";
|
||||
import { constant } from "../reactive/Behavior";
|
||||
import { Epoch } from "../../reactive/ObservableScope";
|
||||
import { constant } from "../../reactive/Behavior";
|
||||
import { type DataPacket } from "../api";
|
||||
import { type Connection, ConnectionState } from "./Connection";
|
||||
import { ConnectionManagerData } from "./ConnectionManager";
|
||||
import { createDataChannel$ } from "./DataChannel";
|
||||
import { exampleTransport, testScope } from "../utils/test";
|
||||
import { testScope } from "../../utils/test";
|
||||
import { exampleTransport } from "./test";
|
||||
|
||||
const encode = (text: string): Uint8Array => new TextEncoder().encode(text);
|
||||
|
||||
@@ -39,15 +39,8 @@ function fakeConnection(
|
||||
return { connection, room, publishData };
|
||||
}
|
||||
|
||||
function member(id: string): RemoteRTCMember {
|
||||
return { id, userId: `@${id}:example.org` } as RemoteRTCMember;
|
||||
}
|
||||
|
||||
function setup(
|
||||
connections: Connection[],
|
||||
members: RemoteRTCMember[] = [member("alice")],
|
||||
): {
|
||||
received: DataMessage[];
|
||||
function setup(connections: Connection[]): {
|
||||
received: DataPacket[];
|
||||
connections$: BehaviorSubject<Epoch<ConnectionManagerData>>;
|
||||
sendData: (topic: string, text: string) => Promise<void>;
|
||||
} {
|
||||
@@ -57,17 +50,15 @@ function setup(
|
||||
const { data$, sendData } = createDataChannel$({
|
||||
scope: testScope(),
|
||||
connectionManager: { connectionManagerData$: connections$ },
|
||||
remoteMembers$: constant(members),
|
||||
connection$: constant(connections[0] ?? null),
|
||||
logger,
|
||||
});
|
||||
const received: DataMessage[] = [];
|
||||
const received: DataPacket[] = [];
|
||||
data$.subscribe((message) => received.push(message));
|
||||
return { received, connections$, sendData };
|
||||
}
|
||||
|
||||
describe("data$", () => {
|
||||
it("delivers a packet from a member with its topic and text", () => {
|
||||
it("delivers a packet with its sender, topic and text", () => {
|
||||
const { connection, room } = fakeConnection("https://sfu");
|
||||
const { received } = setup([connection]);
|
||||
|
||||
@@ -80,18 +71,15 @@ describe("data$", () => {
|
||||
);
|
||||
|
||||
expect(received).toEqual([
|
||||
{ member: member("alice"), topic: "chat", text: "hello" },
|
||||
{ senderId: "alice", topic: "chat", text: "hello" },
|
||||
]);
|
||||
});
|
||||
|
||||
it("drops packets from the server and from identities that are not members", () => {
|
||||
it("drops packets from the server", () => {
|
||||
const { connection, room } = fakeConnection("https://sfu");
|
||||
const { received } = setup([connection]);
|
||||
|
||||
room.emit(RoomEvent.DataReceived, encode("server"), undefined);
|
||||
room.emit(RoomEvent.DataReceived, encode("stranger"), {
|
||||
identity: "mallory",
|
||||
} as RemoteParticipant);
|
||||
|
||||
expect(received).toEqual([]);
|
||||
});
|
||||
@@ -99,10 +87,7 @@ describe("data$", () => {
|
||||
it("listens on every connection, including ones added later", () => {
|
||||
const first = fakeConnection("https://sfu-1");
|
||||
const second = fakeConnection("https://sfu-2");
|
||||
const { received, connections$ } = setup(
|
||||
[first.connection],
|
||||
[member("alice"), member("bob")],
|
||||
);
|
||||
const { received, connections$ } = setup([first.connection]);
|
||||
|
||||
const data = new ConnectionManagerData();
|
||||
data.add(first.connection, []);
|
||||
@@ -116,7 +101,7 @@ describe("data$", () => {
|
||||
identity: "bob",
|
||||
} as RemoteParticipant);
|
||||
|
||||
expect(received.map((m) => [m.member.id, m.text])).toEqual([
|
||||
expect(received.map((m) => [m.senderId, m.text])).toEqual([
|
||||
["alice", "one"],
|
||||
["bob", "two"],
|
||||
]);
|
||||
@@ -10,7 +10,6 @@ import {
|
||||
type Room as LivekitRoom,
|
||||
RoomEvent,
|
||||
} from "livekit-client";
|
||||
import { type Logger } from "matrix-js-sdk/lib/logger";
|
||||
import {
|
||||
distinctUntilChanged,
|
||||
filter,
|
||||
@@ -22,39 +21,34 @@ import {
|
||||
switchMap,
|
||||
} from "rxjs";
|
||||
|
||||
import { type DataMessage, type RemoteRTCMember } from "../api";
|
||||
import { type Behavior } from "../reactive/Behavior";
|
||||
import { type ObservableScope } from "../reactive/ObservableScope";
|
||||
import { type Behavior } from "../../reactive/Behavior";
|
||||
import { type ObservableScope } from "../../reactive/ObservableScope";
|
||||
import { type DataPacket } from "../api";
|
||||
import { type Connection, ConnectionState } from "./Connection";
|
||||
import { type IConnectionManager } from "./ConnectionManager";
|
||||
|
||||
interface Props {
|
||||
scope: ObservableScope;
|
||||
connectionManager: IConnectionManager;
|
||||
/** Used to resolve the sender of each message, by its transport identity. */
|
||||
remoteMembers$: Behavior<RemoteRTCMember[]>;
|
||||
/** The local member's connection, which carries what we send. */
|
||||
connection$: Behavior<Connection | null>;
|
||||
logger: Logger;
|
||||
}
|
||||
|
||||
export interface DataChannel {
|
||||
data$: Observable<DataMessage>;
|
||||
data$: Observable<DataPacket>;
|
||||
sendData: (topic: string, text: string) => Promise<void>;
|
||||
}
|
||||
|
||||
/**
|
||||
* A text channel beside the media: LiveKit's reliable data packets, received
|
||||
* on every connection the client holds and sent on the local member's. The
|
||||
* sender is matched to a member by its identity, so the packet's own claim of
|
||||
* who sent it never reaches the host.
|
||||
* sender is reported by its identity; the packet's own claim of who sent it
|
||||
* never reaches the host.
|
||||
*/
|
||||
export function createDataChannel$({
|
||||
scope,
|
||||
connectionManager,
|
||||
remoteMembers$,
|
||||
connection$,
|
||||
logger,
|
||||
}: Props): DataChannel {
|
||||
const decoder = new TextDecoder();
|
||||
const encoder = new TextEncoder();
|
||||
@@ -68,19 +62,17 @@ export function createDataChannel$({
|
||||
|
||||
const data$ = rooms$.pipe(
|
||||
switchMap((rooms) => merge(...rooms.map(receivedOn))),
|
||||
map(({ payload, participant, topic }): DataMessage | null => {
|
||||
map(({ payload, participant, topic }): DataPacket | null =>
|
||||
// Only the server sends without a participant; a host has no use for it
|
||||
if (participant === undefined) return null;
|
||||
const member = remoteMembers$.value.find(
|
||||
(m) => m.id === participant.identity,
|
||||
);
|
||||
if (member === undefined) {
|
||||
logger.warn(`Dropping data from ${participant.identity}: not a member`);
|
||||
return null;
|
||||
}
|
||||
return { member, topic: topic ?? "", text: decoder.decode(payload) };
|
||||
}),
|
||||
filter((message) => message !== null),
|
||||
participant === undefined
|
||||
? null
|
||||
: {
|
||||
senderId: participant.identity,
|
||||
topic: topic ?? "",
|
||||
text: decoder.decode(payload),
|
||||
},
|
||||
),
|
||||
filter((packet) => packet !== null),
|
||||
scope.bind(),
|
||||
share(),
|
||||
);
|
||||
@@ -7,16 +7,16 @@ Please see LICENSE in the repository root for full details.
|
||||
|
||||
import { type BaseKeyProvider, ExternalE2EEKeyProvider } from "livekit-client";
|
||||
import { type Logger } from "matrix-js-sdk/lib/logger";
|
||||
import { type MatrixRTCSession as JsSdkRTCSession } from "matrix-js-sdk/lib/matrixrtc";
|
||||
import { type Observable } from "rxjs";
|
||||
|
||||
import { E2eeType } from "../encryption";
|
||||
import { E2eeType, type EncryptionSystem } from "../../encryption";
|
||||
import { type MediaKey } from "../api";
|
||||
import { MatrixKeyProvider } from "./MatrixKeyProvider";
|
||||
import { type EncryptionSystem } from "../encryption";
|
||||
|
||||
/** The LiveKit key provider for the encryption the host asked for, or none. */
|
||||
export function createKeyProvider(
|
||||
encryptionSystem: EncryptionSystem,
|
||||
session: JsSdkRTCSession,
|
||||
mediaKeys$: Observable<MediaKey>,
|
||||
logger: Logger,
|
||||
): BaseKeyProvider | undefined {
|
||||
switch (encryptionSystem.kind) {
|
||||
@@ -24,7 +24,7 @@ export function createKeyProvider(
|
||||
return undefined;
|
||||
case E2eeType.PER_PARTICIPANT: {
|
||||
const keyProvider = new MatrixKeyProvider();
|
||||
keyProvider.setRTCSession(session);
|
||||
keyProvider.setMediaKeys(mediaKeys$);
|
||||
return keyProvider;
|
||||
}
|
||||
case E2eeType.SHARED_KEY: {
|
||||
@@ -0,0 +1,112 @@
|
||||
/*
|
||||
Copyright 2026 Element Creations Ltd.
|
||||
|
||||
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
|
||||
Please see LICENSE in the repository root for full details.
|
||||
*/
|
||||
|
||||
import fetchMock from "fetch-mock";
|
||||
import { logger } from "matrix-js-sdk/lib/logger";
|
||||
import { type LivekitTransport } from "matrix-js-sdk/lib/matrixrtc";
|
||||
import { BehaviorSubject } from "rxjs";
|
||||
import { afterEach, describe, expect, it } from "vitest";
|
||||
|
||||
import { ObservableScope } from "../../reactive/ObservableScope";
|
||||
import { testScope } from "../../utils/test";
|
||||
import { MediaConnectionState } from "../api";
|
||||
import { ConnectionState } from "./Connection";
|
||||
import { toMediaConnectionState } from "./connectionState";
|
||||
import { createTransportRegistry, supportsDelegation } from "./LivekitBackend";
|
||||
import { exampleTransport } from "./test";
|
||||
|
||||
const otherTransport: LivekitTransport = {
|
||||
type: "livekit",
|
||||
livekit_service_url: "https://other.example.org",
|
||||
};
|
||||
|
||||
describe("createTransportRegistry", () => {
|
||||
it("connects to a transport once however many members are on it, and drops it with the last", () => {
|
||||
const registry = createTransportRegistry(testScope());
|
||||
const urls = (): string[] =>
|
||||
registry.transports$.value.value.map((t) => t.livekit_service_url);
|
||||
const alice = new ObservableScope();
|
||||
const bob = new ObservableScope();
|
||||
|
||||
registry.follow(alice, new BehaviorSubject(exampleTransport));
|
||||
registry.follow(bob, new BehaviorSubject(exampleTransport));
|
||||
expect(urls()).toEqual([exampleTransport.livekit_service_url]);
|
||||
|
||||
alice.end();
|
||||
expect(urls()).toEqual([exampleTransport.livekit_service_url]);
|
||||
|
||||
bob.end();
|
||||
expect(urls()).toEqual([]);
|
||||
});
|
||||
|
||||
it("follows a member that changes transport", () => {
|
||||
const registry = createTransportRegistry(testScope());
|
||||
const transport$ = new BehaviorSubject<LivekitTransport | undefined>(
|
||||
exampleTransport,
|
||||
);
|
||||
|
||||
registry.follow(testScope(), transport$);
|
||||
transport$.next(otherTransport);
|
||||
expect(registry.transports$.value.value).toEqual([otherTransport]);
|
||||
|
||||
transport$.next(undefined);
|
||||
expect(registry.transports$.value.value).toEqual([]);
|
||||
});
|
||||
|
||||
it("is safe to release after the registry's own scope has ended", () => {
|
||||
const scope = new ObservableScope();
|
||||
const registry = createTransportRegistry(scope);
|
||||
const member = new ObservableScope();
|
||||
registry.follow(member, new BehaviorSubject(exampleTransport));
|
||||
|
||||
scope.end();
|
||||
expect(() => member.end()).not.toThrow();
|
||||
});
|
||||
});
|
||||
|
||||
describe("supportsDelegation", () => {
|
||||
afterEach(() => fetchMock.reset());
|
||||
const homeserver = "https://matrix.example.org";
|
||||
const homeserverEndpoint = `${homeserver}/_matrix/client/unstable/io.element.msc4195/rtc/livekit/delegate_delayed_leave`;
|
||||
const transportEndpoint = `${exampleTransport.livekit_service_url}/delegate_delayed_leave`;
|
||||
|
||||
it.each([
|
||||
["the homeserver supports it", 400, 404, true],
|
||||
["only the transport supports it", 404, 400, true],
|
||||
["neither supports it", 404, 404, false],
|
||||
])("%s", async (_name, homeserverStatus, transportStatus, expected) => {
|
||||
fetchMock.post(homeserverEndpoint, homeserverStatus);
|
||||
fetchMock.post(transportEndpoint, transportStatus);
|
||||
await expect(
|
||||
supportsDelegation(homeserver, exampleTransport, logger),
|
||||
).resolves.toBe(expected);
|
||||
});
|
||||
|
||||
it("assumes no support where the probe itself fails", async () => {
|
||||
fetchMock.post(homeserverEndpoint, { throws: new TypeError("CORS") });
|
||||
fetchMock.post(transportEndpoint, { throws: new TypeError("CORS") });
|
||||
await expect(
|
||||
supportsDelegation(homeserver, exampleTransport, logger),
|
||||
).resolves.toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe("toMediaConnectionState", () => {
|
||||
it("maps every LiveKit state and passes an error through", () => {
|
||||
expect(toMediaConnectionState(ConnectionState.FetchingConfig)).toBe(
|
||||
MediaConnectionState.Preparing,
|
||||
);
|
||||
expect(toMediaConnectionState(ConnectionState.LivekitConnected)).toBe(
|
||||
MediaConnectionState.Connected,
|
||||
);
|
||||
expect(
|
||||
toMediaConnectionState(ConnectionState.LivekitSignalReconnecting),
|
||||
).toBe(MediaConnectionState.Reconnecting);
|
||||
const error = new Error("gone");
|
||||
expect(toMediaConnectionState(error)).toBe(error);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,357 @@
|
||||
/*
|
||||
Copyright 2026 Element Creations Ltd.
|
||||
|
||||
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
|
||||
Please see LICENSE in the repository root for full details.
|
||||
*/
|
||||
|
||||
import { type MatrixClient } from "matrix-js-sdk";
|
||||
import { type Logger } from "matrix-js-sdk/lib/logger";
|
||||
import {
|
||||
isLivekitTransport,
|
||||
type LivekitTransport,
|
||||
type Transport,
|
||||
} from "matrix-js-sdk/lib/matrixrtc";
|
||||
import {
|
||||
BehaviorSubject,
|
||||
combineLatest,
|
||||
distinctUntilChanged,
|
||||
map,
|
||||
type Observable,
|
||||
of,
|
||||
ReplaySubject,
|
||||
switchMap,
|
||||
} from "rxjs";
|
||||
|
||||
import {
|
||||
type MemberMedia,
|
||||
type ResolvedTransport,
|
||||
type RTCMembership,
|
||||
} from "../../api";
|
||||
import { type MediaQuality } from "../../config";
|
||||
import { MatrixRTCError } from "../../errors";
|
||||
import { type Behavior } from "../../reactive/Behavior";
|
||||
import {
|
||||
type Epoch,
|
||||
type ObservableScope,
|
||||
trackEpoch,
|
||||
} from "../../reactive/ObservableScope";
|
||||
import { mapScoped } from "../../utils/mapScoped";
|
||||
import {
|
||||
type BackendConnection,
|
||||
type MediaBackend,
|
||||
type MediaBackendContext,
|
||||
type TransportCapabilities,
|
||||
} from "../api";
|
||||
import { type Connection } from "./Connection";
|
||||
import {
|
||||
LivekitConnectionFactory,
|
||||
ResolvedConnection,
|
||||
} from "./ConnectionFactory";
|
||||
import {
|
||||
createConnectionManager$,
|
||||
type LocalTransport,
|
||||
} from "./ConnectionManager";
|
||||
import { toMediaConnectionState } from "./connectionState";
|
||||
import { createDataChannel$ } from "./DataChannel";
|
||||
import { createKeyProvider } from "./KeyProvider";
|
||||
import { createLivekitMemberMedia } from "./LivekitMemberMedia";
|
||||
import { createLivekitLocalMedia } from "./LocalMedia";
|
||||
import { getSFUConfigWithOpenID, type TokenEndpoint } from "./openIDSFU";
|
||||
|
||||
export interface LivekitBackendOptions {
|
||||
/** `getOpenIdToken` for the token exchange, `baseUrl` for the delegation probe and handover. */
|
||||
client: Pick<MatrixClient, "getOpenIdToken" | "baseUrl">;
|
||||
mediaQuality?: MediaQuality;
|
||||
/** Which token service endpoint the local transport uses; remote ones try both. */
|
||||
tokenEndpoint: TokenEndpoint;
|
||||
}
|
||||
|
||||
/**
|
||||
* MatrixRTC media over LiveKit SFUs: one room per transport, the local one
|
||||
* published on, every remote one subscribed to, with the token service
|
||||
* protocol in front of each.
|
||||
*/
|
||||
export function createLivekitBackend(
|
||||
scope: ObservableScope,
|
||||
{
|
||||
roomId,
|
||||
ownMembershipIdentity,
|
||||
localMedia,
|
||||
encryptionSystem,
|
||||
mediaKeys$,
|
||||
timings,
|
||||
logger: parentLogger,
|
||||
}: MediaBackendContext,
|
||||
{ client, mediaQuality, tokenEndpoint }: LivekitBackendOptions,
|
||||
): MediaBackend {
|
||||
const logger = parentLogger.getChild("[LivekitBackend]");
|
||||
const keyProvider = createKeyProvider(encryptionSystem, mediaKeys$, logger);
|
||||
const transports = createTransportRegistry(scope);
|
||||
const localTransport$ = new ReplaySubject<LocalTransport>(1);
|
||||
|
||||
const connectionManager = createConnectionManager$({
|
||||
scope,
|
||||
connectionFactory: new LivekitConnectionFactory(
|
||||
client,
|
||||
roomId,
|
||||
localMedia,
|
||||
keyProvider,
|
||||
mediaQuality,
|
||||
),
|
||||
localTransport$,
|
||||
remoteTransports$: transports.transports$,
|
||||
logger,
|
||||
ownMembershipIdentity,
|
||||
});
|
||||
|
||||
let local: LivekitTransport | undefined;
|
||||
const prepareLocalTransport = async (
|
||||
transport: Transport,
|
||||
): Promise<TransportCapabilities> => {
|
||||
if (!isLivekitTransport(transport))
|
||||
throw new MatrixRTCError(`Not a LiveKit transport: ${transport.type}`);
|
||||
const [sfuConfig, canDelegateDelayedLeave] = await Promise.all([
|
||||
getSFUConfigWithOpenID(
|
||||
client,
|
||||
ownMembershipIdentity,
|
||||
transport.livekit_service_url,
|
||||
roomId,
|
||||
{ tokenEndpoint },
|
||||
logger,
|
||||
),
|
||||
supportsDelegation(client.baseUrl, transport, logger),
|
||||
]);
|
||||
local = transport;
|
||||
localTransport$.next({ transport, sfuConfig });
|
||||
return { canDelegateDelayedLeave };
|
||||
};
|
||||
|
||||
// The token this issues is discarded; the request is what hands the leave over
|
||||
const delegateDelayedLeave = async (delayId: string): Promise<void> => {
|
||||
if (local === undefined)
|
||||
throw new MatrixRTCError("No local transport to delegate the leave to");
|
||||
await getSFUConfigWithOpenID(
|
||||
client,
|
||||
ownMembershipIdentity,
|
||||
local.livekit_service_url,
|
||||
roomId,
|
||||
{
|
||||
tokenEndpoint,
|
||||
delayEndpointBaseUrl: client.baseUrl,
|
||||
delayId,
|
||||
delayTimeoutMs: timings.delegatedDelayedLeave.delay_ms,
|
||||
},
|
||||
logger,
|
||||
);
|
||||
};
|
||||
|
||||
const mediaFor$ = (
|
||||
memberScope: ObservableScope,
|
||||
membership$: Behavior<RTCMembership>,
|
||||
): Behavior<MemberMedia | null> => {
|
||||
const identity = membership$.value.rtcBackendIdentity;
|
||||
const transport$ = memberScope.behavior(
|
||||
membership$.pipe(
|
||||
map((membership) => membership.getTransport()),
|
||||
map((transport) =>
|
||||
isLivekitTransport(transport) ? transport : undefined,
|
||||
),
|
||||
distinctUntilChanged(
|
||||
(a, b) => a?.livekit_service_url === b?.livekit_service_url,
|
||||
),
|
||||
),
|
||||
);
|
||||
transports.follow(memberScope, transport$);
|
||||
|
||||
const source$ = memberScope.behavior(
|
||||
combineLatest([
|
||||
transport$,
|
||||
connectionManager.connectionManagerData$,
|
||||
]).pipe(
|
||||
map(([transport, { value: connections }]) => {
|
||||
if (transport === undefined) return null;
|
||||
const connection = connections.getConnectionForTransport(transport);
|
||||
const participant = connections
|
||||
.getParticipantsForTransport(transport)
|
||||
.find((p) => p.identity === identity);
|
||||
return connection && participant
|
||||
? { participant, room: connection.livekitRoom }
|
||||
: null;
|
||||
}),
|
||||
distinctUntilChanged(
|
||||
(a, b) => a?.participant === b?.participant && a?.room === b?.room,
|
||||
),
|
||||
),
|
||||
);
|
||||
source$.pipe(memberScope.bind()).subscribe((source) => {
|
||||
logger.info(
|
||||
`${identity}: LiveKit participant ${source ? `matched (${source.participant.sid})` : "missing"}`,
|
||||
);
|
||||
});
|
||||
return memberScope.behavior(
|
||||
mapScoped(memberScope, source$, (mediaScope, { participant, room }) =>
|
||||
createLivekitMemberMedia(
|
||||
mediaScope,
|
||||
participant,
|
||||
room,
|
||||
encryptionSystem,
|
||||
),
|
||||
).pipe(map((media) => media ?? null)),
|
||||
);
|
||||
};
|
||||
|
||||
const connections$ = scope.behavior<BackendConnection[]>(
|
||||
connectionManager.connectionManagerData$.pipe(
|
||||
switchMap(({ value }) => {
|
||||
const connections = value.getConnections();
|
||||
if (connections.length === 0) return of([]);
|
||||
return combineLatest(connections.map(backendConnection$));
|
||||
}),
|
||||
),
|
||||
);
|
||||
|
||||
const localMedia_ = createLivekitLocalMedia({
|
||||
scope,
|
||||
connectionManager,
|
||||
localTransport$,
|
||||
localMedia,
|
||||
encryptionSystem,
|
||||
logger,
|
||||
});
|
||||
|
||||
const { data$, sendData } = createDataChannel$({
|
||||
scope,
|
||||
connectionManager,
|
||||
connection$: localMedia_.connection$,
|
||||
});
|
||||
|
||||
return {
|
||||
transportType: "livekit",
|
||||
prepareLocalTransport,
|
||||
delegateDelayedLeave,
|
||||
local: localMedia_,
|
||||
mediaFor$,
|
||||
connections$,
|
||||
sendData,
|
||||
data$,
|
||||
};
|
||||
}
|
||||
|
||||
function backendConnection$(
|
||||
connection: Connection,
|
||||
): Observable<BackendConnection> {
|
||||
const resolved$: Observable<ResolvedTransport | undefined> =
|
||||
connection instanceof ResolvedConnection
|
||||
? connection.resolved$
|
||||
: of(undefined);
|
||||
return combineLatest([connection.state$, resolved$]).pipe(
|
||||
map(([state, resolved]): BackendConnection => ({
|
||||
transport: connection.transport,
|
||||
state: toMediaConnectionState(state),
|
||||
resolved,
|
||||
})),
|
||||
);
|
||||
}
|
||||
|
||||
export interface TransportRegistry {
|
||||
/** Keeps a member's transport registered for as long as its scope lives. */
|
||||
follow(
|
||||
scope: ObservableScope,
|
||||
transport$: Behavior<LivekitTransport | undefined>,
|
||||
): void;
|
||||
/** Every transport some member is on, for the connection manager. */
|
||||
transports$: Behavior<Epoch<LivekitTransport[]>>;
|
||||
}
|
||||
|
||||
/**
|
||||
* The remote transports to connect to, counted per member so that a transport
|
||||
* stays while any member names it. A release after the scope has ended only
|
||||
* touches the map, so it is safe in any order of teardown.
|
||||
*/
|
||||
export function createTransportRegistry(
|
||||
scope: ObservableScope,
|
||||
): TransportRegistry {
|
||||
const counts = new Map<
|
||||
string,
|
||||
{ transport: LivekitTransport; count: number }
|
||||
>();
|
||||
const transports = new BehaviorSubject<LivekitTransport[]>([]);
|
||||
const publish = (): void =>
|
||||
transports.next([...counts.values()].map(({ transport }) => transport));
|
||||
|
||||
const register = (transport: LivekitTransport): (() => void) => {
|
||||
const url = transport.livekit_service_url;
|
||||
const entry = counts.get(url) ?? { transport, count: 0 };
|
||||
entry.count++;
|
||||
if (entry.count === 1) {
|
||||
counts.set(url, entry);
|
||||
publish();
|
||||
}
|
||||
return () => {
|
||||
if (--entry.count > 0) return;
|
||||
counts.delete(url);
|
||||
publish();
|
||||
};
|
||||
};
|
||||
|
||||
return {
|
||||
follow: (memberScope, transport$) => {
|
||||
let release: (() => void) | undefined;
|
||||
transport$.pipe(memberScope.bind()).subscribe((transport) => {
|
||||
release?.();
|
||||
release = transport && register(transport);
|
||||
});
|
||||
memberScope.onEnd(() => release?.());
|
||||
},
|
||||
transports$: scope.behavior(transports.pipe(trackEpoch())),
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Whether the SFU can take over restarting the delayed leave. Either the
|
||||
* homeserver or the transport has to support it. Each endpoint is hit without
|
||||
* credentials and read for a 404, not retried: many servers predate the
|
||||
* endpoint altogether and answer with a CORS failure that a retry loop would
|
||||
* only repeat.
|
||||
*/
|
||||
export async function supportsDelegation(
|
||||
homeserverUrl: string,
|
||||
transport: LivekitTransport,
|
||||
logger: Logger,
|
||||
): Promise<boolean> {
|
||||
const [homeserver, sfu] = await Promise.all([
|
||||
endpointExists(
|
||||
`${homeserverUrl}/_matrix/client/unstable/io.element.msc4195/rtc/livekit/delegate_delayed_leave`,
|
||||
"homeserver",
|
||||
logger,
|
||||
),
|
||||
endpointExists(
|
||||
`${transport.livekit_service_url}/delegate_delayed_leave`,
|
||||
`transport ${transport.livekit_service_url}`,
|
||||
logger,
|
||||
),
|
||||
]);
|
||||
return homeserver || sfu;
|
||||
}
|
||||
|
||||
async function endpointExists(
|
||||
url: string,
|
||||
serviceName: string,
|
||||
logger: Logger,
|
||||
): Promise<boolean> {
|
||||
try {
|
||||
const res = await fetch(url, { method: "POST" });
|
||||
const supported = res.status !== 404;
|
||||
logger.info(
|
||||
`${serviceName} ${supported ? "supports" : "does not support"} delegation`,
|
||||
);
|
||||
return supported;
|
||||
} catch (e) {
|
||||
logger.warn(
|
||||
`Failed to determine whether ${serviceName} supports delegation, assuming no support`,
|
||||
e,
|
||||
);
|
||||
return false;
|
||||
}
|
||||
}
|
||||
+2
-2
@@ -18,8 +18,8 @@ import {
|
||||
} from "livekit-client";
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
|
||||
import { ObservableScope } from "../reactive/ObservableScope";
|
||||
import { type AudioMediaTrack } from "../api";
|
||||
import { ObservableScope } from "../../reactive/ObservableScope";
|
||||
import { type AudioMediaTrack } from "../../api";
|
||||
import {
|
||||
createLivekitMediaTrack,
|
||||
createLocalLivekitMediaTrack,
|
||||
@@ -38,8 +38,8 @@ import {
|
||||
switchMap,
|
||||
} from "rxjs";
|
||||
|
||||
import { type Behavior } from "../reactive/Behavior";
|
||||
import { type ObservableScope } from "../reactive/ObservableScope";
|
||||
import { type Behavior } from "../../reactive/Behavior";
|
||||
import { type ObservableScope } from "../../reactive/ObservableScope";
|
||||
import {
|
||||
type AudioMediaTrack,
|
||||
type LocalAudioMediaTrack,
|
||||
@@ -48,9 +48,9 @@ import {
|
||||
type MediaStreamStats,
|
||||
type MediaTrack,
|
||||
type VideoMediaTrack,
|
||||
} from "../api";
|
||||
import { MatrixRTCError } from "../errors";
|
||||
import { LazyBehavior } from "../utils/LazyBehavior";
|
||||
} from "../../api";
|
||||
import { MatrixRTCError } from "../../errors";
|
||||
import { LazyBehavior } from "../../utils/LazyBehavior";
|
||||
|
||||
export const mediaSources: Record<Track.Source, MediaSource> = {
|
||||
[Track.Source.Microphone]: "microphone",
|
||||
+2
-2
@@ -15,8 +15,8 @@ import {
|
||||
} from "livekit-client";
|
||||
import { describe, expect, it } from "vitest";
|
||||
|
||||
import { E2eeType } from "../encryption";
|
||||
import { testScope } from "../utils/test";
|
||||
import { E2eeType } from "../../encryption";
|
||||
import { testScope } from "../../utils/test";
|
||||
import { createLivekitMemberMedia } from "./LivekitMemberMedia";
|
||||
|
||||
describe("createLivekitMemberMedia", () => {
|
||||
@@ -18,17 +18,17 @@ import {
|
||||
} from "livekit-client";
|
||||
import { filter, map, type Observable } from "rxjs";
|
||||
|
||||
import { type ObservableScope } from "../reactive/ObservableScope";
|
||||
import { generateItems } from "../reactive/observable";
|
||||
import { E2eeType } from "../encryption";
|
||||
import { type EncryptionSystem } from "../encryption";
|
||||
import { type Behavior } from "../reactive/Behavior";
|
||||
import { type ObservableScope } from "../../reactive/ObservableScope";
|
||||
import { generateItems } from "../../reactive/observable";
|
||||
import { E2eeType } from "../../encryption";
|
||||
import { type EncryptionSystem } from "../../encryption";
|
||||
import { type Behavior } from "../../reactive/Behavior";
|
||||
import {
|
||||
type EncryptionError,
|
||||
type LocalMemberMedia,
|
||||
type MediaSource,
|
||||
type MemberMedia,
|
||||
} from "../api";
|
||||
} from "../../api";
|
||||
import {
|
||||
createLivekitMediaTrack,
|
||||
createLocalLivekitMediaTrack,
|
||||
@@ -0,0 +1,272 @@
|
||||
/*
|
||||
Copyright 2026 Element Creations Ltd.
|
||||
|
||||
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
|
||||
Please see LICENSE in the repository root for full details.
|
||||
*/
|
||||
|
||||
import { observeParticipantEvents } from "@livekit/components-core";
|
||||
import {
|
||||
type LocalParticipant,
|
||||
type LocalTrackPublication,
|
||||
ParticipantEvent,
|
||||
type Room as LivekitRoom,
|
||||
} from "livekit-client";
|
||||
import { type Logger } from "matrix-js-sdk/lib/logger";
|
||||
import {
|
||||
BehaviorSubject,
|
||||
combineLatest,
|
||||
distinctUntilChanged,
|
||||
filter,
|
||||
firstValueFrom,
|
||||
map,
|
||||
NEVER,
|
||||
type Observable,
|
||||
of,
|
||||
switchMap,
|
||||
} from "rxjs";
|
||||
|
||||
import {
|
||||
type LocalAudioMediaTrack,
|
||||
type LocalMediaInputs,
|
||||
type LocalMemberMedia,
|
||||
type LocalVideoMediaTrack,
|
||||
type MediaSource,
|
||||
type PublishRequest,
|
||||
} from "../../api";
|
||||
import { type EncryptionSystem } from "../../encryption";
|
||||
import { FailToStartLivekitConnection, toMatrixRTCError } from "../../errors";
|
||||
import { type Behavior } from "../../reactive/Behavior";
|
||||
import { type ObservableScope } from "../../reactive/ObservableScope";
|
||||
import { mapScoped } from "../../utils/mapScoped";
|
||||
import { type LocalMediaBackend, MediaConnectionState } from "../api";
|
||||
import { type Connection } from "./Connection";
|
||||
import {
|
||||
type IConnectionManager,
|
||||
type LocalTransport,
|
||||
} from "./ConnectionManager";
|
||||
import { createLocalLivekitMemberMedia } from "./LivekitMemberMedia";
|
||||
import { livekitSources } from "./LivekitMediaTrack";
|
||||
import { type DesiredMedia, Publisher } from "./Publisher";
|
||||
import { toMediaConnectionState } from "./connectionState";
|
||||
|
||||
interface Props {
|
||||
scope: ObservableScope;
|
||||
connectionManager: IConnectionManager;
|
||||
localTransport$: Observable<LocalTransport>;
|
||||
localMedia: LocalMediaInputs;
|
||||
encryptionSystem: EncryptionSystem;
|
||||
logger: Logger;
|
||||
}
|
||||
|
||||
export interface LivekitLocalMedia extends LocalMediaBackend {
|
||||
/** The connection the local member publishes on; the data channel sends on it. */
|
||||
connection$: Behavior<Connection | null>;
|
||||
}
|
||||
|
||||
/**
|
||||
* The local member's side of the LiveKit backend: one `Publisher` per
|
||||
* connection to the local transport, driven by the publish intent and the
|
||||
* requests the host made, both of which outlive any one connection.
|
||||
*/
|
||||
export function createLivekitLocalMedia({
|
||||
scope,
|
||||
connectionManager,
|
||||
localTransport$,
|
||||
localMedia,
|
||||
encryptionSystem,
|
||||
logger: parentLogger,
|
||||
}: Props): LivekitLocalMedia {
|
||||
const logger = parentLogger.getChild("[LocalMedia]");
|
||||
|
||||
const connection$ = scope.behavior(
|
||||
combineLatest([
|
||||
connectionManager.connectionManagerData$,
|
||||
localTransport$,
|
||||
]).pipe(
|
||||
map(([{ value: connections }, { transport }]) =>
|
||||
connections.getConnectionForTransport(transport),
|
||||
),
|
||||
),
|
||||
null,
|
||||
);
|
||||
|
||||
const connectionState$ = scope.behavior<MediaConnectionState | Error>(
|
||||
connection$.pipe(
|
||||
switchMap(
|
||||
(connection) =>
|
||||
connection?.state$.pipe(map(toMediaConnectionState)) ??
|
||||
of(MediaConnectionState.Initialized),
|
||||
),
|
||||
),
|
||||
);
|
||||
|
||||
const desired: DesiredMedia = new Map(
|
||||
localMedia.publish.map((request) => [
|
||||
request.source,
|
||||
{ request, enabled: true },
|
||||
]),
|
||||
);
|
||||
const publishing$ = new BehaviorSubject(false);
|
||||
const publisher$ = new BehaviorSubject<Publisher | null>(null);
|
||||
const publishError$ = new BehaviorSubject<Error | null>(null);
|
||||
|
||||
scope.reconcile(connection$, async (connection) => {
|
||||
if (connection === null) return;
|
||||
const publisher = new Publisher(
|
||||
connection.livekitRoom,
|
||||
localMedia,
|
||||
desired,
|
||||
logger.getChild(
|
||||
`[Publisher ${connection.transport.livekit_service_url}]`,
|
||||
),
|
||||
);
|
||||
publisher$.next(publisher);
|
||||
return Promise.resolve(async (): Promise<void> => {
|
||||
publisher$.next(null);
|
||||
await publisher.destroy();
|
||||
});
|
||||
});
|
||||
|
||||
scope.reconcile(
|
||||
scope.behavior(combineLatest([publisher$, publishing$])),
|
||||
async ([publisher, publish]) => {
|
||||
if (publisher === null) return;
|
||||
try {
|
||||
if (publish) {
|
||||
publisher.start();
|
||||
await publisher.startPublishing();
|
||||
} else if (publisher.shouldPublish) await publisher.stopPublishing();
|
||||
} catch (e) {
|
||||
if (publishError$.value === null)
|
||||
publishError$.next(
|
||||
new FailToStartLivekitConnection(
|
||||
e instanceof Error ? e.message : String(e),
|
||||
),
|
||||
);
|
||||
else logger.error("Another publish error", e);
|
||||
}
|
||||
},
|
||||
);
|
||||
|
||||
const participant$ = scope.behavior(
|
||||
connection$.pipe(map((c) => c?.livekitRoom.localParticipant ?? null)),
|
||||
);
|
||||
|
||||
const setEnabled = async (
|
||||
source: MediaSource,
|
||||
enabled: boolean,
|
||||
): Promise<boolean> => {
|
||||
const wanted = desired.get(source);
|
||||
if (wanted) wanted.enabled = enabled;
|
||||
// Without a publisher the request waits for the tracks to be created
|
||||
const publisher = publisher$.value;
|
||||
if (publisher === null) return enabled;
|
||||
const result = await publisher.setEnabled(source, enabled);
|
||||
if (wanted && result !== enabled) wanted.enabled = result;
|
||||
return result;
|
||||
};
|
||||
|
||||
const media$ = scope.behavior<LocalMemberMedia | null>(
|
||||
mapScoped(
|
||||
scope,
|
||||
participantAndRoom$(scope, participant$, connection$),
|
||||
(mediaScope, { participant, room }) =>
|
||||
createLocalLivekitMemberMedia(
|
||||
mediaScope,
|
||||
participant,
|
||||
room,
|
||||
encryptionSystem,
|
||||
setEnabled,
|
||||
),
|
||||
).pipe(map((media) => media ?? null)),
|
||||
);
|
||||
|
||||
const publish = async (
|
||||
request: PublishRequest,
|
||||
): Promise<LocalAudioMediaTrack | LocalVideoMediaTrack> => {
|
||||
desired.set(request.source, { request, enabled: true });
|
||||
const publisher = publisher$.value;
|
||||
try {
|
||||
// Before the start, the publisher publishes everything desired itself
|
||||
const publication =
|
||||
(publisher?.started ? await publisher.publish(request) : undefined) ??
|
||||
(await firstValueFrom(publication$(participant$, request.source)));
|
||||
return await firstValueFrom(trackWithId$(media$, publication.trackSid));
|
||||
} catch (e) {
|
||||
desired.delete(request.source);
|
||||
throw toMatrixRTCError(e);
|
||||
}
|
||||
};
|
||||
|
||||
const unpublish = async (id: string): Promise<void> => {
|
||||
const track = media$.value?.tracks$.value.find((t) => t.id === id);
|
||||
if (track === undefined) return;
|
||||
// The screen share audio goes with its video
|
||||
const source =
|
||||
track.source === "screenShareAudio" ? "screenShare" : track.source;
|
||||
desired.delete(source);
|
||||
await publisher$.value?.unpublish(source);
|
||||
};
|
||||
|
||||
return {
|
||||
connection$,
|
||||
connectionState$,
|
||||
media$,
|
||||
setPublishing: (publish) => publishing$.next(publish),
|
||||
publishError$,
|
||||
publish,
|
||||
unpublish,
|
||||
};
|
||||
}
|
||||
|
||||
/** The pair a `MemberMedia` is built on, changing only when one of the two does. */
|
||||
function participantAndRoom$(
|
||||
scope: ObservableScope,
|
||||
participant$: Behavior<LocalParticipant | null>,
|
||||
connection$: Behavior<Connection | null>,
|
||||
): Behavior<{ participant: LocalParticipant; room: LivekitRoom } | null> {
|
||||
return scope.behavior(
|
||||
combineLatest([participant$, connection$]).pipe(
|
||||
map(([participant, connection]) =>
|
||||
participant && connection
|
||||
? { participant, room: connection.livekitRoom }
|
||||
: null,
|
||||
),
|
||||
distinctUntilChanged(
|
||||
(a, b) => a?.participant === b?.participant && a?.room === b?.room,
|
||||
),
|
||||
),
|
||||
);
|
||||
}
|
||||
|
||||
/** The publication of a source, once the current participant has made it. */
|
||||
function publication$(
|
||||
participant$: Behavior<LocalParticipant | null>,
|
||||
source: MediaSource,
|
||||
): Observable<LocalTrackPublication> {
|
||||
return participant$.pipe(
|
||||
switchMap((participant) =>
|
||||
participant === null
|
||||
? NEVER
|
||||
: observeParticipantEvents(
|
||||
participant,
|
||||
ParticipantEvent.LocalTrackPublished,
|
||||
).pipe(
|
||||
map(() => participant.getTrackPublication(livekitSources[source])),
|
||||
),
|
||||
),
|
||||
filter((publication) => publication !== undefined),
|
||||
);
|
||||
}
|
||||
|
||||
function trackWithId$(
|
||||
media$: Behavior<LocalMemberMedia | null>,
|
||||
id: string,
|
||||
): Observable<LocalAudioMediaTrack | LocalVideoMediaTrack> {
|
||||
return media$.pipe(
|
||||
switchMap((media) => media?.tracks$ ?? of([])),
|
||||
map((tracks) => tracks.find((track) => track.id === id)),
|
||||
filter((track) => track !== undefined),
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,51 @@
|
||||
/*
|
||||
Copyright 2024 New Vector Ltd.
|
||||
|
||||
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
|
||||
Please see LICENSE in the repository root for full details.
|
||||
*/
|
||||
|
||||
import { KeyProviderEvent } from "livekit-client";
|
||||
import { Subject } from "rxjs";
|
||||
import { describe, expect, test, vi } from "vitest";
|
||||
|
||||
import { type MediaKey } from "../api";
|
||||
import { MatrixKeyProvider } from "./MatrixKeyProvider";
|
||||
|
||||
describe("matrixKeyProvider", () => {
|
||||
test("initializes", () => {
|
||||
const keyProvider = new MatrixKeyProvider();
|
||||
expect(keyProvider).toBeTruthy();
|
||||
});
|
||||
|
||||
test("follows the key stream it is given", async () => {
|
||||
// jsdom has no HKDF; the derived material is not what is under test
|
||||
vi.spyOn(crypto.subtle, "importKey").mockResolvedValue({} as CryptoKey);
|
||||
const keyProvider = new MatrixKeyProvider();
|
||||
const setKey = vi.fn();
|
||||
keyProvider.on(KeyProviderEvent.SetKey, setKey);
|
||||
const keys$ = new Subject<MediaKey>();
|
||||
|
||||
keyProvider.setMediaKeys(keys$);
|
||||
keys$.next({ participantId: "alice", index: 2, key: new Uint8Array(16) });
|
||||
await vi.waitFor(() =>
|
||||
expect(setKey.mock.lastCall?.[0]).toMatchObject({
|
||||
participantIdentity: "alice",
|
||||
keyIndex: 2,
|
||||
}),
|
||||
);
|
||||
});
|
||||
|
||||
test("stops following a stream when given another", () => {
|
||||
const keyProvider = new MatrixKeyProvider();
|
||||
const first$ = new Subject<MediaKey>();
|
||||
const second$ = new Subject<MediaKey>();
|
||||
|
||||
keyProvider.setMediaKeys(first$);
|
||||
expect(first$.observed).toBe(true);
|
||||
|
||||
keyProvider.setMediaKeys(second$);
|
||||
expect(first$.observed).toBe(false);
|
||||
expect(second$.observed).toBe(true);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,46 @@
|
||||
/*
|
||||
Copyright 2023, 2024 New Vector Ltd.
|
||||
|
||||
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
|
||||
Please see LICENSE in the repository root for full details.
|
||||
*/
|
||||
|
||||
import { BaseKeyProvider } from "livekit-client";
|
||||
import { logger as rootLogger, type Logger } from "matrix-js-sdk/lib/logger";
|
||||
import { type Observable, type Subscription } from "rxjs";
|
||||
|
||||
import { type MediaKey } from "../api";
|
||||
|
||||
/** LiveKit's key provider, fed from the session's per-participant media keys. */
|
||||
export class MatrixKeyProvider extends BaseKeyProvider {
|
||||
private subscription?: Subscription;
|
||||
private logger: Logger;
|
||||
public constructor() {
|
||||
super({ ratchetWindowSize: 10, keyringSize: 256 });
|
||||
this.logger = rootLogger.getChild("[MatrixKeyProvider]");
|
||||
}
|
||||
|
||||
public setMediaKeys(mediaKeys$: Observable<MediaKey>): void {
|
||||
this.subscription?.unsubscribe();
|
||||
this.subscription = mediaKeys$.subscribe(this.onMediaKey);
|
||||
}
|
||||
|
||||
private onMediaKey = ({ key, index, participantId }: MediaKey): void => {
|
||||
crypto.subtle
|
||||
.importKey("raw", key, "HKDF", false, ["deriveBits", "deriveKey"])
|
||||
.then(
|
||||
(keyMaterial) => {
|
||||
this.onSetEncryptionKey(keyMaterial, participantId, index);
|
||||
this.logger.debug(
|
||||
`Sent new key to livekit participantId=${participantId} encryptionKeyIndex=${index}`,
|
||||
);
|
||||
},
|
||||
(e) => {
|
||||
this.logger.error(
|
||||
`Failed to create key material from buffer for livekit participantId=${participantId} encryptionKeyIndex=${index}`,
|
||||
e,
|
||||
);
|
||||
},
|
||||
);
|
||||
};
|
||||
}
|
||||
@@ -14,13 +14,13 @@ import {
|
||||
} from "livekit-client";
|
||||
import { type Logger } from "matrix-js-sdk/lib/logger";
|
||||
|
||||
import { ObservableScope } from "../reactive/ObservableScope";
|
||||
import { ObservableScope } from "../../reactive/ObservableScope";
|
||||
import {
|
||||
type LocalMediaInputs,
|
||||
type MediaSource,
|
||||
type PublishRequest,
|
||||
} from "../api";
|
||||
import { livekitSources, mediaSources } from "../media/LivekitMediaTrack";
|
||||
} from "../../api";
|
||||
import { livekitSources, mediaSources } from "./LivekitMediaTrack";
|
||||
import {
|
||||
audioCaptureOptions,
|
||||
screenShareCaptureOptions,
|
||||
@@ -0,0 +1,28 @@
|
||||
/*
|
||||
Copyright 2026 Element Creations Ltd.
|
||||
|
||||
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
|
||||
Please see LICENSE in the repository root for full details.
|
||||
*/
|
||||
|
||||
import { MediaConnectionState } from "../api";
|
||||
import { ConnectionState } from "./Connection";
|
||||
|
||||
const states: Record<ConnectionState, MediaConnectionState> = {
|
||||
[ConnectionState.Initialized]: MediaConnectionState.Initialized,
|
||||
[ConnectionState.FetchingConfig]: MediaConnectionState.Preparing,
|
||||
[ConnectionState.Stopped]: MediaConnectionState.Stopped,
|
||||
[ConnectionState.LivekitDisconnected]: MediaConnectionState.Disconnected,
|
||||
[ConnectionState.LivekitConnecting]: MediaConnectionState.Connecting,
|
||||
[ConnectionState.LivekitConnected]: MediaConnectionState.Connected,
|
||||
[ConnectionState.LivekitReconnecting]: MediaConnectionState.Reconnecting,
|
||||
[ConnectionState.LivekitSignalReconnecting]:
|
||||
MediaConnectionState.Reconnecting,
|
||||
};
|
||||
|
||||
/** A connection's state as the session sees it; an error passes through. */
|
||||
export function toMediaConnectionState(
|
||||
state: ConnectionState | Error,
|
||||
): MediaConnectionState | Error {
|
||||
return state instanceof Error ? state : states[state];
|
||||
}
|
||||
@@ -15,7 +15,7 @@ import {
|
||||
VideoPreset as VideoPresetClass,
|
||||
} from "livekit-client";
|
||||
|
||||
import { defaultMediaQuality, type MediaQuality } from "../config";
|
||||
import { defaultMediaQuality, type MediaQuality } from "../../config";
|
||||
|
||||
/**
|
||||
* Find the closest matching VideoPreset for a given height.
|
||||
@@ -18,9 +18,9 @@ import fetchMock from "fetch-mock";
|
||||
import { MatrixError } from "matrix-js-sdk";
|
||||
|
||||
import { getSFUConfigWithOpenID, type OpenIDClientParts } from "./openIDSFU";
|
||||
import { testJWTToken } from "../utils/test-fixtures";
|
||||
import { ownMemberMock } from "../utils/test";
|
||||
import { FailToGetOpenIdToken } from "../errors";
|
||||
import { testJWTToken } from "../../utils/test-fixtures";
|
||||
import { ownMemberMock } from "../../utils/test";
|
||||
import { FailToGetOpenIdToken } from "../../errors";
|
||||
|
||||
const sfuUrl = "https://sfu.example.org";
|
||||
|
||||
@@ -30,7 +30,6 @@ describe("getSFUConfigWithOpenID", () => {
|
||||
fetchMock.catch(404);
|
||||
matrixClient = {
|
||||
getOpenIdToken: vitest.fn(),
|
||||
getDeviceId: vitest.fn(),
|
||||
};
|
||||
});
|
||||
afterEach(() => {
|
||||
@@ -13,9 +13,11 @@ import {
|
||||
import { type CallMembershipIdentityParts } from "matrix-js-sdk/lib/matrixrtc/EncryptionManager";
|
||||
import { type Logger } from "matrix-js-sdk/lib/logger";
|
||||
|
||||
import { FailToGetOpenIdToken, NoMatrix2AuthorizationService } from "../errors";
|
||||
import { doNetworkOperationWithRetry } from "../utils/network";
|
||||
import { MatrixRTCMode } from "../config";
|
||||
import {
|
||||
FailToGetOpenIdToken,
|
||||
NoMatrix2AuthorizationService,
|
||||
} from "../../errors";
|
||||
import { doNetworkOperationWithRetry } from "../../utils/network";
|
||||
|
||||
/**
|
||||
* Configuration and access tokens provided by the SFU on successful authentication.
|
||||
@@ -62,10 +64,14 @@ interface SFUJWTPayload {
|
||||
}
|
||||
|
||||
// The bits we need from MatrixClient
|
||||
export type OpenIDClientParts = Pick<
|
||||
MatrixClient,
|
||||
"getOpenIdToken" | "getDeviceId"
|
||||
>;
|
||||
export type OpenIDClientParts = Pick<MatrixClient, "getOpenIdToken">;
|
||||
|
||||
/**
|
||||
* Which token service endpoint to use: the MSC4195 one, or the legacy one
|
||||
* with the old identity format. Undefined tries the MSC4195 one and falls
|
||||
* back to the legacy one, for transports we only subscribe on.
|
||||
*/
|
||||
export type TokenEndpoint = "msc4195" | "legacy";
|
||||
|
||||
/**
|
||||
* Gets a bearer token from the homeserver and then use it to authenticate
|
||||
@@ -76,8 +82,8 @@ export type OpenIDClientParts = Pick<
|
||||
* @param serviceUrl The URL of the livekit SFU service
|
||||
* @param roomId The room id used in the jwt request. This is NOT the livekit_alias. The jwt service will provide the alias. It maps matrix room ids <-> Livekit aliases.
|
||||
* @param opts Additional options to modify which endpoint with which data will be used to acquire the jwt token.
|
||||
* @param opts.matrixRTCMode Determines which version of the JWT endpoint to use, which affects whether the
|
||||
* RTC backend identity is based on string concatenation (legacy) or a hash (Matrix 2.0).
|
||||
* @param opts.tokenEndpoint Which version of the JWT endpoint to use, which affects whether the
|
||||
* RTC backend identity is based on string concatenation (legacy) or a hash (MSC4195).
|
||||
* This function by default uses whatever is possible with the current jwt service installed next to the SFU.
|
||||
* For remote connections this does not matter, since we will not publish there we can rely on the newest option.
|
||||
* @param opts.delayEndpointBaseUrl The URL of the matrix homeserver.
|
||||
@@ -92,7 +98,7 @@ export async function getSFUConfigWithOpenID(
|
||||
serviceUrl: string,
|
||||
roomId: string,
|
||||
opts?: {
|
||||
matrixRTCMode?: MatrixRTCMode;
|
||||
tokenEndpoint?: TokenEndpoint;
|
||||
delayEndpointBaseUrl?: string;
|
||||
delayId?: string;
|
||||
/** How long the SFU waits before sending the delegated leave. Needed with `delayId`. */
|
||||
@@ -113,9 +119,9 @@ export async function getSFUConfigWithOpenID(
|
||||
logger?.debug("Got openID token", openIdToken);
|
||||
let sfuConfig: { url: string; jwt: string } | undefined;
|
||||
|
||||
const tryBothJwtEndpoints = opts?.matrixRTCMode === undefined; // This is for SFUs where we do not publish.
|
||||
const tryBothJwtEndpoints = opts?.tokenEndpoint === undefined; // This is for SFUs where we do not publish.
|
||||
|
||||
const forceMatrix2Jwt = opts?.matrixRTCMode === MatrixRTCMode.Matrix_2_0;
|
||||
const forceMatrix2Jwt = opts?.tokenEndpoint === "msc4195";
|
||||
|
||||
// We want to start using the new endpoint (with optional delay delegation)
|
||||
// if we can use both or if we are forced to use the new one.
|
||||
@@ -12,7 +12,7 @@ import {
|
||||
type VideoCaptureOptions,
|
||||
} from "livekit-client";
|
||||
|
||||
import { type PublishRequest, type VideoCaptureSettings } from "../api";
|
||||
import { type PublishRequest, type VideoCaptureSettings } from "../../api";
|
||||
|
||||
export type PublishRequestFor<S extends PublishRequest["source"]> = Extract<
|
||||
PublishRequest,
|
||||
@@ -0,0 +1,36 @@
|
||||
/*
|
||||
Copyright 2026 Element Creations Ltd.
|
||||
|
||||
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
|
||||
Please see LICENSE in the repository root for full details.
|
||||
*/
|
||||
|
||||
/** LiveKit test doubles shared by the backend's unit tests. Not part of the package. */
|
||||
|
||||
import {
|
||||
type RemoteParticipant,
|
||||
type RemoteTrackPublication,
|
||||
} from "livekit-client";
|
||||
import { type LivekitTransport } from "matrix-js-sdk/lib/matrixrtc";
|
||||
|
||||
import { mockEmitter } from "../../utils/test";
|
||||
|
||||
export const exampleTransport: LivekitTransport = {
|
||||
type: "livekit",
|
||||
livekit_service_url: "https://lk.example.org",
|
||||
};
|
||||
|
||||
export function mockRemoteParticipant(
|
||||
participant: Partial<RemoteParticipant>,
|
||||
): RemoteParticipant {
|
||||
return {
|
||||
isLocal: false,
|
||||
setVolume() {},
|
||||
getTrackPublication: () =>
|
||||
({}) as Partial<RemoteTrackPublication> as RemoteTrackPublication,
|
||||
// this will only get used for `getTrackPublications().length`
|
||||
getTrackPublications: () => [0],
|
||||
...mockEmitter(),
|
||||
...participant,
|
||||
} as RemoteParticipant;
|
||||
}
|
||||
@@ -0,0 +1,212 @@
|
||||
/*
|
||||
Copyright 2026 Element Creations Ltd.
|
||||
|
||||
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
|
||||
Please see LICENSE in the repository root for full details.
|
||||
*/
|
||||
|
||||
import { logger } from "matrix-js-sdk/lib/logger";
|
||||
import { Status, type Transport } from "matrix-js-sdk/lib/matrixrtc";
|
||||
import { BehaviorSubject, type Observable, Subject, throwError } from "rxjs";
|
||||
import { describe, expect, it, vi } from "vitest";
|
||||
|
||||
import { defaultSessionTimings } from "../config";
|
||||
import { MatrixRTCError } from "../errors";
|
||||
import { type LocalMediaBackend, MediaConnectionState } from "../backend/api";
|
||||
import { type HomeserverConnected } from "./HomeserverConnected";
|
||||
import {
|
||||
createLocalMembership$,
|
||||
type LocalMembership,
|
||||
type PreparedTransport,
|
||||
PublishState,
|
||||
TransportState,
|
||||
} from "./LocalMember";
|
||||
import { flushPromises, testScope } from "../utils/test";
|
||||
|
||||
const transport: Transport = { type: "livekit", livekit_service_url: "u" };
|
||||
|
||||
describe("createLocalMembership$", () => {
|
||||
it("waits for the transport, then reports the media connection and the join", async () => {
|
||||
const { membership, local, prepared$ } = setup();
|
||||
expect(membership.state$.value).toBe(TransportState.Waiting);
|
||||
|
||||
prepared$.next({ transport, canDelegateDelayedLeave: false });
|
||||
expect(membership.state$.value).toEqual({
|
||||
media: { connection: MediaConnectionState.Initialized },
|
||||
matrix: Status.Connected,
|
||||
});
|
||||
|
||||
local.connectionState$.next(MediaConnectionState.Connected);
|
||||
expect(membership.state$.value).toEqual({
|
||||
media: PublishState.WaitingForUser,
|
||||
matrix: Status.Connected,
|
||||
});
|
||||
|
||||
membership.requestJoinAndPublish();
|
||||
await flushPromises();
|
||||
expect(membership.state$.value).toEqual({
|
||||
media: PublishState.Publishing,
|
||||
matrix: Status.Connected,
|
||||
});
|
||||
expect(membership.connected$.value).toBe(true);
|
||||
});
|
||||
|
||||
it("publishes only while joined and the homeserver is reachable", async () => {
|
||||
const { membership, local, homeserver } = setup();
|
||||
expect(local.setPublishing).toHaveBeenLastCalledWith(false);
|
||||
|
||||
membership.requestJoinAndPublish();
|
||||
expect(local.setPublishing).toHaveBeenLastCalledWith(true);
|
||||
|
||||
homeserver.combined$.next([false, "sync"]);
|
||||
expect(local.setPublishing).toHaveBeenLastCalledWith(false);
|
||||
homeserver.combined$.next([true, null]);
|
||||
expect(local.setPublishing).toHaveBeenLastCalledWith(true);
|
||||
|
||||
membership.requestDisconnect();
|
||||
expect(local.setPublishing).toHaveBeenLastCalledWith(false);
|
||||
// A sync outage after leaving must not bring the media back
|
||||
homeserver.combined$.next([false, "sync"]);
|
||||
homeserver.combined$.next([true, null]);
|
||||
expect(local.setPublishing).toHaveBeenLastCalledWith(false);
|
||||
await flushPromises();
|
||||
});
|
||||
|
||||
it("joins with the delegated timings and hands over each delay id where the backend can", async () => {
|
||||
const { membership, prepared$, delayId$, joinMatrixRTC, delegate } =
|
||||
setup();
|
||||
prepared$.next({ transport, canDelegateDelayedLeave: true });
|
||||
membership.requestJoinAndPublish();
|
||||
await flushPromises();
|
||||
expect(joinMatrixRTC).toHaveBeenCalledWith(
|
||||
transport,
|
||||
defaultSessionTimings.delegatedDelayedLeave,
|
||||
);
|
||||
|
||||
delayId$.next("delay-1");
|
||||
await flushPromises();
|
||||
expect(delegate).toHaveBeenCalledWith("delay-1");
|
||||
delayId$.next("delay-2");
|
||||
await flushPromises();
|
||||
expect(delegate).toHaveBeenLastCalledWith("delay-2");
|
||||
});
|
||||
|
||||
it("never hands the leave over where the backend cannot take it", async () => {
|
||||
const { membership, prepared$, delayId$, joinMatrixRTC, delegate } =
|
||||
setup();
|
||||
prepared$.next({ transport, canDelegateDelayedLeave: false });
|
||||
membership.requestJoinAndPublish();
|
||||
delayId$.next("delay-1");
|
||||
await flushPromises();
|
||||
expect(joinMatrixRTC).toHaveBeenCalledWith(
|
||||
transport,
|
||||
defaultSessionTimings.delayedLeave,
|
||||
);
|
||||
expect(delegate).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("reports a transport that could not be prepared as fatal", () => {
|
||||
const error = new MatrixRTCError("no token");
|
||||
const { membership } = setup(throwError(() => error));
|
||||
expect(membership.state$.value).toBe(error);
|
||||
});
|
||||
|
||||
it("reports a connection loss and a publish failure in the media state", () => {
|
||||
const { membership, local, prepared$ } = setup();
|
||||
prepared$.next({ transport, canDelegateDelayedLeave: false });
|
||||
local.connectionState$.next(MediaConnectionState.Connected);
|
||||
membership.requestJoinAndPublish();
|
||||
|
||||
local.connectionState$.next(MediaConnectionState.Reconnecting);
|
||||
expect(membership.state$.value).toEqual({
|
||||
media: { connection: MediaConnectionState.Reconnecting },
|
||||
matrix: Status.Connected,
|
||||
});
|
||||
expect(membership.disconnectReason$.value).toBe("media");
|
||||
expect(membership.reconnecting$.value).toBe(true);
|
||||
|
||||
const failure = new MatrixRTCError("no camera");
|
||||
local.publishError$.next(failure);
|
||||
const state = membership.state$.value;
|
||||
expect(typeof state === "object" && "media" in state && state.media).toBe(
|
||||
failure,
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
function setup(preparedTransportWithErrors$?: Observable<PreparedTransport>): {
|
||||
membership: LocalMembership;
|
||||
local: FakeLocalMediaBackend;
|
||||
homeserver: FakeHomeserverConnected;
|
||||
prepared$: Subject<PreparedTransport>;
|
||||
delayId$: BehaviorSubject<string | null>;
|
||||
joinMatrixRTC: ReturnType<typeof vi.fn>;
|
||||
delegate: ReturnType<typeof vi.fn>;
|
||||
} {
|
||||
const prepared$ = new Subject<PreparedTransport>();
|
||||
const local = fakeLocalMediaBackend();
|
||||
const homeserver = fakeHomeserverConnected();
|
||||
const delayId$ = new BehaviorSubject<string | null>(null);
|
||||
const joinMatrixRTC = vi.fn();
|
||||
const delegate = vi.fn(async () => Promise.resolve());
|
||||
const membership = createLocalMembership$({
|
||||
scope: testScope(),
|
||||
local,
|
||||
delegateDelayedLeave: delegate,
|
||||
preparedTransport$: preparedTransportWithErrors$ ?? prepared$,
|
||||
homeserverConnected: homeserver,
|
||||
joinMatrixRTC,
|
||||
membershipManagerError$: new Subject(),
|
||||
matrixRTCSession: {
|
||||
leaveRoomSession: vi.fn(async () => Promise.resolve(true)),
|
||||
},
|
||||
delayId$,
|
||||
timings: defaultSessionTimings,
|
||||
logger,
|
||||
});
|
||||
return {
|
||||
membership,
|
||||
local,
|
||||
homeserver,
|
||||
prepared$,
|
||||
delayId$,
|
||||
joinMatrixRTC,
|
||||
delegate,
|
||||
};
|
||||
}
|
||||
|
||||
type FakeLocalMediaBackend = Omit<
|
||||
LocalMediaBackend,
|
||||
"connectionState$" | "publishError$" | "setPublishing"
|
||||
> & {
|
||||
connectionState$: BehaviorSubject<MediaConnectionState | Error>;
|
||||
publishError$: BehaviorSubject<Error | null>;
|
||||
setPublishing: ReturnType<typeof vi.fn<LocalMediaBackend["setPublishing"]>>;
|
||||
};
|
||||
|
||||
function fakeLocalMediaBackend(): FakeLocalMediaBackend {
|
||||
return {
|
||||
connectionState$: new BehaviorSubject<MediaConnectionState | Error>(
|
||||
MediaConnectionState.Initialized,
|
||||
),
|
||||
media$: new BehaviorSubject(null),
|
||||
setPublishing: vi.fn<LocalMediaBackend["setPublishing"]>(),
|
||||
publishError$: new BehaviorSubject<Error | null>(null),
|
||||
publish: vi.fn<LocalMediaBackend["publish"]>(),
|
||||
unpublish: vi.fn<LocalMediaBackend["unpublish"]>(),
|
||||
};
|
||||
}
|
||||
|
||||
type FakeHomeserverConnected = Omit<HomeserverConnected, "combined$"> & {
|
||||
combined$: BehaviorSubject<HomeserverConnected["combined$"]["value"]>;
|
||||
};
|
||||
|
||||
function fakeHomeserverConnected(): FakeHomeserverConnected {
|
||||
return {
|
||||
combined$: new BehaviorSubject<HomeserverConnected["combined$"]["value"]>([
|
||||
true,
|
||||
null,
|
||||
]),
|
||||
rtsSession$: new BehaviorSubject(Status.Connected),
|
||||
};
|
||||
}
|
||||
+54
-274
@@ -5,20 +5,12 @@ SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
|
||||
Please see LICENSE in the repository root for full details.
|
||||
*/
|
||||
|
||||
import { observeParticipantEvents } from "@livekit/components-core";
|
||||
import {
|
||||
type LocalParticipant,
|
||||
type LocalTrackPublication,
|
||||
ParticipantEvent,
|
||||
} from "livekit-client";
|
||||
import { type MatrixClient } from "matrix-js-sdk";
|
||||
import { type Logger } from "matrix-js-sdk/lib/logger";
|
||||
import {
|
||||
type LivekitTransport,
|
||||
type MatrixRTCSession as JsSdkRTCSession,
|
||||
type Status as RTCSessionStatus,
|
||||
type Transport,
|
||||
} from "matrix-js-sdk/lib/matrixrtc";
|
||||
import { type CallMembershipIdentityParts } from "matrix-js-sdk/lib/matrixrtc/EncryptionManager";
|
||||
import { deepCompare } from "matrix-js-sdk/lib/utils";
|
||||
import {
|
||||
BehaviorSubject,
|
||||
@@ -26,8 +18,6 @@ import {
|
||||
combineLatest,
|
||||
concat,
|
||||
distinctUntilChanged,
|
||||
filter,
|
||||
firstValueFrom,
|
||||
map,
|
||||
NEVER,
|
||||
type Observable,
|
||||
@@ -40,29 +30,20 @@ import {
|
||||
|
||||
import { type Behavior } from "../reactive/Behavior";
|
||||
import { type ObservableScope } from "../reactive/ObservableScope";
|
||||
import { type DelayedLeaveTimings, type SessionTimings } from "../config";
|
||||
import { type DisconnectReason } from "../api";
|
||||
import {
|
||||
type DelayedLeaveTimings,
|
||||
type MatrixRTCMode,
|
||||
type SessionTimings,
|
||||
} from "../config";
|
||||
import {
|
||||
type DisconnectReason,
|
||||
type MediaSource,
|
||||
type PublishRequest,
|
||||
} from "../api";
|
||||
import {
|
||||
FailToStartLivekitConnection,
|
||||
type MatrixRTCError,
|
||||
MembershipManagerError,
|
||||
toMatrixRTCError,
|
||||
} from "../errors";
|
||||
import {
|
||||
type LocalMediaBackend,
|
||||
type MediaBackend,
|
||||
MediaConnectionState,
|
||||
type TransportCapabilities,
|
||||
} from "../backend/api";
|
||||
import { type HomeserverConnected } from "./HomeserverConnected";
|
||||
import { type Connection, ConnectionState } from "./Connection";
|
||||
import { type IConnectionManager } from "./ConnectionManager";
|
||||
import { type LocalTransport } from "./LocalTransport";
|
||||
import { getSFUConfigWithOpenID } from "./openIDSFU";
|
||||
import { livekitSources } from "../media/LivekitMediaTrack";
|
||||
import { type DesiredMedia, type Publisher } from "./Publisher";
|
||||
|
||||
export enum TransportState {
|
||||
Waiting = "transport_waiting",
|
||||
@@ -74,7 +55,7 @@ export enum PublishState {
|
||||
}
|
||||
|
||||
export type LocalMemberMediaState =
|
||||
| { connection: ConnectionState | MatrixRTCError }
|
||||
| { connection: MediaConnectionState | MatrixRTCError }
|
||||
| PublishState
|
||||
| MatrixRTCError;
|
||||
|
||||
@@ -83,27 +64,27 @@ export type LocalMemberState =
|
||||
| TransportState.Waiting
|
||||
| { media: LocalMemberMediaState; matrix: MatrixRTCError | RTCSessionStatus };
|
||||
|
||||
/** The local transport, found and prepared, with what the join needs to know about it. */
|
||||
export interface PreparedTransport extends TransportCapabilities {
|
||||
transport: Transport;
|
||||
}
|
||||
|
||||
interface Props {
|
||||
scope: ObservableScope;
|
||||
connectionManager: IConnectionManager;
|
||||
localTransport$: Observable<LocalTransport>;
|
||||
local: LocalMediaBackend;
|
||||
delegateDelayedLeave: MediaBackend["delegateDelayedLeave"];
|
||||
preparedTransport$: Observable<PreparedTransport>;
|
||||
homeserverConnected: HomeserverConnected;
|
||||
createPublisher: (connection: Connection) => Publisher;
|
||||
joinMatrixRTC: (
|
||||
transport: LivekitTransport,
|
||||
transport: Transport,
|
||||
delayedLeave: DelayedLeaveTimings,
|
||||
) => void;
|
||||
/** The membership manager giving up on keeping the membership alive. */
|
||||
membershipManagerError$: Observable<unknown>;
|
||||
matrixRTCSession: Pick<JsSdkRTCSession, "leaveRoomSession">;
|
||||
/** The id of the delayed leave event, for the SFU to take over. */
|
||||
/** The id of the delayed leave event, for the backend to take over. */
|
||||
delayId$: Behavior<string | null>;
|
||||
client: Pick<MatrixClient, "getOpenIdToken" | "getDeviceId" | "baseUrl">;
|
||||
roomId: string;
|
||||
ownMembershipIdentity: CallMembershipIdentityParts;
|
||||
matrixRTCMode: MatrixRTCMode;
|
||||
timings: SessionTimings;
|
||||
desired: DesiredMedia;
|
||||
logger: Logger;
|
||||
}
|
||||
|
||||
@@ -112,133 +93,64 @@ export interface LocalMembership {
|
||||
requestDisconnect: () => void;
|
||||
joinRequested$: Behavior<boolean>;
|
||||
state$: Behavior<LocalMemberState>;
|
||||
participant$: Behavior<LocalParticipant | null>;
|
||||
connection$: Behavior<Connection | null>;
|
||||
/** Fully connected: to the homeserver, the session and the transport. */
|
||||
connected$: Behavior<boolean>;
|
||||
/** Connected once, and currently not. */
|
||||
reconnecting$: Behavior<boolean>;
|
||||
disconnectReason$: Behavior<DisconnectReason | null>;
|
||||
/** Resolves with the publication, once a publisher has made it. */
|
||||
publish: (request: PublishRequest) => Promise<LocalTrackPublication>;
|
||||
unpublish: (source: MediaSource) => Promise<void>;
|
||||
/** Mutes or unmutes one of our publications; the result is the state that resulted. */
|
||||
setEnabled: (source: MediaSource, enabled: boolean) => Promise<boolean>;
|
||||
}
|
||||
|
||||
/**
|
||||
* The local member's state machine: waits for the transport, publishes on
|
||||
* its connection once asked to join, and enters and leaves the MatrixRTC
|
||||
* session in step.
|
||||
* The local member's state machine: waits for the transport, publishes once
|
||||
* asked to join, and enters and leaves the MatrixRTC session in step.
|
||||
*/
|
||||
export function createLocalMembership$({
|
||||
scope,
|
||||
connectionManager,
|
||||
localTransport$: localTransportWithErrors$,
|
||||
local,
|
||||
delegateDelayedLeave,
|
||||
preparedTransport$: preparedTransportWithErrors$,
|
||||
homeserverConnected,
|
||||
createPublisher,
|
||||
joinMatrixRTC,
|
||||
membershipManagerError$,
|
||||
matrixRTCSession,
|
||||
delayId$,
|
||||
client,
|
||||
roomId,
|
||||
ownMembershipIdentity,
|
||||
matrixRTCMode,
|
||||
timings,
|
||||
desired,
|
||||
logger: parentLogger,
|
||||
}: Props): LocalMembership {
|
||||
const logger = parentLogger.getChild("[LocalMember]");
|
||||
|
||||
const fatalTransportError$ = new Subject<MatrixRTCError>();
|
||||
const localTransport$ = localTransportWithErrors$.pipe(
|
||||
const preparedTransport$ = preparedTransportWithErrors$.pipe(
|
||||
catchError((e: unknown) => {
|
||||
fatalTransportError$.next(toMatrixRTCError(e));
|
||||
return NEVER;
|
||||
}),
|
||||
);
|
||||
|
||||
// Whether the SFU can take over restarting the delayed leave, so that a
|
||||
// client that vanishes is removed by the SFU rather than by the timeout.
|
||||
// Either the homeserver or the transport has to support it.
|
||||
const homeserverSupportsDelegation = checkDelegationSupport(
|
||||
`${client.baseUrl}/_matrix/client/unstable/io.element.msc4195/rtc/livekit/delegate_delayed_leave`,
|
||||
"homeserver",
|
||||
logger,
|
||||
);
|
||||
const joinParams$ = scope.behavior(
|
||||
localTransport$.pipe(
|
||||
switchMap(async ({ transport }) => ({
|
||||
transport,
|
||||
delegationSupported:
|
||||
(await homeserverSupportsDelegation) ||
|
||||
(await checkDelegationSupport(
|
||||
`${transport.livekit_service_url}/delegate_delayed_leave`,
|
||||
`transport ${transport.livekit_service_url}`,
|
||||
logger,
|
||||
)),
|
||||
})),
|
||||
),
|
||||
null,
|
||||
);
|
||||
|
||||
const connection$ = scope.behavior(
|
||||
combineLatest([
|
||||
connectionManager.connectionManagerData$,
|
||||
localTransport$,
|
||||
]).pipe(
|
||||
map(([{ value: connections }, { transport }]) =>
|
||||
connections.getConnectionForTransport(transport),
|
||||
),
|
||||
),
|
||||
const transport$ = scope.behavior<PreparedTransport | null>(
|
||||
preparedTransport$,
|
||||
null,
|
||||
);
|
||||
|
||||
const joinRequested$ = new BehaviorSubject(false);
|
||||
const publisher$ = new BehaviorSubject<Publisher | null>(null);
|
||||
const publishError$ = new BehaviorSubject<MatrixRTCError | null>(null);
|
||||
const matrixError$ = new BehaviorSubject<MatrixRTCError | null>(null);
|
||||
|
||||
scope.reconcile(connection$, async (connection) => {
|
||||
if (connection === null) return;
|
||||
const publisher = createPublisher(connection);
|
||||
publisher$.next(publisher);
|
||||
return Promise.resolve(async (): Promise<void> => {
|
||||
publisher$.next(null);
|
||||
await publisher.destroy();
|
||||
});
|
||||
});
|
||||
// Nothing leaves this device while it may already have been dropped from
|
||||
// the session: the member would show as away while still being heard
|
||||
combineLatest(
|
||||
[joinRequested$, homeserverConnected.combined$],
|
||||
(join, [connected]) => join && connected,
|
||||
)
|
||||
.pipe(distinctUntilChanged(), scope.bind())
|
||||
.subscribe((publish) => local.setPublishing(publish));
|
||||
|
||||
scope.reconcile(
|
||||
scope.behavior(combineLatest([publisher$, joinRequested$])),
|
||||
async ([publisher, shouldPublish]) => {
|
||||
if (publisher === null) return;
|
||||
try {
|
||||
if (shouldPublish) {
|
||||
publisher.start();
|
||||
await publisher.startPublishing();
|
||||
} else if (publisher.shouldPublish) await publisher.stopPublishing();
|
||||
} catch (e) {
|
||||
if (publishError$.value === null)
|
||||
publishError$.next(
|
||||
new FailToStartLivekitConnection(
|
||||
e instanceof Error ? e.message : String(e),
|
||||
),
|
||||
);
|
||||
else logger.error("Another publish error", e);
|
||||
}
|
||||
},
|
||||
);
|
||||
|
||||
scope.reconcile(
|
||||
scope.behavior(combineLatest([joinParams$, joinRequested$])),
|
||||
async ([joinParams, shouldJoin]) => {
|
||||
if (joinParams === null || !shouldJoin) return;
|
||||
scope.behavior(combineLatest([transport$, joinRequested$])),
|
||||
async ([prepared, shouldJoin]) => {
|
||||
if (prepared === null || !shouldJoin) return;
|
||||
try {
|
||||
joinMatrixRTC(
|
||||
joinParams.transport,
|
||||
joinParams.delegationSupported
|
||||
prepared.transport,
|
||||
prepared.canDelegateDelayedLeave
|
||||
? timings.delegatedDelayedLeave
|
||||
: timings.delayedLeave,
|
||||
);
|
||||
@@ -256,31 +168,14 @@ export function createLocalMembership$({
|
||||
},
|
||||
);
|
||||
|
||||
// Hand the delayed leave to the SFU. The token this issues is discarded;
|
||||
// the request is what triggers the delegation.
|
||||
scope.reconcile(
|
||||
scope.behavior(combineLatest([joinParams$, delayId$])),
|
||||
async ([joinParams, delayId]) => {
|
||||
if (!joinParams?.delegationSupported || delayId === null) return;
|
||||
scope.behavior(combineLatest([transport$, delayId$])),
|
||||
async ([prepared, delayId]) => {
|
||||
if (!prepared?.canDelegateDelayedLeave || delayId === null) return;
|
||||
try {
|
||||
await getSFUConfigWithOpenID(
|
||||
client,
|
||||
ownMembershipIdentity,
|
||||
joinParams.transport.livekit_service_url,
|
||||
roomId,
|
||||
{
|
||||
matrixRTCMode,
|
||||
delayEndpointBaseUrl: client.baseUrl,
|
||||
delayId,
|
||||
delayTimeoutMs: timings.delegatedDelayedLeave.delay_ms,
|
||||
},
|
||||
logger,
|
||||
);
|
||||
await delegateDelayedLeave(delayId);
|
||||
} catch (e) {
|
||||
logger.error(
|
||||
`Failed to delegate the leave to ${joinParams.transport.livekit_service_url}`,
|
||||
e,
|
||||
);
|
||||
logger.error("Failed to delegate the leave", e);
|
||||
}
|
||||
},
|
||||
);
|
||||
@@ -295,19 +190,15 @@ export function createLocalMembership$({
|
||||
);
|
||||
});
|
||||
|
||||
const connectionState$ = connection$.pipe(
|
||||
switchMap((connection) => connection?.state$ ?? of(null)),
|
||||
);
|
||||
|
||||
const mediaState$ = scope.behavior<LocalMemberMediaState>(
|
||||
combineLatest([connectionState$, joinRequested$]).pipe(
|
||||
combineLatest([local.connectionState$, joinRequested$]).pipe(
|
||||
map(([connectionState, shouldPublish]) => {
|
||||
if (connectionState !== ConnectionState.LivekitConnected)
|
||||
if (connectionState !== MediaConnectionState.Connected)
|
||||
return {
|
||||
connection:
|
||||
connectionState instanceof Error
|
||||
? toMatrixRTCError(connectionState)
|
||||
: (connectionState ?? ConnectionState.Initialized),
|
||||
: connectionState,
|
||||
};
|
||||
return shouldPublish
|
||||
? PublishState.Publishing
|
||||
@@ -322,18 +213,18 @@ export function createLocalMembership$({
|
||||
of(TransportState.Waiting),
|
||||
race(
|
||||
fatalTransportError$,
|
||||
localTransport$.pipe(
|
||||
preparedTransport$.pipe(
|
||||
switchMap(() =>
|
||||
combineLatest(
|
||||
[
|
||||
mediaState$,
|
||||
homeserverConnected.rtsSession$,
|
||||
matrixError$,
|
||||
publishError$,
|
||||
local.publishError$,
|
||||
],
|
||||
(media, sessionStatus, matrixError, publishError) => ({
|
||||
matrix: matrixError ?? sessionStatus,
|
||||
media: publishError ?? media,
|
||||
media: publishError ? toMatrixRTCError(publishError) : media,
|
||||
}),
|
||||
),
|
||||
),
|
||||
@@ -344,14 +235,13 @@ export function createLocalMembership$({
|
||||
|
||||
const disconnectReason$ = scope.behavior(
|
||||
combineLatest(
|
||||
[homeserverConnected.combined$, connectionState$],
|
||||
[homeserverConnected.combined$, local.connectionState$],
|
||||
(
|
||||
[homeserverConnected, reason],
|
||||
connectionState,
|
||||
): DisconnectReason | null => {
|
||||
if (!homeserverConnected) return reason ?? "sync";
|
||||
if (connectionState !== ConnectionState.LivekitConnected)
|
||||
return "media";
|
||||
if (connectionState !== MediaConnectionState.Connected) return "media";
|
||||
return null;
|
||||
},
|
||||
),
|
||||
@@ -369,123 +259,13 @@ export function createLocalMembership$({
|
||||
false,
|
||||
);
|
||||
|
||||
const participant$ = scope.behavior(
|
||||
connection$.pipe(map((c) => c?.livekitRoom.localParticipant ?? null)),
|
||||
);
|
||||
|
||||
// Nothing leaves this device while it may already have been dropped from
|
||||
// the session: the member would show as away while still being heard
|
||||
combineLatest([participant$, homeserverConnected.combined$])
|
||||
.pipe(scope.bind())
|
||||
.subscribe(([participant, [connected]]) => {
|
||||
if (participant === null) return;
|
||||
for (const { track } of participant.trackPublications.values()) {
|
||||
if (!track) continue;
|
||||
if (connected && track.isUpstreamPaused)
|
||||
track.resumeUpstream().catch((e) => {
|
||||
logger.error(`Failed to resume the ${track.kind} track`, e);
|
||||
});
|
||||
else if (!connected && !track.isUpstreamPaused)
|
||||
track.pauseUpstream().catch((e) => {
|
||||
logger.error(`Failed to pause the ${track.kind} track`, e);
|
||||
});
|
||||
}
|
||||
});
|
||||
|
||||
const publish = async (
|
||||
request: PublishRequest,
|
||||
): Promise<LocalTrackPublication> => {
|
||||
desired.set(request.source, { request, enabled: true });
|
||||
const publisher = publisher$.value;
|
||||
try {
|
||||
// Before the start, the publisher publishes everything desired itself
|
||||
if (publisher?.started) {
|
||||
const publication = await publisher.publish(request);
|
||||
if (publication) return publication;
|
||||
}
|
||||
return await firstValueFrom(publication$(request.source));
|
||||
} catch (e) {
|
||||
desired.delete(request.source);
|
||||
throw toMatrixRTCError(e);
|
||||
}
|
||||
};
|
||||
|
||||
const publication$ = (
|
||||
source: MediaSource,
|
||||
): Observable<LocalTrackPublication> =>
|
||||
participant$.pipe(
|
||||
switchMap((participant) =>
|
||||
participant === null
|
||||
? NEVER
|
||||
: observeParticipantEvents(
|
||||
participant,
|
||||
ParticipantEvent.LocalTrackPublished,
|
||||
).pipe(
|
||||
map(() =>
|
||||
participant.getTrackPublication(livekitSources[source]),
|
||||
),
|
||||
),
|
||||
),
|
||||
filter((publication) => publication !== undefined),
|
||||
);
|
||||
|
||||
const unpublish = async (source: MediaSource): Promise<void> => {
|
||||
desired.delete(source);
|
||||
await publisher$.value?.unpublish(source);
|
||||
};
|
||||
|
||||
const setEnabled = async (
|
||||
source: MediaSource,
|
||||
enabled: boolean,
|
||||
): Promise<boolean> => {
|
||||
const wanted = desired.get(source);
|
||||
if (wanted) wanted.enabled = enabled;
|
||||
// Without a publisher the request waits for the tracks to be created
|
||||
const publisher = publisher$.value;
|
||||
if (publisher === null) return enabled;
|
||||
const result = await publisher.setEnabled(source, enabled);
|
||||
if (wanted && result !== enabled) wanted.enabled = result;
|
||||
return result;
|
||||
};
|
||||
|
||||
return {
|
||||
requestJoinAndPublish: () => joinRequested$.next(true),
|
||||
requestDisconnect: () => joinRequested$.next(false),
|
||||
joinRequested$,
|
||||
state$,
|
||||
participant$,
|
||||
connection$,
|
||||
connected$,
|
||||
reconnecting$,
|
||||
disconnectReason$,
|
||||
publish,
|
||||
unpublish,
|
||||
setEnabled,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Whether an endpoint exists, by hitting it without credentials and reading
|
||||
* the status. Not retried: many servers predate the endpoint altogether and
|
||||
* answer with a CORS failure that a retry loop would only repeat.
|
||||
*/
|
||||
async function checkDelegationSupport(
|
||||
endpointUrl: string,
|
||||
serviceName: string,
|
||||
logger: Logger,
|
||||
): Promise<boolean> {
|
||||
try {
|
||||
const res = await fetch(endpointUrl, { method: "POST" });
|
||||
const supported = res.status !== 404;
|
||||
logger.info(
|
||||
`${serviceName} ${supported ? "supports" : "does not support"} delegation`,
|
||||
);
|
||||
return supported;
|
||||
} catch (e) {
|
||||
logger.warn(
|
||||
`Failed to determine whether ${serviceName} supports delegation, assuming no support`,
|
||||
e,
|
||||
);
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
@@ -7,74 +7,38 @@ Please see LICENSE in the repository root for full details.
|
||||
|
||||
import { type MatrixClient } from "matrix-js-sdk";
|
||||
import { type Logger } from "matrix-js-sdk/lib/logger";
|
||||
import { type LivekitTransport } from "matrix-js-sdk/lib/matrixrtc";
|
||||
import { type CallMembershipIdentityParts } from "matrix-js-sdk/lib/matrixrtc/EncryptionManager";
|
||||
import { type Transport } from "matrix-js-sdk/lib/matrixrtc";
|
||||
|
||||
import { type MatrixRTCMode } from "../config";
|
||||
import { getSFUConfigWithOpenID, type SFUConfig } from "./openIDSFU";
|
||||
import { RtcTransportAutoDiscovery } from "./RtcTransportAutoDiscovery";
|
||||
import { type MatrixRTCClientOptions } from "../api";
|
||||
import { MatrixRTCTransportMissingError } from "../errors";
|
||||
|
||||
/** The transport the local member publishes on, authenticated with. */
|
||||
export interface LocalTransport {
|
||||
transport: LivekitTransport;
|
||||
sfuConfig: SFUConfig;
|
||||
}
|
||||
|
||||
export function isLocalTransport(
|
||||
obj: LivekitTransport | LocalTransport,
|
||||
): obj is LocalTransport {
|
||||
return "transport" in obj && "sfuConfig" in obj;
|
||||
}
|
||||
|
||||
interface Props {
|
||||
client: Pick<
|
||||
MatrixClient,
|
||||
| "getDomain"
|
||||
| "_unstable_getRTCTransports"
|
||||
| "getOpenIdToken"
|
||||
| "getDeviceId"
|
||||
>;
|
||||
ownMembershipIdentity: CallMembershipIdentityParts;
|
||||
roomId: string;
|
||||
matrixRTCMode: MatrixRTCMode;
|
||||
/** Skip discovery and use this transport. */
|
||||
transportUrl?: string;
|
||||
/** Use this transport when the homeserver advertises none. */
|
||||
fallbackTransportUrl?: string;
|
||||
logger: Logger;
|
||||
}
|
||||
import { RtcTransportAutoDiscovery } from "./RtcTransportAutoDiscovery";
|
||||
|
||||
/**
|
||||
* The transport the local member publishes on: the one the host names, else
|
||||
* the homeserver's preferred one, else the host's fallback; authenticated
|
||||
* with, so that the session can be joined with a token in hand.
|
||||
* The transport the local member advertises: the one the host names, else the
|
||||
* homeserver's first of the type the backend serves, else the host's fallback.
|
||||
* The two URL options are LiveKit service URLs, and turning one into a
|
||||
* membership transport is the one piece of LiveKit knowledge outside the
|
||||
* backend. It stays here, for hosts that configure a URL rather than a
|
||||
* transport, and goes once the options take a `Transport`.
|
||||
*/
|
||||
export async function getLocalTransport({
|
||||
client,
|
||||
ownMembershipIdentity,
|
||||
roomId,
|
||||
matrixRTCMode,
|
||||
transportUrl,
|
||||
fallbackTransportUrl,
|
||||
logger,
|
||||
}: Props): Promise<LocalTransport> {
|
||||
const transport: LivekitTransport | null = transportUrl
|
||||
export async function discoverLocalTransport(
|
||||
client: Pick<MatrixClient, "getDomain" | "_unstable_getRTCTransports">,
|
||||
transportType: string,
|
||||
{
|
||||
transportUrl,
|
||||
fallbackTransportUrl,
|
||||
}: Pick<MatrixRTCClientOptions, "transportUrl" | "fallbackTransportUrl">,
|
||||
logger: Logger,
|
||||
): Promise<Transport> {
|
||||
const transport = transportUrl
|
||||
? { type: "livekit", livekit_service_url: transportUrl }
|
||||
: await new RtcTransportAutoDiscovery({
|
||||
client,
|
||||
transportType,
|
||||
fallbackTransportUrl,
|
||||
logger,
|
||||
}).discoverPreferredTransport();
|
||||
if (transport === null)
|
||||
throw new MatrixRTCTransportMissingError(client.getDomain() ?? "");
|
||||
const sfuConfig = await getSFUConfigWithOpenID(
|
||||
client,
|
||||
ownMembershipIdentity,
|
||||
transport.livekit_service_url,
|
||||
roomId,
|
||||
{ matrixRTCMode },
|
||||
logger,
|
||||
);
|
||||
return { transport, sfuConfig };
|
||||
return transport;
|
||||
}
|
||||
|
||||
@@ -1,72 +0,0 @@
|
||||
/*
|
||||
Copyright 2024 New Vector Ltd.
|
||||
|
||||
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
|
||||
Please see LICENSE in the repository root for full details.
|
||||
*/
|
||||
|
||||
import { describe, expect, test, vi } from "vitest";
|
||||
import {
|
||||
type MatrixRTCSession,
|
||||
MatrixRTCSessionEvent,
|
||||
} from "matrix-js-sdk/lib/matrixrtc";
|
||||
import { KeyProviderEvent } from "livekit-client";
|
||||
|
||||
import { MatrixKeyProvider } from "./MatrixKeyProvider";
|
||||
|
||||
function mockRTCSession(): MatrixRTCSession {
|
||||
return {
|
||||
on: vi.fn(),
|
||||
off: vi.fn(),
|
||||
reemitEncryptionKeys: vi.fn(),
|
||||
} as unknown as MatrixRTCSession;
|
||||
}
|
||||
|
||||
describe("matrixKeyProvider", () => {
|
||||
test("initializes", () => {
|
||||
const keyProvider = new MatrixKeyProvider();
|
||||
expect(keyProvider).toBeTruthy();
|
||||
});
|
||||
|
||||
test("listens for key requests and emits existing keys", () => {
|
||||
const keyProvider = new MatrixKeyProvider();
|
||||
|
||||
const session = mockRTCSession();
|
||||
|
||||
keyProvider.setRTCSession(session);
|
||||
|
||||
expect(session.on).toHaveBeenCalledWith(
|
||||
MatrixRTCSessionEvent.EncryptionKeyChanged,
|
||||
expect.any(Function),
|
||||
);
|
||||
expect(session.off).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test("stops listening when session changes", () => {
|
||||
const keyProvider = new MatrixKeyProvider();
|
||||
|
||||
const session1 = mockRTCSession();
|
||||
const session2 = mockRTCSession();
|
||||
|
||||
keyProvider.setRTCSession(session1);
|
||||
expect(session1.off).not.toHaveBeenCalled();
|
||||
|
||||
keyProvider.setRTCSession(session2);
|
||||
expect(session1.off).toHaveBeenCalledWith(
|
||||
MatrixRTCSessionEvent.EncryptionKeyChanged,
|
||||
expect.any(Function),
|
||||
);
|
||||
});
|
||||
|
||||
test("emits existing keys", () => {
|
||||
const keyProvider = new MatrixKeyProvider();
|
||||
const setKeyListener = vi.fn();
|
||||
keyProvider.on(KeyProviderEvent.SetKey, setKeyListener);
|
||||
|
||||
const session = mockRTCSession();
|
||||
|
||||
keyProvider.setRTCSession(session);
|
||||
|
||||
expect(session.reemitEncryptionKeys).toHaveBeenCalled();
|
||||
});
|
||||
});
|
||||
@@ -1,75 +0,0 @@
|
||||
/*
|
||||
Copyright 2023, 2024 New Vector Ltd.
|
||||
|
||||
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
|
||||
Please see LICENSE in the repository root for full details.
|
||||
*/
|
||||
|
||||
import { BaseKeyProvider } from "livekit-client";
|
||||
import {
|
||||
type MatrixRTCSession,
|
||||
MatrixRTCSessionEvent,
|
||||
} from "matrix-js-sdk/lib/matrixrtc";
|
||||
import { logger as rootLogger, type Logger } from "matrix-js-sdk/lib/logger";
|
||||
import { type CallMembershipIdentityParts } from "matrix-js-sdk/lib/matrixrtc/EncryptionManager";
|
||||
|
||||
export class MatrixKeyProvider extends BaseKeyProvider {
|
||||
private rtcSession?: MatrixRTCSession;
|
||||
private logger: Logger;
|
||||
public constructor() {
|
||||
super({ ratchetWindowSize: 10, keyringSize: 256 });
|
||||
this.logger = rootLogger.getChild("[MatrixKeyProvider]");
|
||||
}
|
||||
|
||||
public setRTCSession(rtcSession: MatrixRTCSession): void {
|
||||
if (this.rtcSession) {
|
||||
this.rtcSession.off(
|
||||
MatrixRTCSessionEvent.EncryptionKeyChanged,
|
||||
this.onEncryptionKeyChanged,
|
||||
);
|
||||
}
|
||||
|
||||
this.rtcSession = rtcSession;
|
||||
|
||||
this.rtcSession.on(
|
||||
MatrixRTCSessionEvent.EncryptionKeyChanged,
|
||||
this.onEncryptionKeyChanged,
|
||||
);
|
||||
|
||||
// The new session could be aware of keys of which the old session wasn't,
|
||||
// so emit key changed events
|
||||
this.rtcSession.reemitEncryptionKeys();
|
||||
}
|
||||
|
||||
private onEncryptionKeyChanged = (
|
||||
encryptionKey: Uint8Array<ArrayBuffer>,
|
||||
encryptionKeyIndex: number,
|
||||
membershipParts: CallMembershipIdentityParts,
|
||||
rtcBackendIdentity: string,
|
||||
): void => {
|
||||
crypto.subtle
|
||||
.importKey("raw", encryptionKey, "HKDF", false, [
|
||||
"deriveBits",
|
||||
"deriveKey",
|
||||
])
|
||||
.then(
|
||||
(keyMaterial) => {
|
||||
this.onSetEncryptionKey(
|
||||
keyMaterial,
|
||||
rtcBackendIdentity,
|
||||
encryptionKeyIndex,
|
||||
);
|
||||
|
||||
this.logger.debug(
|
||||
`Sent new key to livekit room=${this.rtcSession?.room.roomId} participantId=${rtcBackendIdentity} (before hash: ${membershipParts.userId}:${membershipParts.deviceId}) encryptionKeyIndex=${encryptionKeyIndex}`,
|
||||
);
|
||||
},
|
||||
(e) => {
|
||||
this.logger.error(
|
||||
`Failed to create key material from buffer for livekit room=${this.rtcSession?.room.roomId} participantId before hash=${membershipParts.userId}:${membershipParts.deviceId} encryptionKeyIndex=${encryptionKeyIndex}`,
|
||||
e,
|
||||
);
|
||||
},
|
||||
);
|
||||
};
|
||||
}
|
||||
@@ -1,333 +0,0 @@
|
||||
/*
|
||||
Copyright 2025 Element Creations Ltd.
|
||||
|
||||
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
|
||||
Please see LICENSE in the repository root for full details.
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeEach, afterEach, vi } from "vitest";
|
||||
import { logger } from "matrix-js-sdk/lib/logger";
|
||||
import {
|
||||
type CallMembership,
|
||||
type LivekitTransport,
|
||||
} from "matrix-js-sdk/lib/matrixrtc";
|
||||
import { BehaviorSubject, combineLatest, map, type Observable } from "rxjs";
|
||||
|
||||
import { type IConnectionManager } from "./ConnectionManager.ts";
|
||||
import {
|
||||
type RemoteMatrixLivekitMember,
|
||||
createRemoteMatrixLivekitMembers$,
|
||||
} from "./MatrixLivekitMembers.ts";
|
||||
import {
|
||||
Epoch,
|
||||
mapEpoch,
|
||||
ObservableScope,
|
||||
trackEpoch,
|
||||
} from "../reactive/ObservableScope";
|
||||
import { ConnectionManagerData } from "./ConnectionManager.ts";
|
||||
import {
|
||||
flushPromises,
|
||||
mockRtcMembership,
|
||||
mockRemoteParticipant,
|
||||
} from "../utils/test";
|
||||
import { type Connection } from "./Connection.ts";
|
||||
import { constant } from "../reactive/Behavior";
|
||||
import { localRtcMember } from "../utils/test-fixtures";
|
||||
|
||||
let testScope: ObservableScope;
|
||||
|
||||
const fallbackMemberId = (userId: string, deviceId: string): string =>
|
||||
`${userId}:${deviceId}`;
|
||||
|
||||
const transportA: LivekitTransport = {
|
||||
type: "livekit",
|
||||
livekit_service_url: "https://lk.example.org",
|
||||
};
|
||||
|
||||
const transportB: LivekitTransport = {
|
||||
type: "livekit",
|
||||
livekit_service_url: "https://lk.sample.com",
|
||||
};
|
||||
|
||||
const bobMembership = mockRtcMembership("@bob:example.org", "DEV000", {
|
||||
fociPreferred: [transportA],
|
||||
});
|
||||
const carlMembership = mockRtcMembership("@carl:sample.com", "DEV111", {
|
||||
fociPreferred: [transportB],
|
||||
});
|
||||
|
||||
beforeEach(() => {
|
||||
testScope = new ObservableScope();
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
testScope.end();
|
||||
});
|
||||
|
||||
function epochMeWith$<T, U>(
|
||||
source$: Observable<Epoch<U>>,
|
||||
me$: Observable<T>,
|
||||
): Observable<Epoch<T>> {
|
||||
return combineLatest([source$, me$]).pipe(
|
||||
map(([ep, cd]) => {
|
||||
return new Epoch(cd, ep.epoch);
|
||||
}),
|
||||
);
|
||||
}
|
||||
|
||||
test("should signal participant not yet connected to livekit", async () => {
|
||||
const info = vi.spyOn(logger, "info");
|
||||
const mockedMemberships$ = new BehaviorSubject([bobMembership]);
|
||||
const mockConnectionManagerData$ = new BehaviorSubject(
|
||||
new ConnectionManagerData(),
|
||||
);
|
||||
const { memberships$, membershipsWithTransport$ } =
|
||||
createEpochedMemberships$(mockedMemberships$);
|
||||
|
||||
const connectionManagerData$ = epochMeWith$(
|
||||
memberships$,
|
||||
mockConnectionManagerData$,
|
||||
);
|
||||
|
||||
const remoteMatrixLivekitMembers$ = createRemoteMatrixLivekitMembers$({
|
||||
scope: testScope,
|
||||
membershipsWithTransport$: testScope.behavior(membershipsWithTransport$),
|
||||
connectionManager: {
|
||||
connectionManagerData$: connectionManagerData$,
|
||||
} as unknown as IConnectionManager,
|
||||
localUser: localRtcMember,
|
||||
});
|
||||
|
||||
await flushPromises();
|
||||
expect(remoteMatrixLivekitMembers$.value.value).toSatisfy(
|
||||
(data: RemoteMatrixLivekitMember[]) => {
|
||||
expect(data.length).toEqual(1);
|
||||
expect(data[0].membership$.value).toBe(bobMembership);
|
||||
expect(data[0].participant.value$.value).toBe(null);
|
||||
expect(data[0].connection$.value).toBe(null);
|
||||
return true;
|
||||
},
|
||||
);
|
||||
expect(info).toHaveBeenCalledWith(
|
||||
"[RemoteMatrixLivekitMembers] @bob:example.org:DEV000: LiveKit participant missing on no connection",
|
||||
);
|
||||
});
|
||||
|
||||
// Helper to create epoch'ed memberships$ and membershipsWithTransport$ from memberships observable.
|
||||
function createEpochedMemberships$(m$: Observable<CallMembership[]>): {
|
||||
memberships$: Observable<Epoch<CallMembership[]>>;
|
||||
membershipsWithTransport$: Observable<
|
||||
Epoch<{ membership: CallMembership; transport?: LivekitTransport }[]>
|
||||
>;
|
||||
} {
|
||||
const memberships$ = m$.pipe(trackEpoch());
|
||||
const membershipsWithTransport$ = memberships$.pipe(
|
||||
mapEpoch((members) => {
|
||||
return members.map((m) => {
|
||||
const tr = m.getTransport();
|
||||
return {
|
||||
membership: m,
|
||||
transport:
|
||||
tr?.type === "livekit" ? (tr as LivekitTransport) : undefined,
|
||||
};
|
||||
});
|
||||
}),
|
||||
);
|
||||
return {
|
||||
memberships$,
|
||||
membershipsWithTransport$,
|
||||
};
|
||||
}
|
||||
|
||||
test("should signal participant on a connection that is publishing", async () => {
|
||||
const bobParticipantId = fallbackMemberId(
|
||||
bobMembership.userId,
|
||||
bobMembership.deviceId,
|
||||
);
|
||||
|
||||
const { memberships$, membershipsWithTransport$ } = createEpochedMemberships$(
|
||||
constant([bobMembership]),
|
||||
);
|
||||
|
||||
const connection = {
|
||||
transport: bobMembership.getTransport(),
|
||||
} as unknown as Connection;
|
||||
const dataWithPublisher = new ConnectionManagerData();
|
||||
dataWithPublisher.add(connection, [
|
||||
mockRemoteParticipant({ identity: bobParticipantId }),
|
||||
]);
|
||||
|
||||
const connectionManagerData$ = epochMeWith$(
|
||||
memberships$,
|
||||
constant(dataWithPublisher),
|
||||
);
|
||||
|
||||
const remoteMatrixLivekitMembers$ = createRemoteMatrixLivekitMembers$({
|
||||
scope: testScope,
|
||||
membershipsWithTransport$: testScope.behavior(membershipsWithTransport$),
|
||||
connectionManager: {
|
||||
connectionManagerData$: connectionManagerData$,
|
||||
} as unknown as IConnectionManager,
|
||||
localUser: localRtcMember,
|
||||
});
|
||||
|
||||
await flushPromises();
|
||||
expect(remoteMatrixLivekitMembers$.value.value).toSatisfy(
|
||||
(data: RemoteMatrixLivekitMember[]) => {
|
||||
expect(data.length).toEqual(1);
|
||||
expect(data[0].membership$.value).toBe(bobMembership);
|
||||
expect(data[0].participant.value$.value).toSatisfy((participant) => {
|
||||
expect(participant).toBeDefined();
|
||||
expect(participant!.identity).toEqual(bobParticipantId);
|
||||
return true;
|
||||
});
|
||||
expect(data[0].connection$.value).toBe(connection);
|
||||
return true;
|
||||
},
|
||||
);
|
||||
});
|
||||
|
||||
test("should signal participant on a connection that is not publishing", async () => {
|
||||
const { memberships$, membershipsWithTransport$ } = createEpochedMemberships$(
|
||||
constant([bobMembership]),
|
||||
);
|
||||
|
||||
const connection = {
|
||||
transport: bobMembership.getTransport(),
|
||||
} as unknown as Connection;
|
||||
const dataWithPublisher = new ConnectionManagerData();
|
||||
dataWithPublisher.add(connection, []);
|
||||
|
||||
const connectionManagerData$ = epochMeWith$(
|
||||
memberships$,
|
||||
constant(dataWithPublisher),
|
||||
);
|
||||
|
||||
const remoteMatrixLivekitMembers$ = createRemoteMatrixLivekitMembers$({
|
||||
scope: testScope,
|
||||
membershipsWithTransport$: testScope.behavior(membershipsWithTransport$),
|
||||
connectionManager: {
|
||||
connectionManagerData$: connectionManagerData$,
|
||||
} as unknown as IConnectionManager,
|
||||
localUser: localRtcMember,
|
||||
});
|
||||
await flushPromises();
|
||||
expect(remoteMatrixLivekitMembers$.value.value).toSatisfy(
|
||||
(data: RemoteMatrixLivekitMember[]) => {
|
||||
expect(data.length).toEqual(1);
|
||||
expect(data[0].membership$.value).toBe(bobMembership);
|
||||
expect(data[0].participant.value$.value).toBe(null);
|
||||
expect(data[0].connection$.value).toBe(connection);
|
||||
return true;
|
||||
},
|
||||
);
|
||||
});
|
||||
|
||||
describe("Publication edge case", () => {
|
||||
test("bob is publishing in several connections", async () => {
|
||||
const { memberships$, membershipsWithTransport$ } =
|
||||
createEpochedMemberships$(constant([bobMembership, carlMembership]));
|
||||
|
||||
const connectionWithPublisher = new ConnectionManagerData();
|
||||
const bobParticipantId = fallbackMemberId(
|
||||
bobMembership.userId,
|
||||
bobMembership.deviceId,
|
||||
);
|
||||
const connectionA = {
|
||||
transport: transportA,
|
||||
} as unknown as Connection;
|
||||
const connectionB = {
|
||||
transport: transportB,
|
||||
} as unknown as Connection;
|
||||
|
||||
connectionWithPublisher.add(connectionA, [
|
||||
mockRemoteParticipant({ identity: bobParticipantId }),
|
||||
]);
|
||||
connectionWithPublisher.add(connectionB, [
|
||||
mockRemoteParticipant({ identity: bobParticipantId }),
|
||||
]);
|
||||
|
||||
const connectionManagerData$ = epochMeWith$(
|
||||
memberships$,
|
||||
constant(connectionWithPublisher),
|
||||
);
|
||||
|
||||
const remoteMatrixLivekitMembers$ = createRemoteMatrixLivekitMembers$({
|
||||
scope: testScope,
|
||||
membershipsWithTransport$: testScope.behavior(membershipsWithTransport$),
|
||||
connectionManager: {
|
||||
connectionManagerData$: connectionManagerData$,
|
||||
} as unknown as IConnectionManager,
|
||||
localUser: localRtcMember,
|
||||
});
|
||||
await flushPromises();
|
||||
expect(remoteMatrixLivekitMembers$.value.value).toSatisfy(
|
||||
(data: RemoteMatrixLivekitMember[]) => {
|
||||
expect(data.length).toEqual(2);
|
||||
expect(data[0].membership$.value).toBe(bobMembership);
|
||||
expect(data[0].connection$.value).toBe(connectionA);
|
||||
expect(data[0].participant.value$.value).toSatisfy((participant) => {
|
||||
expect(participant).toBeDefined();
|
||||
expect(participant!.identity).toEqual(bobParticipantId);
|
||||
return true;
|
||||
});
|
||||
|
||||
return true;
|
||||
},
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
test("bob is publishing in the wrong connection", async () => {
|
||||
const mockedMemberships$ = new BehaviorSubject([
|
||||
bobMembership,
|
||||
carlMembership,
|
||||
]);
|
||||
|
||||
const { memberships$, membershipsWithTransport$ } =
|
||||
createEpochedMemberships$(mockedMemberships$);
|
||||
|
||||
const connectionWithPublisher = new ConnectionManagerData();
|
||||
|
||||
const bobParticipantId = fallbackMemberId(
|
||||
bobMembership.userId,
|
||||
bobMembership.deviceId,
|
||||
);
|
||||
const connectionA = { transport: transportA } as unknown as Connection;
|
||||
const connectionB = { transport: transportB } as unknown as Connection;
|
||||
|
||||
// Bob is not publishing on A
|
||||
connectionWithPublisher.add(connectionA, []);
|
||||
// Bob is publishing on B but his membership says A
|
||||
connectionWithPublisher.add(connectionB, [
|
||||
mockRemoteParticipant({ identity: bobParticipantId }),
|
||||
]);
|
||||
|
||||
const connectionsWithPublisher$ = new BehaviorSubject(
|
||||
connectionWithPublisher,
|
||||
);
|
||||
const connectionManagerData$ = epochMeWith$(
|
||||
memberships$,
|
||||
connectionsWithPublisher$,
|
||||
);
|
||||
|
||||
const remoteMatrixLivekitMembers$ = createRemoteMatrixLivekitMembers$({
|
||||
scope: testScope,
|
||||
membershipsWithTransport$: testScope.behavior(membershipsWithTransport$),
|
||||
connectionManager: {
|
||||
connectionManagerData$: connectionManagerData$,
|
||||
} as unknown as IConnectionManager,
|
||||
localUser: localRtcMember,
|
||||
});
|
||||
|
||||
await flushPromises();
|
||||
expect(remoteMatrixLivekitMembers$.value.value).toSatisfy(
|
||||
(data: RemoteMatrixLivekitMember[]) => {
|
||||
expect(data.length).toEqual(2);
|
||||
expect(data[0].membership$.value).toBe(bobMembership);
|
||||
expect(data[0].connection$.value).toBe(connectionA);
|
||||
expect(data[0].participant.value$.value).toBe(null);
|
||||
return true;
|
||||
},
|
||||
);
|
||||
});
|
||||
@@ -1,160 +0,0 @@
|
||||
/*
|
||||
Copyright 2025 Element Creations Ltd.
|
||||
|
||||
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
|
||||
Please see LICENSE in the repository root for full details.
|
||||
*/
|
||||
|
||||
import { type RemoteParticipant } from "livekit-client";
|
||||
import {
|
||||
type CallMembership,
|
||||
type LivekitTransport,
|
||||
} from "matrix-js-sdk/lib/matrixrtc";
|
||||
import { combineLatest, filter, map } from "rxjs";
|
||||
import { logger } from "matrix-js-sdk/lib/logger";
|
||||
|
||||
import { type Behavior } from "../reactive/Behavior";
|
||||
import { type IConnectionManager } from "./ConnectionManager";
|
||||
import { Epoch, type ObservableScope } from "../reactive/ObservableScope";
|
||||
import { type Connection } from "./Connection";
|
||||
import { generateItemsWithEpoch } from "../reactive/observable";
|
||||
|
||||
interface RemoteTaggedParticipant {
|
||||
type: "remote";
|
||||
value$: Behavior<RemoteParticipant | null>;
|
||||
}
|
||||
|
||||
export interface MatrixLivekitMember {
|
||||
membership$: Behavior<CallMembership>;
|
||||
connection$: Behavior<Connection | null>;
|
||||
// participantId: string; We do not want a participantId here since it will be generated by the jwt
|
||||
// TODO decide if we can also drop the userId. Its in the matrix membership anyways.
|
||||
userId: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* Represents a remote Matrix call member and their associated LiveKit participation.
|
||||
* `livekitParticipant` can be null if the member is not yet connected to the livekit room
|
||||
* or if it has no livekit transport at all.
|
||||
*/
|
||||
export interface RemoteMatrixLivekitMember extends MatrixLivekitMember {
|
||||
participant: RemoteTaggedParticipant;
|
||||
}
|
||||
|
||||
interface Props {
|
||||
scope: ObservableScope;
|
||||
membershipsWithTransport$: Behavior<
|
||||
Epoch<{ membership: CallMembership; transport?: LivekitTransport }[]>
|
||||
>;
|
||||
connectionManager: IConnectionManager;
|
||||
localUser: { deviceId: string; userId: string };
|
||||
}
|
||||
|
||||
/**
|
||||
* Combines MatrixRTC and Livekit worlds.
|
||||
*
|
||||
* It has a small public interface:
|
||||
* - in (via constructor):
|
||||
* - an observable of CallMembership[] to track the call members (The matrix side)
|
||||
* - a `ConnectionManager` for the lk rooms (The livekit side)
|
||||
* - out (via public Observable):
|
||||
* - `remoteMatrixLivekitMember` an observable of MatrixLivekitMember[] to track the remote members and associated livekit data.
|
||||
*/
|
||||
export function createRemoteMatrixLivekitMembers$({
|
||||
scope,
|
||||
membershipsWithTransport$,
|
||||
connectionManager,
|
||||
localUser,
|
||||
}: Props): Behavior<Epoch<RemoteMatrixLivekitMember[]>> {
|
||||
/**
|
||||
* Behavior of all the remote call members and their associated livekit data (if available).
|
||||
*/
|
||||
return scope.behavior(
|
||||
combineLatest([
|
||||
membershipsWithTransport$,
|
||||
connectionManager.connectionManagerData$,
|
||||
]).pipe(
|
||||
filter((values) =>
|
||||
values.every((value) => value.epoch === values[0].epoch),
|
||||
),
|
||||
map(([ms, data]) => new Epoch([ms.value, data.value] as const, ms.epoch)),
|
||||
generateItemsWithEpoch(
|
||||
"RemoteMatrixLivekitMembers",
|
||||
// Generator function.
|
||||
// creates an array of `{key, data}[]`
|
||||
// Each change in the keys (new key) will result in a call to the factory function.
|
||||
function* ([membershipsWithTransport, managerData]) {
|
||||
for (const { membership, transport } of membershipsWithTransport) {
|
||||
// Exclude the local membership
|
||||
if (
|
||||
membership.userId === localUser.userId &&
|
||||
membership.deviceId === localUser.deviceId
|
||||
)
|
||||
continue;
|
||||
|
||||
const participants = transport
|
||||
? managerData.getParticipantsForTransport(transport)
|
||||
: [];
|
||||
const matches = participants.filter(
|
||||
(p) => p.identity == membership.rtcBackendIdentity,
|
||||
);
|
||||
const participant = matches[0] ?? null;
|
||||
const connection = transport
|
||||
? managerData.getConnectionForTransport(transport)
|
||||
: null;
|
||||
if (matches.length > 1)
|
||||
logger.warn(
|
||||
`[RemoteMatrixLivekitMembers] ${membership.rtcBackendIdentity}: ${matches.length} LiveKit participants match (sids ${matches.map((p) => p.sid).join(", ")}), using ${participant?.sid}`,
|
||||
);
|
||||
|
||||
yield {
|
||||
// This could just be the backend identity without the other keys.
|
||||
// The user ID, device ID, and member ID are included however so
|
||||
// they show up in debug logs.
|
||||
keys: [
|
||||
membership.userId,
|
||||
membership.deviceId,
|
||||
membership.memberId,
|
||||
membership.rtcBackendIdentity,
|
||||
],
|
||||
data: { membership, participant, connection },
|
||||
};
|
||||
}
|
||||
},
|
||||
// Each update where the key of the generator array do not change will result in updates to the `data$` behavior.
|
||||
(scope, data$, userId, _deviceId, _memberId, rtcBackendIdentity) => {
|
||||
const { participant$, ...rest } = scope.splitBehavior(data$);
|
||||
// Log whether the member could be matched to a LiveKit participant,
|
||||
// since a tile shows "waiting for media" for as long as it cannot.
|
||||
participant$.pipe(scope.bind()).subscribe((p) => {
|
||||
const url = data$.value.connection?.transport.livekit_service_url;
|
||||
logger.info(
|
||||
`[RemoteMatrixLivekitMembers] ${rtcBackendIdentity}: LiveKit participant ${p ? `matched (${p.sid})` : "missing"} on ${url ?? "no connection"}`,
|
||||
);
|
||||
});
|
||||
// will only get called once per backend identity.
|
||||
// updates to data$ and as a result to displayName$ and mxcAvatarUrl$ are more frequent.
|
||||
return {
|
||||
userId,
|
||||
participant: { type: "remote" as const, value$: participant$ },
|
||||
...rest,
|
||||
};
|
||||
},
|
||||
),
|
||||
),
|
||||
new Epoch([], -1),
|
||||
);
|
||||
}
|
||||
|
||||
// TODO add back in the callviewmodel pauseWhen(this.pretendToBeDisconnected$)
|
||||
|
||||
// TODO add this to the JS-SDK
|
||||
export function areLivekitTransportsEqual<T extends LivekitTransport>(
|
||||
t1: T | null,
|
||||
t2: T | null,
|
||||
): boolean {
|
||||
if (t1 && t2) {
|
||||
return t1.livekit_service_url === t2.livekit_service_url;
|
||||
}
|
||||
return !t1 && !t2;
|
||||
}
|
||||
@@ -8,6 +8,7 @@ Please see LICENSE in the repository root for full details.
|
||||
import { type MatrixClient, type Room } from "matrix-js-sdk";
|
||||
import { type Logger, logger as rootLogger } from "matrix-js-sdk/lib/logger";
|
||||
import {
|
||||
type MatrixRTCSession as JsSdkRTCSession,
|
||||
MatrixRTCSessionEvent,
|
||||
MatrixRTCSessionManager,
|
||||
MembershipManagerEvent,
|
||||
@@ -15,27 +16,32 @@ import {
|
||||
import { type IMembershipManager } from "matrix-js-sdk/lib/matrixrtc/IMembershipManager";
|
||||
import { type CallMembershipIdentityParts } from "matrix-js-sdk/lib/matrixrtc/EncryptionManager";
|
||||
import { v4 as uuidv4 } from "uuid";
|
||||
import { combineLatest, from, fromEvent, map, type Observable } from "rxjs";
|
||||
import { combineLatest, filter, from, fromEvent, map, Observable } from "rxjs";
|
||||
|
||||
import { defaultSessionTimings, MatrixRTCMode } from "../config";
|
||||
import { E2eeType } from "../encryption";
|
||||
import { type ObservableScope } from "../reactive/ObservableScope";
|
||||
import { filterBehavior, generateItems } from "../reactive/observable";
|
||||
import { mapScoped } from "../utils/mapScoped";
|
||||
import {
|
||||
type DataMessage,
|
||||
type LocalMediaInputs,
|
||||
type LocalRTCMember,
|
||||
type MatrixRTCClient,
|
||||
type MatrixRTCClientOptions,
|
||||
type RemoteRTCMember,
|
||||
} from "../api";
|
||||
import {
|
||||
type MediaBackend,
|
||||
type MediaBackendContext,
|
||||
type MediaKey,
|
||||
} from "../backend/api";
|
||||
import { createLivekitBackend } from "../backend/livekit/LivekitBackend";
|
||||
import { MatrixRTCError } from "../errors";
|
||||
import { LivekitConnectionFactory } from "./ConnectionFactory";
|
||||
import { createHomeserverConnected$ } from "./HomeserverConnected";
|
||||
import { joinJsSdkSession } from "./joinJsSdkSession";
|
||||
import { createKeyProvider } from "./KeyProvider";
|
||||
import { createLocalMembership$ } from "./LocalMember";
|
||||
import { getLocalTransport } from "./LocalTransport";
|
||||
import { createRemoteMatrixLivekitMembers$ } from "./MatrixLivekitMembers";
|
||||
import { createLocalMembership$, type PreparedTransport } from "./LocalMember";
|
||||
import { discoverLocalTransport } from "./LocalTransport";
|
||||
import {
|
||||
createMatrixMemberMetadata$,
|
||||
createRoomMembers$,
|
||||
@@ -45,13 +51,9 @@ import {
|
||||
createRemoteRTCMember,
|
||||
membershipKeys,
|
||||
} from "./Members";
|
||||
import { createConnectionManager$ } from "./ConnectionManager";
|
||||
import { createDataChannel$ } from "./DataChannel";
|
||||
import { type DesiredMedia, Publisher } from "./Publisher";
|
||||
import {
|
||||
createKeyRotationSuppressed$,
|
||||
createMemberships$,
|
||||
membershipsAndTransports$,
|
||||
} from "./SessionBehaviors";
|
||||
import { fatalError, sessionStatus } from "./status";
|
||||
import { createTransportRegistry } from "./Transports";
|
||||
@@ -84,7 +86,6 @@ export function createMatrixRTCClient(
|
||||
options,
|
||||
logger,
|
||||
).getRoomSession(room);
|
||||
const keyProvider = createKeyProvider(encryptionSystem, jsSdkSession, logger);
|
||||
|
||||
const ownMembershipIdentity: CallMembershipIdentityParts = {
|
||||
userId,
|
||||
@@ -98,69 +99,43 @@ export function createMatrixRTCClient(
|
||||
: `${userId}:${deviceId}`,
|
||||
};
|
||||
|
||||
const memberships$ = createMemberships$(scope, jsSdkSession);
|
||||
const { membershipsWithTransport$, transports$ } = membershipsAndTransports$(
|
||||
scope,
|
||||
memberships$,
|
||||
);
|
||||
|
||||
const localTransport$ = from(
|
||||
getLocalTransport({
|
||||
client,
|
||||
ownMembershipIdentity,
|
||||
roomId: room.roomId,
|
||||
matrixRTCMode,
|
||||
transportUrl: options.transportUrl,
|
||||
fallbackTransportUrl: options.fallbackTransportUrl,
|
||||
logger,
|
||||
}),
|
||||
);
|
||||
|
||||
const desired: DesiredMedia = new Map(
|
||||
localMedia.publish.map((request) => [
|
||||
request.source,
|
||||
{ request, enabled: true },
|
||||
]),
|
||||
);
|
||||
|
||||
const connectionManager = createConnectionManager$({
|
||||
scope,
|
||||
connectionFactory: new LivekitConnectionFactory(
|
||||
client,
|
||||
room.roomId,
|
||||
localMedia,
|
||||
keyProvider,
|
||||
options.mediaQuality,
|
||||
),
|
||||
localTransport$,
|
||||
remoteTransports$: transports$,
|
||||
logger,
|
||||
const backend = createBackend(scope, client, options, {
|
||||
roomId: room.roomId,
|
||||
ownMembershipIdentity,
|
||||
localMedia,
|
||||
encryptionSystem,
|
||||
mediaKeys$: mediaKeys$(jsSdkSession),
|
||||
timings,
|
||||
logger,
|
||||
});
|
||||
const transports = createTransportRegistry(scope, connectionManager);
|
||||
|
||||
const memberships$ = createMemberships$(scope, jsSdkSession);
|
||||
const transports = createTransportRegistry(scope, backend.connections$);
|
||||
|
||||
const preparedTransport$ = from(
|
||||
discoverLocalTransport(client, backend.transportType, options, logger).then(
|
||||
async (transport): Promise<PreparedTransport> => ({
|
||||
transport,
|
||||
...(await backend.prepareLocalTransport(transport)),
|
||||
}),
|
||||
),
|
||||
);
|
||||
|
||||
const localMembership = createLocalMembership$({
|
||||
scope,
|
||||
connectionManager,
|
||||
localTransport$,
|
||||
local: backend.local,
|
||||
delegateDelayedLeave: async (delayId) =>
|
||||
backend.delegateDelayedLeave(delayId),
|
||||
preparedTransport$,
|
||||
homeserverConnected: createHomeserverConnected$(
|
||||
scope,
|
||||
client,
|
||||
jsSdkSession,
|
||||
timings.syncDisconnectGracePeriodMs,
|
||||
),
|
||||
createPublisher: (connection) =>
|
||||
new Publisher(
|
||||
connection.livekitRoom,
|
||||
localMedia,
|
||||
desired,
|
||||
logger.getChild(
|
||||
`[Publisher ${connection.transport.livekit_service_url}]`,
|
||||
),
|
||||
),
|
||||
joinMatrixRTC: (transport, delayedLeave) =>
|
||||
joinJsSdkSession(jsSdkSession, ownMembershipIdentity, transport, {
|
||||
encryptMedia: keyProvider !== undefined,
|
||||
encryptMedia: encryptionSystem.kind !== E2eeType.NONE,
|
||||
matrixRTCMode,
|
||||
sendNotificationType: options.sendNotificationType,
|
||||
applicationData: options.applicationData,
|
||||
@@ -182,12 +157,7 @@ export function createMatrixRTCClient(
|
||||
).pipe(map(([delayId]) => delayId ?? null)),
|
||||
jsSdkSession.delayId ?? null,
|
||||
),
|
||||
client,
|
||||
roomId: room.roomId,
|
||||
ownMembershipIdentity,
|
||||
matrixRTCMode,
|
||||
timings,
|
||||
desired,
|
||||
logger,
|
||||
});
|
||||
|
||||
@@ -198,28 +168,30 @@ export function createMatrixRTCClient(
|
||||
createRoomMembers$(scope, room),
|
||||
),
|
||||
transports,
|
||||
encryptionSystem,
|
||||
};
|
||||
|
||||
const remoteMembers$ = scope.behavior<RemoteRTCMember[]>(
|
||||
createRemoteMatrixLivekitMembers$({
|
||||
scope,
|
||||
membershipsWithTransport$,
|
||||
connectionManager,
|
||||
localUser: { userId, deviceId },
|
||||
}).pipe(
|
||||
memberships$.pipe(
|
||||
map(({ value }) => value),
|
||||
generateItems(
|
||||
"MatrixRTCClient remoteMembers",
|
||||
function* (members) {
|
||||
for (const member of members)
|
||||
yield {
|
||||
keys: membershipKeys(member.membership$.value),
|
||||
data: member,
|
||||
};
|
||||
function* (memberships) {
|
||||
for (const membership of memberships) {
|
||||
if (
|
||||
membership.userId === userId &&
|
||||
membership.deviceId === deviceId
|
||||
)
|
||||
continue;
|
||||
yield { keys: membershipKeys(membership), data: membership };
|
||||
}
|
||||
},
|
||||
(memberScope, member$) =>
|
||||
createRemoteRTCMember(memberScope, member$.value, context),
|
||||
(memberScope, membership$) =>
|
||||
createRemoteRTCMember(
|
||||
memberScope,
|
||||
membership$,
|
||||
backend.mediaFor$(memberScope, membership$),
|
||||
context,
|
||||
),
|
||||
),
|
||||
),
|
||||
);
|
||||
@@ -238,17 +210,23 @@ export function createMatrixRTCClient(
|
||||
);
|
||||
const localMember$ = scope.behavior<LocalRTCMember | null>(
|
||||
mapScoped(scope, localMembership$, (memberScope, membership$) =>
|
||||
createLocalRTCMember(memberScope, membership$, localMembership, context),
|
||||
createLocalRTCMember(memberScope, membership$, backend.local, context),
|
||||
).pipe(map((member) => member ?? null)),
|
||||
);
|
||||
|
||||
const { data$, sendData } = createDataChannel$({
|
||||
scope,
|
||||
connectionManager,
|
||||
remoteMembers$,
|
||||
connection$: localMembership.connection$,
|
||||
logger,
|
||||
});
|
||||
// A packet from an identity that is not a member is dropped, so a host only
|
||||
// ever hears from attested members
|
||||
const data$ = backend.data$.pipe(
|
||||
map(({ senderId, topic, text }): DataMessage | null => {
|
||||
const member = remoteMembers$.value.find((m) => m.id === senderId);
|
||||
if (member === undefined) {
|
||||
logger.warn(`Dropping data from ${senderId}: not a member`);
|
||||
return null;
|
||||
}
|
||||
return { member, topic, text };
|
||||
}),
|
||||
filter((message) => message !== null),
|
||||
);
|
||||
|
||||
const status$ = scope.behavior(
|
||||
combineLatest(
|
||||
@@ -280,11 +258,42 @@ export function createMatrixRTCClient(
|
||||
),
|
||||
keyRotationSuppressed$: createKeyRotationSuppressed$(scope, jsSdkSession),
|
||||
connectedTransports$: transports.connected$,
|
||||
sendData,
|
||||
sendData: async (topic, text) => backend.sendData(topic, text),
|
||||
data$,
|
||||
};
|
||||
}
|
||||
|
||||
/** The host's backend, or LiveKit configured from the client options. */
|
||||
function createBackend(
|
||||
scope: ObservableScope,
|
||||
client: MatrixClient,
|
||||
{ backend, mediaQuality, matrixRTCMode }: MatrixRTCClientOptions,
|
||||
context: MediaBackendContext,
|
||||
): MediaBackend {
|
||||
if (backend) return backend(scope, context);
|
||||
return createLivekitBackend(scope, context, {
|
||||
client,
|
||||
mediaQuality,
|
||||
tokenEndpoint:
|
||||
matrixRTCMode === MatrixRTCMode.Matrix_2_0 ? "msc4195" : "legacy",
|
||||
});
|
||||
}
|
||||
|
||||
/** The session's per-participant keys: the ones it already holds on subscribe, then each new one. */
|
||||
function mediaKeys$(session: JsSdkRTCSession): Observable<MediaKey> {
|
||||
return new Observable((subscriber) => {
|
||||
const onKey = (
|
||||
key: Uint8Array<ArrayBuffer>,
|
||||
index: number,
|
||||
_membership: CallMembershipIdentityParts,
|
||||
participantId: string,
|
||||
): void => subscriber.next({ participantId, index, key });
|
||||
session.on(MatrixRTCSessionEvent.EncryptionKeyChanged, onKey);
|
||||
session.reemitEncryptionKeys();
|
||||
return () => session.off(MatrixRTCSessionEvent.EncryptionKeyChanged, onKey);
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* The js-sdk keeps one session manager per slot. The client comes with the
|
||||
* manager for the default slot; any other slot gets a manager of its own,
|
||||
|
||||
+11
-102
@@ -5,46 +5,25 @@ SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
|
||||
Please see LICENSE in the repository root for full details.
|
||||
*/
|
||||
|
||||
import {
|
||||
type LocalParticipant,
|
||||
type Participant,
|
||||
type Room as LivekitRoom,
|
||||
} from "livekit-client";
|
||||
import { type CallMembership } from "matrix-js-sdk/lib/matrixrtc";
|
||||
import {
|
||||
combineLatest,
|
||||
distinctUntilChanged,
|
||||
filter,
|
||||
firstValueFrom,
|
||||
map,
|
||||
of,
|
||||
switchMap,
|
||||
} from "rxjs";
|
||||
import { distinctUntilChanged, map } from "rxjs";
|
||||
|
||||
import { type Behavior } from "../reactive/Behavior";
|
||||
import { type ObservableScope } from "../reactive/ObservableScope";
|
||||
import { type Connection } from "./Connection";
|
||||
import { type createMatrixMemberMetadata$ } from "./MatrixMemberMetadata";
|
||||
import { type RemoteMatrixLivekitMember } from "./MatrixLivekitMembers";
|
||||
import { type EncryptionSystem } from "../encryption";
|
||||
import {
|
||||
type LocalRTCMember,
|
||||
type MemberMedia,
|
||||
type RemoteRTCMember,
|
||||
type RTCMember,
|
||||
} from "../api";
|
||||
import {
|
||||
createLivekitMemberMedia,
|
||||
createLocalLivekitMemberMedia,
|
||||
} from "../media/LivekitMemberMedia";
|
||||
import { mapScoped } from "../utils/mapScoped";
|
||||
import { type LocalMembership } from "./LocalMember";
|
||||
import { type LocalMediaBackend } from "../backend/api";
|
||||
import { type TransportRegistry } from "./Transports";
|
||||
|
||||
/** What every member is built from, besides its own membership and media. */
|
||||
export interface MemberContext {
|
||||
metadata: ReturnType<typeof createMatrixMemberMetadata$>;
|
||||
transports: TransportRegistry;
|
||||
encryptionSystem: EncryptionSystem;
|
||||
}
|
||||
|
||||
/** Everything that tells one membership from another, for keying items. */
|
||||
@@ -61,67 +40,29 @@ export function membershipKeys(
|
||||
|
||||
export function createRemoteRTCMember(
|
||||
scope: ObservableScope,
|
||||
member: RemoteMatrixLivekitMember,
|
||||
membership$: Behavior<CallMembership>,
|
||||
media$: Behavior<MemberMedia | null>,
|
||||
context: MemberContext,
|
||||
): RemoteRTCMember {
|
||||
return {
|
||||
...createRTCMember(scope, member.membership$, context),
|
||||
...createRTCMember(scope, membership$, context),
|
||||
local: false,
|
||||
media$: mediaFor(
|
||||
scope,
|
||||
member.participant.value$,
|
||||
member.connection$,
|
||||
(mediaScope, participant, room) =>
|
||||
createLivekitMemberMedia(
|
||||
mediaScope,
|
||||
participant,
|
||||
room,
|
||||
context.encryptionSystem,
|
||||
),
|
||||
),
|
||||
media$,
|
||||
};
|
||||
}
|
||||
|
||||
export function createLocalRTCMember(
|
||||
scope: ObservableScope,
|
||||
membership$: Behavior<CallMembership>,
|
||||
localMembership: LocalMembership,
|
||||
local: Pick<LocalMediaBackend, "media$" | "publish" | "unpublish">,
|
||||
context: MemberContext,
|
||||
): LocalRTCMember {
|
||||
const media$ = mediaFor(
|
||||
scope,
|
||||
localMembership.participant$,
|
||||
localMembership.connection$,
|
||||
(mediaScope, participant, room) =>
|
||||
createLocalLivekitMemberMedia(
|
||||
mediaScope,
|
||||
participant,
|
||||
room,
|
||||
context.encryptionSystem,
|
||||
localMembership.setEnabled,
|
||||
),
|
||||
);
|
||||
return {
|
||||
...createRTCMember(scope, membership$, context),
|
||||
local: true,
|
||||
media$,
|
||||
publish: async (request) => {
|
||||
const { trackSid } = await localMembership.publish(request);
|
||||
return firstValueFrom(
|
||||
media$.pipe(
|
||||
switchMap((media) => media?.tracks$ ?? of([])),
|
||||
map((tracks) => tracks.find((track) => track.id === trackSid)),
|
||||
filter((track) => track !== undefined),
|
||||
),
|
||||
);
|
||||
},
|
||||
unpublish: async (id) => {
|
||||
const track = media$.value?.tracks$.value.find((t) => t.id === id);
|
||||
if (track === undefined) return;
|
||||
await localMembership.unpublish(
|
||||
track.source === "screenShareAudio" ? "screenShare" : track.source,
|
||||
);
|
||||
},
|
||||
media$: local.media$,
|
||||
publish: async (request) => local.publish(request),
|
||||
unpublish: async (id) => local.unpublish(id),
|
||||
};
|
||||
}
|
||||
|
||||
@@ -151,35 +92,3 @@ function createRTCMember(
|
||||
),
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* The member's media once its participant is known, in a scope that ends when
|
||||
* the participant or the connection changes.
|
||||
*/
|
||||
function mediaFor<
|
||||
P extends Participant | LocalParticipant,
|
||||
M extends MemberMedia,
|
||||
>(
|
||||
scope: ObservableScope,
|
||||
participant$: Behavior<P | null>,
|
||||
connection$: Behavior<Connection | null>,
|
||||
factory: (scope: ObservableScope, participant: P, room: LivekitRoom) => M,
|
||||
): Behavior<M | null> {
|
||||
const source$ = scope.behavior(
|
||||
combineLatest([participant$, connection$]).pipe(
|
||||
map(([participant, connection]) =>
|
||||
participant && connection
|
||||
? { participant, room: connection.livekitRoom }
|
||||
: null,
|
||||
),
|
||||
distinctUntilChanged(
|
||||
(a, b) => a?.participant === b?.participant && a?.room === b?.room,
|
||||
),
|
||||
),
|
||||
);
|
||||
return scope.behavior(
|
||||
mapScoped(scope, source$, (mediaScope, { participant, room }) =>
|
||||
factory(mediaScope, participant, room),
|
||||
).pipe(map((media) => media ?? null)),
|
||||
);
|
||||
}
|
||||
|
||||
@@ -64,6 +64,7 @@ describe("RtcTransportAutoDiscovery", () => {
|
||||
|
||||
const discovery = new RtcTransportAutoDiscovery({
|
||||
client,
|
||||
transportType: "livekit",
|
||||
fallbackTransportUrl: configTransport.livekit_service_url,
|
||||
logger: rootLogger,
|
||||
});
|
||||
@@ -94,6 +95,7 @@ describe("RtcTransportAutoDiscovery", () => {
|
||||
|
||||
const discovery = new RtcTransportAutoDiscovery({
|
||||
client,
|
||||
transportType: "livekit",
|
||||
fallbackTransportUrl: "https://config.example.org",
|
||||
logger: rootLogger,
|
||||
});
|
||||
@@ -117,6 +119,7 @@ describe("RtcTransportAutoDiscovery", () => {
|
||||
|
||||
const discovery = new RtcTransportAutoDiscovery({
|
||||
client,
|
||||
transportType: "livekit",
|
||||
fallbackTransportUrl: configTransport.livekit_service_url,
|
||||
logger: rootLogger,
|
||||
});
|
||||
@@ -133,6 +136,7 @@ describe("RtcTransportAutoDiscovery", () => {
|
||||
|
||||
const discovery = new RtcTransportAutoDiscovery({
|
||||
client,
|
||||
transportType: "livekit",
|
||||
fallbackTransportUrl: undefined,
|
||||
logger: rootLogger,
|
||||
});
|
||||
|
||||
@@ -1,13 +1,10 @@
|
||||
/*
|
||||
Copyright 2026 Element Creations Ltd.
|
||||
|
||||
SPDX-License-IdFentifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
|
||||
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
|
||||
Please see LICENSE in the repository root for full details.
|
||||
*/
|
||||
import {
|
||||
isLivekitTransport,
|
||||
type LivekitTransport,
|
||||
} from "matrix-js-sdk/lib/matrixrtc";
|
||||
import { type Transport } from "matrix-js-sdk/lib/matrixrtc";
|
||||
import { type MatrixClient } from "matrix-js-sdk";
|
||||
import { type Logger } from "matrix-js-sdk/lib/logger";
|
||||
|
||||
@@ -20,42 +17,43 @@ type TransportDiscoveryClient = Pick<
|
||||
|
||||
export interface RtcTransportAutoDiscoveryProps {
|
||||
client: TransportDiscoveryClient;
|
||||
/** A transport to use when the homeserver advertises none. */
|
||||
/** The transport type the media backend serves. */
|
||||
transportType: string;
|
||||
/** A LiveKit service URL to use when the homeserver advertises none. */
|
||||
fallbackTransportUrl?: string;
|
||||
logger: Logger;
|
||||
}
|
||||
|
||||
export class RtcTransportAutoDiscovery {
|
||||
private readonly client: TransportDiscoveryClient;
|
||||
private readonly transportType: string;
|
||||
private readonly fallbackTransportUrl: string | undefined;
|
||||
private readonly logger: Logger;
|
||||
|
||||
public constructor({
|
||||
client,
|
||||
transportType,
|
||||
fallbackTransportUrl,
|
||||
logger,
|
||||
}: RtcTransportAutoDiscoveryProps) {
|
||||
this.client = client;
|
||||
this.transportType = transportType;
|
||||
this.fallbackTransportUrl = fallbackTransportUrl;
|
||||
this.logger = logger.getChild("[RtcTransportAutoDiscovery]");
|
||||
}
|
||||
|
||||
public async discoverPreferredTransport(): Promise<LivekitTransport | null> {
|
||||
public async discoverPreferredTransport(): Promise<Transport | null> {
|
||||
// 1) backend transports
|
||||
const backendTransport = await this.tryBackendTransports();
|
||||
if (backendTransport) {
|
||||
this.logger.info(
|
||||
`Found backend transport: ${backendTransport.livekit_service_url}`,
|
||||
);
|
||||
this.logger.info("Found backend transport:", backendTransport);
|
||||
return backendTransport;
|
||||
}
|
||||
|
||||
// 2) app config URL
|
||||
const configTransport = this.tryConfigTransport();
|
||||
if (configTransport) {
|
||||
this.logger.info(
|
||||
`Found app config transport: ${configTransport.livekit_service_url}`,
|
||||
);
|
||||
this.logger.info("Found app config transport:", configTransport);
|
||||
return configTransport;
|
||||
}
|
||||
|
||||
@@ -63,11 +61,11 @@ export class RtcTransportAutoDiscovery {
|
||||
}
|
||||
|
||||
/**
|
||||
* Fetches the first rtc_foci from the backend.
|
||||
* Fetches the first transport of the backend's type from the homeserver.
|
||||
* This will not throw errors, but instead just log them and return null if the expected config is not found or malformed.
|
||||
* @private
|
||||
*/
|
||||
private async tryBackendTransports(): Promise<LivekitTransport | null> {
|
||||
private async tryBackendTransports(): Promise<Transport | null> {
|
||||
const client = this.client;
|
||||
// MSC4143: Attempt to fetch transports from backend.
|
||||
this.logger.info("First try to use getRTCTransports end point ...");
|
||||
@@ -75,12 +73,14 @@ export class RtcTransportAutoDiscovery {
|
||||
const transportList = await doNetworkOperationWithRetry(async () =>
|
||||
client._unstable_getRTCTransports(),
|
||||
);
|
||||
const first = transportList.find(isLivekitTransport);
|
||||
const first = transportList.find(
|
||||
(transport) => transport.type === this.transportType,
|
||||
);
|
||||
if (first) {
|
||||
return first;
|
||||
} else {
|
||||
this.logger.info(
|
||||
`No livekit transport found in getRTCTransports end point`,
|
||||
`No ${this.transportType} transport found in getRTCTransports end point`,
|
||||
transportList,
|
||||
);
|
||||
}
|
||||
@@ -90,7 +90,7 @@ export class RtcTransportAutoDiscovery {
|
||||
return null;
|
||||
}
|
||||
|
||||
private tryConfigTransport(): LivekitTransport | null {
|
||||
private tryConfigTransport(): Transport | null {
|
||||
const url = this.fallbackTransportUrl;
|
||||
if (url) {
|
||||
return {
|
||||
|
||||
@@ -7,67 +7,18 @@ Please see LICENSE in the repository root for full details.
|
||||
|
||||
import {
|
||||
type CallMembership,
|
||||
type LivekitTransport,
|
||||
type MatrixRTCSession,
|
||||
MatrixRTCSessionEvent,
|
||||
isLivekitTransport,
|
||||
} from "matrix-js-sdk/lib/matrixrtc";
|
||||
import { fromEvent } from "rxjs";
|
||||
|
||||
import {
|
||||
Epoch,
|
||||
mapEpoch,
|
||||
trackEpoch,
|
||||
type ObservableScope,
|
||||
} from "../reactive/ObservableScope";
|
||||
import { type Behavior } from "../reactive/Behavior";
|
||||
|
||||
/**
|
||||
* Tracks the transports used by ourselves, plus all other MatrixRTC session
|
||||
* members.
|
||||
*/
|
||||
export const membershipsAndTransports$ = (
|
||||
scope: ObservableScope,
|
||||
memberships$: Behavior<Epoch<CallMembership[]>>,
|
||||
): {
|
||||
membershipsWithTransport$: Behavior<
|
||||
Epoch<{ membership: CallMembership; transport?: LivekitTransport }[]>
|
||||
>;
|
||||
transports$: Behavior<Epoch<LivekitTransport[]>>;
|
||||
} => {
|
||||
const membershipsWithTransport$: Behavior<
|
||||
Epoch<
|
||||
{
|
||||
membership: CallMembership;
|
||||
transport: LivekitTransport | undefined;
|
||||
}[]
|
||||
>
|
||||
> = scope.behavior(
|
||||
memberships$.pipe(
|
||||
mapEpoch((memberships) => {
|
||||
return memberships.map((membership) => {
|
||||
const transport = membership.getTransport();
|
||||
return {
|
||||
membership,
|
||||
transport: isLivekitTransport(transport) ? transport : undefined,
|
||||
};
|
||||
});
|
||||
}),
|
||||
),
|
||||
);
|
||||
|
||||
const transports$: Behavior<Epoch<LivekitTransport[]>> = scope.behavior(
|
||||
membershipsWithTransport$.pipe(
|
||||
mapEpoch((mts) => mts.flatMap(({ transport: t }) => (t ? [t] : []))),
|
||||
),
|
||||
);
|
||||
|
||||
return {
|
||||
membershipsWithTransport$,
|
||||
transports$,
|
||||
};
|
||||
};
|
||||
|
||||
export const createMemberships$ = (
|
||||
scope: ObservableScope,
|
||||
matrixRTCSession: MatrixRTCSession,
|
||||
|
||||
@@ -0,0 +1,68 @@
|
||||
/*
|
||||
Copyright 2026 Element Creations Ltd.
|
||||
|
||||
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
|
||||
Please see LICENSE in the repository root for full details.
|
||||
*/
|
||||
|
||||
import { type Transport } from "matrix-js-sdk/lib/matrixrtc";
|
||||
import { BehaviorSubject } from "rxjs";
|
||||
import { describe, expect, it } from "vitest";
|
||||
|
||||
import { type ResolvedTransport } from "../api";
|
||||
import { type BackendConnection, MediaConnectionState } from "../backend/api";
|
||||
import { testScope } from "../utils/test";
|
||||
import { createTransportRegistry, transportId } from "./Transports";
|
||||
|
||||
const sfu: Transport = { type: "livekit", livekit_service_url: "https://sfu" };
|
||||
const resolved: ResolvedTransport = {
|
||||
type: "livekit",
|
||||
url: "wss://sfu",
|
||||
token: "jwt",
|
||||
roomAlias: "room",
|
||||
identity: "me",
|
||||
};
|
||||
|
||||
describe("transportId", () => {
|
||||
it("does not depend on the order a sender wrote the keys in", () => {
|
||||
expect(transportId({ type: "livekit", livekit_service_url: "u" })).toBe(
|
||||
transportId({ livekit_service_url: "u", type: "livekit" }),
|
||||
);
|
||||
});
|
||||
|
||||
it("tells transports of the same type apart", () => {
|
||||
expect(transportId(sfu)).not.toBe(
|
||||
transportId({ ...sfu, livekit_service_url: "https://other" }),
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
describe("createTransportRegistry", () => {
|
||||
it("hands out one metadata object per transport", () => {
|
||||
const registry = createTransportRegistry(
|
||||
testScope(),
|
||||
new BehaviorSubject<BackendConnection[]>([]),
|
||||
);
|
||||
expect(registry.get(sfu)).toBe(registry.get({ ...sfu }));
|
||||
expect(registry.get(sfu).id).toBe(transportId(sfu));
|
||||
});
|
||||
|
||||
it("derives the connected transports and what was resolved from the backend's connections", () => {
|
||||
const connections$ = new BehaviorSubject<BackendConnection[]>([]);
|
||||
const registry = createTransportRegistry(testScope(), connections$);
|
||||
const metadata = registry.get(sfu);
|
||||
expect(registry.connected$.value).toEqual([]);
|
||||
expect(metadata.resolved$.value).toBeUndefined();
|
||||
|
||||
connections$.next([
|
||||
{ transport: sfu, state: MediaConnectionState.Connecting, resolved },
|
||||
]);
|
||||
expect(registry.connected$.value).toEqual([]);
|
||||
expect(metadata.resolved$.value).toBe(resolved);
|
||||
|
||||
connections$.next([
|
||||
{ transport: sfu, state: MediaConnectionState.Connected, resolved },
|
||||
]);
|
||||
expect(registry.connected$.value).toEqual([metadata]);
|
||||
});
|
||||
});
|
||||
@@ -5,18 +5,13 @@ SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
|
||||
Please see LICENSE in the repository root for full details.
|
||||
*/
|
||||
|
||||
import {
|
||||
isLivekitTransport,
|
||||
type Transport,
|
||||
} from "matrix-js-sdk/lib/matrixrtc";
|
||||
import { combineLatest, distinctUntilChanged, map, of, switchMap } from "rxjs";
|
||||
import { type Transport } from "matrix-js-sdk/lib/matrixrtc";
|
||||
import { distinctUntilChanged, map } from "rxjs";
|
||||
|
||||
import { constant, type Behavior } from "../reactive/Behavior";
|
||||
import { type Behavior } from "../reactive/Behavior";
|
||||
import { type ObservableScope } from "../reactive/ObservableScope";
|
||||
import { ConnectionState } from "./Connection";
|
||||
import { type IConnectionManager } from "./ConnectionManager";
|
||||
import { type TransportMetadata } from "../api";
|
||||
import { ResolvedConnection } from "./ConnectionFactory";
|
||||
import { type MediaBackend, MediaConnectionState } from "../backend/api";
|
||||
|
||||
export interface TransportRegistry {
|
||||
/** The one `TransportMetadata` for a transport, however many memberships name it. */
|
||||
@@ -27,7 +22,7 @@ export interface TransportRegistry {
|
||||
|
||||
export function createTransportRegistry(
|
||||
scope: ObservableScope,
|
||||
connectionManager: IConnectionManager,
|
||||
connections$: MediaBackend["connections$"],
|
||||
): TransportRegistry {
|
||||
const transports = new Map<string, TransportMetadata>();
|
||||
|
||||
@@ -35,29 +30,19 @@ export function createTransportRegistry(
|
||||
const id = transportId(raw);
|
||||
let transport = transports.get(id);
|
||||
if (transport === undefined) {
|
||||
transport = createTransportMetadata(scope, connectionManager, raw, id);
|
||||
transport = createTransportMetadata(scope, connections$, raw, id);
|
||||
transports.set(id, transport);
|
||||
}
|
||||
return transport;
|
||||
};
|
||||
|
||||
const connected$ = scope.behavior(
|
||||
connectionManager.connectionManagerData$.pipe(
|
||||
switchMap(({ value }) => {
|
||||
const connections = value.getConnections();
|
||||
if (connections.length === 0) return of([]);
|
||||
return combineLatest(
|
||||
connections.map((connection) =>
|
||||
connection.state$.pipe(
|
||||
map((state) =>
|
||||
state === ConnectionState.LivekitConnected
|
||||
? get(connection.transport)
|
||||
: null,
|
||||
),
|
||||
),
|
||||
),
|
||||
).pipe(map((transports) => transports.filter((t) => t !== null)));
|
||||
}),
|
||||
connections$.pipe(
|
||||
map((connections) =>
|
||||
connections
|
||||
.filter(({ state }) => state === MediaConnectionState.Connected)
|
||||
.map(({ transport }) => get(transport)),
|
||||
),
|
||||
),
|
||||
);
|
||||
|
||||
@@ -66,28 +51,31 @@ export function createTransportRegistry(
|
||||
|
||||
function createTransportMetadata(
|
||||
scope: ObservableScope,
|
||||
connectionManager: IConnectionManager,
|
||||
connections$: MediaBackend["connections$"],
|
||||
raw: Transport,
|
||||
id: string,
|
||||
): TransportMetadata {
|
||||
const resolved$ = isLivekitTransport(raw)
|
||||
? scope.behavior(
|
||||
connectionManager.connectionManagerData$.pipe(
|
||||
map(({ value }) => value.getConnectionForTransport(raw)),
|
||||
distinctUntilChanged(),
|
||||
switchMap((connection) =>
|
||||
connection instanceof ResolvedConnection
|
||||
? connection.resolved$
|
||||
: of(undefined),
|
||||
),
|
||||
return {
|
||||
type: raw.type,
|
||||
id,
|
||||
raw,
|
||||
resolved$: scope.behavior(
|
||||
connections$.pipe(
|
||||
map(
|
||||
(connections) =>
|
||||
connections.find((c) => transportId(c.transport) === id)?.resolved,
|
||||
),
|
||||
)
|
||||
: constant(undefined);
|
||||
return { type: raw.type, id, raw, resolved$ };
|
||||
distinctUntilChanged(),
|
||||
),
|
||||
),
|
||||
};
|
||||
}
|
||||
|
||||
function transportId(raw: Transport): string {
|
||||
return isLivekitTransport(raw)
|
||||
? raw.livekit_service_url
|
||||
: JSON.stringify(raw);
|
||||
/**
|
||||
* Transports are plain JSON out of membership events, so the same transport
|
||||
* serialises the same way wherever it appears. Keys are sorted so that the
|
||||
* order a sender wrote them in does not matter.
|
||||
*/
|
||||
export function transportId(raw: Transport): string {
|
||||
return JSON.stringify(raw, Object.keys(raw).sort());
|
||||
}
|
||||
|
||||
@@ -6,10 +6,10 @@ Please see LICENSE in the repository root for full details.
|
||||
*/
|
||||
|
||||
import {
|
||||
type LivekitTransport,
|
||||
type MatrixRTCSession as JsSdkRTCSession,
|
||||
type RTCCallIntent,
|
||||
type RTCNotificationType,
|
||||
type Transport,
|
||||
} from "matrix-js-sdk/lib/matrixrtc";
|
||||
import { type CallMembershipIdentityParts } from "matrix-js-sdk/lib/matrixrtc/EncryptionManager";
|
||||
|
||||
@@ -37,7 +37,7 @@ interface Options {
|
||||
export function joinJsSdkSession(
|
||||
session: JsSdkRTCSession,
|
||||
ownMembershipIdentity: CallMembershipIdentityParts,
|
||||
transport: LivekitTransport,
|
||||
transport: Transport,
|
||||
{
|
||||
encryptMedia,
|
||||
matrixRTCMode,
|
||||
|
||||
@@ -8,7 +8,7 @@ Please see LICENSE in the repository root for full details.
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { Status } from "matrix-js-sdk/lib/matrixrtc";
|
||||
|
||||
import { ConnectionState } from "./Connection";
|
||||
import { MediaConnectionState } from "../backend/api";
|
||||
import { MatrixRTCError } from "../api";
|
||||
import {
|
||||
type LocalMemberState,
|
||||
@@ -22,7 +22,7 @@ const publishing: LocalMemberState = {
|
||||
matrix: Status.Connected,
|
||||
};
|
||||
const connecting: LocalMemberState = {
|
||||
media: { connection: ConnectionState.LivekitConnecting },
|
||||
media: { connection: MediaConnectionState.Connecting },
|
||||
matrix: Status.Connecting,
|
||||
};
|
||||
const error = new MatrixRTCError("gone");
|
||||
|
||||
@@ -7,8 +7,6 @@ Please see LICENSE in the repository root for full details.
|
||||
|
||||
import { mockRtcMembership, mockMatrixRoomMember } from "./test";
|
||||
|
||||
export const localRtcMember = mockRtcMembership("@local:example.org", "1111");
|
||||
|
||||
export const aliceDeviceId = "AAAA";
|
||||
export const aliceUserId = "@alice:example.org";
|
||||
export const aliceRtcMember = mockRtcMembership(aliceUserId, aliceDeviceId);
|
||||
|
||||
+1
-21
@@ -20,7 +20,6 @@ import {
|
||||
import {
|
||||
CallMembership,
|
||||
type LivekitFocusSelection,
|
||||
type LivekitTransport,
|
||||
type MatrixRTCSession,
|
||||
MatrixRTCSessionEvent,
|
||||
type MatrixRTCSessionEventHandlerMap,
|
||||
@@ -36,10 +35,6 @@ import {
|
||||
} from "matrix-js-sdk/lib/matrixrtc/IKeyTransport";
|
||||
import { type CallMembershipIdentityParts } from "matrix-js-sdk/lib/matrixrtc/EncryptionManager";
|
||||
import { TypedEventEmitter } from "matrix-js-sdk/lib/models/typed-event-emitter";
|
||||
import {
|
||||
type RemoteParticipant,
|
||||
type RemoteTrackPublication,
|
||||
} from "livekit-client";
|
||||
import { randomUUID } from "crypto";
|
||||
import EventEmitter from "events";
|
||||
|
||||
@@ -159,7 +154,7 @@ export function mockEmitter<T>(): EmitterMock<T> {
|
||||
};
|
||||
}
|
||||
|
||||
export const exampleTransport: LivekitTransport = {
|
||||
const exampleTransport: Transport = {
|
||||
type: "livekit",
|
||||
livekit_service_url: "https://lk.example.org",
|
||||
};
|
||||
@@ -242,21 +237,6 @@ export function mockMatrixRoom(room: Partial<MatrixRoom>): MatrixRoom {
|
||||
return { ...mockEmitter(), ...room } as Partial<MatrixRoom> as MatrixRoom;
|
||||
}
|
||||
|
||||
export function mockRemoteParticipant(
|
||||
participant: Partial<RemoteParticipant>,
|
||||
): RemoteParticipant {
|
||||
return {
|
||||
isLocal: false,
|
||||
setVolume() {},
|
||||
getTrackPublication: () =>
|
||||
({}) as Partial<RemoteTrackPublication> as RemoteTrackPublication,
|
||||
// this will only get used for `getTrackPublications().length`
|
||||
getTrackPublications: () => [0],
|
||||
...mockEmitter(),
|
||||
...participant,
|
||||
} as RemoteParticipant;
|
||||
}
|
||||
|
||||
export class MockRTCSession extends TypedEventEmitter<
|
||||
MatrixRTCSessionEvent | MembershipManagerEvent | KeyTransportEvents,
|
||||
KeyTransportEventsHandlerMap &
|
||||
|
||||
Reference in New Issue
Block a user