Matrix/element-call-Github
1
0
Fork 0
mirror of https://github.com/vector-im/element-call.git synced 2026-10-16 23:45:50 +00:00
Code Issues Packages Projects Releases Wiki Activity

[SDK] split the clint object into -> slot and participation

Browse Source
This commit is contained in:
Timo K.
2026-10-08 13:53:57 +02:00
parent 9cc9eb0889
commit dc983e5493
17 changed files with 653 additions and 415 deletions
Download Patch File Download Diff File Expand all files Collapse all files
sdk/SdkArchitecture.md
+20 -17
View File
@@ -126,23 +126,26 @@ it, and `leave()` also sends the leave.
## Vocabulary
| Word | Meaning here | Elsewhere |
| ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------- |
| client | a `MatrixRTCClient`: one room, one slot, one local member | `MatrixClient` is always written out |
| membership | one device's MatrixRTC state event, seen by hosts as `RTCMembership`, the js-sdk `CallMembership` behind it | |
| member | an `RTCMember`: a membership plus what the SDK derives from it (display name, transport, media) | Element Call's "matrixLivekitMember" |
| media | what a member sends, once it has arrived on its transport: `tracks$` on the member, `null` while waiting for media, and `encryptionError$` beside it. The backend supplies the same two fields | Element Call's `MediaViewModel` is a view of it |
| track | a `MediaTrack`: one publication of a member, audio or video, with `source`, `kind`, `id` and the behaviors on it. `LocalAudioMediaTrack` and `LocalVideoMediaTrack` add the controls over our own | LiveKit `TrackPublication` and `Track`, which never leave the SDK |
| source | a `MediaSource`: what a track is captured from, `microphone`, `camera`, `screenShare`, `screenShareAudio` or `unknown`. Picked out of `tracks$` with `trackBySource$` | LiveKit `Track.Source` |
| publish request | a `PublishRequest`: a source, where to capture it from and how to encode it. Given at the join in `options.publish`, or later to `publish` on the local member | |
| active | `isActive$` on an audio track: the track carries sound, as the backend measures it. "Speaking" is a call's reading of it on the microphone track | LiveKit "speaking", `isSpeaking` |
| attach, detach | handing a `<video>` or `<audio>` element to a track and taking it back. The SDK owns the stream and the observers on the element | LiveKit `Track.attach` |
| transport | where media is exchanged: a LiveKit service url today, described by `TransportMetadata` | MSC4143 "focus" |
| media backend | a `MediaBackend`: what carries the media, behind `sdk/src/media-backend/api.ts`. One per client, chosen with `options.backend`; LiveKit under `sdk/src/media-backend/livekit/` is the one that exists | "SFU", a LiveKit room, a peer connection |
| application, slot | what the session is for (`m.call`) and which one of them in the room (`ROOM`) | MSC4143 |
The words the SDK is written in. They name the shape after the slot and
participation split in
[`client-into-slot-session-split-plan.md`](./client-into-slot-session-split-plan.md);
the sections below still describe `MatrixRTCClient` until that lands.
Names use `RTC` in capitals, as the js-sdk does: `MatrixRTCClient`, `RTCMember`,
`matrixRTCMode`.
| Word | Meaning here | Elsewhere |
| --------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------- |
| slot | an `RTCSlot`: the namespace in a room that memberships join into, `application` plus `id`. It exists while the room does, with nobody in it too. Holds `members$`, its open or closed `status$`, and `join()` | the js-sdk `MatrixRTCSession` is one per slot; the MSC4143 slot event |
| participation | an `RTCParticipation`: the span between our `join()` of a slot and our `leave()`. Holds the connection and its `status$`, `localMember$`, `remoteMembers$`, the connected transports and the data channel. Made by `slot.join()`, never reused | "session" is not used here, because in the js-sdk it means the slot |
| member | an `RTCMember`: one membership in a slot with what the SDK derives from it, `rtcBackendIdentity`, `userId`, `deviceId`, `memberId`, `displayName$`, `avatarUrl$`, `transport$` and `applicationData$`. No media. What the slot hands out in `members$` | the js-sdk `CallMembership` behind it, which never leaves the SDK; Element Call's old "matrixLivekitMember" |
| local member, remote member | `LocalRTCMember` and `RemoteRTCMember`: an `RTCMember` plus its `MemberMedia`, which only a participation can carry. The local one's tracks carry the controls. What the participation hands out in `localMember$` and `remoteMembers$` | LiveKit `LocalParticipant` and `RemoteParticipant`, which never leave the SDK |
| media | a `MemberMedia`: what a member sends once it has arrived on its transport, `tracks$`, null while waiting, and `encryptionError$` beside it. The media backend supplies exactly these two fields per member | Element Call's `MediaViewModel` is a view of it |
| track | a `MediaTrack`: one publication of a member, audio or video, with `source`, `kind`, `id` and the behaviors on it. `LocalAudioMediaTrack` and `LocalVideoMediaTrack` add the controls over our own | LiveKit `TrackPublication` and `Track`, which never leave the SDK |
| publish request | a `PublishRequest`: a source, where to capture it from and how to encode it. Given at the join in `publish`, or later to `publish` on the participation | |
| attach, detach | handing a `<video>` or `<audio>` element to a track and taking it back. The SDK owns the stream and the observers on the element | LiveKit `Track.attach` |
| transport | an `RTCTransport`: where media is exchanged, a LiveKit service url today. Carries the raw js-sdk `Transport` from the membership and, in `resolved$`, what the backend fetched to connect to it | MSC4143 "focus" |
| media backend | a `MediaBackend`: what carries the media, behind `sdk/src/media-backend/api.ts`. One per participation, chosen with `backend` in the join options; LiveKit under `sdk/src/media-backend/livekit/` is the one that exists | "SFU", a LiveKit room, a peer connection |
Names use `RTC` in capitals, as the js-sdk does: `RTCSlot`, `RTCParticipation`,
`RTCMember`, `matrixRTCMode`.
## Reactive primitives
@@ -181,7 +184,7 @@ context, it is in the view model.
### Creating one
````ts
```ts
import { type MatrixClient, type Room } from "matrix-js-sdk";
import { type RTCNotificationType } from "matrix-js-sdk/lib/matrixrtc";
sdk/dev/main.ts
+38 -28
View File
@@ -15,12 +15,13 @@ import { logger } from "matrix-js-sdk/lib/logger";
import { combineLatest, type Observable, of, switchMap } from "rxjs";
import {
type AudioMediaTrack,
createMatrixRTCClient,
createRTCSlot,
E2eeType,
type LocalRTCMember,
MatrixRTCMode,
ObservableScope,
type RTCMember,
type MatrixRTCClient,
type RemoteRTCMember,
type RTCParticipation,
trackBySource$,
type VideoMediaTrack,
} from "@element-hq/matrixrtc-sdk";
@@ -69,33 +70,34 @@ async function start(
const room = await joinRoom(client, roomIdOrAlias);
const scope = new ObservableScope();
const rtcClient = createMatrixRTCClient(scope, client, room, {
const slot = createRTCSlot(scope, client, room, {
encryptionSystem: { kind: E2eeType.PER_PARTICIPANT },
matrixRTCMode: MatrixRTCMode.Compatibility,
});
const rtcParticipation = slot.join({
publish: [{ source: "microphone" }, { source: "camera" }],
});
rtcClient.status$.pipe(scope.bind()).subscribe((s) => {
rtcParticipation.status$.pipe(scope.bind()).subscribe((s) => {
status.textContent = s;
});
rtcClient.fatalError$.pipe(scope.bind()).subscribe((error) => {
rtcParticipation.fatalError$.pipe(scope.bind()).subscribe((error) => {
if (error !== null) status.textContent = `Error: ${error.message}`;
});
showMembers(scope, rtcClient);
showMessages(scope, rtcClient);
rtcClient.join();
showMembers(scope, rtcParticipation);
showMessages(scope, rtcParticipation);
toggle(buttons.microphone, async (enabled) =>
setPublished(rtcClient, "microphone", enabled),
setPublished(rtcParticipation, "microphone", enabled),
);
toggle(buttons.camera, async (enabled) =>
setPublished(rtcClient, "camera", enabled),
setPublished(rtcParticipation, "camera", enabled),
);
toggle(buttons.screenShare, async (enabled) =>
setPublished(rtcClient, "screenShare", enabled),
setPublished(rtcParticipation, "screenShare", enabled),
);
buttons.leave.hidden = false;
buttons.leave.onclick = (): void => {
rtcClient.leave();
rtcParticipation.leave();
scope.end();
members.replaceChildren();
messages.replaceChildren();
@@ -115,20 +117,20 @@ async function start(
* published and unpublished.
*/
async function setPublished(
rtcClient: MatrixRTCClient,
rtcParticipation: RTCParticipation,
source: "microphone" | "camera" | "screenShare",
enabled: boolean,
): Promise<boolean> {
const member = rtcClient.localMember$.value;
if (member === null) return !enabled;
const track = member.tracks$.value?.find((t) => t.source === source);
const track = rtcParticipation.localMember$.value?.tracks$.value?.find(
(t) => t.source === source,
);
if (track && source === "screenShare") {
if (!enabled) await member.unpublish(track.id);
if (!enabled) await rtcParticipation.unpublish(source);
return enabled;
}
if (track) return track.setEnabled(enabled);
if (!enabled) return false;
await member.publish({ source });
await rtcParticipation.publish({ source });
return true;
}
@@ -148,20 +150,22 @@ function toggle(
/** Sends what the form holds on the "chat" topic, and lists what arrives. */
function showMessages(
scope: ObservableScope,
rtcClient: MatrixRTCClient,
rtcParticipation: RTCParticipation,
): void {
dataForm.hidden = false;
dataForm.onsubmit = (event): void => {
event.preventDefault();
const text = new FormData(dataForm).get("text") as string;
rtcClient.sendData("chat", text).then(
rtcParticipation.sendData("chat", text).then(
() => dataForm.reset(),
(e: unknown) => {
status.textContent = `Error: ${e}`;
},
);
};
rtcClient.data$.pipe(scope.bind()).subscribe(({ member, topic, text }) => {
rtcParticipation.data$
.pipe(scope.bind())
.subscribe(({ member, topic, text }) => {
const line = messages.appendChild(document.createElement("li"));
line.dataset.testid = "message";
line.dataset.topic = topic;
@@ -170,27 +174,33 @@ function showMessages(
});
}
function showMembers(scope: ObservableScope, rtcClient: MatrixRTCClient): void {
function showMembers(
scope: ObservableScope,
rtcParticipation: RTCParticipation,
): void {
const tiles = new Map<string, HTMLElement>();
combineLatest([rtcClient.localMember$, rtcClient.remoteMembers$])
combineLatest([rtcParticipation.localMember$, rtcParticipation.remoteMembers$])
.pipe(scope.bind())
.subscribe(([local, remote]) => {
const current = local === null ? remote : [local, ...remote];
for (const [id, tile] of tiles)
if (!current.some((m) => m.id === id)) {
if (!current.some((m) => m.rtcBackendIdentity === id)) {
tile.remove();
tiles.delete(id);
}
for (const member of current)
if (!tiles.has(member.id)) {
if (!tiles.has(member.rtcBackendIdentity)) {
const tile = memberTile(scope, member);
tiles.set(member.id, tile);
tiles.set(member.rtcBackendIdentity, tile);
members.append(tile);
}
});
}
function memberTile(scope: ObservableScope, member: RTCMember): HTMLElement {
function memberTile(
scope: ObservableScope,
member: LocalRTCMember | RemoteRTCMember,
): HTMLElement {
const tile = document.createElement("section");
tile.dataset.testid = "member";
tile.dataset.userId = member.userId;
sdk/index.ts
+1 -1
View File
@@ -56,7 +56,7 @@ export * from "./src/api";
export * from "./src/media-api";
export * from "./src/media-backend/api";
export { trackBySource$, type TrackOfSource } from "./src/utils/tracks";
export { createMatrixRTCClient } from "./src/MatrixRTCClient";
export { createRTCSlot } from "./src/RTCSlot";
export {
createLivekitBackend,
type LivekitBackendOptions,
sdk/src/MatrixRTCClient.ts → sdk/src/RTCParticipation.ts
+101 -115
View File
@@ -6,29 +6,40 @@ Please see LICENSE in the repository root for full details.
*/
import { type MatrixClient, type Room } from "matrix-js-sdk";
import { type Logger, logger as rootLogger } from "matrix-js-sdk/lib/logger";
import { type Logger } from "matrix-js-sdk/lib/logger";
import {
type MatrixRTCSession as JsSdkRTCSession,
MatrixRTCSessionEvent,
MatrixRTCSessionManager,
MembershipManagerEvent,
} from "matrix-js-sdk/lib/matrixrtc";
import { type IMembershipManager } from "matrix-js-sdk/lib/matrixrtc/IMembershipManager";
import { type CallMembershipIdentityParts } from "matrix-js-sdk/lib/matrixrtc/EncryptionManager";
import { v4 as uuidv4 } from "uuid";
import { combineLatest, filter, from, fromEvent, map, Observable } from "rxjs";
import {
BehaviorSubject,
combineLatest,
filter,
from,
fromEvent,
map,
Observable,
} from "rxjs";
import { defaultSessionTimings, MatrixRTCMode } from "./config";
import { E2eeType } from "./encryption";
import { type Behavior } from "./reactive/Behavior";
import { type ObservableScope } from "./reactive/ObservableScope";
import { filterBehavior, generateItems } from "./reactive/observable";
import { generateItems } from "./reactive/observable";
import { mapScoped } from "./utils/mapScoped";
import {
type DataMessage,
type LocalRTCMember,
type MatrixRTCClient,
type MatrixRTCClientOptions,
type RemoteRTCMember,
type RTCMember,
type RTCParticipation,
type RTCParticipationOptions,
type RTCSlot,
type RTCSlotOptions,
} from "./api";
import {
type MediaBackend,
@@ -36,7 +47,6 @@ import {
type MediaKey,
} from "./media-backend/api";
import { createLivekitBackend } from "./media-backend/livekit/LivekitBackend";
import { MatrixRTCError } from "./errors";
import { createHomeserverConnected$ } from "./matrixrtc/HomeserverConnected";
import { joinJsSdkSession } from "./matrixrtc/joinJsSdkSession";
import {
@@ -44,49 +54,55 @@ import {
type PreparedTransport,
} from "./matrixrtc/LocalMember";
import { discoverLocalTransport } from "./matrixrtc/LocalTransport";
import {
createMatrixMemberMetadata$,
createRoomMembers$,
} from "./matrixrtc/MatrixMemberMetadata";
import {
createLocalRTCMember,
createRemoteRTCMember,
membershipKeys,
} from "./matrixrtc/Members";
import {
createKeyRotationSuppressed$,
createMemberships$,
} from "./matrixrtc/JsRtcSessionBehaviors";
import { fatalError, sessionStatus } from "./matrixrtc/status";
import { createTransportRegistry } from "./matrixrtc/Transports";
import { createKeyRotationSuppressed$ } from "./matrixrtc/JsRtcSessionBehaviors";
import { fatalError, participationStatus } from "./matrixrtc/status";
import { type TransportRegistry } from "./matrixrtc/Transports";
/** What the slot lends a participation. */
export interface ParticipationDeps {
client: MatrixClient;
room: Room;
slot: RTCSlot;
jsSdkSession: JsSdkRTCSession;
members$: Behavior<RTCMember[]>;
transports: TransportRegistry;
slotOptions: Pick<RTCSlotOptions, "matrixRTCMode" | "encryptionSystem">;
logger: Logger;
}
/** The participation, plus what the slot reads from it without exposing it. */
export interface ParticipationInternals {
participation: RTCParticipation;
connections$: MediaBackend["connections$"];
}
/**
* Takes the whole `MatrixClient` rather than a slice of it, and finds the
* MatrixRTC session itself. The js-sdk is the MatrixRTC implementation today;
* when the rust-rtc crate replaces it, the SDK has to bridge the crate to the
* js-sdk client, and only the SDK knows what that bridge needs. Holding the
* client keeps that change inside the SDK.
* Our span in a slot: the backend, the local transport, the join state
* machine and the members with the media it carries for them. Starts joining
* at once and ends with `leave()`, which also ends `scope`.
*/
export function createMatrixRTCClient(
export function createRTCParticipation(
scope: ObservableScope,
client: MatrixClient,
room: Room,
options: MatrixRTCClientOptions,
): MatrixRTCClient {
const logger = rootLogger.getChild("[MatrixRTCClient]");
const userId = client.getUserId();
const deviceId = client.getDeviceId();
if (!(userId && deviceId))
throw new MatrixRTCError("The client has to be logged in");
const { encryptionSystem, matrixRTCMode } = options;
const timings = { ...defaultSessionTimings, ...options.timings };
const jsSdkSession = sessionManager(
scope,
{
client,
options,
logger,
).getRoomSession(room);
room,
slot,
jsSdkSession,
members$,
transports,
slotOptions: { matrixRTCMode, encryptionSystem },
logger: parentLogger,
}: ParticipationDeps,
options: RTCParticipationOptions,
): ParticipationInternals {
const logger = parentLogger.getChild("[RTCParticipation]");
const userId = client.getUserId()!;
const deviceId = client.getDeviceId()!;
const timings = { ...defaultSessionTimings, ...options.timings };
const ownMembershipIdentity: CallMembershipIdentityParts = {
userId,
@@ -100,7 +116,7 @@ export function createMatrixRTCClient(
: `${userId}:${deviceId}`,
};
const backend = createBackend(scope, client, options, {
const backend = createBackend(scope, client, options, matrixRTCMode, {
roomId: room.roomId,
ownMembershipIdentity,
publish: options.publish,
@@ -110,9 +126,6 @@ export function createMatrixRTCClient(
logger,
});
const memberships$ = createMemberships$(scope, jsSdkSession);
const transports = createTransportRegistry(scope, backend.connections$);
const preparedTransport$ = from(
discoverLocalTransport(client, backend.transportType, options, logger).then(
async (transport): Promise<PreparedTransport> => ({
@@ -162,56 +175,40 @@ export function createMatrixRTCClient(
logger,
});
const context = {
metadata: createMatrixMemberMetadata$(
scope,
scope.behavior(memberships$.pipe(map(({ value }) => value))),
createRoomMembers$(scope, room),
),
transports,
};
const remoteMembers$ = scope.behavior<RemoteRTCMember[]>(
memberships$.pipe(
map(({ value }) => value),
members$.pipe(
generateItems(
"MatrixRTCClient remoteMembers",
function* (memberships) {
for (const membership of memberships) {
if (
membership.userId === userId &&
membership.deviceId === deviceId
)
continue;
yield { keys: membershipKeys(membership), data: membership };
}
"RTCParticipation remoteMembers",
function* (members) {
for (const member of members)
if (!member.local) yield { keys: [member] as const, data: member };
},
(memberScope, membership$) =>
(memberScope, member$) =>
createRemoteRTCMember(
memberScope,
membership$,
backend.mediaFor$(memberScope, membership$),
context,
member$.value,
backend.mediaFor(memberScope, member$.value),
),
),
),
);
const localMembership$ = scope.behavior(
memberships$.pipe(
// Matched by the identity we joined with: a stale membership of ours from
// an earlier participation is `local` in the slot but is not this one
const ownMember$ = scope.behavior(
members$.pipe(
map(
({ value }) =>
value.find(
(membership) =>
membership.userId === userId && membership.deviceId === deviceId,
(members) =>
members.find(
(member) =>
member.local &&
member.memberId === ownMembershipIdentity.memberId,
) ?? null,
),
filterBehavior((membership) => membership !== null),
),
);
const localMember$ = scope.behavior<LocalRTCMember | null>(
mapScoped(scope, localMembership$, (memberScope, membership$) =>
createLocalRTCMember(memberScope, membership$, backend.local, context),
mapScoped(scope, ownMember$, (_memberScope, member) =>
createLocalRTCMember(member, backend.local),
).pipe(map((member) => member ?? null)),
);
@@ -219,7 +216,9 @@ export function createMatrixRTCClient(
// ever hears from attested members
const data$ = backend.data$.pipe(
map(({ senderId, topic, text }): DataMessage | null => {
const member = remoteMembers$.value.find((m) => m.id === senderId);
const member = remoteMembers$.value.find(
(m) => m.rtcBackendIdentity === senderId,
);
if (member === undefined) {
logger.warn(`Dropping data from ${senderId}: not a member`);
return null;
@@ -229,21 +228,31 @@ export function createMatrixRTCClient(
filter((message) => message !== null),
);
const left$ = new BehaviorSubject(false);
const status$ = scope.behavior(
combineLatest(
[
localMembership.state$,
localMembership.joinRequested$,
localMembership.connected$,
localMembership.reconnecting$,
left$,
],
sessionStatus,
participationStatus,
),
);
const leave = (): void => {
if (left$.value) return;
left$.next(true);
localMembership.leave();
scope.end();
};
return {
join: localMembership.requestJoinAndPublish,
leave: localMembership.requestDisconnect,
connections$: backend.connections$,
participation: {
slot,
leave,
status$,
connected$: localMembership.connected$,
reconnecting$: localMembership.reconnecting$,
@@ -251,26 +260,24 @@ export function createMatrixRTCClient(
fatalError$: scope.behavior(localMembership.state$.pipe(map(fatalError))),
localMember$,
remoteMembers$,
memberCount$: scope.behavior(
combineLatest(
[localMember$, remoteMembers$],
(local, remote) => remote.length + (local === null ? 0 : 1),
),
),
publish: async (request) => backend.local.publish(request),
unpublish: async (source) => backend.local.unpublish(source),
keyRotationSuppressed$: createKeyRotationSuppressed$(scope, jsSdkSession),
connectedTransports$: transports.connected$,
setAudioOutputDeviceId: async (deviceId) =>
backend.setAudioOutputDeviceId(deviceId),
sendData: async (topic, text) => backend.sendData(topic, text),
data$,
},
};
}
/** The host's backend, or LiveKit configured from the client options. */
/** The host's backend, or LiveKit configured from the participation options. */
function createBackend(
scope: ObservableScope,
client: MatrixClient,
{ backend, mediaQuality, matrixRTCMode }: MatrixRTCClientOptions,
{ backend, mediaQuality }: RTCParticipationOptions,
matrixRTCMode: MatrixRTCMode,
context: MediaBackendContext,
): MediaBackend {
if (backend) return backend(scope, context);
@@ -282,7 +289,7 @@ function createBackend(
});
}
/** The session's per-participant keys: the ones it already holds on subscribe, then each new one. */
/** The slot's per-participant keys: the ones it already holds on subscribe, then each new one. */
function mediaKeys$(session: JsSdkRTCSession): Observable<MediaKey> {
return new Observable((subscriber) => {
const onKey = (
@@ -296,24 +303,3 @@ function mediaKeys$(session: JsSdkRTCSession): Observable<MediaKey> {
return () => session.off(MatrixRTCSessionEvent.EncryptionKeyChanged, onKey);
});
}
/**
* The js-sdk keeps one session manager per slot. The client comes with the
* manager for the default slot; any other slot gets a manager of its own,
* which lives as long as the scope.
*/
function sessionManager(
scope: ObservableScope,
client: MatrixClient,
{ application = "m.call", slot = "ROOM" }: MatrixRTCClientOptions,
logger: Logger,
): MatrixRTCSessionManager {
if (application === "m.call" && slot === "ROOM") return client.matrixRTC;
const manager = new MatrixRTCSessionManager(logger, client, {
application,
id: slot,
});
manager.start();
scope.onEnd(() => manager.stop());
return manager;
}
sdk/src/RTCSlot.ts
+185
View File
@@ -0,0 +1,185 @@
/*
Copyright 2026 Element Creations Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { type MatrixClient, type Room } from "matrix-js-sdk";
import { EventType } from "matrix-js-sdk/lib/@types/event";
import { type Logger, logger as rootLogger } from "matrix-js-sdk/lib/logger";
import {
type MatrixRTCSession as JsSdkRTCSession,
MatrixRTCSessionManager,
type SlotDescription,
} from "matrix-js-sdk/lib/matrixrtc";
import { type MatrixEvent } from "matrix-js-sdk/lib/models/event";
import { RoomStateEvent } from "matrix-js-sdk/lib/models/room-state";
import { BehaviorSubject, filter, fromEvent, map, of, switchMap } from "rxjs";
import { ObservableScope } from "./reactive/ObservableScope";
import { generateItems } from "./reactive/observable";
import {
type RTCMember,
type RTCParticipation,
type RTCParticipationOptions,
type RTCSlot,
type RTCSlotOptions,
} from "./api";
import { MatrixRTCError } from "./errors";
import {
createRTCParticipation,
type ParticipationInternals,
} from "./RTCParticipation";
import { createMemberships$ } from "./matrixrtc/JsRtcSessionBehaviors";
import {
createMatrixMemberMetadata$,
createRoomMembers$,
} from "./matrixrtc/MatrixMemberMetadata";
import {
createRTCMember,
type MemberContext,
membershipKeys,
} from "./matrixrtc/Members";
import { createTransportRegistry } from "./matrixrtc/Transports";
/**
* Takes the whole `MatrixClient` rather than a slice of it, and finds the
* MatrixRTC session itself. The js-sdk is the MatrixRTC implementation today;
* when the rust-rtc crate replaces it, the SDK has to bridge the crate to the
* js-sdk client, and only the SDK knows what that bridge needs. Holding the
* client keeps that change inside the SDK.
*/
export function createRTCSlot(
scope: ObservableScope,
client: MatrixClient,
room: Room,
options: RTCSlotOptions,
): RTCSlot {
const logger = rootLogger.getChild("[RTCSlot]");
const userId = client.getUserId();
const deviceId = client.getDeviceId();
if (!(userId && deviceId))
throw new MatrixRTCError("The client has to be logged in");
const { application = "m.call", id = "ROOM" } = options;
const jsSdkSession = sessionManager(
scope,
client,
{ application, id },
logger,
).getRoomSession(room);
const memberships$ = createMemberships$(scope, jsSdkSession);
const active$ = new BehaviorSubject<ParticipationInternals | null>(null);
const participationScopes = new Set<ObservableScope>();
scope.onEnd(() => {
for (const participationScope of participationScopes)
participationScope.end();
});
const transports = createTransportRegistry(
scope,
scope.behavior(
active$.pipe(switchMap((active) => active?.connections$ ?? of([]))),
),
);
const context: MemberContext = {
metadata: createMatrixMemberMetadata$(
scope,
scope.behavior(memberships$.pipe(map(({ value }) => value))),
createRoomMembers$(scope, room),
),
transports,
own: { userId, deviceId },
};
const members$ = scope.behavior<RTCMember[]>(
memberships$.pipe(
map(({ value }) => value),
generateItems(
"RTCSlot members",
function* (memberships) {
for (const membership of memberships)
yield { keys: membershipKeys(membership), data: membership };
},
(memberScope, membership$) =>
createRTCMember(memberScope, membership$, context),
),
),
);
const status$ = scope.behavior(
fromEvent(room, RoomStateEvent.Events, (event: MatrixEvent) => event).pipe(
filter((event) => event.getType() === EventType.RTCSlot),
map(() => slotStatus(jsSdkSession)),
),
slotStatus(jsSdkSession),
);
const slot: RTCSlot = {
roomId: room.roomId,
application,
id,
status$,
members$,
participation$: scope.behavior(
active$.pipe(map((active) => active?.participation ?? null)),
),
join: (participationOptions: RTCParticipationOptions): RTCParticipation => {
if (active$.value !== null)
throw new MatrixRTCError(
"Already in this slot: leave the participation before joining again",
);
const participationScope = new ObservableScope();
participationScopes.add(participationScope);
const internals = createRTCParticipation(
participationScope,
{
client,
room,
slot,
jsSdkSession,
members$,
transports,
slotOptions: options,
logger,
},
participationOptions,
);
participationScope.onEnd(() => {
participationScopes.delete(participationScope);
if (active$.value === internals) active$.next(null);
});
active$.next(internals);
return internals.participation;
},
};
return slot;
}
function slotStatus(
session: Pick<JsSdkRTCSession, "getRtcSlot">,
): "open" | "closed" | undefined {
return session.getRtcSlot()?.status;
}
/**
* The js-sdk keeps one session manager per slot. The client comes with the
* manager for the default slot; any other slot gets a manager of its own,
* which lives as long as the scope.
*/
function sessionManager(
scope: ObservableScope,
client: MatrixClient,
{ application, id }: SlotDescription,
logger: Logger,
): MatrixRTCSessionManager {
if (application === "m.call" && id === "ROOM") return client.matrixRTC;
const manager = new MatrixRTCSessionManager(logger, client, {
application,
id,
});
manager.start();
scope.onEnd(() => manager.stop());
return manager;
}
sdk/src/api.ts
+118 -90
View File
@@ -6,13 +6,12 @@ Please see LICENSE in the repository root for full details.
*/
/**
* The public types of the SDK: the client, its members and transports. What a
* member sends is in `media-api.ts`. `createMatrixRTCClient` in
* `MatrixRTCClient.ts` is the only way to obtain an implementation of them.
* The public types of the SDK: the slot, the participation, their members and
* transports. What a member sends is in `media-api.ts`. `createRTCSlot` in
* `RTCSlot.ts` is the only way to obtain an implementation of them.
*/
import {
type CallMembership,
type RTCNotificationType,
type Transport,
} from "matrix-js-sdk/lib/matrixrtc";
@@ -20,12 +19,12 @@ import { type Observable } from "rxjs";
import { type Behavior } from "./reactive/Behavior";
import {
type AudioMediaTrack,
type EncryptionError,
type LocalAudioMediaTrack,
type LocalMemberMedia,
type LocalVideoMediaTrack,
type MediaSource,
type MemberMedia,
type PublishRequest,
type VideoMediaTrack,
} from "./media-api";
import { type MediaBackendFactory } from "./media-backend/api";
import { type EncryptionSystem } from "./encryption";
@@ -34,26 +33,72 @@ import {
type MediaQuality,
type SessionTimings,
} from "./config";
import { type MatrixRTCError } from "./errors";
export { MatrixRTCError } from "./errors";
// ---------------------------------------------------------------------------
// MatrixRTCClient
// Slot
export interface MatrixRTCClientOptions {
encryptionSystem: EncryptionSystem;
/** Resolved by the host; the SDK reads neither config.json nor settings. */
export interface RTCSlotOptions {
/** The application the slot belongs to, as named in the membership. Default `m.call`. */
application?: string;
/** The application's slot in the room. Default `ROOM`. */
id?: string;
/** How memberships and keys in this slot are written. Resolved by the host. */
matrixRTCMode: MatrixRTCMode;
/** Published at the join; `publish` on the local member adds to it from then on. */
encryptionSystem: EncryptionSystem;
}
/**
* The namespace in a room that memberships join into. It exists while the
* room does, with nobody in it too. The room has to be the one the client's
* sync loop maintains (`client.getRoom(roomId)`, once the join has synced):
* the detached copy `joinRoom` returns for a room joined just now never
* receives the state the members are read from.
*/
export interface RTCSlot {
readonly roomId: string;
readonly application: string;
readonly id: string;
/** From the slot state event; undefined while the room has none. */
status$: Behavior<"open" | "closed" | undefined>;
/**
* Everyone in the slot, one object per membership, kept for as long as the
* membership is. Our own membership is in here too, with `local: true`,
* once the room has seen it. No media: that is the participation's.
*/
members$: Behavior<RTCMember[]>;
/** The participation we are in, null between `leave()` and the next `join()`. */
participation$: Behavior<RTCParticipation | null>;
/**
* Joins the slot and returns the participation that lasts until its
* `leave()`. Throws while `participation$` is not null: a slot has one local
* member.
*/
join(options: RTCParticipationOptions): RTCParticipation; // Maybe MediaSession,
}
// ---------------------------------------------------------------------------
// Participation
export interface RTCParticipationOptions {
/**
* Published at the join; `publish` on the participation adds to it from then
* on. Kept as an option, rather than only the method, because the LiveKit
* backend reads the initial requests to build the room's default capture
* options before the first connection exists.
*/
publish: PublishRequest[];
/**
* LEGACY
*
* Conside already removing it. just skips code in js-sdk. But rust sdk will not have this feature.
* MSC4075 notification sent with the join. A parameter of the MatrixRTC
* join itself, so it is here even though it is named after calls; reacting
* to a notification (ringing, timeouts, declines) is the application's job.
*/
sendNotificationType?: RTCNotificationType;
/** The application the session belongs to, as named in the membership. Default `m.call`. */
application?: string;
/** The application's slot in the room. Default `ROOM`. */
slot?: string;
/**
* Whatever the application wants to say about itself in the membership,
* under namespaced keys. Opaque to the SDK, apart from `m.call.intent`
@@ -61,7 +106,7 @@ export interface MatrixRTCClientOptions {
* field of its own.
*/
applicationData?: Record<string, unknown>;
/** Session timings the host has configured; the defaults otherwise. */
/** Timings the host has configured; the defaults otherwise. */
timings?: Partial<SessionTimings>;
/** Limits on what is published; LiveKit's defaults otherwise. */
mediaQuality?: MediaQuality;
@@ -76,51 +121,58 @@ export interface MatrixRTCClientOptions {
backend?: MediaBackendFactory;
}
export type ConnectionStatus =
export type ParticipationStatus =
| "waitingForTransport"
| "connecting"
| "connected"
| "reconnecting"
| "disconnected";
| "disconnected"
/** `leave()` was called; the participation is over and stays so. */
| "left";
/** Why the client is not connected: the first failing of its three links. */
/** Why the participation is not connected: the first failing of its three links. */
export type DisconnectReason = "sync" | "membership" | "probablyLeft" | "media";
import { type MatrixRTCError } from "./errors";
export { MatrixRTCError } from "./errors";
/**
* A session in one room. The room has to be the one the client's sync loop
* maintains (`client.getRoom(roomId)`, once the join has synced): the
* detached copy `joinRoom` returns for a room joined just now never receives
* the state the members are read from.
*/
export interface MatrixRTCClient {
join(): void;
/** The span between our `join()` of a slot and our `leave()`. */
export interface RTCParticipation {
readonly slot: RTCSlot;
/** Sends the leave, tears the connections down and ends the participation. */
leave(): void;
/** Collapsed view of the local member's state machine. */
status$: Behavior<ConnectionStatus>;
status$: Behavior<ParticipationStatus>;
connected$: Behavior<boolean>;
reconnecting$: Behavior<boolean>;
/** Null while connected, the first failing link otherwise. */
disconnectReason$: Behavior<DisconnectReason | null>;
/** A transport, Matrix or connection error that stops the session. */
/** A transport, Matrix or connection error that stops the participation. */
fatalError$: Behavior<MatrixRTCError | null>;
/** Null until our own membership has been seen in the room. */
localMember$: Behavior<LocalRTCMember | null>;
/**
* The slot's other members, each with the media this participation carries
* for it. A call view needs nothing but this and `localMember$` to render.
*/
remoteMembers$: Behavior<RemoteRTCMember[]>;
/** `remoteMembers.length`, plus one for the local member once it exists. */
memberCount$: Behavior<number>;
/**
* Whether the session has grown large enough that MatrixRTC has stopped
* Publishes a source and resolves with its track, which also shows up in
* `localMember$.tracks$`. Works from the moment `join()` returns: before the
* transport is connected.
*/
publish(
request: PublishRequest,
): Promise<LocalAudioMediaTrack | LocalVideoMediaTrack>;
/** Removes one of our publications; a screen share takes its audio with it. */
unpublish(source: MediaSource): Promise<void>;
/**
* Whether the slot has grown large enough that MatrixRTC has stopped
* rotating the media encryption key.
*/
keyRotationSuppressed$: Behavior<boolean>;
/** Transports the session currently holds a live connection to. */
/** Transports the participation currently holds a live connection to. */
connectedTransports$: Behavior<TransportMetadata[]>;
/**
@@ -139,9 +191,9 @@ export interface MatrixRTCClient {
*/
sendData(topic: string, text: string): Promise<void>;
/**
* What remote members sent with `sendData`, on every transport the client
* is connected to. A message from an identity that is not a member is
* dropped, so a host only ever hears from attested members.
* What remote members sent with `sendData`, on every transport the
* participation is connected to. A message from an identity that is not a
* member is dropped, so a host only ever hears from attested members.
*/
data$: Observable<DataMessage>;
}
@@ -164,7 +216,7 @@ export interface DataMessage {
export interface TransportMetadata {
/** `"livekit"` today. */
type: string;
/** Stable key, unique per transport in the session: the raw transport, serialised with sorted keys. */
/** Stable key, unique per transport in the slot: the raw transport, serialised with sorted keys. */
id: string;
/** The transport object as it appears in the membership. */
raw: Transport;
@@ -190,63 +242,39 @@ export type ResolvedTransport =
// ---------------------------------------------------------------------------
// Members
/**
* What a membership says, as hosts read it. The js-sdk `CallMembership` is
* behind it; anything beyond this is the js-sdk's API, not the SDK's.
*/
export type RTCMembership = Pick<
CallMembership,
| "userId"
| "deviceId"
| "memberId"
| "rtcBackendIdentity"
| "application"
| "applicationData"
| "getTransport"
| "transports"
| "createdTs"
| "getAbsoluteExpiry"
>;
/** A membership in the slot, with what the SDK derives from it. No media. */
export interface RTCMember {
local: boolean;
/** The identity the media backend knows this member by. */
id: string;
/**
* The identity this member has on the media backend's transport: the
* participant identity the SFU sees, and the sender of its data packets. The
* SDK uses it to match what arrives on the transport back to the member; a
* host needs it only as a stable key per membership or as a debug label.
*
* Today it is read off the membership, where the js-sdk computes it:
* `${userId}:${deviceId}` before sticky events, a hash of `memberId` in
* Matrix 2.0 mode. Eventually the backend will derive it itself, so treat it
* as opaque.
*/
rtcBackendIdentity: string;
userId: string;
deviceId: string;
membership$: Behavior<RTCMembership>;
displayName$: Behavior<string>;
avatarUrl$: Behavior<string | undefined>;
/** The membership's own id; what the Matrix 2.0 identity is derived from. */
memberId: string;
/** Which transport this member is on; undefined when the membership has none. */
transport$: Behavior<TransportMetadata | undefined>;
/**
* The member's tracks, in publication order, once it has shown up on its
* transport; null until then ("waiting for media"). An entry stays the same
* object for as long as the same publication is behind it. Which track is
* which is in its `source`; `trackBySource$` picks one out.
*/
tracks$: Behavior<(AudioMediaTrack | VideoMediaTrack)[] | null>;
/** Emits when the SFU reports a key problem for this member. */
encryptionError$: Observable<EncryptionError>;
/** What the member's application says about itself, e.g. `m.call.intent`. */
applicationData$: Behavior<Record<string, unknown>>;
displayName$: Behavior<string>;
avatarUrl$: Behavior<string | undefined>;
}
export interface RemoteRTCMember extends RTCMember {
export interface RemoteRTCMember extends RTCMember, MemberMedia {
local: false;
}
export interface LocalRTCMember extends RTCMember {
/** Our member: the same as a remote one, with tracks that carry the controls. */
export interface LocalRTCMember extends RTCMember, LocalMemberMedia {
local: true;
tracks$: Behavior<(LocalAudioMediaTrack | LocalVideoMediaTrack)[] | null>;
/**
* Publishes a source and resolves with its track once it is in `tracks$`.
* Before the transport is connected the request is remembered and applied
* once it is. Rejects where the device could not be used, including the
* user closing the picker. One publication per source: publishing a source
* again unmutes it.
*/
publish(
request: PublishRequest,
): Promise<LocalAudioMediaTrack | LocalVideoMediaTrack>;
/** Removes one of our tracks; a screen share takes its audio with it. */
unpublish(id: string): Promise<void>;
}
sdk/src/matrixrtc/LocalMember.test.ts
+9 -17
View File
@@ -46,25 +46,22 @@ describe("createLocalMembership$", () => {
});
local.connectionState$.next(MediaConnectionState.Connected);
expect(membership.state$.value).toEqual({
media: PublishState.WaitingForUser,
matrix: Status.Connected,
});
membership.requestJoinAndPublish();
await flushPromises();
expect(membership.state$.value).toEqual({
media: PublishState.Publishing,
matrix: Status.Connected,
});
expect(membership.connected$.value).toBe(true);
membership.leave();
expect(membership.state$.value).toEqual({
media: PublishState.WaitingForUser,
matrix: Status.Connected,
});
});
it("publishes only while joined and the homeserver is reachable", async () => {
const { membership, local, homeserver } = setup();
expect(local.setPublishing).toHaveBeenLastCalledWith(false);
membership.requestJoinAndPublish();
expect(local.setPublishing).toHaveBeenLastCalledWith(true);
homeserver.combined$.next([false, "sync"]);
@@ -72,7 +69,7 @@ describe("createLocalMembership$", () => {
homeserver.combined$.next([true, null]);
expect(local.setPublishing).toHaveBeenLastCalledWith(true);
membership.requestDisconnect();
membership.leave();
expect(local.setPublishing).toHaveBeenLastCalledWith(false);
// A sync outage after leaving must not bring the media back
homeserver.combined$.next([false, "sync"]);
@@ -82,10 +79,8 @@ describe("createLocalMembership$", () => {
});
it("joins with the delegated timings and hands over each delay id where the backend can", async () => {
const { membership, prepared$, delayId$, joinMatrixRTC, delegate } =
setup();
const { prepared$, delayId$, joinMatrixRTC, delegate } = setup();
prepared$.next({ transport, canDelegateDelayedLeave: true });
membership.requestJoinAndPublish();
await flushPromises();
expect(joinMatrixRTC).toHaveBeenCalledWith(
transport,
@@ -101,10 +96,8 @@ describe("createLocalMembership$", () => {
});
it("never hands the leave over where the backend cannot take it", async () => {
const { membership, prepared$, delayId$, joinMatrixRTC, delegate } =
setup();
const { prepared$, delayId$, joinMatrixRTC, delegate } = setup();
prepared$.next({ transport, canDelegateDelayedLeave: false });
membership.requestJoinAndPublish();
delayId$.next("delay-1");
await flushPromises();
expect(joinMatrixRTC).toHaveBeenCalledWith(
@@ -124,7 +117,6 @@ describe("createLocalMembership$", () => {
const { membership, local, prepared$ } = setup();
prepared$.next({ transport, canDelegateDelayedLeave: false });
local.connectionState$.next(MediaConnectionState.Connected);
membership.requestJoinAndPublish();
local.connectionState$.next(MediaConnectionState.Reconnecting);
expect(membership.state$.value).toEqual({
sdk/src/matrixrtc/LocalMember.ts
+14 -19
View File
@@ -89,9 +89,8 @@ interface Props {
}
export interface LocalMembership {
requestJoinAndPublish: () => void;
requestDisconnect: () => void;
joinRequested$: Behavior<boolean>;
/** Sends the leave and stops publishing. Final: there is no joining again. */
leave: () => void;
state$: Behavior<LocalMemberState>;
/** Fully connected: to the homeserver, the session and the transport. */
connected$: Behavior<boolean>;
@@ -101,8 +100,8 @@ export interface LocalMembership {
}
/**
* The local member's state machine: waits for the transport, publishes once
* asked to join, and enters and leaves the MatrixRTC session in step.
* The local member's state machine: waits for the transport, then publishes
* and enters the MatrixRTC session, and leaves both in step on `leave()`.
*/
export function createLocalMembership$({
scope,
@@ -131,22 +130,22 @@ export function createLocalMembership$({
null,
);
const joinRequested$ = new BehaviorSubject(false);
const joined$ = new BehaviorSubject(true);
const matrixError$ = new BehaviorSubject<MatrixRTCError | null>(null);
// Nothing leaves this device while it may already have been dropped from
// the session: the member would show as away while still being heard
combineLatest(
[joinRequested$, homeserverConnected.combined$],
(join, [connected]) => join && connected,
[joined$, homeserverConnected.combined$],
(joined, [connected]) => joined && connected,
)
.pipe(distinctUntilChanged(), scope.bind())
.subscribe((publish) => local.setPublishing(publish));
scope.reconcile(
scope.behavior(combineLatest([transport$, joinRequested$])),
async ([prepared, shouldJoin]) => {
if (prepared === null || !shouldJoin) return;
scope.behavior(combineLatest([transport$, joined$])),
async ([prepared, joined]) => {
if (prepared === null || !joined) return;
try {
joinMatrixRTC(
prepared.transport,
@@ -191,8 +190,8 @@ export function createLocalMembership$({
});
const mediaState$ = scope.behavior<LocalMemberMediaState>(
combineLatest([local.connectionState$, joinRequested$]).pipe(
map(([connectionState, shouldPublish]) => {
combineLatest([local.connectionState$, joined$]).pipe(
map(([connectionState, joined]) => {
if (connectionState !== MediaConnectionState.Connected)
return {
connection:
@@ -200,9 +199,7 @@ export function createLocalMembership$({
? toMatrixRTCError(connectionState)
: connectionState,
};
return shouldPublish
? PublishState.Publishing
: PublishState.WaitingForUser;
return joined ? PublishState.Publishing : PublishState.WaitingForUser;
}),
distinctUntilChanged(deepCompare),
),
@@ -260,9 +257,7 @@ export function createLocalMembership$({
);
return {
requestJoinAndPublish: () => joinRequested$.next(true),
requestDisconnect: () => joinRequested$.next(false),
joinRequested$,
leave: () => joined$.next(false),
state$,
connected$,
reconnecting$,
sdk/src/matrixrtc/LocalTransport.ts
+2 -2
View File
@@ -9,7 +9,7 @@ import { type MatrixClient } from "matrix-js-sdk";
import { type Logger } from "matrix-js-sdk/lib/logger";
import { type Transport } from "matrix-js-sdk/lib/matrixrtc";
import { type MatrixRTCClientOptions } from "../api";
import { type RTCParticipationOptions } from "../api";
import { MatrixRTCTransportMissingError } from "../errors";
import { TransportAutoDiscovery } from "./TransportAutoDiscovery";
@@ -27,7 +27,7 @@ export async function discoverLocalTransport(
{
transportUrl,
fallbackTransportUrl,
}: Pick<MatrixRTCClientOptions, "transportUrl" | "fallbackTransportUrl">,
}: Pick<RTCParticipationOptions, "transportUrl" | "fallbackTransportUrl">,
logger: Logger,
): Promise<Transport> {
const transport = transportUrl
sdk/src/matrixrtc/Members.ts
+36 -44
View File
@@ -6,6 +6,7 @@ Please see LICENSE in the repository root for full details.
*/
import { type CallMembership } from "matrix-js-sdk/lib/matrixrtc";
import { deepCompare } from "matrix-js-sdk/lib/utils";
import { distinctUntilChanged, map } from "rxjs";
import { type Behavior } from "../reactive/Behavior";
@@ -16,16 +17,15 @@ import {
type RemoteRTCMember,
type RTCMember,
} from "../api";
import {
type LocalMediaBackend,
type MemberMediaFields,
} from "../media-backend/api";
import { type LocalMemberMedia, type MemberMedia } from "../media-api";
import { type TransportRegistry } from "./Transports";
/** What every member is built from, besides its own membership and media. */
/** What every member is built from, besides its own membership. */
export interface MemberContext {
metadata: ReturnType<typeof createMatrixMemberMetadata$>;
transports: TransportRegistry;
/** Whose memberships are `local`. */
own: { userId: string; deviceId: string };
}
/** Everything that tells one membership from another, for keying items. */
@@ -40,50 +40,21 @@ export function membershipKeys(
];
}
export function createRemoteRTCMember(
/** The slot's view of a membership: identity, room state and transport, no media. */
export function createRTCMember(
scope: ObservableScope,
membership$: Behavior<CallMembership>,
media: MemberMediaFields,
context: MemberContext,
): RemoteRTCMember {
{ metadata, transports, own }: MemberContext,
): RTCMember {
const { userId, deviceId, memberId, rtcBackendIdentity } = membership$.value;
return {
...createRTCMember(scope, membership$, context),
local: false,
tracks$: media.tracks$,
encryptionError$: media.encryptionError$,
};
}
export function createLocalRTCMember(
scope: ObservableScope,
membership$: Behavior<CallMembership>,
local: Pick<
LocalMediaBackend,
"tracks$" | "encryptionError$" | "publish" | "unpublish"
>,
context: MemberContext,
): LocalRTCMember {
return {
...createRTCMember(scope, membership$, context),
local: true,
tracks$: local.tracks$,
encryptionError$: local.encryptionError$,
publish: async (request) => local.publish(request),
unpublish: async (id) => local.unpublish(id),
};
}
function createRTCMember(
scope: ObservableScope,
membership$: Behavior<CallMembership>,
{ metadata, transports }: MemberContext,
): Omit<RTCMember, "local" | "tracks$" | "encryptionError$"> {
const { userId, deviceId, rtcBackendIdentity } = membership$.value;
return {
id: rtcBackendIdentity,
local: userId === own.userId && deviceId === own.deviceId,
// Read off the membership for now; the backend will derive it once the
// identity scheme is its own and not the js-sdk's
rtcBackendIdentity,
userId,
deviceId,
membership$,
memberId,
displayName$: scope.behavior(
metadata
.createDisplayNameBehavior$(scope, userId)
@@ -97,5 +68,26 @@ function createRTCMember(
map((transport) => transport && transports.get(transport)),
),
),
applicationData$: scope.behavior(
membership$.pipe(
map((membership) => membership.applicationData),
distinctUntilChanged(deepCompare),
),
),
};
}
/** The participation's view of a remote member: the slot's member plus its media. */
export function createRemoteRTCMember(
member: RTCMember,
media: MemberMedia,
): RemoteRTCMember {
return { ...member, local: false, ...media };
}
export function createLocalRTCMember(
member: RTCMember,
media: LocalMemberMedia,
): LocalRTCMember {
return { ...member, local: true, ...media };
}
sdk/src/matrixrtc/status.test.ts
+19 -11
View File
@@ -15,7 +15,7 @@ import {
PublishState,
TransportState,
} from "./LocalMember";
import { fatalError, sessionStatus } from "./status";
import { fatalError, participationStatus } from "./status";
const publishing: LocalMemberState = {
media: PublishState.Publishing,
@@ -27,31 +27,39 @@ const connecting: LocalMemberState = {
};
const error = new MatrixRTCError("gone");
describe("sessionStatus", () => {
describe("participationStatus", () => {
it.each<[string, LocalMemberState, boolean, boolean, boolean, string]>([
[
"no transport yet",
TransportState.Waiting,
true,
false,
false,
false,
"waitingForTransport",
],
["not asked to join", connecting, false, false, false, "disconnected"],
["joining", connecting, true, false, false, "connecting"],
["connected", publishing, true, true, false, "connected"],
["dropped after connecting", connecting, true, false, true, "reconnecting"],
["a transport error", error, true, false, false, "disconnected"],
["joining", connecting, false, false, false, "connecting"],
["connected", publishing, true, false, false, "connected"],
[
"dropped after connecting",
connecting,
false,
true,
false,
"reconnecting",
],
["a transport error", error, false, false, false, "disconnected"],
[
"a matrix error",
{ ...publishing, matrix: error },
true,
true,
false,
false,
"disconnected",
],
])("%s", (_name, state, joinRequested, connected, reconnecting, expected) => {
expect(sessionStatus(state, joinRequested, connected, reconnecting)).toBe(
["left", publishing, true, false, true, "left"],
["left after an error", error, false, false, true, "left"],
])("%s", (_name, state, connected, reconnecting, left, expected) => {
expect(participationStatus(state, connected, reconnecting, left)).toBe(
expected,
);
});
sdk/src/matrixrtc/status.ts
+7 -6
View File
@@ -5,24 +5,25 @@ SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { type MatrixRTCError, type ConnectionStatus } from "../api";
import { type MatrixRTCError, type ParticipationStatus } from "../api";
import { type LocalMemberState, TransportState } from "./LocalMember";
/** The local member's state, collapsed to what a host shows. */
export function sessionStatus(
export function participationStatus(
state: LocalMemberState,
joinRequested: boolean,
connected: boolean,
reconnecting: boolean,
): ConnectionStatus {
left: boolean,
): ParticipationStatus {
if (left) return "left";
if (fatalError(state) !== null) return "disconnected";
if (state === TransportState.Waiting) return "waitingForTransport";
if (connected) return "connected";
if (reconnecting) return "reconnecting";
return joinRequested ? "connecting" : "disconnected";
return "connecting";
}
/** The error that stops the session, if the state holds one. */
/** The error that stops the participation, if the state holds one. */
export function fatalError(state: LocalMemberState): MatrixRTCError | null {
if (state === TransportState.Waiting) return null;
if (state instanceof Error) return state;
sdk/src/media-api.ts
+35
View File
@@ -16,6 +16,8 @@ import {
type TrackProcessor,
type VideoCodec,
} from "livekit-client";
import { type Observable } from "rxjs";
import { type Behavior } from "./reactive/Behavior";
export interface AudioCaptureSettings {
@@ -42,6 +44,11 @@ export type PublishRequest =
source: "camera";
deviceId?: string;
/** Background blur and the like. */
// REVIEW: this is ugly makes us need to import lk
// We need an interface:
// - super high level: blur + background img
// - lower level: rtc stream insertable stream (copy lk TrackProcessor concept)
// - florian: should be independent of webCodec or WebRtc
processor?: TrackProcessor<Track.Kind.Video>;
capture?: VideoCaptureSettings;
}
@@ -73,8 +80,14 @@ export interface MediaTrack {
id: string;
muted$: Behavior<boolean>;
/** False when the SFU reports the track as unencrypted. */
// REVIEW: does it need to be a behavior
encrypted$: Behavior<boolean>;
/** Polled while subscribed. */
// REVIEW: we want the perf gain of not having it always run.
//
// proposals
// - obs -> on subscribe start polling timer
// - add public method the MediaTrack: for example: actiavteStatsPolling(ms)
stats$: Behavior<MediaStreamStats>;
/**
* Rendering. The view hands its `<video>` or `<audio>` element over; the SDK
@@ -95,6 +108,7 @@ export interface AudioMediaTrack extends MediaTrack {
* "speaking". LiveKit measures it per member, so every audio track of a
* member reports the same value.
*/
// REVIEW: audio Level (could even super-seed isActive "audioLevel != 0")
isActive$: Behavior<boolean>;
/** Route playback through Web Audio, for earpiece pan and gain. Undefined resets. */
setAudioContext(ctx: AudioContext | undefined, plugins?: AudioNode[]): void;
@@ -135,3 +149,24 @@ export interface LocalVideoMediaTrack extends VideoMediaTrack, LocalMediaTrack {
}
export type EncryptionError = "MissingKey" | "InvalidKey";
/**
* What a member sends, once it has arrived on its transport: its tracks, null
* while nothing has arrived for it, and the key errors beside them. The media
* backend supplies exactly these two fields per member.
*/
export interface MemberMedia {
/**
* The member's tracks, in publication order, once it has shown up on its
* transport; null until then ("waiting for media"). An entry stays the same
* object for as long as the same publication is behind it. Which track is
* which is in its `source`; `trackBySource$` picks one out.
*/
tracks$: Behavior<(AudioMediaTrack | VideoMediaTrack)[] | null>;
/** Emits when the SFU reports a key problem for this member. */
encryptionError$: Observable<EncryptionError>;
}
export interface LocalMemberMedia extends MemberMedia {
tracks$: Behavior<(LocalAudioMediaTrack | LocalVideoMediaTrack)[] | null>;
}
sdk/src/media-backend/api.ts
+12 -24
View File
@@ -17,15 +17,13 @@ import { type Transport } from "matrix-js-sdk/lib/matrixrtc";
import { type CallMembershipIdentityParts } from "matrix-js-sdk/lib/matrixrtc/EncryptionManager";
import { type Observable } from "rxjs";
import {
type LocalRTCMember,
type ResolvedTransport,
type RTCMember,
type RTCMembership,
} from "../api";
import { type ResolvedTransport, type RTCMember } from "../api";
import {
type LocalAudioMediaTrack,
type LocalMemberMedia,
type LocalVideoMediaTrack,
type MediaSource,
type MemberMedia,
type PublishRequest,
} from "../media-api";
import { type SessionTimings } from "../config";
@@ -53,13 +51,6 @@ export interface MediaKey {
key: Uint8Array<ArrayBuffer>;
}
/** The fields of a member the backend supplies, as the member exposes them. */
export type MemberMediaFields = Pick<RTCMember, "tracks$" | "encryptionError$">;
export type LocalMemberMediaFields = Pick<
LocalRTCMember,
"tracks$" | "encryptionError$"
>;
export interface MediaBackendContext {
roomId: string;
ownMembershipIdentity: CallMembershipIdentityParts;
@@ -144,15 +135,12 @@ export interface MediaBackend {
/**
* The media of one remote member: its tracks, null while nothing has
* arrived for it on its transport, and its key errors. Built in `scope`,
* which the caller ends with the member. This is also how the backend
* learns which remote transports exist: it follows the membership's
* transport for as long as `scope` lives, and connects to every transport
* some member is on.
* which the caller ends with the member or the participation, whichever
* goes first. This is also how the backend learns which remote transports
* exist: it follows the member's `transport$` for as long as `scope` lives,
* and connects to every transport some member is on.
*/
mediaFor$(
scope: ObservableScope,
membership$: Behavior<RTCMembership>,
): MemberMediaFields;
mediaFor(scope: ObservableScope, member: RTCMember): MemberMedia;
/** Every connection the backend holds. For debugging and devtool purposes.*/
readonly connections$: Behavior<BackendConnection[]>;
@@ -175,7 +163,7 @@ export interface MediaBackend {
}
/** `tracks$` is null until the connection carries a local participant. */
export interface LocalMediaBackend extends LocalMemberMediaFields {
export interface LocalMediaBackend extends LocalMemberMedia {
/**
* State of the connection the local member publishes on. `Initialized`
* while there is none; an `Error` is the connection's failure.
@@ -201,6 +189,6 @@ export interface LocalMediaBackend extends LocalMemberMediaFields {
publish(
request: PublishRequest,
): Promise<LocalAudioMediaTrack | LocalVideoMediaTrack>;
/** Removes one of our tracks by id; a screen share takes its audio with it. */
unpublish(id: string): Promise<void>;
/** Removes one of our publications; a screen share takes its audio with it. */
unpublish(source: MediaSource): Promise<void>;
}
sdk/src/media-backend/livekit/LivekitBackend.ts
+9 -9
View File
@@ -23,11 +23,12 @@ import {
switchMap,
} from "rxjs";
import { type ResolvedTransport, type RTCMembership } from "../../api";
import { type ResolvedTransport, type RTCMember } from "../../api";
import { type MediaQuality } from "../../config";
import { MatrixRTCError } from "../../errors";
import { type Behavior } from "../../reactive/Behavior";
import { type MemberMedia } from "../../media-api";
import {
type Epoch,
type ObservableScope,
@@ -37,7 +38,6 @@ import {
type BackendConnection,
type MediaBackend,
type MediaBackendContext,
type MemberMediaFields,
type TransportCapabilities,
} from "../api";
import { type Connection } from "./Connection";
@@ -147,14 +147,14 @@ export function createLivekitBackend(
);
};
const mediaFor$ = (
const mediaFor = (
memberScope: ObservableScope,
membership$: Behavior<RTCMembership>,
): MemberMediaFields => {
const identity = membership$.value.rtcBackendIdentity;
member: RTCMember,
): MemberMedia => {
const identity = member.rtcBackendIdentity;
const transport$ = memberScope.behavior(
membership$.pipe(
map((membership) => membership.getTransport()),
member.transport$.pipe(
map((transport) => transport?.raw),
map((transport) =>
isLivekitTransport(transport) ? transport : undefined,
),
@@ -224,7 +224,7 @@ export function createLivekitBackend(
prepareLocalTransport,
delegateDelayedLeave,
local: localMedia_,
mediaFor$,
mediaFor,
connections$,
setAudioOutputDeviceId: localMedia_.setAudioOutputDeviceId,
sendData,
sdk/src/media-backend/livekit/LivekitMemberMedia.ts
+8 -4
View File
@@ -23,8 +23,12 @@ import { generateItems } from "../../reactive/observable";
import { E2eeType, type EncryptionSystem } from "../../encryption";
import { type Behavior } from "../../reactive/Behavior";
import { mapScoped } from "../../utils/mapScoped";
import { type EncryptionError, type MediaSource } from "../../media-api";
import { type LocalMemberMediaFields, type MemberMediaFields } from "../api";
import {
type EncryptionError,
type LocalMemberMedia,
type MediaSource,
type MemberMedia,
} from "../../media-api";
import {
createLivekitMediaTrack,
createLocalLivekitMediaTrack,
@@ -45,7 +49,7 @@ export function createLivekitMemberMedia(
scope: ObservableScope,
source$: Behavior<ParticipantSource>,
encryptionSystem: EncryptionSystem,
): MemberMediaFields {
): MemberMedia {
return {
tracks$: memberTracks$(
scope,
@@ -62,7 +66,7 @@ export function createLocalLivekitMemberMedia(
source$: Behavior<ParticipantSource<LocalParticipant>>,
encryptionSystem: EncryptionSystem,
setEnabled: (source: MediaSource, enabled: boolean) => Promise<boolean>,
): LocalMemberMediaFields {
): LocalMemberMedia {
return {
tracks$: memberTracks$(
scope,
sdk/src/media-backend/livekit/LocalMedia.ts
+19 -8
View File
@@ -191,14 +191,11 @@ export function createLivekitLocalMedia({
}
};
const unpublish = async (id: string): Promise<void> => {
const track = tracks$.value?.find((t) => t.id === id);
if (track === undefined) return;
// The screen share audio goes with its video
const source =
track.source === "screenShareAudio" ? "screenShare" : track.source;
desired.delete(source);
await publisher$.value?.unpublish(source);
const unpublish = async (source: MediaSource): Promise<void> => {
const published = publishedSource(source);
if (published === undefined) return;
desired.delete(published);
await publisher$.value?.unpublish(published);
};
const setAudioOutputDeviceId = async (deviceId: string): Promise<void> => {
@@ -268,3 +265,17 @@ function trackWithId$(
filter((track) => track !== undefined),
);
}
/** The publication a source belongs to: the screen share audio goes with its video. */
function publishedSource(
source: MediaSource,
): PublishRequest["source"] | undefined {
switch (source) {
case "screenShareAudio":
return "screenShare";
case "unknown":
return undefined;
default:
return source;
}
}