Matrix/element-call-Github
1
0
Fork 0
mirror of https://github.com/vector-im/element-call.git synced 2026-10-16 23:45:50 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
18e0065086c03031f079a23724e6dcff5e587656
element-call-Github/sdk/MemberMediaPlan.md
T

383 lines
25 KiB
Markdown
Raw Blame History

# `MemberMedia`: a list of tracks instead of a call's four slots
`MemberMedia` in `sdk/src/api.ts` is written in the vocabulary of a video call. It
has four named slots, `microphone$`, `camera$`, `screenShare$` and
`screenShareAudio$`, and two member-level flags, `speaking$` and
`screenShareEnabled$`. A MatrixRTC session is not always a call: a whiteboard that
streams one video track, a translation bot with two audio tracks, a member
publishing two cameras, or a future application the SDK has not heard of all have
media, and none of it fits the four slots. The flags are worse than the slots.
`screenShareEnabled$` says only that the `screenShare$` slot is filled, and
`speaking$` names the microphone's activity after what people do with it in a call.
This plan replaces the slots and the flags with one `tracks$` list, and moves the
activity signal to where it is measured: `isActive$` on the audio track. The public
API changes; behaviour does not; the `playwright/sdk` specs and the dev harness
verify it.
The SDK's consumers of these fields today are the dev harness (`sdk/dev/main.ts`)
and, through it, the specs. Element Call's media view models on this branch still
read the LiveKit participant directly, so nothing under `src/` changes. This is the
cheapest moment to make the change: once Element Call consumes `MemberMedia`, every
view model in `src/state/media/` will be a call site.
> Note for the status table of [`SdkMigration.md`](./SdkMigration.md): it records
> slice 4 (view models take `media$`) as done, but on this branch
> `src/state/media/*.ts` still imports `livekit-client` and `CallViewModel.ts`
> does not import the SDK. Either the table is ahead of this branch or the slice
> lives elsewhere; the table should say which before this plan is added to it.
## What the tracks already know
The four slot getters were one line each: `participant.getTrackPublication(
livekitSources[source])`. That is a lookup by `source` in the participant's
`trackPublications` map. Everything the slot name told the consumer, the track
already carries:
| On the LiveKit publication | On `MediaTrack` today | Set by |
| -------------------------- | ---------------------- | ----------------------------------------------------------------------- |
| `source: Track.Source` | `source: MediaSource` | the publisher, at publish time; the SFU forwards it to every subscriber |
| `kind: Track.Kind` | `kind` | the media stream track |
| `trackSid` | `id` | the SFU, stable for the life of the publication |
| `trackName` | not exposed | the publisher; free-form, empty for everything Element Call publishes |
| `mimeType`, `dimensions` | not exposed | the SFU; `stats$` carries the live values, these are what was announced |
| `isMuted`, `isEncrypted` | `muted$`, `encrypted$` | the publisher and the SFU |
So a flat list is lossless: `tracks$.find((t) => t.source === "camera")` is exactly
what `camera$` returned, and `tracks$.some((t) => t.source === "screenShare")` is
exactly `screenShareEnabled$`, since LiveKit's `isScreenShareEnabled` is defined as
`!!getTrackPublication(Track.Source.ScreenShare)`, presence and not mute state.
One thing the slots lost that the list keeps: a publication with
`Track.Source.Unknown` never appeared in any slot. The list includes it, so
`MediaSource` grows a fifth value, `"unknown"`. A consumer that wants the call
vocabulary filters for the four names it knows; nothing is dropped on the way.
What the list does not try to do: the `trackName` and any application-specific
meaning stay out of the interface for now. Whether a non-call application tags
its tracks through the name, through the membership's `applicationData`, or
through a MatrixRTC-level description is a question for when such an application
exists; see "Other possible call-related refactors".
## The interface
```ts
export type MediaSource =
| "microphone"
| "camera"
| "screenShare"
| "screenShareAudio"
/** Published without a source; the application knows what it is. */
| "unknown";
export interface AudioMediaTrack extends MediaTrack {
kind: "audio";
/**
* Whether the track carries sound right now, as the backend measures it.
* False while muted. A call reads this on the microphone track and calls
* it "speaking".
*/
isActive$: Behavior<boolean>;
setAudioContext(ctx: AudioContext | undefined, plugins?: AudioNode[]): void;
setVolume(volume: number): void;
}
/**
* The media of one member. One entry per published track, in publication
* order; an entry is the same object for as long as the same publication is
* behind it. There is no identity field: the backend identity is the member's
* `id`.
*/
export interface MemberMedia {
local: boolean;
tracks$: Behavior<(AudioMediaTrack | VideoMediaTrack)[]>;
/** Emits when the SFU reports a key problem for this member. */
encryptionError$: Observable<EncryptionError>;
}
export interface LocalMemberMedia extends MemberMedia {
local: true;
switchCamera(): Promise<string | undefined>;
}
```
Removed: `speaking$`, `screenShareEnabled$`, `microphone$`, `camera$`,
`screenShare$`, `screenShareAudio$`. `switchCamera` stays in this plan because it
is a different question (see the refactors section).
### One helper, exported
Every consumer will want "the member's track for a source, as a behavior". The
harness needs it twice already, and Element Call will need it in every media view
model. Rather than let each caller write the `map` + `distinctUntilChanged` chain,
the SDK exports one function from `sdk/src/utils/tracks.ts`:
```ts
/** The first track of a source, undefined while there is none. */
export function trackBySource$<S extends MediaSource>(
scope: ObservableScope,
tracks$: Behavior<MediaTrack[]>,
source: S,
): Behavior<TrackOfSource<S> | undefined>;
```
`TrackOfSource` narrows `"microphone" | "screenShareAudio"` to `AudioMediaTrack`
and `"camera" | "screenShare"` to `VideoMediaTrack`, so the typed getters the slots
gave are not lost. It is a pure function over `Behavior`s, tested with marbles.
Nothing else is added; `some`, `filter` and `find` on the array cover the rest.
## `isActive$`: how it replaces `speaking$`
### What LiveKit measures
The SFU runs speaker detection from the audio-level RTP header extension on the
audio it forwards, and pushes `SpeakerInfo { sid, level, active }` updates to every
client. `livekit-client` turns them into `participant.audioLevel`,
`participant.isSpeaking` and `ParticipantEvent.IsSpeakingChanged` (see
`Room.handleActiveSpeakersUpdate` and `handleSpeakersChanged`). The `sid` is the
participant's; there is no per-track speaking signal in the client API. `speaking$`
today is `IsSpeakingChanged` mapped to `isSpeaking`, nothing more.
### What the SDK exposes
`isActive$` on an audio track is the participant's speaking signal gated by the
track's own mute state:
```
isActive = participant.isSpeaking && !publication.isMuted
```
For the LiveKit backend this maps one participant-level signal onto every audio
track of the member. In practice a member has one audio track, the microphone, and
the mapping is exact. A member that also shares a screen with audio gets the same
value on both tracks while either carries sound; the mute gate keeps a muted
microphone dark while the screen share plays. This is a documented approximation of
the LiveKit adapter, not of the interface: a full-mesh backend has a peer connection
per remote member and measures each track's level itself, and a stats-based
refinement exists for LiveKit too (below). The track is the right place for the
behavior because that is where every backend other than a single SFU would produce
it.
### Why a call loses nothing
"Speaking" is a call's reading of "the member's microphone track is active". The
replacement for `media.speaking$` is:
```ts
trackBySource$(scope, media.tracks$, "microphone").pipe(
switchMap((track) => track?.isActive$ ?? of(false)),
);
```
False when there is no microphone track, which is what `isSpeaking` reported in
that case too. Element Call's 1 s / 60 s hysteresis (`observeSpeaker$`) and the
spotlight speaker selection stay in the call view model, where they are: they are
call UX, and they consume a boolean exactly as before.
What is gained: screen share audio can show activity ("the presenter's video has
sound"), and an application with several audio tracks per member can tell them
apart once the backend can. What is lost: nothing the SFU knew. The signal was
never per track in LiveKit; the interface now says so instead of pretending the
member speaks.
### Later refinement, not in this plan
A genuinely per-track level for LiveKit is available from the RTP statistics the
SDK already polls for `stats$`: `audioLevel` and `totalAudioEnergy` on
`RTCInboundRtpStreamStats`, and on the `media-source` stats for a local track. The
poll is once a second, which the call's hysteresis tolerates but which is too
coarse for an unfiltered indicator, and it costs a stats request per audio track
per second while subscribed. If a second audio track per member ever matters, that
is the first thing to try.
## What changes, file by file
### `sdk/src/api.ts`
The interface above. The comment on `MemberMedia` says the list is in publication
order and that entries are stable per publication; the comment on `isActive$` says
what it measures and names the LiveKit approximation.
### `sdk/src/media/LivekitMediaTrack.ts`
- `createLivekitMediaTrack` stops taking `source` as a parameter and derives it
from `publication.source` through the inverse of `livekitSources`, with
`Track.Source.Unknown` as `"unknown"`.
- Audio tracks get `isActive$`: `observeParticipantEvents(participant,
ParticipantEvent.IsSpeakingChanged)` mapped to `isSpeaking`, combined with
`muted$`, as a `scope.behavior` seeded from `participant.isSpeaking &&
!publication.isMuted`.
- `setVolume` acts on the track instead of the source. Today it branches on
`source` and calls `RemoteParticipant.setVolume(volume, Track.Source.X)`, which
has nothing to do for an `"unknown"` audio track. `RemoteAudioTrack.setVolume`
exists and is per track; the local no-op stays. `track$` is already a behavior
here, so the volume is reapplied when a remote track arrives on subscription, the
same way the audio context is.
### `sdk/src/media/LivekitMemberMedia.ts`
- `speaking$`, `screenShareEnabled$` and the four `track$` calls go; `memberTrack$`
goes with them.
- `tracks$` is built from `observeParticipantMedia(participant)` (it already fires
on publish, unpublish, subscribe and mute) mapped to
`[...participant.trackPublications.values()]`, then through `generateItems` from
`reactive/observable.ts`, keyed by `trackSid`, so each publication gets one
`createLivekitMediaTrack` in a scope that ends when the publication leaves the
map. This is the pattern `MatrixRTCClient.ts` uses for `remoteMembers$`; nothing
new is written for it.
- Publication order is the map's insertion order, which `generateItems` preserves.
### `sdk/src/utils/tracks.ts`
`trackBySource$` as above, exported from `sdk/index.ts`.
### `sdk/dev/main.ts`
`memberTile` renders from `trackBySource$(scope, tracks$, "camera")` and
`"microphone"`, where `tracks$` is `media$` switched to `media?.tracks$ ?? of([])`.
`render` adds one label, `data-active`, from `isActive$` when the track is audio.
The `member`, `video` and `audio` test ids do not change, so the existing specs
keep passing untouched.
### Documentation
- `SdkArchitecture.md`: the `MemberMedia` block in "Media: `MemberMedia`", the
diagram line that lists `speaking$ microphone$ camera$ screenShare$`, and the
harness description that mentions `media$.camera$` and `media$.microphone$`.
- `SdkMigration.md`: a new row in the status table once the branch question above
is settled.
- `MediaBackendPlan.md`: no change; it already treats `media/` as the clean seam and
this plan only makes that seam less call-shaped.
## Tests
### Unit
- `LivekitMediaTrack.test.ts`, existing file: `isActive$` follows
`IsSpeakingChanged`; a muted publication reports `false` while the participant
speaks; `source` is derived from the publication, including `"unknown"`;
`setVolume` reaches the remote track, and again when a track arrives later.
- `LivekitMemberMedia.test.ts`, new: `tracks$` lists every publication in order;
the same object is returned for a publication across unrelated media events (a
mute on another track); a publication replaced under the same source yields a new
object and ends the old one's scope; an unknown-source publication appears with
`source: "unknown"`; screen share presence is derivable with `some`.
- `tracks.test.ts`, new: `trackBySource$` with marbles, including the transition
from undefined to a track and back.
The fakes come from `sdk/src/utils/test.ts` (`mockRemoteParticipant`) and the
`fakes()` helper already in `LivekitMediaTrack.test.ts`; the latter moves to
`test.ts` if the new file needs it, rather than being copied.
### End-to-end
`playwright/sdk/media.spec.ts` gains one test: the peer's `audio` element reaches
`data-active="true"`, then `"false"` after the Microphone button is clicked. The
fake capture device Playwright gives Chromium emits a tone, which the SFU's speaker
detection picks up, so the signal is real end to end. Firefox's fake device also
produces sound; if its level stays under the SFU threshold the test is skipped
there with a comment saying why, not loosened. The existing five tests run as they
are.
## Order of work
1. `api.ts`, `tracks.ts` and the unit tests for `trackBySource$`.
2. `LivekitMediaTrack.ts`: source from the publication, `isActive$`, per-track
volume, with its tests.
3. `LivekitMemberMedia.ts`: `tracks$`, remove the rest, with its new test file.
4. `dev/main.ts` and the new spec.
5. Documentation.
Steps 1 to 3 do not compile against the harness until step 4, so they land as one
PR; it is under 400 lines because the media adapter is small. `pnpm lint`,
`pnpm test` and the `playwright/sdk` project are the gates. `knip` will not
complain about `trackBySource$` since the harness uses it.
## Other possible call-related refactors
Everything else in `sdk/src/api.ts` and `sdk/src/session/` that only makes sense
for a call. None of it is in this plan; each would be its own slice. The first
three are the ones this plan makes visible.
| What | Why it is call-specific | What instead |
| --------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `MediaSource` itself | Four call names plus `"unknown"`. LiveKit's `Track.Source` has the same four, so the type is faithful to the wire today, but a whiteboard or a bot has no name for its track. | Keep `source` for the backend's notion; add an optional `name` (LiveKit `trackName`) or a description in `applicationData`, once an application needs it. |
| `LocalRTCMember.sharingScreen$`, `toggleScreenSharing`, `screenShareError$`, `dismissScreenShareError` | A screen share is one publication among others. `sharingScreen$` is `participant.isScreenShareEnabled`, now `tracks$.some(...)` on the local media. | A generic `publish(source, capture)` / `unpublish(id)` on the local member, with its error on the returned promise; `sharingScreen$` derived by the host from `tracks$`. |
| `LocalMemberMedia.switchCamera` | "Camera" is the call's name for the local video track with a `facingMode$`. | `switchFacingMode()` on the local `VideoMediaTrack`, next to `facingMode$`. |
| `MatrixRTCClient.setMicrophoneEnabled` / `setCameraEnabled` | Two fixed publications the client knows by name. | `setEnabled(enabled)` on the local track, or `publish`/`unpublish`; `LocalMember.setEnabled` and `Publisher.setEnabled` already take a `Track.Source` and would take a track id. |
| `LocalMediaInputs` (`microphoneEnabled`, `cameraEnabled`, `audioInputDeviceId$`, `videoInputDeviceId$`, `videoProcessor$`) | Describes exactly one microphone and one camera. `audioOutputDeviceId$` is about playback and is not call-specific. | A list of desired local publications, each with its device, processor and enabled state; `Publisher.desired.microphone$` / `camera$` become entries in it. |
| `CaptureSettings { audio, camera, screenShare }` and `MediaQuality { video, screen_share }` | Keyed by the four sources. | Per-publication capture and encoding options, passed with `publish`. `config.ts` keeps the host-facing `MediaQuality` shape for `config.json`, mapped at the edge. |
| `AudioMediaTrack.setVolume` branching on source | Covered by this plan (per-track volume); listed so the row above it is complete. | Done here. |
| `Track.Source` plumbing in `Publisher.ts` (`pauseUpstreams`, `resumeUpstreams`, the processor on the camera) and `LocalMember.ts` | Internal, but the three sources are enumerated by hand in five places. | Iterate the local publications; attach the processor to every local video track whose publication asks for one. |
| `applicationData` carrying only `m.call.intent`; `application` defaulting to `m.call` | The js-sdk's membership format carries one key. The default is a convenience for the one application that exists. | Pass the record through once the js-sdk can; keep the default but say so in the SDK README. Already an open item in `SdkArchitecture.md`. |
| `membership$: Behavior<CallMembership>` on `RTCMember` | The js-sdk's type name. The SDK does not get to rename it, but every host sees the word "call" in a session API. | Re-export under an SDK alias (`RTCMembership`) and narrow to what hosts read: `userId`, `deviceId`, `memberId`, `rtcBackendIdentity`, `getTransport()`, `applicationData`. |
| `UnknownCallError` in `errors.ts` | Name only. | Rename with the two `Livekit*` errors in slice 11 of `SdkMigration.md`. |
| `observeSharingScreen$` in `LocalMember.ts` | Reads `participant.isScreenShareEnabled`. | Goes with `sharingScreen$`, or reads `tracks$` from the local media in the meantime. |
| `sendNotificationType` | Kept on purpose; see "Call-specific things kept out of the client" in `SdkArchitecture.md`. | Nothing. |
## Plan for the refactors above
Four slices, in dependency order. Each is one PR, green on its own, with the
`playwright/sdk` specs and the dev harness as the proof that behaviour is kept.
Element Call follows on its own branch after each.
### 1. Local track controls
The local member's tracks become objects a host can act on, instead of methods
named after a call's devices.
- **Add** `LocalAudioMediaTrack` and `LocalVideoMediaTrack` (`local: true`) in
`api.ts`, with `setEnabled(enabled)`, `setDevice(deviceId)` and, on video,
`switchFacingMode()` and `setProcessor(processor)`. `tracks$` on
`LocalMemberMedia` is typed to them.
- **Remove** `LocalMemberMedia.switchCamera`, and `audioInputDeviceId$`,
`videoInputDeviceId$`, `videoProcessor$` from `LocalMediaInputs`. The host
subscribes to its settings and calls the track; the SDK stops watching
behaviors it does not own. `audioOutputDeviceId$` stays: playback is not a
track.
- **Keep** `setMicrophoneEnabled` / `setCameraEnabled` for now: they also cover
"no track yet", which slice 2 solves.
- **Touches** `LivekitMediaTrack.ts`, `Publisher.ts` (the processor and device
code moves onto the track), `MatrixRTCClient.ts`, the harness buttons.
### 2. `publish` and `unpublish`
One way to put media on the session, whatever it is.
- **Add** to `LocalRTCMember`: `publish(request): Promise<LocalMediaTrack>` and
`unpublish(id): Promise<void>`. A request is `{ source, deviceId?, capture? }`
for `"microphone"` and `"camera"`, `{ source: "screenShare", audio?, capture?
}` for a screen. Failures reject the promise with a `MatrixRTCError`.
- **Remove** `sharingScreen$`, `toggleScreenSharing`, `screenShareError$`,
`dismissScreenShareError`, `observeSharingScreen$`, and the two
`set*Enabled` methods on the client. "Sharing" is `tracks$.some(...)` on the
local media; "enable" is `publish` or `track.setEnabled`.
- **Replace** `LocalMediaInputs.microphoneEnabled` / `cameraEnabled` with
`publish: PublishRequest[]`, applied once the transport is up, as today.
`CaptureSettings` becomes the `capture` of a request; `MediaQuality` keeps its
`config.json` shape and is mapped to per-request capture at the edge.
- **Touches** `LocalMember.ts` (the screen share block and `setEnabled`),
`Publisher.ts` (iterates requests instead of enumerating three sources),
`livekitOptions.ts`, `lifecycle.spec.ts` and `media.spec.ts` for the new
buttons.
### 3. Names that say "call"
No behaviour change; one PR of renames and type narrowing.
- `RTCMember.membership$` is typed as an SDK `RTCMembership`: the subset of
`CallMembership` hosts read (`userId`, `deviceId`, `memberId`,
`rtcBackendIdentity`, `getTransport()`, `applicationData`). The js-sdk object
is still what is behind it.
- `UnknownCallError` is renamed with the two `Livekit*` errors, as slice 11 of
`SdkMigration.md` already plans; `index.ts` drops its alias.
- `applicationData` is passed through whole once the js-sdk accepts more than
`m.call.intent`; until then the README says which key survives.
### Not planned
- `MediaSource` keeps its five values. A `name` on `MediaTrack` (LiveKit's
`trackName`) is a one-line addition when the first non-call application asks
for it; adding it now would be a guess at that application's needs.
- `sendNotificationType` stays, as `SdkArchitecture.md` already decided.
Reference in New Issue View Git Blame Copy Permalink