Compare commits

..
Author SHA1 Message Date
Johannes Marbach cb0490e3f0 Render tiles for unmappable LiveKit identities 2026-09-14 10:30:45 +02:00
Johannes Marbach fe91162869 Merge pull request #4257 from element-hq/matthew/log-livekit-ice
Always log LiveKit ICE candidates and states to the rageshake
2026-09-14 08:31:42 +02:00
Johannes Marbach 74328d7e42 Merge pull request #4258 from element-hq/renovate/livekit-client
Update dependency livekit-client to v2.22.3
2026-09-14 08:03:12 +02:00
renovate[bot] 6046337877 Update dependency livekit-client to v2.22.3 2026-09-14 02:37:28 +00:00
Matthew Hodgson 16b2c31ebb Always log LiveKit ICE candidates and states to the rageshake
LiveKit's livekit-ice logger records each local ICE candidate, ICE
candidate errors and ICE connection state changes, but only at debug,
which we cap to info unless extended LiveKit logs are switched on. That
means a 'could not establish pc connection' rageshake carries no way to
tell 'no relay candidates at all' from 'relay candidates that never
paired'. Keep that one logger at debug regardless of the toggle.
2026-09-11 18:30:33 +01:00
fkwpandClaude Opus 5 a58a5f615f Add AGENTS.md documenting the repo's conventions (#4256)
* Add AGENTS.md documenting the repo's conventions

Codifies what is currently tribal knowledge or only discoverable by reading CI
config: the view model / view contract, the rule that nothing reads the page,
Clean Code ordering, the three test layers, and reuse-before-you-build.

The root AGENTS.md is the always-loaded minimum and routes to docs/agents/, so
an agent loads only the detail its task needs.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Allow either hand at the commit, and measure continuous redraws

Committing from the agent rules out running it where there is no git
identity, a sandbox included. Either is now fine: commit on the user's
word where the setup allows it, or write the message and hand it over.

Redrawing continuously is the other addition. A level meter reported a
fresh value every animation frame, so its whole menu reconciled sixty
times a second, silence included, and nobody noticed until review. A PR
that adds anything of the kind now says what it costs, or better, counts
the redraws — commits rather than render calls, since React runs a
component it then discards.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-11 15:12:15 +02:00
Robin 2cac2d7e8b Merge pull request #4231 from element-hq/matthew/blur-processor-ended-track
Don't attach the background blur processor to an ended camera track
2026-09-10 21:27:11 +02:00
Robin 1695d3f518 Use flushPromises for readability 2026-09-10 21:15:48 +02:00
Robin 96a2e9b481 Merge pull request #4230 from element-hq/matthew/connection-stop-during-connect
Don't report an aborted connect as a connection error
2026-09-10 20:25:12 +02:00
Robin 23ba8f3204 Merge pull request #4249 from element-hq/matthew/audio-renderer-warn-spam
Stop MatrixAudioRenderer warning on every render
2026-09-10 19:56:56 +02:00
Matthew HodgsonandClaude Fable 5.1 b284bb00c0 Cover the stopProcessor failure path and trackProcessorSync
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GSB8Qb89KDoTNRV83dWJHa
2026-09-10 18:28:57 +01:00
Johannes Marbach cb0ae89143 Merge pull request #4253 from element-hq/johannes/jwt-0.7.0
Bump lk-jwt-service to 0.7.0 in the docker setup and integrate it as an app service
2026-09-10 17:46:02 +02:00
Johannes Marbach 344aac21f9 Force undelegated mode in tests 2026-09-10 14:15:38 +02:00
Johannes Marbach 21d7f24f61 Bump lk-jwt-service to 0.7.0 in the docker setup and integrate it as an app service 2026-09-10 12:26:49 +02:00
Matthew Hodgson dbd2423e7b Stop MatrixAudioRenderer warning on every render
The "has no matching matrix call member" warning fired on every re-render
(i.e. every active speaker update) and also for the local participant's own
track, which is never rendered here anyway. Skip local tracks and warn once
per unexpected identity.
2026-09-09 11:20:49 +01:00
Matthew Hodgson 2e5b5cb186 Don't attach the background blur processor to an ended camera track
If the camera track has already ended by the time the blur processor is
applied to it, MediaStreamTrackProcessor cannot be constructed and
setProcessor rejects. Both call sites used `void`, so this surfaced as

    Unhandled promise rejection: TypeError: Failed to construct
    'MediaStreamTrackProcessor': Input track cannot be ended

(element-call-rageshakes#17225). Skip ended tracks, and catch and log
processor attach/detach failures instead of leaking them.
2026-09-03 14:50:28 +01:00
Matthew Hodgson 8060d10fcc Don't report an aborted connect as a connection error
When the ConnectionManager stops a Connection while livekitRoom.connect()
is still pending (which happens on every join, because the local
membership emits twice in quick succession and the connection set is
recomputed), livekit-client rejects the pending connect with "Client
initiated disconnect". Connection.start() then treated that as a failure:
it pushed an UnknownCallError into the state of a connection that was
already stopped and rethrew, and since start() is fire-and-forget the
throw surfaced as

    Unhandled promise rejection: Error: Failed to connect to Livekit server

in every rageshake, right after a "livekitRoom.connect FAILED ... Client
initiated disconnect" line. Nothing was actually wrong: the replacement
connection connects fine a moment later.

Mark the connection as stopped before disconnecting, and have start()
return quietly when the rejection is the abort we asked for.
2026-09-03 14:48:23 +01:00
30 changed files with 897 additions and 97 deletions
+17
View File
@@ -0,0 +1,17 @@
id: "lk-jwt-service-othersite"
as_token: "ec_dev_as_token_othersite_0123456789ab"
hs_token: "ec_dev_hs_token_othersite_0123456789ab"
sender_localpart: "_lk_jwt_service"
namespaces:
users:
- exclusive: false
regex: "@.*:homeserver-1" # Cover all local users
# Required for Synapse to call POST /_matrix/app/v1/ping on this service.
url: "http://auth-server-1:16080"
# Enable membership look-ups via /is_joined.
io.element.msc4502.scopes:
["urn:matrix:client:io.element.msc4502:rooms:is_joined"]
# Route /_matrix/client/(v*|unstable/*)/rtc/livekit/* C-S and S-S requests to this
# service.
io.element.msc4512.proxy_prefix: "rtc/livekit"
io.element.msc4512.proxy_url: "http://auth-server-1:16080"
+17
View File
@@ -0,0 +1,17 @@
id: "lk-jwt-service"
as_token: "ec_dev_as_token_main_0123456789ab"
hs_token: "ec_dev_hs_token_main_0123456789ab"
sender_localpart: "_lk_jwt_service"
namespaces:
users:
- exclusive: false
regex: "@.*:homeserver" # Cover all local users
# Required for Synapse to call POST /_matrix/app/v1/ping on this service.
url: "http://auth-server:6080"
# Enable membership look-ups via /is_joined.
io.element.msc4502.scopes:
["urn:matrix:client:io.element.msc4502:rooms:is_joined"]
# Route /_matrix/client/(v*|unstable/*)/rtc/livekit/* C-S and S-S requests to this
# service.
io.element.msc4512.proxy_prefix: "rtc/livekit"
io.element.msc4512.proxy_url: "http://auth-server:6080"
+7
View File
@@ -42,6 +42,13 @@ experimental_features:
msc4354_enabled: true
# MatrixRTC
msc4143_enabled: true
# Enable membership look-up via /is_joined.
msc4502_enabled: true
# Enable C-S & S-S request proxying for application services.
msc4512_enabled: true
app_service_config_files:
- /data/cfg/app-service.yaml
# The maximum allowed duration by which sent events can be delayed, as
# per MSC4140. Must be a positive value if set. Defaults to no
+7
View File
@@ -42,6 +42,13 @@ experimental_features:
msc4354_enabled: true
# MatrixRTC
msc4143_enabled: true
# Enable membership look-up via /is_joined.
msc4502_enabled: true
# Enable C-S & S-S request proxying for application services.
msc4512_enabled: true
app_service_config_files:
- /data/cfg/app-service.yaml
# The maximum allowed duration by which sent events can be delayed, as
# per MSC4140. Must be a positive value if set. Defaults to no
@@ -42,6 +42,13 @@ experimental_features:
msc4354_enabled: true
# MatrixRTC
msc4143_enabled: true
# Enable membership look-up via /is_joined.
msc4502_enabled: true
# Enable C-S & S-S request proxying for application services.
msc4512_enabled: true
app_service_config_files:
- /data/cfg/app-service.yaml
# The maximum allowed duration by which sent events can be delayed, as
# per MSC4140. Must be a positive value if set. Defaults to no
+7
View File
@@ -42,6 +42,13 @@ experimental_features:
msc4354_enabled: true
# MatrixRTC
msc4143_enabled: true
# Enable membership look-up via /is_joined.
msc4502_enabled: true
# Enable C-S & S-S request proxying for application services.
msc4512_enabled: true
app_service_config_files:
- /data/cfg/app-service.yaml
# The maximum allowed duration by which sent events can be delayed, as
# per MSC4140. Must be a positive value if set. Defaults to no
+28 -4
View File
@@ -3,7 +3,7 @@ networks:
services:
auth-service:
image: ghcr.io/element-hq/lk-jwt-service:0.4.4
image: ghcr.io/element-hq/lk-jwt-service:0.7.0
pull_policy: always
hostname: auth-server
environment:
@@ -15,6 +15,14 @@ services:
# a self-signed certificate
- LIVEKIT_INSECURE_SKIP_VERIFY_TLS=YES_I_KNOW_WHAT_I_AM_DOING
- LIVEKIT_FULL_ACCESS_HOMESERVERS=*
# Registers this instance as an application service on the main homeserver
- LIVEKIT_AS_REGISTRATION_FILE=/etc/lk-jwt-service/app-service.yaml
- LIVEKIT_HS_SERVER_NAME=synapse.m.localhost
# Neither homeserver serves /.well-known/matrix/client, so the C-S API
# location has to be given explicitly.
- LIVEKIT_CS_API_URL_OVERRIDES=synapse.m.localhost=http://homeserver:8008,synapse.othersite.m.localhost=http://homeserver-1:18008
volumes:
- ./backend/app-service.yaml:/etc/lk-jwt-service/app-service.yaml:Z
deploy:
restart_policy:
condition: on-failure
@@ -25,7 +33,7 @@ services:
- ecbackend
auth-service-1:
image: ghcr.io/element-hq/lk-jwt-service:0.4.4
image: ghcr.io/element-hq/lk-jwt-service:0.7.0
pull_policy: always
hostname: auth-server-1
environment:
@@ -37,6 +45,14 @@ services:
# a self-signed certificate
- LIVEKIT_INSECURE_SKIP_VERIFY_TLS=YES_I_KNOW_WHAT_I_AM_DOING
- LIVEKIT_FULL_ACCESS_HOMESERVERS=*
# Registers this instance as an application service on the other homeserver.
- LIVEKIT_AS_REGISTRATION_FILE=/etc/lk-jwt-service/app-service.yaml
- LIVEKIT_HS_SERVER_NAME=synapse.othersite.m.localhost
# Neither homeserver serves /.well-known/matrix/client, so the C-S API
# location has to be given explicitly.
- LIVEKIT_CS_API_URL_OVERRIDES=synapse.m.localhost=http://homeserver:8008,synapse.othersite.m.localhost=http://homeserver-1:18008
volumes:
- ./backend/app-service-othersite.yaml:/etc/lk-jwt-service/app-service.yaml:Z
deploy:
restart_policy:
condition: on-failure
@@ -94,7 +110,10 @@ services:
synapse:
hostname: homeserver
image: ghcr.io/element-hq/synapse:latest
# develop (not latest) is required for application service C-S/S-S proxying
# (MSC4512) and membership look-ups (MSC4502), which are not in a stable
# release yet.
image: ghcr.io/element-hq/synapse:develop
pull_policy: always
environment:
- SYNAPSE_CONFIG_PATH=/data/cfg/homeserver.yaml
@@ -107,12 +126,16 @@ services:
volumes:
- ./backend/synapse_tmp:/data:Z
- ./backend/dev_homeserver.yaml:/data/cfg/homeserver.yaml:Z
- ./backend/app-service.yaml:/data/cfg/app-service.yaml:Z
networks:
- ecbackend
synapse-1:
hostname: homeserver-1
image: ghcr.io/element-hq/synapse:latest
# develop (not latest) is required for application service C-S/S-S proxying
# (MSC4512) and membership look-ups (MSC4502), which are not in a stable
# release yet.
image: ghcr.io/element-hq/synapse:develop
pull_policy: always
environment:
- SYNAPSE_CONFIG_PATH=/data/cfg/homeserver.yaml
@@ -125,6 +148,7 @@ services:
volumes:
- ./backend/synapse_tmp_othersite:/data:Z
- ./backend/dev_homeserver-othersite.yaml:/data/cfg/homeserver.yaml:Z
- ./backend/app-service-othersite.yaml:/data/cfg/app-service.yaml:Z
networks:
- ecbackend
+8 -2
View File
@@ -3,12 +3,18 @@
services:
synapse:
# Pin to a SHA so that upstream cannot break our tests. Renovate handles regular updates.
image: ghcr.io/element-hq/synapse:latest@sha256:da325af40104051923899e5f5a2f1d537e6e3ccf2f0f38285689ae9bbfdb190a
# develop (not latest) is required for application service C-S/S-S proxying
# (MSC4512) and membership look-ups (MSC4502), which are not in a stable
# release yet.
image: ghcr.io/element-hq/synapse:develop@sha256:337921fd22be310d453344b2265f6c9942315446595df3cf81866f71ca2055a7
volumes:
- ./backend/playwright_homeserver.yaml:/data/cfg/homeserver.yaml:Z
synapse-1:
# Pin to a SHA so that upstream cannot break our tests. Renovate handles regular updates.
image: ghcr.io/element-hq/synapse:latest@sha256:da325af40104051923899e5f5a2f1d537e6e3ccf2f0f38285689ae9bbfdb190a
# develop (not latest) is required for application service C-S/S-S proxying
# (MSC4512) and membership look-ups (MSC4502), which are not in a stable
# release yet.
image: ghcr.io/element-hq/synapse:develop@sha256:337921fd22be310d453344b2265f6c9942315446595df3cf81866f71ca2055a7
volumes:
- ./backend/playwright_homeserver-othersite.yaml:/data/cfg/homeserver.yaml:Z
element-web:
+1
View File
@@ -273,6 +273,7 @@
"mute_for_me": "Mute for me",
"muted_for_me": "Muted for me",
"screen_share_volume": "Screen share volume",
"unknown_participant": "Unknown participant",
"volume": "Volume",
"waiting_for_media": "Waiting for media..."
}
+8
View File
@@ -7,6 +7,8 @@ Please see LICENSE in the repository root for full details.
import { expect, test } from "@playwright/test";
import { SpaHelpers } from "./spa-helpers";
// Skip test for Firefox, due to page.keyboard.press("Tab") not reliable on headless mode
test.skip(
({ browserName }) => browserName === "firefox",
@@ -17,6 +19,12 @@ test("can only interact with header and footer while reconnecting", async ({
page,
}) => {
await page.goto("/");
// The reconnecting state is entered via the probablyLeft timer, which
// mirrors the delayed leave event's timeout. With delegation that timeout
// is one hour, putting it out of reach of the clock fast-forward below.
// Keep the leave client-managed so its short timeout applies.
await SpaHelpers.disableLeaveDelegation(page);
await page.getByTestId("home_callName").click();
await page.getByTestId("home_callName").fill("Test call");
await page.getByTestId("home_displayName").click();
+11 -1
View File
@@ -19,6 +19,7 @@ async function setupTwoUserSpaCall(
browser: Browser,
page: Page,
browserName: string,
opts: { disableGuestLeaveDelegation?: boolean } = {},
): Promise<{ guestPage: Page }> {
test.skip(
browserName === "firefox",
@@ -52,6 +53,9 @@ async function setupTwoUserSpaCall(
await guestPage.goto("/");
if (opts.disableGuestLeaveDelegation)
await SpaHelpers.disableLeaveDelegation(guestPage);
let pevaraHasSentStickyEvent = false;
const pevaraResolver = Promise.withResolvers<void>();
@@ -102,7 +106,13 @@ test("One to One rejoin after improper leave does not crash EC", async ({
page,
browserName,
}) => {
const { guestPage } = await setupTwoUserSpaCall(browser, page, browserName);
// With delegation, the backend sends the guest's delayed leave event within
// moments of the improper leave, so the stale membership this test needs
// would be cleaned up before the rejoin. Keep the guest's leave
// client-managed so the stale membership lingers.
const { guestPage } = await setupTwoUserSpaCall(browser, page, browserName, {
disableGuestLeaveDelegation: true,
});
await SpaHelpers.expectVideoTilesCount(page, 2);
await SpaHelpers.expectVideoTilesCount(guestPage, 2);
+19
View File
@@ -111,6 +111,24 @@ async function setRtcModeFromSettings(
await page.getByTestId("modal_close").click();
}
/**
* Makes the delayed-leave delegation support probes fail so that the client
* manages its delayed leave event itself instead of delegating it to the
* backend.
*
* Must be installed before the page joins a call.
*/
async function disableLeaveDelegation(page: Page): Promise<void> {
// Covers both the transport probe (<livekit_service_url>/delegate_delayed_leave)
// and the homeserver probe (MSC4195, .../rtc/livekit/delegate_delayed_leave).
await page.route("**/delegate_delayed_leave", async (route) =>
route.fulfill({
status: 404,
headers: { "Access-Control-Allow-Origin": "*" },
}),
);
}
/**
* Expect a certain number of video tiles to be present and visible.
*/
@@ -133,5 +151,6 @@ export const SpaHelpers = {
createCall,
getCallInviteLink,
joinCallFromInviteLink,
disableLeaveDelegation,
expectVideoTilesCount,
};
+14 -14
View File
@@ -39,13 +39,13 @@ importers:
version: 11.7.12
'@livekit/components-core':
specifier: ^0.12.0
version: 0.12.15(livekit-client@2.22.2(@types/dom-mediacapture-record@1.0.22))(tslib@2.8.1)
version: 0.12.15(livekit-client@2.22.3(@types/dom-mediacapture-record@1.0.22))(tslib@2.8.1)
'@livekit/components-react':
specifier: ^2.0.0
version: 2.9.24(livekit-client@2.22.2(@types/dom-mediacapture-record@1.0.22))(react-dom@19.2.8(react@19.2.8))(react@19.2.8)(tslib@2.8.1)
version: 2.9.24(livekit-client@2.22.3(@types/dom-mediacapture-record@1.0.22))(react-dom@19.2.8(react@19.2.8))(react@19.2.8)(tslib@2.8.1)
'@livekit/track-processors':
specifier: ^0.7.1
version: 0.7.2(@types/dom-mediacapture-transform@0.1.11)(livekit-client@2.22.2(@types/dom-mediacapture-record@1.0.22))
version: 0.7.2(@types/dom-mediacapture-transform@0.1.11)(livekit-client@2.22.3(@types/dom-mediacapture-record@1.0.22))
'@mediapipe/tasks-vision':
specifier: ^0.10.18
version: 0.10.35
@@ -183,7 +183,7 @@ importers:
version: 5.88.1(@emnapi/core@1.11.2)(@emnapi/runtime@1.11.2)(@types/node@24.13.3)(typescript@5.9.3)
livekit-client:
specifier: ^2.18.1
version: 2.22.2(@types/dom-mediacapture-record@1.0.22)
version: 2.22.3(@types/dom-mediacapture-record@1.0.22)
lodash-es:
specifier: ^4.17.21
version: 4.18.1
@@ -4605,8 +4605,8 @@ packages:
lines-and-columns@1.2.4:
resolution: {integrity: sha512-7ylylesZQ/PV29jhEDl3Ufjo6ZX7gCqJr5F7PKrqc93v7fzSymt1BpwEU8nAUXs8qzzvqhbjhK5QZg6Mt/HkBg==}
livekit-client@2.22.2:
resolution: {integrity: sha512-o3yLYoH2eHVR/DmIZzgC96FMvtR4+g/DjeqNwS5zp12EjqrqiRULxm6Oq/YsTE97WnA6fhy23+a2hGkbce5SPA==}
livekit-client@2.22.3:
resolution: {integrity: sha512-jw9zBKXY5Gtr5MZ7vEON3QhMNccuDvYHck1PFSyG1aaateQPqgKZFBMgZkFZaXHIf9RV4MDW5xpTK2b/+qbwOg==}
peerDependencies:
'@types/dom-mediacapture-record': ^1
@@ -7079,21 +7079,21 @@ snapshots:
'@jridgewell/resolve-uri': 3.1.2
'@jridgewell/sourcemap-codec': 1.6.0
'@livekit/components-core@0.12.15(livekit-client@2.22.2(@types/dom-mediacapture-record@1.0.22))(tslib@2.8.1)':
'@livekit/components-core@0.12.15(livekit-client@2.22.3(@types/dom-mediacapture-record@1.0.22))(tslib@2.8.1)':
dependencies:
'@floating-ui/dom': 1.7.6
livekit-client: 2.22.2(@types/dom-mediacapture-record@1.0.22)
livekit-client: 2.22.3(@types/dom-mediacapture-record@1.0.22)
loglevel: 1.9.1
rxjs: 7.8.2
tslib: 2.8.1
'@livekit/components-react@2.9.24(livekit-client@2.22.2(@types/dom-mediacapture-record@1.0.22))(react-dom@19.2.8(react@19.2.8))(react@19.2.8)(tslib@2.8.1)':
'@livekit/components-react@2.9.24(livekit-client@2.22.3(@types/dom-mediacapture-record@1.0.22))(react-dom@19.2.8(react@19.2.8))(react@19.2.8)(tslib@2.8.1)':
dependencies:
'@livekit/components-core': 0.12.15(livekit-client@2.22.2(@types/dom-mediacapture-record@1.0.22))(tslib@2.8.1)
'@livekit/components-core': 0.12.15(livekit-client@2.22.3(@types/dom-mediacapture-record@1.0.22))(tslib@2.8.1)
clsx: 2.1.1
events: 3.3.0
jose: 6.2.9
livekit-client: 2.22.2(@types/dom-mediacapture-record@1.0.22)
livekit-client: 2.22.3(@types/dom-mediacapture-record@1.0.22)
react: 19.2.8
react-dom: 19.2.8(react@19.2.8)
tslib: 2.8.1
@@ -7105,11 +7105,11 @@ snapshots:
dependencies:
'@bufbuild/protobuf': 1.10.1
'@livekit/track-processors@0.7.2(@types/dom-mediacapture-transform@0.1.11)(livekit-client@2.22.2(@types/dom-mediacapture-record@1.0.22))':
'@livekit/track-processors@0.7.2(@types/dom-mediacapture-transform@0.1.11)(livekit-client@2.22.3(@types/dom-mediacapture-record@1.0.22))':
dependencies:
'@mediapipe/tasks-vision': 0.10.35
'@types/dom-mediacapture-transform': 0.1.11
livekit-client: 2.22.2(@types/dom-mediacapture-record@1.0.22)
livekit-client: 2.22.3(@types/dom-mediacapture-record@1.0.22)
'@matrix-org/matrix-sdk-crypto-wasm@18.5.0': {}
@@ -10180,7 +10180,7 @@ snapshots:
lines-and-columns@1.2.4: {}
livekit-client@2.22.2(@types/dom-mediacapture-record@1.0.22):
livekit-client@2.22.3(@types/dom-mediacapture-record@1.0.22):
dependencies:
'@livekit/mutex': 1.1.1
'@livekit/protocol': 1.50.4
+4
View File
@@ -25,6 +25,7 @@ import {
import {
setLogExtension as setLKLogExtension,
setLogLevel as setLKLogLevel,
LoggerNames as LKLoggerNames,
} from "livekit-client";
import { getUrlParams } from "./UrlParams";
@@ -226,6 +227,9 @@ export class Initializer {
enableExtendedLivekitLogs.value$.subscribe((enabled) => {
setLKLogLevel(enabled ? "trace" : "info");
// ICE candidate types and connection states are what you need to diagnose
// "could not establish pc connection", so always keep those in the rageshake
setLKLogLevel("debug", LKLoggerNames.ICE);
});
window.setLKLogLevel = setLKLogLevel;
+53
View File
@@ -24,10 +24,12 @@ import { testAudioContext } from "../useAudioContext.test";
import * as MediaDevicesContext from "../MediaDevicesContext";
import { LivekitRoomAudioRenderer } from "./MatrixAudioRenderer";
import {
mockLocalParticipant,
mockMediaDevices,
mockRemoteParticipant,
mockTrack,
} from "../utils/test";
import { logger as rootLogger } from "matrix-js-sdk/lib/logger";
import { initializeWidget } from "../widget";
initializeWidget();
export const TestAudioContextConstructor = vi.fn(
@@ -131,6 +133,57 @@ it("should render for member", () => {
expect(queryAllByTestId("audio")).toHaveLength(1);
});
function spyOnWarn(): ReturnType<typeof vi.fn> {
const warn = vi.fn();
vi.spyOn(rootLogger, "getChild").mockReturnValue({
warn,
} as unknown as typeof rootLogger);
return warn;
}
it("should not render or warn for the local participant", () => {
const warn = spyOnWarn();
const local = mockLocalParticipant({ identity: "@alice:DEV0" });
vi.mocked(useTracks).mockReturnValue([mockTrack(local)]);
const { queryAllByTestId } = render(
<MediaDevicesProvider value={mockMediaDevices({})}>
<LivekitRoomAudioRenderer
validIdentities={[]}
livekitRoom={{ remoteParticipants: new Map() } as unknown as Room}
url={""}
/>
</MediaDevicesProvider>,
);
expect(queryAllByTestId("audio")).toHaveLength(0);
expect(warn).not.toHaveBeenCalled();
});
it("should warn only once per unexpected participant", () => {
const warn = spyOnWarn();
const { rerender } = renderTestComponent(
[{ userId: "@bob", deviceId: "DEV0" }],
["@alice:DEV0"],
[
{
participantId: "@alice:DEV0",
kind: Track.Kind.Audio,
source: Track.Source.Microphone,
},
],
);
expect(warn).toHaveBeenCalledTimes(1);
rerender(
<MediaDevicesProvider value={mockMediaDevices({})}>
<LivekitRoomAudioRenderer
validIdentities={[]}
livekitRoom={{ remoteParticipants: new Map() } as unknown as Room}
url={""}
/>
</MediaDevicesProvider>,
);
expect(warn).toHaveBeenCalledTimes(1);
});
it("should not render without member", () => {
const { container, queryAllByTestId } = renderTestComponent(
[{ userId: "@bob", deviceId: "DEV0" }],
+17 -9
View File
@@ -8,7 +8,7 @@ Please see LICENSE in the repository root for full details.
import { getTrackReferenceId } from "@livekit/components-core";
import { type Room as LivekitRoom } from "livekit-client";
import { type RemoteAudioTrack, Track } from "livekit-client";
import { useEffect, useMemo, useState, type ReactNode } from "react";
import { useEffect, useMemo, useRef, useState, type ReactNode } from "react";
import {
useTracks,
AudioTrack,
@@ -60,6 +60,9 @@ export function LivekitRoomAudioRenderer({
muted,
}: MatrixAudioRendererProps): ReactNode {
const logger = rootLogger.getChild("[MatrixAudioRenderer]");
// Identities we have already warned about, so that re-renders (which happen
// on every active speaker update) don't repeat the warning.
const warnedIdentities = useRef(new Set<string>());
const tracks = useTracks(
[
Track.Source.Microphone,
@@ -72,22 +75,27 @@ export function LivekitRoomAudioRenderer({
room: livekitRoom,
},
)
// Only keep audio tracks
.filter((ref) => ref.publication.kind === Track.Kind.Audio)
// Only keep remote audio tracks (we never render our own audio)
.filter(
(ref) =>
ref.publication.kind === Track.Kind.Audio && !ref.participant.isLocal,
)
// Only keep tracks from participants that are in the validIdentities list
.filter((ref) => {
const isValid = validIdentities.includes(ref.participant.identity);
if (!isValid) {
// TODO make sure to also skip the warn logging for the local identity
const { identity } = ref.participant;
const isValid = validIdentities.includes(identity);
if (!isValid && !warnedIdentities.current.has(identity)) {
warnedIdentities.current.add(identity);
// Log that there is an invalid identity, that means that someone is publishing audio that is not expected to be in the call.
logger.warn(
`Audio track ${ref.participant.identity} from ${url} has no matching matrix call member`,
`Audio track ${identity} from ${url} has no matching matrix call member`,
`current members: ${validIdentities.join()}`,
`track will not get rendered`,
);
return false;
} else if (isValid) {
warnedIdentities.current.delete(identity);
}
return true;
return isValid;
});
// This component is also (in addition to the "only play audio for connected members" logic above)
+87
View File
@@ -0,0 +1,87 @@
/*
Copyright 2026 Element Creations Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { describe, expect, it, vi } from "vitest";
import { type LocalVideoTrack } from "livekit-client";
import {
type BackgroundOptions,
type ProcessorWrapper,
} from "@livekit/track-processors";
import { applyProcessor, trackProcessorSync } from "./TrackProcessorContext";
import { constant } from "../state/Behavior";
import { flushPromises, testScope } from "../utils/test";
const processor = {} as ProcessorWrapper<BackgroundOptions>;
function mockTrack(
readyState: MediaStreamTrackState,
current?: ProcessorWrapper<BackgroundOptions>,
): LocalVideoTrack {
return {
mediaStreamTrack: { readyState },
getProcessor: vi.fn().mockReturnValue(current),
setProcessor: vi.fn().mockResolvedValue(undefined),
stopProcessor: vi.fn().mockResolvedValue(undefined),
} as unknown as LocalVideoTrack;
}
describe("applyProcessor", () => {
it("attaches the processor to a live track", () => {
const track = mockTrack("live");
applyProcessor(track, processor);
expect(track.setProcessor).toHaveBeenCalledWith(processor);
});
it("does not attach the processor to an ended track", () => {
const track = mockTrack("ended");
applyProcessor(track, processor);
expect(track.setProcessor).not.toHaveBeenCalled();
});
it("does not surface a rejected setProcessor", async () => {
const track = mockTrack("live");
vi.mocked(track.setProcessor).mockRejectedValue(
new TypeError("Input track cannot be ended"),
);
const unhandled = vi.fn();
process.on("unhandledRejection", unhandled);
applyProcessor(track, processor);
await flushPromises();
process.off("unhandledRejection", unhandled);
expect(unhandled).not.toHaveBeenCalled();
});
it("stops the processor when none is wanted", () => {
const track = mockTrack("live", processor);
applyProcessor(track, undefined);
expect(track.stopProcessor).toHaveBeenCalled();
});
it("does not surface a rejected stopProcessor", async () => {
const track = mockTrack("live", processor);
vi.mocked(track.stopProcessor).mockRejectedValue(new Error("nope"));
const unhandled = vi.fn();
process.on("unhandledRejection", unhandled);
applyProcessor(track, undefined);
await flushPromises();
process.off("unhandledRejection", unhandled);
expect(unhandled).not.toHaveBeenCalled();
});
});
describe("trackProcessorSync", () => {
it("applies the processor to the current track", () => {
const track = mockTrack("live");
trackProcessorSync(
testScope(),
constant(track),
constant({ supported: true, processor }),
);
expect(track.setProcessor).toHaveBeenCalledWith(processor);
});
});
+31 -13
View File
@@ -19,6 +19,7 @@ import {
useMemo,
} from "react";
import { type LocalVideoTrack } from "livekit-client";
import { logger } from "matrix-js-sdk/lib/logger";
import { combineLatest, map, type Observable } from "rxjs";
import { useObservable } from "observable-hooks";
@@ -65,6 +66,34 @@ export function useTrackProcessorObservable$(): Observable<ProcessorState> {
return state$;
}
/**
* Attaches or detaches the processor so that the track matches the desired
* state, without throwing.
*/
export function applyProcessor(
videoTrack: LocalVideoTrack,
processor: ProcessorWrapper<BackgroundOptions> | undefined,
): void {
if (processor && !videoTrack.getProcessor()) {
// A MediaStreamTrackProcessor cannot be constructed on an ended track
// (e.g. the camera was stopped while the processor was being applied),
// and setProcessor rejects with a TypeError. The track is going away
// anyway, so there is nothing to attach to.
if (videoTrack.mediaStreamTrack.readyState === "ended") {
logger.debug("Not attaching video processor to an ended track");
return;
}
videoTrack.setProcessor(processor).catch((e) => {
logger.warn("Failed to attach video processor", e);
});
}
if (!processor && videoTrack.getProcessor()) {
videoTrack.stopProcessor().catch((e) => {
logger.warn("Failed to stop video processor", e);
});
}
}
/**
* Updates your video tracks to always use the given processor.
*/
@@ -78,13 +107,7 @@ export const trackProcessorSync = (
.subscribe(([videoTrack, processorState]) => {
if (!processorState) return;
if (!videoTrack) return;
const { processor } = processorState;
if (processor && !videoTrack.getProcessor()) {
void videoTrack.setProcessor(processor);
}
if (!processor && videoTrack.getProcessor()) {
void videoTrack.stopProcessor();
}
applyProcessor(videoTrack, processorState.processor);
});
};
@@ -94,12 +117,7 @@ export const useTrackProcessorSync = (
const { processor } = useTrackProcessor();
useEffect(() => {
if (!videoTrack) return;
if (processor && !videoTrack.getProcessor()) {
void videoTrack.setProcessor(processor);
}
if (!processor && videoTrack.getProcessor()) {
void videoTrack.stopProcessor();
}
applyProcessor(videoTrack, processor);
}, [processor, videoTrack]);
};
+143 -7
View File
@@ -52,8 +52,10 @@ import {
aliceRtcMember,
aliceUserId,
bob,
bobDeviceId,
bobId,
bobRtcMember,
bobUserId,
local,
localId,
localRtcMember,
@@ -118,6 +120,19 @@ const daveId = `${dave.userId}:${daveRtcMember.deviceId}`;
const bobParticipant = mockRemoteParticipant({ identity: bobId });
const daveParticipant = mockRemoteParticipant({ identity: daveId });
// A LiveKit participant that no MatrixRTC membership accounts for
const rogueParticipant = mockRemoteParticipant({ identity: "rogue" });
const rogueId = `unknown:${exampleTransport.livekit_service_url}:rogue`;
const otherTransport: LivekitTransport = {
type: "livekit",
livekit_service_url: "https://lk.other.example.org",
livekit_alias: "!alias:other.example.org",
};
const bobOnOtherFocusRtcMember = mockRtcMembership(bobUserId, bobDeviceId, {
fociPreferred: [otherTransport],
});
export interface GridLayoutSummary {
type: "grid";
spotlight?: string[];
@@ -450,6 +465,113 @@ describe.each(modes)("CallViewModel (%s mode)", (mode) => {
});
});
test("LiveKit participants without a membership get an unknown participant tile", () => {
withTestScheduler(({ behavior, expectObservable }) => {
// A participant nobody's membership accounts for connects on frame 1 and
// disconnects on frame 2
const participantInputMarbles = "aba";
// It gets a tile at the end of the grid for as long as it is connected
const expectedLayoutMarbles = " aba";
withCallViewModel(
{
remoteParticipants$: behavior(participantInputMarbles, {
a: [aliceParticipant, bobParticipant],
b: [aliceParticipant, bobParticipant, rogueParticipant],
}),
rtcMembers$: constant([localRtcMember, aliceRtcMember, bobRtcMember]),
},
(vm) => {
expectObservable(summarizeLayout$(vm.layout$)).toBe(
expectedLayoutMarbles,
{
a: {
type: "grid",
spotlight: undefined,
grid: [`${localId}:0`, `${aliceId}:0`, `${bobId}:0`],
},
b: {
type: "grid",
spotlight: undefined,
grid: [`${localId}:0`, `${aliceId}:0`, `${bobId}:0`, rogueId],
},
},
);
},
);
});
});
test("an unknown participant takes the call out of one-on-one layout", () => {
withTestScheduler(({ behavior, expectObservable }) => {
// A participant nobody's membership accounts for connects on frame 1 and
// disconnects on frame 2
const participantInputMarbles = "aba";
// The one-on-one layout would hide the unknown participant, so we must
// fall back to the grid while they are present
const expectedLayoutMarbles = " aba";
withCallViewModel(
{
remoteParticipants$: behavior(participantInputMarbles, {
a: [aliceParticipant],
b: [aliceParticipant, rogueParticipant],
}),
rtcMembers$: constant([localRtcMember, aliceRtcMember]),
},
(vm) => {
expectObservable(summarizeLayout$(vm.layout$)).toBe(
expectedLayoutMarbles,
{
a: {
type: "one-on-one-desktop",
pip: `${localId}:0`,
spotlight: `${aliceId}:0`,
},
b: {
type: "grid",
spotlight: undefined,
grid: [`${localId}:0`, `${aliceId}:0`, rogueId],
},
},
);
},
);
});
});
test("a member seen on a focus other than their own is not an unknown participant", () => {
withTestScheduler(({ expectObservable }) => {
// Bob publishes to another focus, but also connects to ours to subscribe
// (the mock reports the same participants on every connection). Matching
// by identity rather than per transport keeps him from being flagged.
const expectedLayoutMarbles = "a";
withCallViewModel(
{
remoteParticipants$: constant([aliceParticipant, bobParticipant]),
rtcMembers$: constant([
localRtcMember,
aliceRtcMember,
bobOnOtherFocusRtcMember,
]),
},
(vm) => {
expectObservable(summarizeLayout$(vm.layout$)).toBe(
expectedLayoutMarbles,
{
a: {
type: "grid",
spotlight: undefined,
grid: [`${localId}:0`, `${aliceId}:0`, `${bobId}:0`],
},
},
);
},
);
});
});
test("one-on-one mobile layout shows local tile when video is enabled", () => {
withTestScheduler(({ behavior, schedule, expectObservable }) => {
// Local participant enables their video, then disables it
@@ -1211,15 +1333,18 @@ describe.each(modes)("CallViewModel (%s mode)", (mode) => {
});
});
test("participants must have a MatrixRTCSession to be visible", () => {
test("participants without a MatrixRTC membership are shown as unknown, without media", () => {
withTestScheduler(({ behavior, expectObservable }) => {
// iterate through a number of combinations of participants and MatrixRTC memberships
// Bob never has an MatrixRTC membership
const participantInputMarbles = "abcd-c";
// Bob even tries to share his screen at the end
const bobSharingInputMarbles = " n---yn";
// Bob should never be visible
const expectedLayoutMarbles = " a-bc-b";
// Bob gets an unknown participant tile (MSC4143) rather than a tile of
// his own, and his screen share never shows up. His presence also keeps
// the call out of the one-on-one layout, which would hide him.
const expectedLayoutMarbles = " abcd-c";
const bobUnknownId = `unknown:${exampleTransport.livekit_service_url}:${bobId}`;
withCallViewModel(
{
@@ -1251,14 +1376,25 @@ describe.each(modes)("CallViewModel (%s mode)", (mode) => {
grid: [`${localId}:0`],
},
b: {
type: "one-on-one-desktop",
pip: `${localId}:0`,
spotlight: `${aliceId}:0`,
type: "grid",
spotlight: undefined,
grid: [`${localId}:0`, bobUnknownId],
},
// Tiles keep their positions, so Bob stays ahead of the others
c: {
type: "grid",
spotlight: undefined,
grid: [`${localId}:0`, `${aliceId}:0`, `${daveId}:0`],
grid: [`${localId}:0`, bobUnknownId, `${aliceId}:0`],
},
d: {
type: "grid",
spotlight: undefined,
grid: [
`${localId}:0`,
bobUnknownId,
`${aliceId}:0`,
`${daveId}:0`,
],
},
},
);
+92 -19
View File
@@ -95,6 +95,7 @@ import {
layoutShallowEquals,
type Alignment,
type GridLayoutMedia,
type GridMediaViewModel,
type Layout,
type LayoutMedia,
type OneOnOneDesktopLayoutMedia,
@@ -162,6 +163,10 @@ import {
createRingingMedia,
type RingingMediaViewModel,
} from "../media/RingingMediaViewModel.ts";
import {
createUnknownParticipantMedia,
type UnknownParticipantMediaViewModel,
} from "../media/UnknownParticipantMediaViewModel.ts";
import { type GridTileViewModel } from "../TileViewModel.ts";
//TODO
@@ -869,6 +874,61 @@ export function createCallViewModel$(
),
);
/**
* Media for LiveKit participants that cannot be mapped to any MatrixRTC
* membership. MSC4143 asks us to surface these rather than ignore them, as
* they can signal an eavesdropping or impersonation attack.
*
* Participants are matched to memberships by identity alone rather than per
* transport, since a member legitimately shows up as a subscribe-only
* participant on every focus other than the one they publish to.
*/
const unknownParticipantMedia$ = scope.behavior<
UnknownParticipantMediaViewModel[]
>(
combineLatest([
memberships$,
connectionManager.connectionManagerData$,
]).pipe(
// Memberships and connections derive from the same epoch of session
// state; comparing them across epochs would flag members whose
// connection hasn't caught up yet.
filter(([memberships, data]) => memberships.epoch === data.epoch),
map(([memberships, data]) => {
const knownIdentities = new Set(
memberships.value.map((m) => m.rtcBackendIdentity),
);
return data.value.getConnections().flatMap((connection) =>
data.value
.getParticipantsForTransport(connection.transport)
.filter((p) => !knownIdentities.has(p.identity))
.map((p) => ({
focusUrl: connection.transport.livekit_service_url,
identity: p.identity,
})),
);
}),
generateItems(
"CallViewModel unknownParticipantMedia$",
function* (unknownParticipants) {
for (const { focusUrl, identity } of unknownParticipants)
yield { keys: [focusUrl, identity], data: undefined };
},
(_scope, _data$, focusUrl, identity) => {
logger.warn(
`LiveKit participant ${identity} on ${focusUrl} matches no MatrixRTC membership`,
);
return createUnknownParticipantMedia({
id: `unknown:${focusUrl}:${identity}`,
rtcBackendIdentity: identity,
focusUrl,
});
},
),
),
[],
);
const ringingMedia$ = scope.behavior<RingingMediaViewModel | null>(
ringAttempts$.pipe(
switchMap(({ intent, recipient, outcome$ }) =>
@@ -1015,21 +1075,29 @@ export function createCallViewModel$(
),
);
const grid$ = scope.behavior<UserMediaViewModel[]>(
userMedia$.pipe(
switchMap((mediaItems) => {
const bins = mediaItems.map((m) =>
m.bin$.pipe(map((bin) => [m, bin] as const)),
);
// Sort the media by bin order and generate a tile for each one
return bins.length === 0
? of([])
: combineLatest(bins, (...bins) =>
bins.sort(([, bin1], [, bin2]) => bin1 - bin2).map(([m]) => m),
);
}),
distinctUntilChanged(shallowArrayEquals),
),
const sortedUserMedia$: Observable<UserMediaViewModel[]> = userMedia$.pipe(
switchMap((mediaItems) => {
const bins = mediaItems.map((m) =>
m.bin$.pipe(map((bin) => [m, bin] as const)),
);
// Sort the media by bin order and generate a tile for each one
return bins.length === 0
? of([])
: combineLatest(bins, (...bins) =>
bins.sort(([, bin1], [, bin2]) => bin1 - bin2).map(([m]) => m),
);
}),
);
const grid$ = scope.behavior<GridMediaViewModel[]>(
combineLatest(
[sortedUserMedia$, unknownParticipantMedia$],
// Unknown participants have no media to sort by, so they go last
(userMedia, unknownParticipants) => [
...userMedia,
...unknownParticipants,
],
).pipe(distinctUntilChanged(shallowArrayEquals)),
);
/**
@@ -1189,10 +1257,15 @@ export function createCallViewModel$(
local: LocalUserMediaViewModel;
remote: UserMediaViewModel | RingingMediaViewModel;
} | null> = scope.behavior(
combineLatest([userMedia$, screenShares$]).pipe(
switchMap(([userMedia, screenShares]) => {
// One-on-one layout only supports 2 user media, no screen shares
if (userMedia.length <= 2 && screenShares.length === 0) {
combineLatest([userMedia$, screenShares$, unknownParticipantMedia$]).pipe(
switchMap(([userMedia, screenShares, unknownParticipants]) => {
// One-on-one layout only supports 2 user media, no screen shares, and
// must never hide an unknown participant
if (
userMedia.length <= 2 &&
screenShares.length === 0 &&
unknownParticipants.length === 0
) {
const local = userMedia.find(
(vm): vm is WrappedUserMediaViewModel & LocalUserMediaViewModel =>
vm.type === "user" && vm.local,
@@ -394,6 +394,34 @@ describe("Start connection states", () => {
expect(connectedState).toEqual(ConnectionState.LivekitConnected);
});
it("stopping while connecting does not report an error", async () => {
setupTest();
const connection = setupRemoteConnection();
const capturedStates: (ConnectionState | Error)[] = [];
const s = connection.state$.subscribe((value) => {
capturedStates.push(value);
});
onTestFinished(() => s.unsubscribe());
// livekit-client rejects a pending connect() when disconnect() is called.
const pendingConnect = Promise.withResolvers<void>();
fakeLivekitRoom.connect.mockReturnValue(pendingConnect.promise);
fakeLivekitRoom.disconnect.mockResolvedValue(undefined);
const started = connection.start();
await vi.waitFor(() => expect(fakeLivekitRoom.connect).toHaveBeenCalled());
const stopping = connection.stop();
pendingConnect.reject(new Error("Client initiated disconnect"));
await stopping;
// start() resolves rather than rejecting (it is not awaited by the
// ConnectionManager, so a rejection would be unhandled).
await expect(started).resolves.toBeUndefined();
expect(capturedStates.at(-1)).toEqual(ConnectionState.Stopped);
expect(capturedStates.some((st) => st instanceof Error)).toBe(false);
});
it("shutting down the scope should stop the connection", async () => {
setupTest();
vi.useFakeTimers();
@@ -373,6 +373,15 @@ export class Connection {
// If we were stopped while connecting, don't proceed to update state.
if (this.stopped) return;
} catch (error) {
if (this.stopped) {
// stop() was called while we were connecting, which makes the pending
// connect reject. That is the abort we asked for, not a failure, so
// don't record an error state on a stopped connection or rethrow it
// (start() is not awaited by the ConnectionManager, so a throw here
// becomes an unhandled promise rejection).
this.logger.debug(`Connect aborted because the connection was stopped`);
return;
}
this.logger.debug(`Failed to connect to LiveKit room: ${error}`);
this._state$.next(
error instanceof ElementCallError
@@ -411,9 +420,11 @@ export class Connection {
`stop: disconnecing from lk room ${this.transport.livekit_service_url}`,
);
if (this.stopped) return;
// Mark as stopped before disconnecting so that a connect() aborted by the
// disconnect sees the flag and does not report the abort as an error.
this.stopped = true;
await this.livekitRoom.disconnect();
this._state$.next(ConnectionState.Stopped);
this.stopped = true;
this.logger.debug(
`stop: DONE disconnecing from lk room ${this.transport.livekit_service_url}`,
);
+13 -12
View File
@@ -8,12 +8,15 @@ Please see LICENSE in the repository root for full details.
import { BehaviorSubject } from "rxjs";
import { logger } from "matrix-js-sdk/lib/logger";
import { GridTileViewModel, SpotlightTileViewModel } from "./TileViewModel";
import {
type GridTileMediaViewModel,
GridTileViewModel,
SpotlightTileViewModel,
} from "./TileViewModel";
import { fillGaps } from "../utils/iter";
import { debugTileLayout } from "../settings/settings";
import { type MediaViewModel } from "./media/MediaViewModel";
import { type UserMediaViewModel } from "./media/UserMediaViewModel";
import { type RingingMediaViewModel } from "./media/RingingMediaViewModel";
type SpotlightBackground = "solid" | "transparent";
@@ -68,10 +71,8 @@ class SpotlightTileData {
}
class GridTileData {
private readonly media$: BehaviorSubject<
UserMediaViewModel | RingingMediaViewModel
>;
public get media(): UserMediaViewModel | RingingMediaViewModel {
private readonly media$: BehaviorSubject<GridTileMediaViewModel>;
public get media(): GridTileMediaViewModel {
return this.media$.value;
}
public set media(value: UserMediaViewModel) {
@@ -80,7 +81,7 @@ class GridTileData {
public readonly vm: GridTileViewModel;
public constructor(media: UserMediaViewModel | RingingMediaViewModel) {
public constructor(media: GridTileMediaViewModel) {
this.media$ = new BehaviorSubject(media);
this.vm = new GridTileViewModel(this.media$);
}
@@ -140,7 +141,7 @@ export class TileStoreBuilder {
: null;
private readonly prevGridByMedia: Map<
MediaViewModel,
GridTileMediaViewModel,
[GridTileData, number]
> = new Map(
this.prevGrid.map((entry, i) => [entry.media, [entry, i]] as const),
@@ -205,9 +206,7 @@ export class TileStoreBuilder {
* Sets up a grid tile for the given media. If this is never called for some
* media, then that media will have no grid tile.
*/
public registerGridTile(
media: UserMediaViewModel | RingingMediaViewModel,
): void {
public registerGridTile(media: GridTileMediaViewModel): void {
if (DEBUG_ENABLED)
logger.debug(
`[TileStore, ${this.generation}] register grid tile: ${media.displayName$.value}`,
@@ -215,8 +214,10 @@ export class TileStoreBuilder {
if (this.spotlight !== null) {
// We actually *don't* want spotlight speakers to appear in both the
// spotlight and the grid, so they're filtered out here
// spotlight and the grid, so they're filtered out here. (Unknown
// participants never make it into the spotlight.)
if (
media.type !== "unknown participant" &&
!(media.type === "user" && media.local) &&
this.spotlight.media.includes(media)
)
+7 -3
View File
@@ -10,8 +10,14 @@ import { BehaviorSubject } from "rxjs";
import { type Behavior } from "./Behavior";
import { type MediaViewModel } from "./media/MediaViewModel";
import { type RingingMediaViewModel } from "./media/RingingMediaViewModel";
import { type UnknownParticipantMediaViewModel } from "./media/UnknownParticipantMediaViewModel";
import { type UserMediaViewModel } from "./media/UserMediaViewModel";
export type GridTileMediaViewModel =
| UserMediaViewModel
| RingingMediaViewModel
| UnknownParticipantMediaViewModel;
let nextId = 0;
function createId(): string {
return (nextId++).toString();
@@ -23,9 +29,7 @@ export class GridTileViewModel {
public readonly showOutline$: Behavior<boolean> = this._showOutline$;
public constructor(
public readonly media$: Behavior<
UserMediaViewModel | RingingMediaViewModel
>,
public readonly media$: Behavior<GridTileMediaViewModel>,
) {}
public setShowOutline(value: boolean): void {
+8 -3
View File
@@ -10,6 +10,7 @@ import { type BehaviorSubject } from "rxjs";
import { type LocalUserMediaViewModel } from "./media/LocalUserMediaViewModel.ts";
import { type MediaViewModel } from "./media/MediaViewModel.ts";
import { type RingingMediaViewModel } from "./media/RingingMediaViewModel.ts";
import { type UnknownParticipantMediaViewModel } from "./media/UnknownParticipantMediaViewModel.ts";
import { type UserMediaViewModel } from "./media/UserMediaViewModel.ts";
import {
type GridTileViewModel,
@@ -18,25 +19,29 @@ import {
import { type Behavior } from "./Behavior.ts";
import { shallowEquals as arrayShallowEquals } from "../utils/array.ts";
export type GridMediaViewModel =
| UserMediaViewModel
| UnknownParticipantMediaViewModel;
export interface GridLayoutMedia {
type: "grid";
edgeToEdge: false;
spotlight?: MediaViewModel[];
grid: UserMediaViewModel[];
grid: GridMediaViewModel[];
}
export interface SpotlightLandscapeLayoutMedia {
type: "spotlight-landscape";
edgeToEdge: boolean;
spotlight: MediaViewModel[];
grid: UserMediaViewModel[];
grid: GridMediaViewModel[];
}
export interface SpotlightPortraitLayoutMedia {
type: "spotlight-portrait";
edgeToEdge: false;
spotlight: MediaViewModel[];
grid: UserMediaViewModel[];
grid: GridMediaViewModel[];
}
export interface SpotlightExpandedLayoutMedia {
@@ -0,0 +1,57 @@
/*
Copyright 2026 Element Creations Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { constant } from "../Behavior";
import { type BaseMediaViewModel } from "./MediaViewModel";
/**
* Media representing a LiveKit participant that cannot be mapped to any
* MatrixRTC member of the session.
*
* MSC4143 asks clients to surface such streams rather than silently ignore
* them, because they can signal an eavesdropping or impersonation attack. We
* therefore give them a tile, but never render their audio or video.
*
* There is no Matrix user behind this media, so `userId` and `displayName$`
* carry the LiveKit identity instead, purely so that debugging output stays
* meaningful.
*/
export interface UnknownParticipantMediaViewModel extends BaseMediaViewModel {
type: "unknown participant";
/**
* The LiveKit identity under which the participant connected. Exposed for
* debugging.
*/
rtcBackendIdentity: string;
/**
* The URL of the LiveKit focus on which the participant was seen. Exposed for
* debugging.
*/
focusUrl: string;
}
export interface UnknownParticipantMediaInputs {
id: string;
rtcBackendIdentity: string;
focusUrl: string;
}
export function createUnknownParticipantMedia({
id,
rtcBackendIdentity,
focusUrl,
}: UnknownParticipantMediaInputs): UnknownParticipantMediaViewModel {
return {
type: "unknown participant",
id,
userId: rtcBackendIdentity,
displayName$: constant(rtcBackendIdentity),
mxcAvatarUrl$: constant(undefined),
rtcBackendIdentity,
focusUrl,
};
}
+98
View File
@@ -0,0 +1,98 @@
/*
Copyright 2026 Element Creations Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { expect, within } from "storybook/test";
import { type JSX, useMemo } from "react";
import type { Meta, StoryObj } from "@storybook/react-vite";
import { GridTile } from "./GridTile";
import { GridTileViewModel } from "../state/TileViewModel";
import { constant } from "../state/Behavior";
import { createUnknownParticipantMedia } from "../state/media/UnknownParticipantMediaViewModel";
interface UnknownParticipantTileStoryProps {
width: number;
height: number;
rtcBackendIdentity: string;
showNameTags: boolean;
showOutline: boolean;
}
/**
* Renders a GridTile for a LiveKit participant that maps to no MatrixRTC
* member, driven by primitive props so that Storybook can document them.
*/
function UnknownParticipantTile({
width,
height,
rtcBackendIdentity,
...props
}: UnknownParticipantTileStoryProps): JSX.Element {
const vm = useMemo(
() =>
new GridTileViewModel(
constant(
createUnknownParticipantMedia({
id: `unknown:https://rtc.example.org:${rtcBackendIdentity}`,
rtcBackendIdentity,
focusUrl: "https://rtc.example.org",
}),
),
),
[rtcBackendIdentity],
);
return (
<GridTile
vm={vm}
onOpenProfile={null}
targetWidth={width}
targetHeight={height}
style={{ width, height }}
showSpeakingIndicators={false}
showRingingStatus={false}
focusable
{...props}
/>
);
}
const meta = {
component: UnknownParticipantTile,
argTypes: {
width: { control: { type: "range", min: 80, max: 800, step: 10 } },
height: { control: { type: "range", min: 80, max: 600, step: 10 } },
},
} satisfies Meta<typeof UnknownParticipantTile>;
export default meta;
type Story = StoryObj<typeof meta>;
export const UnknownParticipant: Story = {
args: {
width: 400,
height: 300,
rtcBackendIdentity: "@rogue:example.org:DEVICE",
showNameTags: true,
showOutline: false,
},
play: async ({ canvasElement }) => {
const canvas = within(canvasElement);
await expect(canvas.getByText("Unknown participant")).toBeInTheDocument();
await expect(
canvas.queryByText("@rogue:example.org:DEVICE"),
).not.toBeInTheDocument();
},
};
/** Narrow tiles hide the name tag, so the label goes with it. */
export const UnknownParticipantNarrow: Story = {
args: {
...UnknownParticipant.args,
width: 90,
height: 90,
},
};
+31
View File
@@ -32,6 +32,7 @@ import {
createRingingMedia,
type RingingMediaViewModel,
} from "../state/media/RingingMediaViewModel";
import { createUnknownParticipantMedia } from "../state/media/UnknownParticipantMediaViewModel";
global.IntersectionObserver = class MockIntersectionObserver {
public observe(): void {}
@@ -165,3 +166,33 @@ test("GridTile displays ringing media", async () => {
act(() => pickupState$.next("decline"));
screen.getByText("Call ended");
});
test("GridTile displays an unknown participant", async () => {
const vm = createUnknownParticipantMedia({
id: "unknown:https://rtc-example.org:rogue",
rtcBackendIdentity: "rogue",
focusUrl: "https://rtc-example.org",
});
const { container } = render(
<ReactionsSenderProvider vm={callVm} rtcSession={fakeRtcSession}>
<GridTile
vm={new GridTileViewModel(constant(vm))}
onOpenProfile={() => {}}
targetWidth={300}
targetHeight={200}
showSpeakingIndicators
showNameTags
showRingingStatus
showOutline
focusable
/>
</ReactionsSenderProvider>,
);
expect(await axe(container)).toHaveNoViolations();
// The tile is labelled as unknown rather than after its LiveKit identity
screen.getByText("Unknown participant");
expect(screen.queryByText("rogue")).toBeNull();
// There is no user to show an avatar for
expect(container.querySelector("[data-style]")).toBeNull();
});
+48
View File
@@ -51,6 +51,7 @@ import { type LocalUserMediaViewModel } from "../state/media/LocalUserMediaViewM
import { type RemoteUserMediaViewModel } from "../state/media/RemoteUserMediaViewModel";
import { type UserMediaViewModel } from "../state/media/UserMediaViewModel";
import { type RingingMediaViewModel } from "../state/media/RingingMediaViewModel";
import { type UnknownParticipantMediaViewModel } from "../state/media/UnknownParticipantMediaViewModel";
import { RingingStatus } from "./RingingStatus";
interface TileProps {
@@ -97,6 +98,44 @@ const RingingMediaTile: FC<RingingMediaTileProps> = ({
);
};
interface UnknownParticipantTileProps extends Omit<
TileProps,
"displayName" | "mxcAvatarUrl"
> {
vm: UnknownParticipantMediaViewModel;
}
/**
* A blank tile standing in for a LiveKit participant that cannot be mapped to
* any MatrixRTC member. MSC4143 asks clients to surface such streams rather
* than ignore them. No media is rendered for them.
*/
const UnknownParticipantTile: FC<UnknownParticipantTileProps> = ({
vm,
className,
...props
}) => {
const { t } = useTranslation();
return (
<MediaView
className={classNames(className, styles.tile)}
video={undefined}
userId={vm.userId}
unencryptedWarning={false}
videoEnabled={false}
mirror={false}
showAvatar={false}
displayName={t("video_tile.unknown_participant")}
mxcAvatarUrl={undefined}
rtcBackendIdentity={vm.rtcBackendIdentity}
focusUrl={vm.focusUrl}
{...props}
/>
);
};
UnknownParticipantTile.displayName = "UnknownParticipantTile";
interface UserMediaTileProps extends TileProps {
vm: UserMediaViewModel;
showSpeakingIndicators: boolean;
@@ -454,6 +493,15 @@ export const GridTile: FC<GridTileProps> = ({
{...props}
/>
);
} else if (media.type === "unknown participant") {
return (
<UnknownParticipantTile
ref={ref}
vm={media}
className={classNames(className, { [styles.outline]: showOutline })}
{...props}
/>
);
} else if (media.local) {
return (
<LocalUserMediaTile
+17 -9
View File
@@ -56,6 +56,11 @@ interface Props extends ComponentProps<typeof animated.div> {
displayName: string;
mxcAvatarUrl: string | undefined;
avatarStyle?: "solid" | "translucent";
/**
* Whether to show an avatar when there is no video. Off for media that has no
* Matrix user behind it.
*/
showAvatar?: boolean;
background?: "solid" | "transparent";
focusable: boolean;
primaryButton?: ReactNode;
@@ -95,6 +100,7 @@ export const MediaView: FC<Props> = ({
displayName,
mxcAvatarUrl,
avatarStyle = "solid",
showAvatar = true,
background = "solid",
focusable,
primaryButton,
@@ -176,15 +182,17 @@ export const MediaView: FC<Props> = ({
<div className={styles.speakingBorder} />
</div>
)}
<Avatar
id={userId}
name={displayName}
size={avatarSize}
src={mxcAvatarUrl}
data-style={avatarStyle}
className={styles.avatar}
style={{ display: video && videoEnabled ? "none" : "initial" }}
/>
{showAvatar && (
<Avatar
id={userId}
name={displayName}
size={avatarSize}
src={mxcAvatarUrl}
data-style={avatarStyle}
className={styles.avatar}
style={{ display: video && videoEnabled ? "none" : "initial" }}
/>
)}
{video?.publication !== undefined && (
<VideoTrack
trackRef={video}