Compare commits

...
Author SHA1 Message Date
Matthew Hodgson dbd2423e7b Stop MatrixAudioRenderer warning on every render
The "has no matching matrix call member" warning fired on every re-render
(i.e. every active speaker update) and also for the local participant's own
track, which is never rendered here anyway. Skip local tracks and warn once
per unexpected identity.
2026-09-09 11:20:49 +01:00
Robin b90ed5bdf7 Merge pull request #4247 from element-hq/fix-cd
Fix call.element.dev not updating after branch rename
2026-09-09 09:07:51 +02:00
Robin 1d7e2288f7 Fix call.element.dev not updating after branch rename 2026-09-09 08:49:08 +02:00
Robin 61eeb8171a Merge pull request #4242 from element-hq/delayed-leave-timings
Adapt delayed leave timings when delegation is available
2026-09-09 08:19:58 +02:00
Robin cff8766aa6 Delegate delayed leaves in LocalMember rather than LocalTransport
IMO this is where the delegation calls should have lived all along, since the leave event is part of the membership lifecycle, and we otherwise end up with an awkward hack to ignore transport updates.

Doing this now ensures that the client won't send any delegation requests if delegation is unsupported, and prepares the code for a future change in which we use the dedicated delegation endpoint from the CS API.
2026-09-08 21:22:52 +02:00
Robin 5cfe7ea4df Base JWT service endpoint choice directly on the MatrixRTC mode
Since we removed the extra 'legacy' mode, meaning there's nothing simplified anymore by translating things over to a JWT service version.

(Also, simplify some things by making the mode static for the duration of a call.)
2026-09-08 21:22:52 +02:00
Robin d38daef567 Reflect session config always being defined 2026-09-08 21:22:52 +02:00
Robin 09782a4d84 Give JWT token fetch procedures more appropriate names 2026-09-08 21:22:52 +02:00
Robin 839e36abd4 Make delegation support check fail immediately in case of CORS errors 2026-09-08 21:22:52 +02:00
Johannes Marbach 00d0adc3c4 Merge pull request #4222 from element-hq/renovate/all-minor-patch
Update all non-major dependencies
2026-09-08 11:40:20 +02:00
Johannes Marbach ddc5a50529 Disable new react compiler rules 2026-09-08 11:29:36 +02:00
renovate[bot] 27734a1a02 Update all non-major dependencies 2026-09-08 06:13:17 +00:00
Johannes Marbach 14c61db43d Merge pull request #4241 from element-hq/matthew/encryption-mismatch-logging
Log track encryption flag and cryptor events for remote tracks
2026-09-07 13:30:50 +02:00
Johannes Marbach 65e1b845a7 Apply suggestion from @Johennes 2026-09-07 13:18:00 +02:00
Robin 6a9583a2ce Adapt delayed leave timings when delegation is available
Splits the config options for the timings of a delayed leave event into two sets: one for when delegation is available (as you can relax the timings and get more stable calls this way), and another for when it's unavailable (as we must continue to gracefully downgrade even after Matrix 2.0 is fully rolled out).

This works by bluntly hitting the delegation endpoints without auth before joining to check for a 404.
2026-09-07 13:10:54 +02:00
Matthew Hodgson eddd95ba41 Log track encryption flag and cryptor events for remote tracks
Extends the [RemoteTracks] logging from #4235 with the per-publication
encryption flag and the ParticipantEncryptionStatusChanged /
EncryptionError room events.

If a publisher encrypts frames while the subscriber believes the
publication is unencrypted, livekit-client bypasses the cryptor and hands
raw ciphertext to the decoder, which is audible as loud noise bursts.
The reverse mismatch (or a missing/invalid key) drops frames instead.
Neither case is visible in a rageshake today.
2026-09-07 11:53:01 +01:00
Johannes Marbach a376280c91 Merge pull request #4240 from element-hq/renovate/livekit-client
Update dependency livekit-client to v2.22.2
2026-09-07 08:07:34 +02:00
renovate[bot] 1e9a5cfa20 Update ghcr.io/element-hq/element-web:develop Docker digest to 6ed62b2 (#4239)
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
2026-09-07 02:04:08 +00:00
renovate[bot] d204b36b55 Update dependency livekit-client to v2.22.2 2026-09-07 01:56:21 +00:00
renovate[bot] a03f23e720 Update ghcr.io/element-hq/element-web:develop Docker digest to a4e9c8c (#4238)
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
2026-09-06 02:02:35 +00:00
renovate[bot] 210d94af18 Update ghcr.io/element-hq/element-web:develop Docker digest to 3221e92 (#4237)
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
2026-09-05 02:55:22 +00:00
renovate[bot]andRobin 70f6e785ce Update ghcr.io/element-hq/element-web:develop Docker digest to 724b468 (#4226)
* Update ghcr.io/element-hq/element-web:develop Docker digest to a9bd290

* Fix Element Web integration tests

---------

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: Robin <robin@robin.town>
2026-09-04 09:18:19 +00:00
Johannes Marbach 0ad31cfc4d Merge pull request #4232 from element-hq/matthew/screenshare-watchdog
Surface screen share failures instead of silently doing nothing
2026-09-04 07:54:23 +02:00
Johannes Marbach bae8da3ddc Reformat 2026-09-04 07:44:16 +02:00
Johannes Marbach b22eba7af9 Add further tests 2026-09-04 07:39:58 +02:00
Robin cd7510e054 Merge pull request #4235 from element-hq/matthew/remote-track-logging
Log remote participant and track events per LiveKit connection
2026-09-03 20:22:27 +02:00
Robin 054913d17d Use testScope in ObservableScope tests 2026-09-03 20:11:18 +02:00
Matthew Hodgson dee4b05695 Diagnostics for duplicate participant matches and mid-delivery scope teardown
Warn when ConnectionManagerData merges participants from a second
connection to the same SFU URL, when more than one LiveKit participant
matches a member's backend identity, and when a scope is ended while one
of its behaviors is still delivering a value (which strands the later
bound subscribers on the old value). The per-member match log now also
names the connection the participant came from.
2026-09-03 18:43:18 +01:00
Matthew Hodgson b1393998a7 Report re-entrancy depth and tag torn behaviors
A single re-entrant emission can only strand a contiguous run of a
behavior's subscribers, so a tile that is both 'speaking' and 'waiting
for media' (a middle subscriber stranded) implies a nested re-entry.
Report the nesting depth and log each new depth once instead of only the
first re-entry, and tag splitBehavior-derived behaviors with their field
name so the warning identifies which behavior tore.
2026-09-03 18:08:53 +01:00
Matthew Hodgson 2ab2197c01 Warn when a behavior is re-entered while delivering a value
rxjs delivers a nested emission to every subscriber and then resumes
delivering the outer, older value to the remaining subscribers, which
leaves them permanently out of sync. Log the first occurrence per
behavior with a stack trace so the re-entrant path can be identified
from a rageshake.
2026-09-03 17:26:38 +01:00
Matthew Hodgson 0b95fd5a65 Log remote member participant matching and tile waiting state
A tile shows "Waiting for media" for as long as its MatrixRTC member
cannot be matched to a LiveKit participant. Log each transition of that
match and of the tile's waitingForMedia state so rageshakes can tie a
stuck tile to the LiveKit participant and track events.
2026-09-03 17:07:08 +01:00
Matthew Hodgson f8f5caaa09 Log remote participant and track events per LiveKit connection
Rageshakes contained nothing about the state of remote tracks, so a tile
showing the wrong mute or video state for a member could not be
diagnosed. Log connect/disconnect, publish/unpublish, subscribe/
unsubscribe, subscription failures, remote mute/unmute and stream state
changes on each Connection's logger, and remove the listeners when the
connection scope ends.
2026-09-03 16:50:30 +01:00
Matthew Hodgson bf5482db8c Surface screen share failures instead of silently doing nothing
toggleScreenSharing only had `.catch(logger.error)`, so a getDisplayMedia
request that hangs (element-call-rageshakes#17152: Element Desktop on
Windows, the user pressed the screen share button 14 times in 25 seconds
and the log shows nothing but the toggle lines and livekit-client's
"waiting for pending publication promise timed out") left the user with
a button that does nothing and us with no evidence of why.

Log when a toggle is requested and when it completes or fails, with the
elapsed time, so a hang is visible in the logs. Explicit failures other
than the user cancelling the picker show as a non-modal "Could not start
screen sharing" toast. Nothing is inferred from a toggle taking a long
time: the user may simply be choosing what to share.
2026-09-03 15:01:35 +01:00
Johannes Marbach d765267c43 Merge pull request #4229 from element-hq/matthew/fix-rageshake-flush
Fix periodic rageshake log flush never running
2026-09-03 07:29:51 +02:00
Matthew Hodgson 698a98a9c8 Test that logs are flushed to IndexedDB periodically 2026-09-02 18:55:18 +01:00
Matthew Hodgson 2db24e2868 Fix periodic rageshake log flush never running
The throttled flush callback returned this.flush instead of calling it
(regressed in #2607), so logs were only persisted to IndexedDB on
rageshake submission or beforeunload. When the host removes the widget
iframe at hangup, the whole call's logs were lost, so a rageshake filed
from a later call carries nothing from the affected one.
2026-09-02 18:46:12 +01:00
40 changed files with 2789 additions and 2074 deletions
+2 -2
View File
@@ -5,7 +5,7 @@
> Every PR must have a linked issue
> that a maintainer has reviewed and approved **before you started writing code**.
> PRs that don't meet this requirement will not be reviewed.
> See [CONTRIBUTING.md](https://github.com/element-hq/element-call/blob/livekit/CONTRIBUTING.md) for ElementCall decided for this approach.
> See [CONTRIBUTING.md](https://github.com/element-hq/element-call/blob/main/CONTRIBUTING.md) for ElementCall decided for this approach.
## Content
@@ -39,7 +39,7 @@ Uncomment the markdown table below and fill in the last line:
## Checklist
- [ ] A linked, pre-approved issue exists for this feature or UI change.
- [ ] I have read [CONTRIBUTING.md](https://github.com/element-hq/element-call/blob/livekit/CONTRIBUTING.md) in full.
- [ ] I have read [CONTRIBUTING.md](https://github.com/element-hq/element-call/blob/main/CONTRIBUTING.md) in full.
- [ ] Pull request includes screenshots or videos for any UI changes.
- [ ] Tests written for new code (and existing touched code where feasible).
- [ ] Linter and other CI checks pass.
+5 -5
View File
@@ -6,7 +6,7 @@ on:
- opened
- labeled
push:
branches: [livekit, full-mesh]
branches: [main]
jobs:
build_full_element_call:
# Use the full package vite build
@@ -22,8 +22,8 @@ jobs:
SENTRY_AUTH_TOKEN: ${{ secrets.SENTRY_AUTH_TOKEN }}
CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
deploy_develop:
# Deploy livekit branch to call.element.dev after build completes
if: github.ref == 'refs/heads/livekit'
# Deploy main branch to call.element.dev after build completes
if: github.ref == 'refs/heads/main'
needs: build_full_element_call
runs-on: ubuntu-latest
steps:
@@ -43,8 +43,8 @@ jobs:
}
})
docker_for_develop:
# Build docker and publish docker for livekit branch after build completes
if: github.ref == 'refs/heads/livekit'
# Build docker and publish docker for main branch after build completes
if: github.ref == 'refs/heads/main'
needs: build_full_element_call
permissions:
contents: write
+11
View File
@@ -39,6 +39,17 @@
"jsdoc/check-property-names": "error",
"jsdoc/require-param-description": "warn",
"react/display-name": "error",
// TODO: These React Compiler-based rules joined oxlint's correctness
// category in v1.81 and flag many pre-existing patterns, including hooks
// like useLatest/useInitial/useReactiveState that intentionally access
// refs during render. Re-evaluate them when adopting the React Compiler.
"react/immutability": "off",
"react/no-deriving-state-in-effects": "off",
"react/preserve-manual-memoization": "off",
"react/refs": "off",
"react/set-state-in-effect": "off",
"react/static-components": "off",
"react/use-memo": "off",
// TODO: Re-enable once oxlint supports lint rules that rely on TypeScript type-awareness.
// "rxjs/no-exposed-subjects": "error",
// "rxjs/finnish": [
-2
View File
@@ -9,8 +9,6 @@
"matrix_rtc_session": {
"wait_for_key_rotation_ms": 3000,
"membership_event_expiry_ms": 180000000,
"delayed_leave_event_delay_ms": 18000,
"delayed_leave_event_restart_ms": 4000,
"network_error_retry_ms": 100
}
}
-2
View File
@@ -13,8 +13,6 @@
"matrix_rtc_session": {
"wait_for_key_rotation_ms": 3000,
"membership_event_expiry_ms": 180000000,
"delayed_leave_event_delay_ms": 18000,
"delayed_leave_event_restart_ms": 4000,
"network_error_retry_ms": 100
}
}
-2
View File
@@ -9,8 +9,6 @@
"matrix_rtc_session": {
"wait_for_key_rotation_ms": 3000,
"membership_event_expiry_ms": 180000000,
"delayed_leave_event_delay_ms": 18000,
"delayed_leave_event_restart_ms": 4000,
"network_error_retry_ms": 100
},
"posthog": {
-2
View File
@@ -9,8 +9,6 @@
"matrix_rtc_session": {
"wait_for_key_rotation_ms": 3000,
"membership_event_expiry_ms": 180000000,
"delayed_leave_event_delay_ms": 18000,
"delayed_leave_event_restart_ms": 4000,
"network_error_retry_ms": 100
}
}
+2 -2
View File
@@ -13,7 +13,7 @@ services:
- ./backend/playwright_homeserver-othersite.yaml:/data/cfg/homeserver.yaml:Z
element-web:
# Pin to a SHA so that upstream cannot break our tests. Renovate handles regular updates.
image: ghcr.io/element-hq/element-web:develop@sha256:c107ec33fa349cb5c14fd03a4e13352d64ef264416dac96d273aa07b4004171f
image: ghcr.io/element-hq/element-web:develop@sha256:6ed62b2963ae098a1f79521eb516834dcf29452168e281cc359d9cfb4380adf6
element-web-1:
# Pin to a SHA so that upstream cannot break our tests. Renovate handles regular updates.
image: ghcr.io/element-hq/element-web:develop@sha256:c107ec33fa349cb5c14fd03a4e13352d64ef264416dac96d273aa07b4004171f
image: ghcr.io/element-hq/element-web:develop@sha256:6ed62b2963ae098a1f79521eb516834dcf29452168e281cc359d9cfb4380adf6
+1
View File
@@ -116,6 +116,7 @@
"peer_connection_timeout_description": "Connection to the media server timed out. Try switching to a different network or disabling your VPN. If the problem persists, see our <0>troubleshooting guide</0> or contact your server administrator.",
"room_creation_restricted": "Failed to create call",
"room_creation_restricted_description": "Call creation might be restricted to authorized users only. Try again later, or contact your server admin if the problem persists.",
"screen_share_failed": "Could not start screen sharing",
"sticky_events_required": "Homeserver does not support Matrix 2.0 calls",
"sticky_events_required_description": "This deployment is configured to use Matrix 2.0 call mode, but the homeserver does not advertise support for sticky events (MSC4354). Ask your server admin to upgrade, or switch the deployment to a compatible mode.",
"unexpected_ec_error": "An unexpected error occurred (<0>Error Code:</0> <1>{{ errorCode }}</1>). Please contact your server admin."
+2 -2
View File
@@ -81,7 +81,7 @@
"@vitejs/plugin-react": "^6.0.2",
"@vitest/browser-playwright": "^4.1.5",
"@vitest/coverage-v8": "^4.0.18",
"@vitest/ui": "4.1.10",
"@vitest/ui": "4.1.11",
"babel-plugin-react-compiler": "^1.0.0",
"classnames": "^2.3.1",
"copy-to-clipboard": "^3.3.3",
@@ -101,7 +101,7 @@
"node-stdlib-browser": "^1.3.1",
"normalize.css": "^8.0.1",
"observable-hooks": "^4.2.3",
"oxfmt": "^0.61.0",
"oxfmt": "^0.66.0",
"oxlint": "^1.70.0",
"oxlint-tsgolint": "^0.25.0",
"pako": "^2.0.4",
+1 -1
View File
@@ -153,7 +153,7 @@ export const widgetTest = test.extend<MyFixtures>({
// Accept the DM invite from brooks
// This how playwright record selects the DM invite in the room list
await ewPage2.getByRole("option", { name: "Open room" }).click();
await ewPage2.getByRole("button", { name: "Open room" }).click();
await ewPage2.getByRole("button", { name: "Start chatting" }).click();
}
+1475 -1476
View File
File diff suppressed because it is too large Load Diff
+55 -33
View File
@@ -24,6 +24,35 @@ export enum MatrixRTCMode {
Matrix_2_0 = "matrix_2_0",
}
export interface DelayedLeaveTimings {
/**
* The delay (in milliseconds) with which delayed leave events are sent.
*
* If the server receives no keep-alives from the client for any longer than
* this duration, it will send the leave event, automatically removing the
* user from the call.
*/
delay_ms?: number;
/**
* How frequently (in milliseconds) the client sends keep-alives to the server
* to restart the timer for a delayed leave event. Should be less than
* {@link DelayedLeaveTimings.delay_ms}.
*/
restart_ms?: number;
/**
* The time (in milliseconds) after which we consider a delayed event restart HTTP request to have failed.
* Setting this to a lower value will result in more frequent retries, but then we will also give up earlier.
*
* In the presence of network packet loss (hurting TCP connections), the custom delayedEventRestartLocalTimeoutMs
* helps by keeping more delayed event reset candidates in flight,
* improving the chances of a successful reset. (its is equivalent to the js-sdk `localTimeout` configuration,
* but only applies to calls to the `_unstable_updateDelayedEvent` endpoint with a body of `{action:"restart"}`.)
*/
restart_timeout_ms?: number;
}
export interface ConfigOptions {
/**
* The Posthog endpoint to which analytics data will be sent.
@@ -184,29 +213,6 @@ export interface ConfigOptions {
*/
wait_for_key_rotation_ms?: number;
/**
* The duration (in milliseconds) after the most recent keep-alive (delayed leave event restart)
* that the server waits before sending the leave MatrixRTC membership event.
*/
delayed_leave_event_delay_ms?: number;
/**
* The time (in milliseconds) after which we consider a delayed event restart http request to have failed.
* Setting this to a lower value will result in more frequent retries but also a higher chance of failiour.
*
* In the presence of network packet loss (hurting TCP connections), the custom delayedEventRestartLocalTimeoutMs
* helps by keeping more delayed event reset candidates in flight,
* improving the chances of a successful reset. (its is equivalent to the js-sdk `localTimeout` configuration,
* but only applies to calls to the `_unstable_updateDelayedEvent` endpoint with a body of `{action:"restart"}`.)
*/
delayed_leave_event_restart_local_timeout_ms?: number;
/**
* The time interval (in milliseconds) at which the client sends membership keep-alive
* messages to the server by restarting the timer for the delayed leave event.
*/
delayed_leave_event_restart_ms?: number;
/**
* How long we wait before retrying after a network error on any of the requests.
*/
@@ -231,9 +237,28 @@ export interface ConfigOptions {
* Defaults to the js-sdk default (undefined). Which means that rotation will always happen.
*/
key_rotation_participant_limit?: number;
/**
* Timing options for delayed leave events, which are used to remove a user
* from a call when they lose connection.
*/
delayed_leave?: DelayedLeaveTimings;
/**
* Timing options for delayed leave events, in cases where the ability to
* send the event can be delegated to the SFU.
*
* We recommend setting {@link DelayedLeaveTimings.delay_ms} >>
* {@link sync_disconnect_grace_period_ms} here.
*/
delegated_delayed_leave?: DelayedLeaveTimings;
};
}
export interface ResolvedDelayedLeaveTimings extends DelayedLeaveTimings {
delay_ms: number; // Required
}
// Overrides members from ConfigOptions that are always provided by the
// default config and are therefore non-optional.
export interface ResolvedConfigOptions extends ConfigOptions {
@@ -257,19 +282,15 @@ export interface ResolvedConfigOptions extends ConfigOptions {
>
>;
};
matrix_rtc_session: {
wait_for_key_rotation_ms?: number;
delayed_leave_event_delay_ms: number;
delayed_leave_event_restart_local_timeout_ms?: number;
delayed_leave_event_restart_ms?: number;
matrix_rtc_session: ConfigOptions["matrix_rtc_session"] & {
network_error_retry_ms: number;
membership_event_expiry_ms?: number;
key_rotation_participant_limit?: number;
delayed_leave: ResolvedDelayedLeaveTimings;
delegated_delayed_leave: ResolvedDelayedLeaveTimings;
};
}
export const DEFAULT_CONFIG: ResolvedConfigOptions = {
sync_disconnect_grace_period_ms: 10000,
sync_disconnect_grace_period_ms: 10_000,
ssla: "https://static.element.io/legal/element-software-and-services-license-agreement-uk-1.pdf",
media_quality: {
video_codec: "vp8",
@@ -285,7 +306,8 @@ export const DEFAULT_CONFIG: ResolvedConfigOptions = {
},
},
matrix_rtc_session: {
delayed_leave_event_delay_ms: 10000,
network_error_retry_ms: 1000,
network_error_retry_ms: 1_000,
delayed_leave: { delay_ms: 18_000, restart_ms: 4_000 },
delegated_delayed_leave: { delay_ms: 3_600_000, restart_ms: 300_000 },
},
};
+53
View File
@@ -24,10 +24,12 @@ import { testAudioContext } from "../useAudioContext.test";
import * as MediaDevicesContext from "../MediaDevicesContext";
import { LivekitRoomAudioRenderer } from "./MatrixAudioRenderer";
import {
mockLocalParticipant,
mockMediaDevices,
mockRemoteParticipant,
mockTrack,
} from "../utils/test";
import { logger as rootLogger } from "matrix-js-sdk/lib/logger";
import { initializeWidget } from "../widget";
initializeWidget();
export const TestAudioContextConstructor = vi.fn(
@@ -131,6 +133,57 @@ it("should render for member", () => {
expect(queryAllByTestId("audio")).toHaveLength(1);
});
function spyOnWarn(): ReturnType<typeof vi.fn> {
const warn = vi.fn();
vi.spyOn(rootLogger, "getChild").mockReturnValue({
warn,
} as unknown as typeof rootLogger);
return warn;
}
it("should not render or warn for the local participant", () => {
const warn = spyOnWarn();
const local = mockLocalParticipant({ identity: "@alice:DEV0" });
vi.mocked(useTracks).mockReturnValue([mockTrack(local)]);
const { queryAllByTestId } = render(
<MediaDevicesProvider value={mockMediaDevices({})}>
<LivekitRoomAudioRenderer
validIdentities={[]}
livekitRoom={{ remoteParticipants: new Map() } as unknown as Room}
url={""}
/>
</MediaDevicesProvider>,
);
expect(queryAllByTestId("audio")).toHaveLength(0);
expect(warn).not.toHaveBeenCalled();
});
it("should warn only once per unexpected participant", () => {
const warn = spyOnWarn();
const { rerender } = renderTestComponent(
[{ userId: "@bob", deviceId: "DEV0" }],
["@alice:DEV0"],
[
{
participantId: "@alice:DEV0",
kind: Track.Kind.Audio,
source: Track.Source.Microphone,
},
],
);
expect(warn).toHaveBeenCalledTimes(1);
rerender(
<MediaDevicesProvider value={mockMediaDevices({})}>
<LivekitRoomAudioRenderer
validIdentities={[]}
livekitRoom={{ remoteParticipants: new Map() } as unknown as Room}
url={""}
/>
</MediaDevicesProvider>,
);
expect(warn).toHaveBeenCalledTimes(1);
});
it("should not render without member", () => {
const { container, queryAllByTestId } = renderTestComponent(
[{ userId: "@bob", deviceId: "DEV0" }],
+17 -9
View File
@@ -8,7 +8,7 @@ Please see LICENSE in the repository root for full details.
import { getTrackReferenceId } from "@livekit/components-core";
import { type Room as LivekitRoom } from "livekit-client";
import { type RemoteAudioTrack, Track } from "livekit-client";
import { useEffect, useMemo, useState, type ReactNode } from "react";
import { useEffect, useMemo, useRef, useState, type ReactNode } from "react";
import {
useTracks,
AudioTrack,
@@ -60,6 +60,9 @@ export function LivekitRoomAudioRenderer({
muted,
}: MatrixAudioRendererProps): ReactNode {
const logger = rootLogger.getChild("[MatrixAudioRenderer]");
// Identities we have already warned about, so that re-renders (which happen
// on every active speaker update) don't repeat the warning.
const warnedIdentities = useRef(new Set<string>());
const tracks = useTracks(
[
Track.Source.Microphone,
@@ -72,22 +75,27 @@ export function LivekitRoomAudioRenderer({
room: livekitRoom,
},
)
// Only keep audio tracks
.filter((ref) => ref.publication.kind === Track.Kind.Audio)
// Only keep remote audio tracks (we never render our own audio)
.filter(
(ref) =>
ref.publication.kind === Track.Kind.Audio && !ref.participant.isLocal,
)
// Only keep tracks from participants that are in the validIdentities list
.filter((ref) => {
const isValid = validIdentities.includes(ref.participant.identity);
if (!isValid) {
// TODO make sure to also skip the warn logging for the local identity
const { identity } = ref.participant;
const isValid = validIdentities.includes(identity);
if (!isValid && !warnedIdentities.current.has(identity)) {
warnedIdentities.current.add(identity);
// Log that there is an invalid identity, that means that someone is publishing audio that is not expected to be in the call.
logger.warn(
`Audio track ${ref.participant.identity} from ${url} has no matching matrix call member`,
`Audio track ${identity} from ${url} has no matching matrix call member`,
`current members: ${validIdentities.join()}`,
`track will not get rendered`,
);
return false;
} else if (isValid) {
warnedIdentities.current.delete(identity);
}
return true;
return isValid;
});
// This component is also (in addition to the "only play audio for connected members" logic above)
+4 -4
View File
@@ -148,7 +148,7 @@ describe("getSFUConfigWithOpenID", () => {
// Verify, that the request contains the expected delay parameters
if (
body.delay_id === "mock_delay_id" &&
body.delay_timeout === 10000 &&
body.delay_timeout === 3600000 &&
body.delay_cs_api_url === "https://homeserverserver.org/cs_api"
) {
return {
@@ -229,7 +229,7 @@ describe("getSFUConfigWithOpenID", () => {
expect(calls[0][0]).toStrictEqual("https://sfu.example.org/get_token");
expect(calls[0][1]).toStrictEqual({
// check if it uses correct delayID!
body: '{"room_id":"!example_room_id","slot_id":"m.call#ROOM","member":{"id":"@alice:example.org:DEVICE","claimed_user_id":"@alice:example.org","claimed_device_id":"DEVICE"},"delay_id":"mock_delay_id","delay_timeout":10000,"delay_cs_api_url":"https://matrix.homeserverserver.org"}',
body: '{"room_id":"!example_room_id","slot_id":"m.call#ROOM","member":{"id":"@alice:example.org:DEVICE","claimed_user_id":"@alice:example.org","claimed_device_id":"DEVICE"},"delay_id":"mock_delay_id","delay_timeout":3600000,"delay_cs_api_url":"https://matrix.homeserverserver.org"}',
method: "POST",
headers: {
"Content-Type": "application/json",
@@ -239,7 +239,7 @@ describe("getSFUConfigWithOpenID", () => {
expect(calls[1][0]).toStrictEqual("https://sfu.example.org/sfu/get");
expect(calls[1][1]).toStrictEqual({
body: '{"room":"!example_room_id","device_id":"DEVICE","delay_id":"mock_delay_id","delay_timeout":10000,"delay_cs_api_url":"https://matrix.homeserverserver.org"}',
body: '{"room":"!example_room_id","device_id":"DEVICE","delay_id":"mock_delay_id","delay_timeout":3600000,"delay_cs_api_url":"https://matrix.homeserverserver.org"}',
headers: {
"Content-Type": "application/json",
},
@@ -284,7 +284,7 @@ describe("getSFUConfigWithOpenID", () => {
expect(calls[0][0]).toStrictEqual("https://sfu.example.org/get_token");
expect(calls[0][1]).toStrictEqual({
// check if it uses correct delayID!
body: '{"room_id":"!example_room_id","slot_id":"m.call#ROOM","member":{"id":"@alice:example.org:DEVICE","claimed_user_id":"@alice:example.org","claimed_device_id":"DEVICE"},"delay_id":"mock_delay_id","delay_timeout":10000,"delay_cs_api_url":"https://matrix.homeserverserver.org"}',
body: '{"room_id":"!example_room_id","slot_id":"m.call#ROOM","member":{"id":"@alice:example.org:DEVICE","claimed_user_id":"@alice:example.org","claimed_device_id":"DEVICE"},"delay_id":"mock_delay_id","delay_timeout":3600000,"delay_cs_api_url":"https://matrix.homeserverserver.org"}',
method: "POST",
headers: {
"Content-Type": "application/json",
+15 -19
View File
@@ -19,7 +19,7 @@ import {
} from "../utils/errors";
import { doNetworkOperationWithRetry } from "../utils/matrix";
import { Config } from "../config/Config";
import { JwtEndpointVersion } from "../state/CallViewModel/localMember/LocalTransport";
import { MatrixRTCMode } from "../config/ConfigOptions";
/**
* Configuration and access tokens provided by the SFU on successful authentication.
@@ -80,11 +80,10 @@ export type OpenIDClientParts = Pick<
* @param serviceUrl The URL of the livekit SFU service
* @param roomId The room id used in the jwt request. This is NOT the livekit_alias. The jwt service will provide the alias. It maps matrix room ids <-> Livekit aliases.
* @param opts Additional options to modify which endpoint with which data will be used to acquire the jwt token.
* @param opts.forceJwtEndpoint This will use the old jwt endpoint which will create the rtc backend identity based on string concatenation
* instead of a hash.
* @param opts.matrixRTCMode Determines which version of the JWT endpoint to use, which affects whether the
* RTC backend identity is based on string concatenation (legacy) or a hash (Matrix 2.0).
* This function by default uses whatever is possible with the current jwt service installed next to the SFU.
* For remote connections this does not matter, since we will not publish there we can rely on the newest option.
* For our own connection we can only use the hashed version if we also send the new matrix2.0 sticky events.
* @param opts.delayEndpointBaseUrl The URL of the matrix homeserver.
* @param opts.delayId The delay id used for the jwt service to manage.
* @param logger optional logger.
@@ -97,7 +96,7 @@ export async function getSFUConfigWithOpenID(
serviceUrl: string,
roomId: string,
opts?: {
forceJwtEndpoint?: JwtEndpointVersion;
matrixRTCMode?: MatrixRTCMode;
delayEndpointBaseUrl?: string;
delayId?: string;
},
@@ -116,19 +115,18 @@ export async function getSFUConfigWithOpenID(
logger?.debug("Got openID token", openIdToken);
let sfuConfig: { url: string; jwt: string } | undefined;
const tryBothJwtEndpoints = opts?.forceJwtEndpoint === undefined; // This is for SFUs where we do not publish.
const tryBothJwtEndpoints = opts?.matrixRTCMode === undefined; // This is for SFUs where we do not publish.
const forceMatrix2Jwt =
opts?.forceJwtEndpoint === JwtEndpointVersion.Matrix_2_0;
const forceMatrix2Jwt = opts?.matrixRTCMode === MatrixRTCMode.Matrix_2_0;
// We want to start using the new endpoint (with optional delay delegation)
// if we can use both or if we are forced to use the new one.
if (tryBothJwtEndpoints || forceMatrix2Jwt) {
try {
logger?.info(
`Trying to get JWT with delegation for focus ${serviceUrl}...`,
`Trying to get JWT via default endpoint for focus ${serviceUrl}...`,
);
const sfuConfig = await getLiveKitJWTWithDelayDelegation(
const sfuConfig = await getLiveKitJWT(
membership,
serviceUrl,
roomId,
@@ -154,7 +152,7 @@ export async function getSFUConfigWithOpenID(
logger?.info(
`Trying to get JWT with legacy endpoint for focus ${serviceUrl}...`,
);
sfuConfig = await getLiveKitJWT(
sfuConfig = await getLiveKitJWTLegacy(
membership.deviceId,
serviceUrl,
roomId,
@@ -188,7 +186,7 @@ function extractFullConfigFromToken(sfuConfig: {
};
}
async function getLiveKitJWT(
async function getLiveKitJWTLegacy(
deviceId: string,
livekitServiceURL: string,
matrixRoomId: string,
@@ -204,11 +202,10 @@ async function getLiveKitJWT(
let bodyDalayParts: IDelayParams = {};
// Also check for empty string
if (delayId && delayEndpointBaseUrl) {
const delayTimeoutMs =
Config.get().matrix_rtc_session?.delayed_leave_event_delay_ms;
bodyDalayParts = {
delay_id: delayId,
delay_timeout: delayTimeoutMs,
delay_timeout:
Config.get().matrix_rtc_session.delegated_delayed_leave.delay_ms,
delay_cs_api_url: delayEndpointBaseUrl,
};
}
@@ -264,7 +261,7 @@ class NotSupportedError extends Error {
}
}
export async function getLiveKitJWTWithDelayDelegation(
export async function getLiveKitJWT(
membership: CallMembershipIdentityParts,
livekitServiceURL: string,
matrixRoomId: string,
@@ -288,11 +285,10 @@ export async function getLiveKitJWTWithDelayDelegation(
let bodyDalayParts = {};
// Also check for empty string
if (delayId && delayEndpointBaseUrl) {
const delayTimeoutMs =
Config.get().matrix_rtc_session?.delayed_leave_event_delay_ms;
bodyDalayParts = {
delay_id: delayId,
delay_timeout: delayTimeoutMs,
delay_timeout:
Config.get().matrix_rtc_session.delegated_delayed_leave.delay_ms,
delay_cs_api_url: delayEndpointBaseUrl,
};
}
+20 -1
View File
@@ -135,7 +135,9 @@ export const ActiveCall: FC<ActiveCallProps> = (props) => {
encryptionSystem: props.e2eeSystem,
autoLeaveWhenOthersLeft,
waitForCallPickup: waitForCallPickup && sendNotificationType === "ring",
matrixRTCMode$: matrixRTCModeSetting.value$,
// We merely sample the current mode here, so the user would need to
// manually rejoin to switch to a different one.
matrixRTCMode: matrixRTCModeSetting.value$.value,
},
reactionsReader.raisedHands$,
reactionsReader.reactions$,
@@ -270,6 +272,7 @@ export const InCallView: FC<InCallViewProps> = ({
const audioParticipants = useBehavior(vm.livekitRoomItems$);
const participantCount = useBehavior(vm.participantCount$);
const reconnecting = useBehavior(vm.reconnecting$);
const screenShareError = useBehavior(vm.screenShareError$);
const layout = useBehavior(vm.layout$);
const edgeToEdge = useBehavior(vm.edgeToEdge$);
const overflowing = useBehavior(vm.overflowing$);
@@ -399,6 +402,21 @@ export const InCallView: FC<InCallViewProps> = ({
);
}
const onDismissScreenShareToast = useCallback(
() => vm.dismissScreenShareError(),
[vm],
);
const screenShareToast = (
<Toast
onDismiss={onDismissScreenShareToast}
open={screenShareError !== null}
autoDismiss={5000}
modal={false}
>
{t("error.screen_share_failed")}
</Toast>
);
// The reconnecting toast cannot be dismissed
const onDismissReconnectingToast = useCallback(() => {}, []);
// We need to use a non-modal toast to avoid trapping focus within the toast.
@@ -634,6 +652,7 @@ export const InCallView: FC<InCallViewProps> = ({
<ReactionsAudioRenderer vm={vm} muted={muteAllAudio} />
<RingingAudioRenderer vm={ringingVm} muted={muteAllAudio} />
{reconnectingToast}
{screenShareToast}
{earpieceOverlay}
<ReactionsOverlay vm={vm} />
{footer}
+10 -10
View File
@@ -171,7 +171,7 @@ exports[`InCallView > rendering > renders 1`] = `
class="_settingsLogoContainer_20b7b4"
>
<button
aria-labelledby="_r_8_"
aria-labelledby="_r_b_"
class="_icon-button_1215g_8 _settingsOnlyShowWide_20b7b4"
data-kind="secondary"
data-testid="settings-bottom-left"
@@ -306,7 +306,7 @@ exports[`InCallView > rendering > renders 1`] = `
class="_buttons_20b7b4"
>
<button
aria-labelledby="_r_d_"
aria-labelledby="_r_g_"
class="_button_1nw83_8 _settingsOnlyShowNarrow_20b7b4 _has-icon_1nw83_60 _icon-only_1nw83_53"
data-kind="secondary"
data-size="lg"
@@ -331,7 +331,7 @@ exports[`InCallView > rendering > renders 1`] = `
aria-busy="false"
aria-checked="false"
aria-disabled="true"
aria-labelledby="_r_i_"
aria-labelledby="_r_l_"
class="_button_1nw83_8 _has-icon_1nw83_60 _icon-only_1nw83_53"
data-kind="primary"
data-size="lg"
@@ -356,7 +356,7 @@ exports[`InCallView > rendering > renders 1`] = `
aria-busy="false"
aria-checked="false"
aria-disabled="true"
aria-labelledby="_r_n_"
aria-labelledby="_r_q_"
class="_button_1nw83_8 _has-icon_1nw83_60 _icon-only_1nw83_53"
data-kind="primary"
data-size="lg"
@@ -381,7 +381,7 @@ exports[`InCallView > rendering > renders 1`] = `
aria-disabled="false"
aria-expanded="false"
aria-haspopup="true"
aria-labelledby="_r_s_"
aria-labelledby="_r_v_"
class="_button_1nw83_8 _raiseHand_20b7b4 _has-icon_1nw83_60 _icon-only_1nw83_53"
data-kind="secondary"
data-size="lg"
@@ -404,7 +404,7 @@ exports[`InCallView > rendering > renders 1`] = `
</svg>
</button>
<button
aria-labelledby="_r_14_"
aria-labelledby="_r_17_"
class="_button_1nw83_8 _has-icon_1nw83_60 _icon-only_1nw83_53 _destructive_1nw83_110"
data-kind="primary"
data-size="lg"
@@ -432,8 +432,8 @@ exports[`InCallView > rendering > renders 1`] = `
data-size="lg"
>
<input
aria-labelledby="_r_1a_"
name="_r_19_"
aria-labelledby="_r_1d_"
name="_r_1c_"
type="radio"
value="spotlight"
/>
@@ -452,9 +452,9 @@ exports[`InCallView > rendering > renders 1`] = `
/>
</svg>
<input
aria-labelledby="_r_1f_"
aria-labelledby="_r_1i_"
checked=""
name="_r_19_"
name="_r_1c_"
type="radio"
value="grid"
/>
+52
View File
@@ -0,0 +1,52 @@
/*
Copyright 2026 Element Creations Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { afterEach, expect, it, vi } from "vitest";
import { init as initRageshake } from "./rageshake";
afterEach(() => {
vi.useRealTimers();
vi.unstubAllGlobals();
});
it("flushes logs to IndexedDB periodically without an explicit flush", async () => {
vi.useFakeTimers();
const add = vi.fn();
const txn = {
oncomplete: undefined as (() => void) | undefined,
onerror: undefined,
objectStore: (name: string) =>
name === "logs"
? {
add: (entry: unknown): void => {
add(entry);
queueMicrotask(() => txn.oncomplete?.());
},
}
: { put: vi.fn() },
};
const open = (): unknown => {
const req = {
result: { transaction: () => txn },
onsuccess: undefined as (() => void) | undefined,
};
queueMicrotask(() => req.onsuccess?.());
return req;
};
vi.stubGlobal("indexedDB", { open });
await initRageshake();
global.mx_rage_logger.log(1, "test", "hello from the buffer");
expect(add).not.toHaveBeenCalled();
await vi.advanceTimersByTimeAsync(2000);
expect(add).toHaveBeenCalledOnce();
expect(add.mock.calls[0][0]).toMatchObject({
lines: expect.stringContaining("hello from the buffer"),
});
});
+7 -4
View File
@@ -204,10 +204,13 @@ class IndexedDBLogStore {
// Throttled function to flush logs. We use throttle rather
// than debounce as we want logs to be written regularly, otherwise
// if there's a constant stream of logging, we'd never write anything.
private throttledFlush = throttle(() => this.flush, MAX_FLUSH_INTERVAL_MS, {
leading: false,
trailing: true,
});
private throttledFlush = throttle(
() => {
this.flush().catch((e) => logger.error("Failed to flush logs", e));
},
MAX_FLUSH_INTERVAL_MS,
{ leading: false, trailing: true },
);
/**
* Flush logs to disk.
+49 -55
View File
@@ -54,7 +54,6 @@ import { type IMembershipManager } from "matrix-js-sdk/lib/matrixrtc/IMembership
import {
createToggle$,
filterBehavior,
generateItem,
generateItems,
pauseWhen,
} from "../../utils/observable";
@@ -64,7 +63,10 @@ import {
showReactions,
} from "../../settings/settings";
import { Config } from "../../config/Config";
import { MatrixRTCMode } from "../../config/ConfigOptions";
import {
MatrixRTCMode,
type ResolvedDelayedLeaveTimings,
} from "../../config/ConfigOptions";
import { isFirefox, platform } from "../../Platform";
import { setPipEnabled$ } from "../../controls";
import { TileStore } from "../TileStore";
@@ -110,7 +112,6 @@ import {
} from "./localMember/LocalMember.ts";
import {
createLocalTransport$,
JwtEndpointVersion,
type LocalTransport,
} from "./localMember/LocalTransport.ts";
import {
@@ -188,7 +189,7 @@ export interface CallViewModelOptions {
/** Optional value overriding the connection factory, for testing purposes. */
connectionFactory?: ConnectionFactory;
/** The version & compatibility mode of MatrixRTC that we should use. */
matrixRTCMode$?: Behavior<MatrixRTCMode>;
matrixRTCMode?: MatrixRTCMode;
/** Optional behavior overriding for the screensharing, for testing */
toggleScreensharing?: () => void;
}
@@ -267,6 +268,11 @@ export interface CallViewModel {
* Whether we are sharing our screen.
*/
sharingScreen$: Behavior<boolean>;
/**
* The last error from toggling screen sharing, until dismissed.
*/
screenShareError$: Behavior<Error | null>;
dismissScreenShareError: () => void;
// UI interactions
/**
@@ -445,10 +451,8 @@ export function createCallViewModel$(
const configMatrixRTCMode = Config.get().matrix_rtc_mode as
| MatrixRTCMode
| undefined;
const matrixRTCMode$ =
configMatrixRTCMode !== undefined
? constant(configMatrixRTCMode)
: (options.matrixRTCMode$ ?? constant(MatrixRTCMode.Compatibility));
const matrixRTCMode =
configMatrixRTCMode ?? options.matrixRTCMode ?? MatrixRTCMode.Compatibility;
// Each hbar seperates a block of input variables required for the CallViewModel to function.
// The outputs of this block is written under the hbar.
@@ -482,38 +486,16 @@ export function createCallViewModel$(
memberId: uuidv4(),
};
const localTransport$ = scope.behavior(
matrixRTCMode$.pipe(
generateItem(
"CallViewModel localTransport$",
// Re-create LocalTransport whenever the mode changes
(mode) => ({ keys: [mode], data: undefined }),
(scope, _data$, mode) =>
options.localTransport ??
createLocalTransport$({
scope: scope,
memberships$: memberships$,
ownMembershipIdentity,
client,
delayId$: scope.behavior(
(
fromEvent(
matrixRTCSession,
MembershipManagerEvent.DelayIdChanged,
// The type of reemitted event includes the original emitted as the second arg.
) as Observable<[string | undefined, IMembershipManager]>
).pipe(map(([delayId]) => delayId ?? null)),
matrixRTCSession.delayId ?? null,
),
roomId: matrixRoom.roomId,
forceJwtEndpoint:
mode === MatrixRTCMode.Matrix_2_0
? JwtEndpointVersion.Matrix_2_0
: JwtEndpointVersion.Legacy,
}),
),
),
);
const localTransport =
options.localTransport ??
createLocalTransport$({
scope: scope,
memberships$: memberships$,
ownMembershipIdentity,
client,
roomId: matrixRoom.roomId,
matrixRTCMode,
});
const connectionFactory =
options.connectionFactory ??
@@ -531,8 +513,7 @@ export function createCallViewModel$(
scope: scope,
connectionFactory: connectionFactory,
localTransport$: scope.behavior(
localTransport$.pipe(
switchMap((t) => t.active$),
localTransport.active$.pipe(
catchError((e: unknown) => {
logger.info(
"could not pass local transport to createConnectionManager$. localTransport$ threw an error",
@@ -557,16 +538,6 @@ export function createCallViewModel$(
localUser: { userId, deviceId },
});
const connectOptions$ = scope.behavior(
matrixRTCMode$.pipe(
map((mode) => ({
encryptMedia: livekitKeyProvider !== undefined,
// TODO. This might need to get called again on each change of matrixRTCMode...
matrixRTCMode: mode,
})),
),
);
const localMembership = createLocalMembership$({
scope,
homeserverConnected: createHomeserverConnected$(
@@ -575,12 +546,19 @@ export function createCallViewModel$(
matrixRTCSession,
),
muteStates,
joinMatrixRTC: (transport: LivekitTransportConfig) => {
joinMatrixRTC: (
transport: LivekitTransportConfig,
delayedLeaveTimings: ResolvedDelayedLeaveTimings,
) => {
return enterRTCSession(
matrixRTCSession,
ownMembershipIdentity,
transport,
connectOptions$.value,
{
encryptMedia: livekitKeyProvider !== undefined,
matrixRTCMode,
delayedLeaveTimings,
},
);
},
createPublisherFactory: (connection: Connection) => {
@@ -595,9 +573,23 @@ export function createCallViewModel$(
);
},
connectionManager,
client,
matrixRTCSession,
localTransport$,
localTransport,
roomId: matrixRoom.roomId,
baseUrl: client.baseUrl,
ownMembershipIdentity,
delayId$: scope.behavior(
(
fromEvent(
matrixRTCSession,
MembershipManagerEvent.DelayIdChanged,
// The type of reemitted event includes the original emitted as the second arg.
) as Observable<[string | undefined, IMembershipManager]>
).pipe(map(([delayId]) => delayId ?? null)),
matrixRTCSession.delayId ?? null,
),
matrixRTCMode,
logger: logger.getChild(`[${Date.now()}]`),
});
@@ -1858,6 +1850,8 @@ export function createCallViewModel$(
reconnecting$: localMembership.reconnecting$,
livekitRoomItems$,
connected$: localMembership.connected$,
screenShareError$: localMembership.screenShareError$,
dismissScreenShareError: localMembership.dismissScreenShareError,
};
}
@@ -238,7 +238,7 @@ export function withCallViewModel(mode: MatrixRTCMode) {
);
},
},
matrixRTCMode$: constant(mode),
matrixRTCMode: mode,
...options,
},
raisedHands$,
@@ -19,21 +19,28 @@ import {
beforeAll,
afterAll,
beforeEach,
afterEach,
} from "vitest";
import { BehaviorSubject, map, of } from "rxjs";
import { logger } from "matrix-js-sdk/lib/logger";
import { type LocalParticipant, type LocalTrack } from "livekit-client";
import fetchMock from "fetch-mock";
import { PosthogAnalytics } from "../../../analytics/PosthogAnalytics";
import { MatrixRTCMode } from "../../../config/ConfigOptions";
import {
MatrixRTCMode,
type ResolvedDelayedLeaveTimings,
} from "../../../config/ConfigOptions";
import { type HomeserverDisconnectReason } from "./HomeserverConnected";
import {
flushPromises,
mockConfig,
mockLivekitRoom,
mockLocalParticipant,
mockMuteStates,
withTestScheduler,
ownMemberMock,
testScope,
} from "../../../utils/test";
import {
TransportState,
@@ -41,6 +48,7 @@ import {
enterRTCSession,
PublishState,
TrackState,
watchScreenShareToggle,
} from "./LocalMember";
import {
FailToGetOpenIdToken,
@@ -56,6 +64,7 @@ import {
type LocalTransport,
type LocalTransportWithSFUConfig,
} from "./LocalTransport";
import * as openIDSFU from "../../../livekit/openIDSFU";
initializeWidget();
@@ -68,112 +77,137 @@ vi.mock("@livekit/components-core", () => ({
.mockReturnValue(of({ isScreenShareEnabled: false })),
}));
describe("LocalMembership", () => {
describe("enterRTCSession", () => {
it("It joins the correct Session", () => {
mockConfig({
livekit: { livekit_service_url: "http://my-default-service-url.com" },
});
const mockedSession = vi.mocked({
room: {
roomId: "roomId",
client: {
getDomain: vi.fn().mockReturnValue("example.org"),
getOpenIdToken: vi.fn().mockResolvedValue({
access_token: "ACCCESS_TOKEN",
token_type: "Bearer",
matrix_server_name: "localhost",
expires_in: 10000,
}),
},
},
memberships: [],
joinRTCSession: vi.fn(),
}) as unknown as MatrixRTCSession;
enterRTCSession(
mockedSession,
ownMemberMock,
{
livekit_alias: "roomId",
livekit_service_url: "http://my-livekit-service-url.com",
type: "livekit",
},
{
encryptMedia: true,
matrixRTCMode: MATRIX_RTC_MODE,
},
);
expect(mockedSession.joinRTCSession).toHaveBeenLastCalledWith(
{
deviceId: "DEVICE",
memberId: "@alice:example.org:DEVICE",
userId: "@alice:example.org",
},
[],
{
livekit_alias: "roomId",
livekit_service_url: "http://my-livekit-service-url.com",
type: "livekit",
},
expect.objectContaining({ manageMediaKeys: true }),
);
});
it("passes keyRotationParticipantLimit from config to joinRTCSession", () => {
mockConfig({
livekit: { livekit_service_url: "http://my-default-service-url.com" },
matrix_rtc_session: {
delayed_leave_event_delay_ms: 0,
network_error_retry_ms: 0,
key_rotation_participant_limit: 50,
},
});
const mockedSession = vi.mocked({
room: {
roomId: "roomId",
client: {
getDomain: vi.fn().mockReturnValue("example.org"),
getOpenIdToken: vi.fn().mockResolvedValue({
access_token: "ACCCESS_TOKEN",
token_type: "Bearer",
matrix_server_name: "localhost",
expires_in: 10000,
}),
},
},
memberships: [],
joinRTCSession: vi.fn(),
}) as unknown as MatrixRTCSession;
enterRTCSession(
mockedSession,
ownMemberMock,
{
livekit_alias: "roomId",
livekit_service_url: "http://my-livekit-service-url.com",
type: "livekit",
},
{
encryptMedia: true,
matrixRTCMode: MATRIX_RTC_MODE,
},
);
expect(mockedSession.joinRTCSession).toHaveBeenLastCalledWith(
expect.any(Object),
[],
expect.any(Object),
expect.objectContaining({
keyRotationParticipantLimit: 50,
}),
);
});
describe("watchScreenShareToggle", () => {
it("reports nothing when the toggle completes", async () => {
const onError = vi.fn();
watchScreenShareToggle(Promise.resolve(), true, logger, onError);
await flushPromises();
expect(onError).not.toHaveBeenCalled();
});
it("reports failures other than the user cancelling", async () => {
const onError = vi.fn();
const e = new Error("NotReadableError");
watchScreenShareToggle(Promise.reject(e), true, logger, onError);
await flushPromises();
expect(onError).toHaveBeenCalledWith(e);
});
it("does not report the user cancelling the picker", async () => {
const onError = vi.fn();
const cancelled = new DOMException("Permission denied", "NotAllowedError");
watchScreenShareToggle(Promise.reject(cancelled), true, logger, onError);
await flushPromises();
expect(onError).not.toHaveBeenCalled();
});
});
const timings: ResolvedDelayedLeaveTimings = {
delay_ms: 10000,
restart_ms: 4000,
restart_timeout_ms: 1000,
};
const delegatedTimings: ResolvedDelayedLeaveTimings = {
delay_ms: timings.delay_ms * 10,
restart_ms: timings.restart_ms! * 10,
restart_timeout_ms: timings.restart_timeout_ms! * 10,
};
const mockedClient = {
getDomain: vi.fn().mockReturnValue("example.org"),
getDeviceId: vi.fn().mockReturnValue("AAAA"),
getOpenIdToken: vi.fn().mockResolvedValue({
access_token: "ACCCESS_TOKEN",
token_type: "Bearer",
matrix_server_name: "localhost",
expires_in: 10000,
}),
};
describe("enterRTCSession", () => {
const transport: LivekitTransportConfig = {
livekit_alias: "roomId",
livekit_service_url: "http://my-livekit-service-url.com",
type: "livekit",
};
const options = {
encryptMedia: true,
matrixRTCMode: MATRIX_RTC_MODE,
delayedLeaveTimings: timings,
};
const mockedSession = vi.mocked({
room: {
roomId: "roomId",
client: mockedClient,
},
memberships: [],
joinRTCSession: vi.fn(),
}) as unknown as MatrixRTCSession;
beforeEach(() =>
mockConfig({
livekit: { livekit_service_url: "http://my-default-service-url.com" },
}),
);
it("It joins the correct Session", () => {
enterRTCSession(mockedSession, ownMemberMock, transport, options);
expect(mockedSession.joinRTCSession).toHaveBeenLastCalledWith(
{
deviceId: "DEVICE",
memberId: "@alice:example.org:DEVICE",
userId: "@alice:example.org",
},
[],
transport,
expect.objectContaining({ manageMediaKeys: true }),
);
});
it("passes keyRotationParticipantLimit from config to joinRTCSession", () => {
mockConfig({
livekit: { livekit_service_url: "http://my-default-service-url.com" },
matrix_rtc_session: {
network_error_retry_ms: 0,
key_rotation_participant_limit: 50,
delayed_leave: timings,
delegated_delayed_leave: timings,
},
});
enterRTCSession(mockedSession, ownMemberMock, transport, options);
expect(mockedSession.joinRTCSession).toHaveBeenLastCalledWith(
expect.any(Object),
[],
expect.any(Object),
expect.objectContaining({
keyRotationParticipantLimit: 50,
}),
);
});
it("uses the specified delayed leave timings", () => {
enterRTCSession(mockedSession, ownMemberMock, transport, options);
expect(mockedSession.joinRTCSession).toHaveBeenLastCalledWith(
expect.anything(),
expect.anything(),
expect.anything(),
expect.objectContaining({
delayedLeaveEventRestartMs: timings.restart_ms,
delayedLeaveEventDelayMs: timings.delay_ms,
delayedLeaveEventRestartLocalTimeoutMs: timings.restart_timeout_ms,
}),
);
});
});
describe("LocalMembership", () => {
const defaultCreateLocalMemberValues = {
options: constant({
encryptMedia: false,
@@ -199,8 +233,30 @@ describe("LocalMembership", () => {
rtsSession$: constant(RTCMemberStatus.Connected),
},
roomId: "!test-room-id:example.org",
baseUrl: "https://matrix.example.org",
ownMembershipIdentity: ownMemberMock,
client: mockedClient,
delayId$: constant(null),
matrixRTCMode: MATRIX_RTC_MODE,
};
beforeEach(() => {
mockConfig({
livekit: { livekit_service_url: "http://my-default-service-url.com" },
matrix_rtc_session: {
network_error_retry_ms: 1000,
delayed_leave: timings,
delegated_delayed_leave: delegatedTimings,
},
});
fetchMock.catch(404);
});
afterEach(async () => {
void (await fetchMock.flush());
fetchMock.reset();
});
it("throws error on missing RTC config error", () => {
withTestScheduler(({ scope, hot, behavior, expectObservable }) => {
const localTransport$ = scope.behavior<null | LivekitTransportConfig>(
@@ -227,9 +283,8 @@ describe("LocalMembership", () => {
scope,
...defaultCreateLocalMemberValues,
connectionManager: mockConnectionManager,
localTransport$: behavior("a", { a: aLocalTransport }),
localTransport: aLocalTransport,
});
localMembership.requestJoinAndPublish();
expectObservable(localMembership.localMemberState$).toBe("ne", {
n: TransportState.Waiting,
@@ -272,9 +327,8 @@ describe("LocalMembership", () => {
scope,
...defaultCreateLocalMemberValues,
connectionManager: mockConnectionManager,
localTransport$: behavior("a", { a: aLocalTransport }),
localTransport: aLocalTransport,
});
localMembership.requestJoinAndPublish();
expectObservable(localMembership.localMemberState$).toBe("n-e", {
n: TransportState.Waiting,
@@ -290,8 +344,8 @@ describe("LocalMembership", () => {
const scope = new ObservableScope();
const aLocalTransport: LocalTransport = {
advertised$: new BehaviorSubject(aTransport),
active$: new BehaviorSubject(aTransportWithSFUConfig),
advertised$: constant(aTransport),
active$: constant(aTransportWithSFUConfig),
};
const mockConnectionManager = {
@@ -309,7 +363,7 @@ describe("LocalMembership", () => {
leaveRoomSession: vi.fn(),
},
connectionManager: mockConnectionManager,
localTransport$: new BehaviorSubject(aLocalTransport),
localTransport: aLocalTransport,
});
const expextedLog =
"'not connected yet' while updating the call intent (this is expected on startup)";
@@ -370,12 +424,90 @@ describe("LocalMembership", () => {
livekitRoom: mockLivekitRoom({}),
} as unknown as Connection;
const authCallSpy = vi
.spyOn(openIDSFU, "getSFUConfigWithOpenID")
.mockImplementation(() => mockedClient.getOpenIdToken());
afterEach(() => authCallSpy.mockClear());
it.each([
["no", null, timings],
[
"homeserver",
"https://matrix.example.org/_matrix/client/unstable/io.element.msc4195/rtc/livekit/delegate_delayed_leave",
delegatedTimings,
],
["transport", "/a/delegate_delayed_leave", delegatedTimings],
])(
"joins session with %s delegation support",
async (_serviceName, delegationUrl, delayedLeaveTimings) => {
const scope = testScope();
const joinMatrixRTC = vi.fn();
const delayId$ = new BehaviorSubject<string | null>(null);
if (delegationUrl !== null)
fetchMock.post(delegationUrl, () => ({ status: 401, body: {} }));
const localMembership = createLocalMembership$({
scope,
...defaultCreateLocalMemberValues,
connectionManager: {
connectionManagerData$: constant(
new Epoch(new ConnectionManagerData()),
),
},
joinMatrixRTC,
localTransport: {
advertised$: constant(aTransport),
active$: constant(aTransportWithSFUConfig),
},
delayId$,
});
localMembership.requestJoinAndPublish();
void (await fetchMock.flush());
await flushPromises();
// Joins with timings appropriate for the level of delegation support
expect(joinMatrixRTC).toHaveBeenCalledWith(
aTransport,
delayedLeaveTimings,
);
expect(authCallSpy).not.toHaveBeenCalled();
delayId$.next("leave1");
await flushPromises();
if (delegationUrl === null) {
expect(authCallSpy).not.toHaveBeenCalled();
} else {
// Delegation is supported in this test case, so go on to check that
// LocalMember actually performs delegation
const expectDelegation = (delayId: string) =>
expect(authCallSpy).toHaveBeenLastCalledWith(
mockedClient,
ownMemberMock,
"a",
"!test-room-id:example.org",
{
matrixRTCMode: MATRIX_RTC_MODE,
delayEndpointBaseUrl: "https://matrix.example.org",
delayId,
},
expect.anything(),
);
expectDelegation("leave1");
delayId$.next("leave2"); // Can change delegated leaves
await flushPromises();
expectDelegation("leave2");
}
},
);
it("recreates publisher if new connection is used, always unpublish and end tracks", async () => {
const scope = new ObservableScope();
const activeTransport$ = new BehaviorSubject(aTransportWithSFUConfig);
const aLocalTransport: LocalTransport = {
advertised$: new BehaviorSubject(aTransport),
advertised$: constant(aTransport),
active$: activeTransport$,
};
@@ -413,7 +545,7 @@ describe("LocalMembership", () => {
connectionManager: {
connectionManagerData$: constant(new Epoch(connectionManagerData)),
},
localTransport$: new BehaviorSubject(aLocalTransport),
localTransport: aLocalTransport,
});
await flushPromises();
activeTransport$.next({
@@ -444,7 +576,7 @@ describe("LocalMembership", () => {
const publishers: Publisher[] = [];
const tracks$ = new BehaviorSubject<LocalTrack[]>([]);
const publishing$ = new BehaviorSubject<boolean>(false);
const publishing$ = constant<boolean>(false);
defaultCreateLocalMemberValues.createPublisherFactory.mockImplementation(
() => {
const p = {
@@ -468,8 +600,8 @@ describe("LocalMembership", () => {
>;
const aLocalTransport: LocalTransport = {
advertised$: new BehaviorSubject(aTransport),
active$: new BehaviorSubject(aTransportWithSFUConfig),
advertised$: constant(aTransport),
active$: constant(aTransportWithSFUConfig),
};
const connectionManagerData = new ConnectionManagerData();
@@ -481,7 +613,7 @@ describe("LocalMembership", () => {
connectionManager: {
connectionManagerData$: constant(new Epoch(connectionManagerData)),
},
localTransport$: new BehaviorSubject(aLocalTransport),
localTransport: aLocalTransport,
});
await flushPromises();
expect(publisherFactory).toHaveBeenCalledOnce();
@@ -509,7 +641,7 @@ describe("LocalMembership", () => {
new BehaviorSubject<null | LocalTransportWithSFUConfig>(null);
const aLocalTransport: LocalTransport = {
advertised$: new BehaviorSubject(aTransport),
advertised$: constant(aTransport),
active$: activeTransport$,
};
@@ -552,7 +684,7 @@ describe("LocalMembership", () => {
connectionManager: {
connectionManagerData$,
},
localTransport$: new BehaviorSubject(aLocalTransport),
localTransport: aLocalTransport,
});
await flushPromises();
@@ -687,10 +819,10 @@ describe("LocalMembership", () => {
connectionManager: {
connectionManagerData$: constant(new Epoch(connectionManagerData)),
},
localTransport$: new BehaviorSubject({
advertised$: new BehaviorSubject(aTransport),
active$: new BehaviorSubject(aTransportWithSFUConfig),
}),
localTransport: {
advertised$: constant(aTransport),
active$: constant(aTransportWithSFUConfig),
},
});
await flushPromises();
@@ -727,10 +859,10 @@ describe("LocalMembership", () => {
connectionManager: {
connectionManagerData$: constant(new Epoch(connectionManagerData)),
},
localTransport$: new BehaviorSubject({
advertised$: new BehaviorSubject(aTransport),
active$: new BehaviorSubject(aTransportWithSFUConfig),
}),
localTransport: {
advertised$: constant(aTransport),
active$: constant(aTransportWithSFUConfig),
},
});
await flushPromises();
@@ -769,18 +901,19 @@ describe("LocalMembership", () => {
scope,
...defaultCreateLocalMemberValues,
homeserverConnected: {
combined$: new BehaviorSubject<
[boolean, HomeserverDisconnectReason | null]
>([true, null]),
combined$: constant<[boolean, HomeserverDisconnectReason | null]>([
true,
null,
]),
rtsSession$: constant(RTCMemberStatus.Connected),
},
connectionManager: {
connectionManagerData$: constant(new Epoch(connectionManagerData)),
},
localTransport$: new BehaviorSubject({
advertised$: new BehaviorSubject(aTransport),
active$: new BehaviorSubject(aTransportWithSFUConfig),
}),
localTransport: {
advertised$: constant(aTransport),
active$: constant(aTransportWithSFUConfig),
},
});
await flushPromises();
@@ -821,10 +954,10 @@ describe("LocalMembership", () => {
connectionManager: {
connectionManagerData$: constant(new Epoch(connectionManagerData)),
},
localTransport$: new BehaviorSubject({
advertised$: new BehaviorSubject(aTransport),
active$: new BehaviorSubject(aTransportWithSFUConfig),
}),
localTransport: {
advertised$: constant(aTransport),
active$: constant(aTransportWithSFUConfig),
},
});
await flushPromises();
@@ -854,4 +987,98 @@ describe("LocalMembership", () => {
scope.end();
});
});
describe("toggleScreenSharing", () => {
let originalMediaDevices: MediaDevices | undefined;
beforeAll(() => {
mockConfig();
// Screen sharing is only offered when getDisplayMedia is available.
originalMediaDevices = navigator.mediaDevices;
Object.defineProperty(navigator, "mediaDevices", {
configurable: true,
value: { getDisplayMedia: vi.fn() },
});
});
afterAll(() => {
Object.defineProperty(navigator, "mediaDevices", {
configurable: true,
value: originalMediaDevices,
});
});
const createMembershipWithConnection = (
connection: Connection | null,
): {
scope: ObservableScope;
localMembership: ReturnType<typeof createLocalMembership$>;
} => {
const scope = new ObservableScope();
const connectionManagerData = new ConnectionManagerData();
if (connection) connectionManagerData.add(connection, []);
const localMembership = createLocalMembership$({
scope,
...defaultCreateLocalMemberValues,
connectionManager: {
connectionManagerData$: constant(new Epoch(connectionManagerData)),
},
localTransport: {
advertised$: constant(aTransport),
active$: constant(aTransportWithSFUConfig),
},
});
return { scope, localMembership };
};
it("surfaces a failure and clears it on dismiss", async () => {
const error = new Error("NotReadableError");
const setScreenShareEnabled = vi.fn().mockRejectedValue(error);
const connection = {
state$: constant(ConnectionState.LivekitConnected),
transport: aTransport,
livekitRoom: mockLivekitRoom({
localParticipant: mockLocalParticipant({
isScreenShareEnabled: false,
setScreenShareEnabled,
}),
}),
} as unknown as Connection;
const { scope, localMembership } =
createMembershipWithConnection(connection);
await flushPromises();
expect(localMembership.toggleScreenSharing).not.toBeNull();
expect(localMembership.screenShareError$.value).toBeNull();
localMembership.toggleScreenSharing!();
await flushPromises();
expect(setScreenShareEnabled).toHaveBeenCalledWith(
true,
expect.any(Object),
undefined,
);
expect(localMembership.screenShareError$.value).toBe(error);
localMembership.dismissScreenShareError();
expect(localMembership.screenShareError$.value).toBeNull();
scope.end();
});
it("does nothing when there is no local participant", async () => {
// No connection means participant$ never resolves to a participant.
const { scope, localMembership } = createMembershipWithConnection(null);
await flushPromises();
expect(localMembership.toggleScreenSharing).not.toBeNull();
localMembership.toggleScreenSharing!();
await flushPromises();
expect(localMembership.screenShareError$.value).toBeNull();
scope.end();
});
});
});
@@ -15,6 +15,7 @@ import {
MediaDeviceFailure,
} from "livekit-client";
import { observeParticipantEvents } from "@livekit/components-core";
import { type MatrixClient } from "matrix-js-sdk";
import {
Status as RTCSessionStatus,
type LivekitTransport,
@@ -62,7 +63,10 @@ import {
screenShareCodec,
parseResolution,
} from "../../../settings/settings.ts";
import { MatrixRTCMode } from "../../../config/ConfigOptions.ts";
import {
MatrixRTCMode,
type ResolvedDelayedLeaveTimings,
} from "../../../config/ConfigOptions.ts";
import { Config } from "../../../config/Config.ts";
import {
ConnectionState,
@@ -72,6 +76,8 @@ import {
import { type HomeserverConnected } from "./HomeserverConnected.ts";
import { type LocalTransport } from "./LocalTransport.ts";
import { areLivekitTransportsEqual } from "../remoteMembers/MatrixLivekitMembers.ts";
import { or$ } from "../../../utils/observable.ts";
import { getSFUConfigWithOpenID } from "../../../livekit/openIDSFU.ts";
export enum TransportState {
/** Not even a transport is available to the LocalMembership */
@@ -133,14 +139,22 @@ interface Props {
muteStates: MuteStates;
connectionManager: IConnectionManager;
createPublisherFactory: (connection: Connection) => Publisher;
joinMatrixRTC: (transport: LivekitTransportConfig) => void;
joinMatrixRTC: (
transport: LivekitTransportConfig,
delayedLeaveTimings: ResolvedDelayedLeaveTimings,
) => void;
homeserverConnected: HomeserverConnected;
roomId: string;
localTransport$: Behavior<LocalTransport>;
ownMembershipIdentity: CallMembershipIdentityParts;
localTransport: LocalTransport;
client: Pick<MatrixClient, "getDeviceId" | "getOpenIdToken">;
matrixRTCSession: Pick<
MatrixRTCSession,
"updateCallIntent" | "leaveRoomSession"
>;
baseUrl: string;
delayId$: Behavior<string | null>;
matrixRTCMode: MatrixRTCMode;
logger: Logger;
}
@@ -155,10 +169,12 @@ interface Props {
* @param props.createPublisherFactory Factory to create a publisher once we have a connection.
* @param props.joinMatrixRTC Callback to join the matrix RTC session once we have a transport.
* @param props.homeserverConnected The homeserver connected state.
* @param props.localTransport$ The transport to advertise in our membership.
* @param props.localTransport The transport to advertise in our membership.
* @param props.logger The logger to use.
* @param props.muteStates The mute states for video and audio.
* @param props.matrixRTCSession The matrix RTC session to join.
* @param props.baseUrl Base URL of the homeserver.
* @param props.delayId$ ID of the delayed leave event to delegate to the SFU.
* @param props.roomId The room ID used as the call identifier in analytics events.
* @returns
* - publisher: The handle to create tracks and publish them to the room.
@@ -170,14 +186,19 @@ interface Props {
export const createLocalMembership$ = ({
scope,
connectionManager,
localTransport$,
localTransport,
homeserverConnected,
createPublisherFactory,
joinMatrixRTC,
logger: parentLogger,
muteStates,
client,
matrixRTCSession,
baseUrl,
roomId,
ownMembershipIdentity,
delayId$,
matrixRTCMode,
}: Props): {
/**
* This request to start audio and video tracks.
@@ -196,6 +217,11 @@ export const createLocalMembership$ = ({
* Callback to toggle screen sharing. If null, screen sharing is not possible.
*/
toggleScreenSharing: (() => void) | null;
/**
* The last error from toggling screen sharing, until dismissed.
*/
screenShareError$: Behavior<Error | null>;
dismissScreenShareError: () => void;
// tracks$: Behavior<LocalTrack[]>;
participant$: Behavior<LocalParticipant | null>;
connection$: Behavior<Connection | null>;
@@ -231,25 +257,69 @@ export const createLocalMembership$ = ({
return of(null);
};
// This is the transport that we will advertise in our membership.
const advertisedTransport$ = localTransport$.pipe(
switchMap((lt) => lt.advertised$),
async function checkDelegationSupport(
endpointUrl: string,
serviceName: string,
): Promise<boolean> {
logger.info(`Checking whether ${serviceName} supports delegation…`);
try {
// Bluntly hit the endpoint without auth to check for a 404. Unfortunately
// we can't wrap this in a retry loop, as many servers don't just disable
// delegation support, but in fact are from a time before the endpoint
// existed at all, therefore we can hit CORS errors which would just gum
// up the retry loop. (May be revisited after Matrix 2.0.)
const res = await fetch(endpointUrl, { method: "POST" });
if (res.status === 404) {
logger.warn(`${serviceName} does not support delegation`);
return false;
} else {
logger.info(`${serviceName} supports delegation`);
return true;
}
} catch (e) {
logger.warn(
`Failed to determine whether ${serviceName} supports delegation, assuming no support`,
e,
);
return false;
}
}
const homeserverSupportsDelegation = checkDelegationSupport(
baseUrl +
"/_matrix/client/unstable/io.element.msc4195/rtc/livekit/delegate_delayed_leave",
"homeserver",
);
// The transport that we will advertise in our membership, paired with info as
// to whether delayed event delegation is supported
const joinParams$ = localTransport.advertised$.pipe(
catchError(handleTransportError),
distinctUntilChanged(areLivekitTransportsEqual),
switchMap((transport) => {
if (transport === null) return of(null);
const transportSupportsDelegation = checkDelegationSupport(
transport.livekit_service_url + "/delegate_delayed_leave",
`transport ${transport.livekit_service_url}`,
);
return or$(
from(homeserverSupportsDelegation),
from(transportSupportsDelegation),
).pipe(
map((delegationSupported) => ({ transport, delegationSupported })),
startWith(null),
);
}),
);
// Unwrap the local transport and set the state of the LocalMembership to error in case the transport is an error.
const activeTransport$ = scope.behavior(
localTransport$.pipe(
switchMap((lt) => {
return combineLatest([lt.active$, lt.advertised$]).pipe(
map(([active, advertised]) => {
// Our policy is to not publish to another transport if our prefered transport is miss-configured
if (advertised == null) return null;
combineLatest([localTransport.active$, localTransport.advertised$]).pipe(
map(([active, advertised]) => {
// Our policy is to not publish to another transport if our prefered transport is miss-configured
if (advertised == null) return null;
return active?.transport ?? null;
}),
);
return active?.transport ?? null;
}),
catchError(handleTransportError),
distinctUntilChanged(areLivekitTransportsEqual),
@@ -612,16 +682,20 @@ export const createLocalMembership$ = ({
// Keep matrix rtc session in sync with advertisedTransport$, connectRequested$
scope.reconcile(
scope.behavior(
combineLatest([advertisedTransport$, joinAndPublishRequested$]),
),
async ([transport, shouldConnect]) => {
if (!transport) return;
scope.behavior(combineLatest([joinParams$, joinAndPublishRequested$])),
async ([joinParams, shouldConnect]) => {
if (!joinParams) return;
// if shouldConnect=false we will do the disconnect as the cleanup from the previous reconcile iteration.
if (!shouldConnect) return;
const sessionConfig = Config.get().matrix_rtc_session;
try {
joinMatrixRTC(transport);
joinMatrixRTC(
joinParams.transport,
joinParams.delegationSupported
? sessionConfig.delegated_delayed_leave
: sessionConfig.delayed_leave,
);
} catch (error) {
logger.error("Error entering RTC session", error);
if (error instanceof Error)
@@ -648,6 +722,34 @@ export const createLocalMembership$ = ({
),
);
// Delegate delayed leaves to the SFU
scope.reconcile(
scope.behavior(combineLatest([joinParams$, delayId$])),
async ([joinParams, delayId]) => {
if (joinParams?.delegationSupported && delayId !== null) {
try {
// This will technically cause the service to issue a new JWT token,
// but it's safe to discard. We're only interested in triggering
// delegation.
await getSFUConfigWithOpenID(
client,
ownMembershipIdentity,
joinParams.transport.livekit_service_url,
roomId,
{ matrixRTCMode, delayEndpointBaseUrl: baseUrl, delayId },
logger,
);
} catch (e) {
// TODO: Surface this to the user as a service interruption?
logger.error(
`Failed to delegate leave to ${joinParams.transport.livekit_service_url}`,
e,
);
}
}
},
);
// Pause upstream of all local media tracks when we're disconnected from
// MatrixRTC, because it can be an unpleasant surprise for the app to say
// 'reconnecting' and yet still be transmitting your media to others.
@@ -706,6 +808,7 @@ export const createLocalMembership$ = ({
),
);
const screenShareError$ = new BehaviorSubject<Error | null>(null);
let toggleScreenSharing: (() => void) | null = null;
if (
"getDisplayMedia" in (navigator.mediaDevices ?? {}) &&
@@ -777,13 +880,18 @@ export const createLocalMembership$ = ({
// We also allow screen sharing to be toggled even if the connection
// is still initializing or publishing tracks, because there's no
// technical reason to disallow this. LiveKit will publish if it can.
participant$.value
?.setScreenShareEnabled(
const participant = participant$.value;
if (!participant) return;
watchScreenShareToggle(
participant.setScreenShareEnabled(
targetScreenshareState,
screenshareSettings,
publishOptions,
)
.catch(logger.error);
),
targetScreenshareState,
logger,
(e) => screenShareError$.next(e),
);
};
}
@@ -802,11 +910,44 @@ export const createLocalMembership$ = ({
),
sharingScreen$,
toggleScreenSharing,
screenShareError$,
dismissScreenShareError: () => screenShareError$.next(null),
connection$: localConnection$,
internalLoggerRef: logger,
};
};
/**
* Logs the outcome of a screen share toggle and reports failures.
*
* getDisplayMedia may legitimately take a long time (the user is choosing
* what to share) or never settle at all, so nothing is inferred from silence:
* the request and its completion are logged with the elapsed time so that a
* hang is visible in the logs, and only an explicit rejection is reported.
*
* The user cancelling the picker rejects with a NotAllowedError; that is
* logged but not reported.
*/
export function watchScreenShareToggle(
toggle: Promise<unknown>,
enable: boolean,
logger: Logger,
onError: (e: Error) => void,
): void {
const what = `Screen share ${enable ? "start" : "stop"}`;
const started = Date.now();
const elapsed = (): string => `${Date.now() - started} ms`;
logger.info(`${what} requested`);
toggle.then(
() => logger.info(`${what} completed in ${elapsed()}`),
(e: unknown) => {
logger.error(`${what} failed after ${elapsed()}:`, e);
if (e instanceof DOMException && e.name === "NotAllowedError") return;
onError(e instanceof Error ? e : new Error(String(e)));
},
);
}
export function observeSharingScreen$(p: Participant): Observable<boolean> {
return observeParticipantEvents(
p,
@@ -820,6 +961,7 @@ export function observeSharingScreen$(p: Participant): Observable<boolean> {
interface EnterRTCSessionOptions {
encryptMedia: boolean;
matrixRTCMode: MatrixRTCMode;
delayedLeaveTimings: ResolvedDelayedLeaveTimings;
}
/**
@@ -832,6 +974,7 @@ interface EnterRTCSessionOptions {
* @param rtcSession - The MatrixRTCSession to join.
* @param ownMembershipIdentity - Options for entering the RTC session.
* @param transport - The LivekitTransport to use for this session.
* @param delayedLeaveTimings - The preferred timings for delayed leave events.
* @param options - `encryptMedia`: Whether to encrypt media `matrixRTCMode`: The Matrix RTC mode to use.
* @throws If the widget could not send ElementWidgetActions.JoinCall action.
*/
@@ -840,9 +983,8 @@ export function enterRTCSession(
rtcSession: MatrixRTCSession,
ownMembershipIdentity: CallMembershipIdentityParts,
transport: LivekitTransportConfig,
options: EnterRTCSessionOptions,
{ encryptMedia, matrixRTCMode, delayedLeaveTimings }: EnterRTCSessionOptions,
): void {
const { encryptMedia, matrixRTCMode } = options;
PosthogAnalytics.instance.eventCallEnded.cacheStartCall(new Date());
PosthogAnalytics.instance.eventCallStarted.track(rtcSession.room.roomId);
@@ -850,7 +992,11 @@ export function enterRTCSession(
// have started tracking by the time calls start getting created.
// groupCallOTelMembership?.onJoinCall();
const { matrix_rtc_session: matrixRtcSessionConfig } = Config.get();
const {
sync_disconnect_grace_period_ms: gracePeriod,
matrix_rtc_session: sessionConfig,
} = Config.get();
const retryInterval = sessionConfig.network_error_retry_ms;
const { sendNotificationType: notificationType, callIntent } = getUrlParams();
const multiSFU =
matrixRTCMode === MatrixRTCMode.Compatibility ||
@@ -868,16 +1014,11 @@ export function enterRTCSession(
};
}
// Calculates `maximumNetworkErrorRetryCount`. The connection is failed if EITHER:
// - The /sync loop is unresponsive for > `gracePeriod` ms, or
// - A delayed leave event is emitted (after `leaveDelay` ms period).
// Note: Use leaveDelay >> gracePeriod for delegated leave events.
const gracePeriod = Config.get().sync_disconnect_grace_period_ms;
const leaveDelay = matrixRtcSessionConfig?.delayed_leave_event_delay_ms;
const retryInterval = matrixRtcSessionConfig?.network_error_retry_ms;
// Set maximumNetworkErrorRetryCount such that we will consider the client
// disconnected as soon as either it fails to sync for longer than the grace
// period, or it is likely that a delayed leave event has been sent.
// Math.min is used to account for the respective worst case: /sync not available or leave event emitted.
const maxWaitTime = Math.min(gracePeriod, leaveDelay);
const maxWaitTime = Math.min(gracePeriod, delayedLeaveTimings.delay_ms);
const maximumNetworkErrorRetryCount =
Math.ceil(maxWaitTime / retryInterval) + 1;
@@ -892,18 +1033,14 @@ export function enterRTCSession(
notificationType,
callIntent,
manageMediaKeys: encryptMedia,
delayedLeaveEventRestartMs:
matrixRtcSessionConfig?.delayed_leave_event_restart_ms,
delayedLeaveEventDelayMs:
matrixRtcSessionConfig?.delayed_leave_event_delay_ms,
delayedLeaveEventRestartMs: delayedLeaveTimings.restart_ms,
delayedLeaveEventDelayMs: delayedLeaveTimings.delay_ms,
delayedLeaveEventRestartLocalTimeoutMs:
matrixRtcSessionConfig?.delayed_leave_event_restart_local_timeout_ms,
networkErrorRetryMs: matrixRtcSessionConfig?.network_error_retry_ms,
makeKeyDelay: matrixRtcSessionConfig?.wait_for_key_rotation_ms,
membershipEventExpiryMs:
matrixRtcSessionConfig?.membership_event_expiry_ms,
keyRotationParticipantLimit:
matrixRtcSessionConfig?.key_rotation_participant_limit,
delayedLeaveTimings.restart_timeout_ms,
networkErrorRetryMs: sessionConfig.network_error_retry_ms,
makeKeyDelay: sessionConfig.wait_for_key_rotation_ms,
membershipEventExpiryMs: sessionConfig.membership_event_expiry_ms,
keyRotationParticipantLimit: sessionConfig.key_rotation_participant_limit,
unstableSendStickyEvents: matrixRTCMode === MatrixRTCMode.Matrix_2_0,
maximumNetworkErrorRetryCount: maximumNetworkErrorRetryCount,
},
@@ -14,11 +14,8 @@ import {
type MockedObject,
vi,
} from "vitest";
import {
type CallMembership,
type LivekitTransportConfig,
} from "matrix-js-sdk/lib/matrixrtc";
import { BehaviorSubject, filter, lastValueFrom } from "rxjs";
import { type CallMembership } from "matrix-js-sdk/lib/matrixrtc";
import { lastValueFrom } from "rxjs";
import fetchMock from "fetch-mock";
import {
@@ -27,11 +24,7 @@ import {
ownMemberMock,
testScope,
} from "../../../utils/test";
import {
createLocalTransport$,
JwtEndpointVersion,
type LocalTransportWithSFUConfig,
} from "./LocalTransport";
import { createLocalTransport$ } from "./LocalTransport";
import { constant } from "../../Behavior";
import { Epoch, ObservableScope } from "../../ObservableScope";
import {
@@ -41,6 +34,7 @@ import {
import * as openIDSFU from "../../../livekit/openIDSFU";
import { customLivekitUrl } from "../../../settings/settings";
import { testJWTToken } from "../../../utils/test-fixtures";
import { MatrixRTCMode } from "../../../config/ConfigOptions";
describe("LocalTransport", () => {
const openIdResponse: openIDSFU.SFUConfig = {
@@ -61,14 +55,12 @@ describe("LocalTransport", () => {
// eslint-disable-next-line @typescript-eslint/naming-convention
_unstable_getRTCTransports: async () => Promise.resolve([]),
getDomain: () => "example.org",
baseUrl: "example.org",
// These won't be called in this error path but satisfy the type
getOpenIdToken: vi.fn(),
getDeviceId: vi.fn(),
},
ownMembershipIdentity: ownMemberMock,
forceJwtEndpoint: JwtEndpointVersion.Legacy,
delayId$: constant("delay_id_mock"),
matrixRTCMode: MatrixRTCMode.Compatibility,
});
await flushPromises();
@@ -100,7 +92,6 @@ describe("LocalTransport", () => {
roomId: "!example_room_id",
memberships$: constant(new Epoch<CallMembership[]>([])),
client: {
baseUrl: "https://example.org",
getDomain: () => "example.org",
// eslint-disable-next-line @typescript-eslint/naming-convention
_unstable_getRTCTransports: async () => Promise.resolve([]),
@@ -108,8 +99,7 @@ describe("LocalTransport", () => {
getDeviceId: vi.fn(),
},
ownMembershipIdentity: ownMemberMock,
forceJwtEndpoint: JwtEndpointVersion.Legacy,
delayId$: constant("delay_id_mock"),
matrixRTCMode: MatrixRTCMode.Compatibility,
});
active$.subscribe(
(o) => observations.push(o),
@@ -147,11 +137,9 @@ describe("LocalTransport", () => {
getDomain: () => "example.org",
getOpenIdToken: vi.fn(),
getDeviceId: vi.fn(),
baseUrl: "https://example.org",
},
ownMembershipIdentity: ownMemberMock,
forceJwtEndpoint: JwtEndpointVersion.Legacy,
delayId$: constant("delay_id_mock"),
matrixRTCMode: MatrixRTCMode.Compatibility,
});
openIdResolver.resolve?.({
@@ -194,11 +182,9 @@ describe("LocalTransport", () => {
ownMembershipIdentity: ownMemberMock,
scope: testScope(),
roomId: "!example_room_id",
forceJwtEndpoint: JwtEndpointVersion.Legacy,
delayId$: constant(null),
matrixRTCMode: MatrixRTCMode.Compatibility,
memberships$: constant(new Epoch<CallMembership[]>([])),
client: {
baseUrl: "https://example.org",
getDomain: vi.fn().mockReturnValue("example.org"),
// eslint-disable-next-line @typescript-eslint/naming-convention
_unstable_getRTCTransports: vi.fn().mockResolvedValue([]),
@@ -306,12 +292,10 @@ describe("LocalTransport", () => {
scope: testScope(),
ownMembershipIdentity: ownMemberMock,
roomId: "!example_room_id",
forceJwtEndpoint: JwtEndpointVersion.Legacy,
delayId$: constant(null),
matrixRTCMode: MatrixRTCMode.Compatibility,
memberships$: constant(new Epoch<CallMembership[]>([])),
client: {
getDomain: () => "example.org",
baseUrl: "https://example.org",
// eslint-disable-next-line @typescript-eslint/naming-convention
_unstable_getRTCTransports: async () => Promise.resolve([]),
// These won't be called in this error path but satisfy the type
@@ -329,84 +313,4 @@ describe("LocalTransport", () => {
);
});
});
it("should not update advertised/active transport on delayID changes, but delay Id delegation should be called", async () => {
// For simplicity, we'll just use the config livekit
customLivekitUrl.setValue("https://lk.example.org");
const authCallSpy = vi
.spyOn(openIDSFU, "getSFUConfigWithOpenID")
.mockResolvedValue(openIdResponse);
const delayId$ = new BehaviorSubject<string | null>(null);
const { advertised$, active$ } = createLocalTransport$({
scope: testScope(),
ownMembershipIdentity: ownMemberMock,
roomId: "!example_room_id",
// We want multi-sdu
forceJwtEndpoint: JwtEndpointVersion.Legacy,
delayId$: delayId$,
memberships$: constant(new Epoch<CallMembership[]>([])),
client: {
getDomain: () => "example.org",
baseUrl: "https://example.org",
// eslint-disable-next-line @typescript-eslint/naming-convention
_unstable_getRTCTransports: async () => Promise.resolve([]),
// These won't be called in this error path but satisfy the type
getOpenIdToken: vi.fn(),
getDeviceId: vi.fn(),
},
});
const advertisedValues: LivekitTransportConfig[] = [];
const activeValues: LocalTransportWithSFUConfig[] = [];
advertised$
.pipe(filter((v) => v !== null))
.subscribe((t) => advertisedValues.push(t));
active$
.pipe(filter((v) => v !== null))
.subscribe((t) => activeValues.push(t));
await flushPromises();
// we have now an active and an advertised
expect(advertisedValues.length).toEqual(1);
expect(activeValues.length).toEqual(1);
expect(advertisedValues[0]!.livekit_service_url).toEqual(
"https://lk.example.org",
);
expect(activeValues[0]!.transport.livekit_service_url).toEqual(
"https://lk.example.org",
);
expect(authCallSpy).toHaveBeenCalledTimes(2);
// Now emits 3 new delays id
delayId$.next("delay_id_1");
await flushPromises();
delayId$.next("delay_id_2");
await flushPromises();
delayId$.next("delay_id_3");
await flushPromises();
// No new emissions should've happened, it is the same transport.
expect(advertisedValues.length).toEqual(1);
expect(activeValues.length).toEqual(1);
// Still we should have updated the delayID to auth
expect(authCallSpy).toHaveBeenCalledTimes(
4 * 2 /* 2 calls for each delayId ?? why */,
);
expect(authCallSpy).toHaveBeenLastCalledWith(
expect.anything(),
expect.anything(),
expect.anything(),
expect.anything(),
expect.objectContaining({
delayId: "delay_id_3",
}),
expect.anything(),
);
});
});
@@ -10,14 +10,7 @@ import {
type LivekitTransportConfig,
} from "matrix-js-sdk/lib/matrixrtc";
import { type MatrixClient } from "matrix-js-sdk";
import {
combineLatest,
distinctUntilChanged,
from,
map,
of,
switchMap,
} from "rxjs";
import { distinctUntilChanged, from, map, of, switchMap } from "rxjs";
import { logger as rootLogger, type Logger } from "matrix-js-sdk/lib/logger";
import { type CallMembershipIdentityParts } from "matrix-js-sdk/lib/matrixrtc/EncryptionManager";
@@ -37,6 +30,7 @@ import {
import { areLivekitTransportsEqual } from "../remoteMembers/MatrixLivekitMembers.ts";
import { customLivekitUrl } from "../../../settings/settings.ts";
import { RtcTransportAutoDiscovery } from "./RtcTransportAutoDiscovery.ts";
import { type MatrixRTCMode } from "../../../config/ConfigOptions.ts";
/*
* It figures out “which LiveKit focus URL/alias the local user should use,”
@@ -46,20 +40,11 @@ interface Props {
scope: ObservableScope;
ownMembershipIdentity: CallMembershipIdentityParts;
memberships$: Behavior<Epoch<CallMembership[]>>;
client: Pick<
MatrixClient,
"getDomain" | "baseUrl" | "_unstable_getRTCTransports"
> &
client: Pick<MatrixClient, "getDomain" | "_unstable_getRTCTransports"> &
OpenIDClientParts;
// Used by the jwt service to create the livekit room and compute the livekit alias.
roomId: string;
forceJwtEndpoint: JwtEndpointVersion;
delayId$: Behavior<string | null>;
}
export enum JwtEndpointVersion {
Legacy = "legacy",
Matrix_2_0 = "matrix_2_0",
matrixRTCMode: MatrixRTCMode;
}
// TODO livekit_alias-cleanup
@@ -122,8 +107,7 @@ export const createLocalTransport$ = ({
ownMembershipIdentity,
client,
roomId,
forceJwtEndpoint,
delayId$,
matrixRTCMode,
}: Props): LocalTransport => {
const logger = rootLogger.getChild("[LocalTransport]");
@@ -138,40 +122,36 @@ export const createLocalTransport$ = ({
transportDiscovery.discoverPreferredTransport(),
);
const preferredConfig$ = customLivekitUrl.value$
.pipe(
switchMap((customUrl) => {
if (customUrl) {
return of({
type: "livekit",
livekit_service_url: customUrl,
} as LivekitTransportConfig);
} else {
return discoveredTransport$;
}
}),
)
.pipe(
map((config) => {
if (!config) {
// Bubbled up from the preferredConfig$ observable.
throw new MatrixRTCTransportMissingError(client.getDomain() ?? "");
}
return config;
}),
distinctUntilChanged(areLivekitTransportsEqual),
);
const preferredConfig$ = customLivekitUrl.value$.pipe(
switchMap((customUrl) => {
if (customUrl) {
return of({
type: "livekit",
livekit_service_url: customUrl,
} as LivekitTransportConfig);
} else {
return discoveredTransport$;
}
}),
map((config) => {
if (!config) {
// Bubbled up from the preferredConfig$ observable.
throw new MatrixRTCTransportMissingError(client.getDomain() ?? "");
}
return config;
}),
distinctUntilChanged(areLivekitTransportsEqual),
);
const preferredTransport$ = combineLatest([preferredConfig$, delayId$]).pipe(
switchMap(async ([transport, delayId]) => {
const preferredTransport$ = preferredConfig$.pipe(
switchMap(async (transport) => {
try {
return await doOpenIdAndJWTFromUrl(
transport,
forceJwtEndpoint,
matrixRTCMode,
ownMembershipIdentity,
roomId,
client,
delayId ?? undefined,
logger,
);
} catch (e) {
@@ -193,21 +173,7 @@ export const createLocalTransport$ = ({
),
null,
),
active$: scope.behavior(
preferredTransport$.pipe(
// XXX: WORK AROUND due to a reconnection glitch.
// To remove when we have a proper way to refresh the delegation event ID without refreshing
// the whole credentials.
// We deliberately hide any changes to the SFU config because we
// do not want the app to reconnect whenever the JWT
// token changes due to us delegating a new delayed event. The
// initial SFU config for the transport is all the app needs.
distinctUntilChanged((prev, next) =>
areLivekitTransportsEqual(prev.transport, next.transport),
),
),
null,
),
active$: scope.behavior(preferredTransport$, null),
};
};
@@ -219,25 +185,19 @@ export const createLocalTransport$ = ({
* use we don't want to risk any issues by re-using a token.
*
* @param transport The transport to authenticate with.
* @param forceJwtEndpoint Whether to force the JWT endpoint to be used.
* @param matrixRTCMode Whether to force the JWT endpoint to be used.
* @param membership The identity of the local member.
* @param roomId The room ID to use for the JWT.
* @param client The client to use for the OpenID token.
* @param delayId The delayId to use for the JWT.
*
* @throws FailToGetOpenIdToken, NoMatrix2AuthorizationService
*/
async function doOpenIdAndJWTFromUrl(
transport: LivekitTransportConfig,
forceJwtEndpoint: JwtEndpointVersion,
matrixRTCMode: MatrixRTCMode,
membership: CallMembershipIdentityParts,
roomId: string,
client: Pick<
MatrixClient,
"getDomain" | "baseUrl" | "_unstable_getRTCTransports"
> &
OpenIDClientParts,
delayId?: string,
client: Pick<MatrixClient, "_unstable_getRTCTransports"> & OpenIDClientParts,
logger?: Logger,
): Promise<LocalTransportWithSFUConfig> {
const sfuConfig = await getSFUConfigWithOpenID(
@@ -245,11 +205,7 @@ async function doOpenIdAndJWTFromUrl(
membership,
transport.livekit_service_url,
roomId,
{
forceJwtEndpoint: forceJwtEndpoint,
delayEndpointBaseUrl: client.baseUrl,
delayId,
},
{ matrixRTCMode },
logger,
);
return {
@@ -19,14 +19,17 @@ import {
import {
type LocalParticipant,
type RemoteParticipant,
type RemoteTrack,
type RemoteTrackPublication,
type Room as LivekitRoom,
RoomEvent,
Track,
ConnectionState as LivekitConnectionState,
} from "livekit-client";
import fetchMock from "fetch-mock";
import EventEmitter from "events";
import { type IOpenIDToken } from "matrix-js-sdk";
import { logger } from "matrix-js-sdk/lib/logger";
import { logger, type Logger } from "matrix-js-sdk/lib/logger";
import { type LivekitTransportConfig } from "matrix-js-sdk/lib/matrixrtc";
import {
@@ -509,6 +512,97 @@ describe("remote participants", () => {
});
});
describe("remote track logging", () => {
it("logs remote participant and track events on the connection logger", () => {
setupTest();
const info = vi.fn();
const warn = vi.fn();
const testLogger = {
getChild: (): unknown => testLogger,
info,
debug: vi.fn(),
warn,
error: vi.fn(),
} as unknown as Logger;
new Connection(
{
client,
roomId: ROOM_ID,
transport: livekitFocus,
scope: testScope,
ownMembershipIdentity: ownMemberMock,
livekitRoomFactory: () => fakeLivekitRoom,
},
testLogger,
);
info.mockClear(); // drop the constructor log line
const bob = mockRemoteParticipant({
identity: "@bob:example.org:DEV111",
sid: "PA_bob",
});
const pub = {
kind: "audio",
source: "microphone",
trackSid: "TR_mic",
isMuted: false,
isEncrypted: true,
} as unknown as RemoteTrackPublication;
const messages = (): string[] => info.mock.calls.map((c) => c[0] as string);
fakeLivekitRoom.emit(RoomEvent.ParticipantConnected, bob);
fakeLivekitRoom.emit(
RoomEvent.TrackSubscribed,
{} as RemoteTrack,
pub,
bob,
);
fakeLivekitRoom.emit(RoomEvent.TrackMuted, pub, bob);
fakeLivekitRoom.emit(
RoomEvent.TrackStreamStateChanged,
pub,
Track.StreamState.Paused,
bob,
);
expect(messages()).toEqual([
"Participant connected: @bob:example.org:DEV111 (PA_bob)",
"Subscribed: audio microphone TR_mic of @bob:example.org:DEV111 encrypted=true muted=false",
"Muted: audio microphone TR_mic of @bob:example.org:DEV111 encrypted=true",
"Stream paused: audio microphone TR_mic of @bob:example.org:DEV111 encrypted=true",
]);
// Encryption status changes are logged; cryptor errors are warnings
fakeLivekitRoom.emit(
RoomEvent.ParticipantEncryptionStatusChanged,
false,
bob,
);
expect(messages().at(-1)).toBe(
"Encryption status of @bob:example.org:DEV111: encrypted=false",
);
const cryptorError = new Error("missing key at index 3");
fakeLivekitRoom.emit(RoomEvent.EncryptionError, cryptorError, bob);
expect(warn).toHaveBeenCalledWith(
"Encryption error for @bob:example.org:DEV111:",
cryptorError,
);
// Local mute events are already logged by the Publisher
fakeLivekitRoom.emit(RoomEvent.TrackMuted, pub, {
...fakeLocalParticipant,
isLocal: true,
} as unknown as LocalParticipant);
expect(messages().filter((m) => m.startsWith("Muted"))).toHaveLength(1);
// Listeners are removed when the scope ends
testScope.end();
fakeLivekitRoom.emit(RoomEvent.ParticipantDisconnected, bob);
expect(
messages().filter((m) => m.startsWith("Participant disconnected")),
).toHaveLength(0);
});
});
//
// NOT USED ANYMORE ?
//
@@ -13,8 +13,14 @@ import {
import {
ConnectionError,
ConnectionErrorReason,
type Participant,
type RemoteParticipant,
type RemoteTrackPublication,
type Room as LivekitRoom,
RoomEvent,
type SubscriptionError,
type Track,
type TrackPublication,
} from "livekit-client";
import { type LivekitTransportConfig } from "matrix-js-sdk/lib/matrixrtc";
import { BehaviorSubject, map } from "rxjs";
@@ -164,6 +170,7 @@ export class Connection {
// Only tracks remote participants
connectedParticipantsObserver(this.livekitRoom),
);
this.logRemoteTrackEvents();
scope.onEnd(() => {
this.logger.info(`Connection scope ended, stopping connection`);
@@ -171,6 +178,114 @@ export class Connection {
});
}
/**
* Logs the lifecycle of remote participants and their tracks as seen by
* livekit-client, so that rageshakes show what a tile should have been
* rendering (published, subscribed, muted, paused) for each remote member.
*/
private logRemoteTrackEvents(): void {
const room = this.livekitRoom;
const log = this.logger.getChild("[RemoteTracks]");
// The encryption flag matters: if the publisher encrypts but this client
// believes the track is unencrypted, the cryptor is bypassed and raw
// ciphertext reaches the decoder (audible as loud noise bursts).
const track = (pub: TrackPublication, p: Participant): string =>
`${pub.kind} ${pub.source} ${pub.trackSid} of ${p.identity} encrypted=${pub.isEncrypted}`;
const onParticipantConnected = (p: RemoteParticipant): void =>
log.info(`Participant connected: ${p.identity} (${p.sid})`);
const onParticipantDisconnected = (p: RemoteParticipant): void =>
log.info(`Participant disconnected: ${p.identity} (${p.sid})`);
const onTrackPublished = (
pub: RemoteTrackPublication,
p: RemoteParticipant,
): void => log.info(`Published: ${track(pub, p)} muted=${pub.isMuted}`);
const onTrackUnpublished = (
pub: RemoteTrackPublication,
p: RemoteParticipant,
): void => log.info(`Unpublished: ${track(pub, p)}`);
const onTrackSubscribed = (
_t: Track,
pub: RemoteTrackPublication,
p: RemoteParticipant,
): void => log.info(`Subscribed: ${track(pub, p)} muted=${pub.isMuted}`);
const onTrackUnsubscribed = (
_t: Track,
pub: RemoteTrackPublication,
p: RemoteParticipant,
): void => log.info(`Unsubscribed: ${track(pub, p)}`);
const onTrackSubscriptionFailed = (
trackSid: string,
p: RemoteParticipant,
reason?: SubscriptionError,
): void =>
log.warn(
`Subscription failed: ${trackSid} of ${p.identity} reason=${reason}`,
);
// Mute events also fire for the local participant, which the Publisher
// already logs.
const onTrackMuted = (pub: TrackPublication, p: Participant): void => {
if (!p.isLocal) log.info(`Muted: ${track(pub, p)}`);
};
const onTrackUnmuted = (pub: TrackPublication, p: Participant): void => {
if (!p.isLocal) log.info(`Unmuted: ${track(pub, p)}`);
};
const onTrackStreamStateChanged = (
pub: RemoteTrackPublication,
state: Track.StreamState,
p: RemoteParticipant,
): void => log.info(`Stream ${state}: ${track(pub, p)}`);
const onEncryptionStatusChanged = (
encrypted: boolean,
p?: Participant,
): void =>
log.info(
`Encryption status of ${p?.identity ?? "unknown participant"}: encrypted=${encrypted}`,
);
// livekit-client throttles these per cryptor; they indicate frames being
// dropped (missing/invalid key).
const onEncryptionError = (error: Error, p?: Participant): void =>
log.warn(
`Encryption error for ${p?.identity ?? "unknown participant"}:`,
error,
);
room
.on(RoomEvent.ParticipantConnected, onParticipantConnected)
.on(RoomEvent.ParticipantDisconnected, onParticipantDisconnected)
.on(RoomEvent.TrackPublished, onTrackPublished)
.on(RoomEvent.TrackUnpublished, onTrackUnpublished)
.on(RoomEvent.TrackSubscribed, onTrackSubscribed)
.on(RoomEvent.TrackUnsubscribed, onTrackUnsubscribed)
.on(RoomEvent.TrackSubscriptionFailed, onTrackSubscriptionFailed)
.on(RoomEvent.TrackMuted, onTrackMuted)
.on(RoomEvent.TrackUnmuted, onTrackUnmuted)
.on(RoomEvent.TrackStreamStateChanged, onTrackStreamStateChanged)
.on(
RoomEvent.ParticipantEncryptionStatusChanged,
onEncryptionStatusChanged,
)
.on(RoomEvent.EncryptionError, onEncryptionError);
this.scope.onEnd(() => {
room
.off(RoomEvent.ParticipantConnected, onParticipantConnected)
.off(RoomEvent.ParticipantDisconnected, onParticipantDisconnected)
.off(RoomEvent.TrackPublished, onTrackPublished)
.off(RoomEvent.TrackUnpublished, onTrackUnpublished)
.off(RoomEvent.TrackSubscribed, onTrackSubscribed)
.off(RoomEvent.TrackUnsubscribed, onTrackUnsubscribed)
.off(RoomEvent.TrackSubscriptionFailed, onTrackSubscriptionFailed)
.off(RoomEvent.TrackMuted, onTrackMuted)
.off(RoomEvent.TrackUnmuted, onTrackUnmuted)
.off(RoomEvent.TrackStreamStateChanged, onTrackStreamStateChanged)
.off(
RoomEvent.ParticipantEncryptionStatusChanged,
onEncryptionStatusChanged,
)
.off(RoomEvent.EncryptionError, onEncryptionError);
});
}
/**
* Starts the connection.
*
@@ -14,7 +14,7 @@ import { logger } from "matrix-js-sdk/lib/logger";
import { Epoch, mapEpoch, ObservableScope } from "../../ObservableScope.ts";
import {
createConnectionManager$,
type ConnectionManagerData,
ConnectionManagerData,
} from "./ConnectionManager.ts";
import { type ConnectionFactory } from "./ConnectionFactory.ts";
import { type Connection } from "./Connection.ts";
@@ -203,6 +203,24 @@ describe("connections$ stream", () => {
});
});
describe("ConnectionManagerData", () => {
test("warns when a second connection to the same URL is merged", () => {
const warn = vi.spyOn(logger, "warn").mockImplementation(() => {});
const data = new ConnectionManagerData(logger);
const connection = { transport: TRANSPORT_1 } as unknown as Connection;
const p = (identity: string): RemoteParticipant =>
({ identity }) as unknown as RemoteParticipant;
data.add(connection, [p("a")]);
data.add({ ...connection } as Connection, [p("b")]);
expect(warn).toHaveBeenCalledWith(
expect.stringContaining(
"second connection to https://lk.example.org: existing [a], adding [b]",
),
);
expect(data.getParticipantsForTransport(TRANSPORT_1)).toHaveLength(2);
});
});
describe("connectionManagerData$ stream", () => {
// Used in test to control fake connections' remoteParticipants$ streams
let fakeRemoteParticipantsStreams: Map<string, Behavior<RemoteParticipant[]>>;
@@ -30,7 +30,7 @@ export class ConnectionManagerData {
{ connection: Connection; participants: RemoteParticipant[] }
> = new Map();
public constructor() {}
public constructor(private readonly logger?: Logger) {}
public add(connection: Connection, participants: RemoteParticipant[]): void {
const key = this.getKey(connection.transport);
@@ -38,6 +38,11 @@ export class ConnectionManagerData {
if (!existing) {
this.store.set(key, { connection, participants });
} else {
// Transports are deduplicated by URL upstream, so this should never
// happen; if it does, members may be matched against the wrong room.
this.logger?.warn(
`Merging participants from a second connection to ${key}: existing [${existing.participants.map((p) => p.identity).join(", ")}], adding [${participants.map((p) => p.identity).join(", ")}]`,
);
existing.participants.push(...participants);
}
}
@@ -239,7 +244,7 @@ export function createConnectionManager$({
lists.reduce((data, { connection, participants }) => {
data.add(connection, participants);
return data;
}, new ConnectionManagerData()),
}, new ConnectionManagerData(logger)),
epoch,
),
),
@@ -5,7 +5,8 @@ SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { describe, test, expect, beforeEach, afterEach } from "vitest";
import { describe, test, expect, beforeEach, afterEach, vi } from "vitest";
import { logger } from "matrix-js-sdk/lib/logger";
import {
type CallMembership,
type LivekitTransport,
@@ -77,6 +78,7 @@ function epochMeWith$<T, U>(
}
test("should signal participant not yet connected to livekit", async () => {
const info = vi.spyOn(logger, "info");
const mockedMemberships$ = new BehaviorSubject([bobMembership]);
const mockConnectionManagerData$ = new BehaviorSubject(
new ConnectionManagerData(),
@@ -108,6 +110,9 @@ test("should signal participant not yet connected to livekit", async () => {
return true;
},
);
expect(info).toHaveBeenCalledWith(
"[RemoteMatrixLivekitMembers] @bob:example.org:DEV000: LiveKit participant missing on no connection",
);
});
// Helper to create epoch'ed memberships$ and membershipsWithTransport$ from memberships observable.
@@ -11,6 +11,7 @@ import {
type LivekitTransportConfig,
} from "matrix-js-sdk/lib/matrixrtc";
import { combineLatest, filter, map } from "rxjs";
import { logger } from "matrix-js-sdk/lib/logger";
import { type Behavior } from "../../Behavior";
import { type IConnectionManager } from "./ConnectionManager";
@@ -110,13 +111,17 @@ export function createRemoteMatrixLivekitMembers$({
const participants = transport
? managerData.getParticipantsForTransport(transport)
: [];
const participant =
participants.find(
(p) => p.identity == membership.rtcBackendIdentity,
) ?? null;
const matches = participants.filter(
(p) => p.identity == membership.rtcBackendIdentity,
);
const participant = matches[0] ?? null;
const connection = transport
? managerData.getConnectionForTransport(transport)
: null;
if (matches.length > 1)
logger.warn(
`[RemoteMatrixLivekitMembers] ${membership.rtcBackendIdentity}: ${matches.length} LiveKit participants match (sids ${matches.map((p) => p.sid).join(", ")}), using ${participant?.sid}`,
);
yield {
// This could just be the backend identity without the other keys.
@@ -133,8 +138,16 @@ export function createRemoteMatrixLivekitMembers$({
}
},
// Each update where the key of the generator array do not change will result in updates to the `data$` behavior.
(scope, data$, userId, _deviceId, _memberId, _rtcBackendIdentity) => {
(scope, data$, userId, _deviceId, _memberId, rtcBackendIdentity) => {
const { participant$, ...rest } = scope.splitBehavior(data$);
// Log whether the member could be matched to a LiveKit participant,
// since a tile shows "waiting for media" for as long as it cannot.
participant$.pipe(scope.bind()).subscribe((p) => {
const url = data$.value.connection?.transport.livekit_service_url;
logger.info(
`[RemoteMatrixLivekitMembers] ${rtcBackendIdentity}: LiveKit participant ${p ? `matched (${p.sid})` : "missing"} on ${url ?? "no connection"}`,
);
});
// will only get called once per backend identity.
// updates to data$ and as a result to displayName$ and mxcAvatarUrl$ are more frequent.
return {
+59 -5
View File
@@ -16,7 +16,7 @@ import {
ObservableScope,
trackEpoch,
} from "./ObservableScope";
import { withTestScheduler } from "../utils/test";
import { testScope, withTestScheduler } from "../utils/test";
describe("Epoch", () => {
it("should map the value correctly", () => {
@@ -87,7 +87,7 @@ describe("Epoch", () => {
});
it("behavior test", () => {
const scope = new ObservableScope();
const scope = testScope();
const s$ = new Subject();
const behavior$ = scope.behavior(s$, 0);
behavior$.subscribe((value) => {
@@ -115,7 +115,7 @@ describe("Reconcile", () => {
it("should wait clean up before processing next", async () => {
vi.useFakeTimers();
const scope = new ObservableScope();
const scope = testScope();
const behavior$ = new BehaviorSubject<number>(0);
const setup = vi.fn().mockImplementation(async () => await sleep(100));
@@ -149,7 +149,7 @@ describe("Reconcile", () => {
it("should skip intermediates values that are not setup", async () => {
vi.useFakeTimers();
const scope = new ObservableScope();
const scope = testScope();
const behavior$ = new BehaviorSubject<number>(0);
const setup = vi
@@ -194,7 +194,7 @@ describe("Reconcile", () => {
it("should wait for setup to complete before starting cleanup", async () => {
vi.useFakeTimers();
const scope = new ObservableScope();
const scope = testScope();
const behavior$ = new BehaviorSubject<number>(0);
const setup = vi
@@ -237,3 +237,57 @@ describe("Reconcile", () => {
expect(setup).toHaveBeenCalledWith(1);
});
});
describe("behavior", () => {
it("warns with the tag and nesting depth, logging each new depth once", () => {
const warn = vi.spyOn(logger, "warn").mockImplementation(() => {});
const scope = testScope();
const source$ = new Subject<number>();
const behavior$ = scope.behavior(source$, 0, "tagged");
// Re-enter once on value 1 (depth 2), then a nested re-entry on value 2
// (depth 3), which is the shape needed to strand a middle subscriber.
behavior$.subscribe((v) => {
if (v === 1) source$.next(2);
});
behavior$.subscribe((v) => {
if (v === 2) source$.next(3);
});
const seen: number[] = [];
behavior$.subscribe((v) => seen.push(v));
source$.next(1);
const messages = warn.mock.calls.map((c) => c[0] as string);
expect(messages).toEqual([
expect.stringContaining("Behavior (tagged) re-entered at depth 2"),
expect.stringContaining("Behavior (tagged) re-entered at depth 3"),
]);
warn.mock.calls.forEach((c) => expect(c[1]).toEqual(expect.any(String)));
// The behavior settles on the newest value while the last subscriber is
// left stranded on the oldest.
expect(behavior$.value).toBe(3);
expect(seen.at(-1)).toBe(1);
});
it("warns when the scope is ended while a behavior is mid-delivery", () => {
const warn = vi.spyOn(logger, "warn").mockImplementation(() => {});
const scope = new ObservableScope();
const source$ = new Subject<number>();
const behavior$ = scope.behavior(source$, 0);
behavior$.subscribe((v) => {
if (v === 1) scope.end();
});
// A derived behavior is bound to the scope, so ending the scope
// mid-delivery unsubscribes it before the value reaches it.
const derived$ = scope.behavior(behavior$);
source$.next(1);
expect(warn).toHaveBeenCalledWith(
expect.stringContaining("Scope ended while 1 of its behaviors"),
expect.any(String),
);
expect(behavior$.value).toBe(1);
expect(derived$.value).toBe(0);
});
});
+50 -4
View File
@@ -20,6 +20,8 @@ import {
takeUntil,
} from "rxjs";
import { logger } from "matrix-js-sdk/lib/logger";
import { type Behavior } from "./Behavior";
type MonoTypeOperator = <T>(o: Observable<T>) => Observable<T>;
@@ -34,6 +36,12 @@ const nothing = Symbol("nothing");
* A scope which limits the execution lifetime of its bound Observables.
*/
export class ObservableScope {
/**
* Number of this scope's behaviors currently mid-delivery. Used to detect a
* scope being torn down synchronously from within one of its own emissions.
*/
private delivering = 0;
private readonly ended$ = new BehaviorSubject(false);
private readonly bindImpl: MonoTypeOperator = takeUntil(
@@ -66,6 +74,9 @@ export class ObservableScope {
public behavior<T>(
setValue$: Observable<T>,
initialValue: T | typeof nothing = nothing,
// An optional label used only in the re-entrancy warning below, so that a
// torn behavior can be identified from a rageshake.
tag?: string,
): Behavior<T> {
const subject$ = new BehaviorSubject(initialValue);
// Push values from the Observable into the BehaviorSubject.
@@ -73,11 +84,37 @@ export class ObservableScope {
// they will no longer re-emit their current value upon subscription. We want
// to support Observables that complete (for example `of({})`), so we have to
// take care to not propagate the completion event.
// If a subscriber synchronously causes this same behavior to emit again,
// rxjs delivers the nested value to every subscriber first and then
// resumes delivering the outer (older) value to the remaining subscribers,
// leaving them permanently out of sync with the others. A single such
// re-entry can only strand a contiguous run of subscribers; stranding a
// subscriber in the middle of the list needs a nested (deeper) re-entry, so
// we report the depth and log each new depth (with a stack trace) rather
// than only the first occurrence, to make a nested re-entry visible.
let depth = 0;
let maxReportedDepth = 1;
setValue$.pipe(this.bind(), distinctUntilChanged()).subscribe({
next(value) {
subject$.next(value);
next: (value) => {
if (depth > 0 && depth + 1 > maxReportedDepth) {
maxReportedDepth = depth + 1;
logger.warn(
`Behavior${tag ? ` (${tag})` : ""} re-entered at depth ${
depth + 1
} while delivering a value; later subscribers will be left with a stale value`,
new Error().stack,
);
}
depth++;
this.delivering++;
try {
subject$.next(value);
} finally {
depth--;
this.delivering--;
}
},
error(err: unknown) {
error: (err: unknown) => {
subject$.error(err);
},
});
@@ -90,6 +127,11 @@ export class ObservableScope {
* Ends the scope, causing any bound Observables to complete.
*/
public end(): void {
if (this.delivering > 0)
logger.warn(
`Scope ended while ${this.delivering} of its behaviors were still delivering a value; later subscribers will be left with a stale value`,
new Error().stack,
);
this.ended$.next(true);
}
@@ -171,7 +213,11 @@ export class ObservableScope {
return Object.fromEntries(
Object.keys(input$.value).map((key) => [
`${key}$`,
this.behavior(input$.pipe(map((input) => input[key as keyof T]))),
this.behavior(
input$.pipe(map((input) => input[key as keyof T])),
nothing,
key,
),
]),
) as SplitBehavior<T>;
}
+16 -10
View File
@@ -8,6 +8,7 @@ Please see LICENSE in the repository root for full details.
import { type RemoteParticipant } from "livekit-client";
import { combineLatest, map, of, switchMap } from "rxjs";
import { logger } from "matrix-js-sdk/lib/logger";
import { type Behavior } from "../Behavior";
import { createVolumeControls, type VolumeControls } from "../VolumeControls";
@@ -45,6 +46,20 @@ export function createRemoteUserMedia(
statsType: "inbound-rtp",
});
const waitingForMedia$ = scope.behavior(
combineLatest(
[inputs.livekitRoom$, inputs.participant$],
(livekitRoom, participant) =>
// If livekitRoom is undefined, the user is not attempting to publish on
// any transport and so we shouldn't expect a participant. (They might
// be a subscribe-only bot for example.)
livekitRoom !== undefined && participant === null,
),
);
waitingForMedia$.pipe(scope.bind()).subscribe((waiting) => {
logger.info(`[RemoteUserMedia ${inputs.id}] waitingForMedia=${waiting}`);
});
return {
...baseUserMedia,
...createVolumeControls(scope, {
@@ -68,15 +83,6 @@ export function createRemoteUserMedia(
),
),
),
waitingForMedia$: scope.behavior(
combineLatest(
[inputs.livekitRoom$, inputs.participant$],
(livekitRoom, participant) =>
// If livekitRoom is undefined, the user is not attempting to publish on
// any transport and so we shouldn't expect a participant. (They might
// be a subscribe-only bot for example.)
livekitRoom !== undefined && participant === null,
),
),
waitingForMedia$,
};
}
+22 -1
View File
@@ -9,9 +9,30 @@ import { expect, test } from "vitest";
import { type Observable, of, Subject, switchMap } from "rxjs";
import { withTestScheduler } from "./test";
import { filterBehavior, generateItems, pauseWhen } from "./observable";
import { or$, filterBehavior, generateItems, pauseWhen } from "./observable";
import { type Behavior } from "../state/Behavior";
const yesNo = {
y: true,
n: false,
};
test("or$", () => {
withTestScheduler(({ behavior, expectObservable }) => {
const input1Marbles = "ny--n--";
const input2Marbles = "n-y--n-";
const input3Marbles = "n--y--n";
const outputMarbles = "nyyyyyn";
expectObservable(
or$(
behavior(input1Marbles, yesNo),
behavior(input2Marbles, yesNo),
behavior(input3Marbles, yesNo),
),
).toBe(outputMarbles, yesNo);
});
});
test("pauseWhen", () => {
withTestScheduler(({ behavior, expectObservable }) => {
const inputMarbles = " abcdefgh-i-jk-";
+4 -38
View File
@@ -114,13 +114,11 @@ export function getValue<T>(state$: Observable<T>): T {
}
/**
* Creates an Observable that has a value of true whenever all its inputs are
* true.
*
* @public
* Creates an Observable that has a value of true whenever some of its inputs
* are true.
*/
export function and$(...inputs: Observable<boolean>[]): Observable<boolean> {
return combineLatest(inputs, (...flags) => flags.every((flag) => flag));
export function or$(...inputs: Observable<boolean>[]): Observable<boolean> {
return combineLatest(inputs, (...flags) => flags.some((flag) => flag));
}
/**
@@ -229,38 +227,6 @@ export function filterBehavior<T, S extends T>(
);
}
/**
* Maps a changing input value to an item whose lifetime is tied to a certain
* computed key. The item may capture some dynamic data from the input.
*/
export function generateItem<
Input,
Keys extends [unknown, ...unknown[]],
Data,
Item,
>(
name: string,
generator: (input: Input) => { keys: readonly [...Keys]; data: Data },
factory: (
scope: ObservableScope,
data$: Behavior<Data>,
...keys: Keys
) => Item,
): OperatorFunction<Input, Item> {
return (input$) =>
input$.pipe(
generateItemsInternal(
name,
function* (input) {
yield generator(input);
},
factory,
(items) => items,
),
map(([item]) => item),
);
}
function generateItemsInternal<
Input,
Keys extends [unknown, ...unknown[]],
+1 -1
View File
@@ -174,7 +174,7 @@ export function getBasicCallViewModelEnvironment(
setE2EEEnabled: async () => Promise.resolve(),
}),
connectionState$: constant(ConnectionState.Connected),
matrixRTCMode$: constant(MatrixRTCMode.Compatibility),
matrixRTCMode: MatrixRTCMode.Compatibility,
...callViewModelOptions,
},
handRaisedSubject$,
-2
View File
@@ -27,8 +27,6 @@ export default defineConfig((env) =>
data: {
matrix_rtc_session: {
wait_for_key_rotation_ms: 5000,
delayed_leave_event_restart_ms: 4000,
delayed_leave_event_delay_ms: 18000,
},
},
},