Compare commits

..

18 Commits

Author SHA1 Message Date
Timo
3572de499d fix setLocalStorageItemReactive 2025-06-05 19:26:17 +02:00
Timo
5332970dcf add onAudioTrackReady callback for controls 2025-06-05 19:00:08 +02:00
Timo
9ff6bcb583 Use html audio element to call setsink id for reactions and call sounds. 2025-06-05 18:59:37 +02:00
Robin
45fd7fdc7f Always render audio from the current set of participants
We forgot to tell React that we need the audio renderer to react to changes in the set of MatrixRTC participants; instead we had it referencing rtcSession.memberships non-reactively.

Now, I'm not 100% confident that this is going to fix the "speaking from the void" issues observed in the wild, because I can't reproduce them and, in my testing, the InCallView component always seemed to be rendered redundantly when the MatrixRTC participants change, even though we hadn't explicitly stated that it needs to react. (This makes sense as we haven't memoized the component.) But it's worth a shot.
2025-06-05 18:59:25 +02:00
Robin
047b3ffded Avoid reactivity bugs in how we track external state (#3316)
* Avoid reactivity bugs in how we track external state

Many of our hooks which attempt to bridge external state from an EventEmitter or EventTarget into React had subtle bugs which could cause them to fail to react to certain updates. The conditions necessary for triggering these bugs are explained by the tests that I've included.

In the majority of cases, I don't think we were triggering these bugs in practice. They could've become problems if we refactored our components in certain ways. The one concrete case I'm aware of in which we actually triggered such a bug was the race condition with the useRoomEncryptionSystem shared secret logic (addressed by a1110af6d5).

But, particularly with all the weird reactivity issues we're debugging this week, I think we need to eliminate the possibility that any of the bugs in these hooks are the cause of our current headaches.

* Reuse useTypedEventEmitterState in useLocalStorage

* Fix type error
2025-06-05 18:59:08 +02:00
Robin
a9ab93304a Ignore spurious 'devicechange' events
This gives us the additional insurance of breaking the Safari media acquisition loop at the source by admitting that they can be spurious in practice. Safari, why!?
2025-06-05 18:55:44 +02:00
Robin
8b75a266ea Break loop in acquiring media on Safari 2025-06-05 18:55:35 +02:00
Timo
87f02b28a5 Merge pull request #3293 from element-hq/toger5/more-disconnect-logging (#3297)
Fix creating two lk rooms if there is no local store setup (fixes a resulting disconnect bug)
2025-05-28 09:07:50 +02:00
Timo
24cb61c24b Merge pull request #3291 from element-hq:toger5/backport-disable-device-switching-and-update-state
Backport: Disable device switching when in controlled audio devices mode
2025-05-23 17:56:49 +02:00
Timo
8f424452fc Disable device switching when in controlled audio devices mode (#3290)
* Disable device switching when in controlled audio devices mode

* Temporarily switch matrix-js-sdk to robin/embedded-no-update-state

To allow us to test this change on Element X, which does not yet support the update_state action.

* Also add a check for controlled audio devices in useAudioContext

* use develop branch

* fix tests

---------

Co-authored-by: Robin <robin@robin.town>
2025-05-23 17:55:28 +02:00
Robin
c2ce1fd382 Merge pull request #3288 from element-hq/robin/backport-audio-controls
Audio device controls for mobile native audio device selection
2025-05-22 14:32:12 -04:00
Robin
52895ed599 Audio device controls for mobile native audio device selection
Backport of 0971a15c40.
2025-05-22 14:11:46 -04:00
Robin
0719320ceb Merge pull request #3286 from element-hq/robin/backport-reset-develop
Reset to develop branch of matrix-js-sdk
2025-05-22 13:57:06 -04:00
Robin
593a50289a Reset to develop branch of matrix-js-sdk
Now that the toger5/add-room-key-fallback-on-encryption-manager-not-supported branch has been merged, we can reset to develop. (And need to, actually, because that branch is deleted.)
2025-05-22 13:12:23 -04:00
Robin
60881d7b11 Merge pull request #3281 from element-hq/robin/backport-reintroduce-update-state
Improve the reliability of state changes in widget mode
2025-05-21 14:36:24 -04:00
Robin
5f65c51cf3 Reference matrix-js-sdk by branch name
Rather than by commit, which makes it hard to tell whether we're using mainline matrix-js-sdk or not.
2025-05-21 14:27:15 -04:00
Robin
4876bddea4 Update matrix-widget-api to support update_state action 2025-05-21 14:27:15 -04:00
Robin
a5a737f830 Update matrix-js-sdk to support update_state action 2025-05-21 14:27:15 -04:00
38 changed files with 789 additions and 281 deletions

View File

@@ -6,7 +6,7 @@ services:
image: ghcr.io/element-hq/lk-jwt-service:latest-ci
hostname: auth-server
environment:
- LIVEKIT_JWT_PORT=8080
- LK_JWT_PORT=8080
- LIVEKIT_URL=wss://matrix-rtc.m.localhost/livekit/sfu
- LIVEKIT_KEY=devkey
- LIVEKIT_SECRET=secret

View File

@@ -1,7 +1,23 @@
# Global JS controls
A few aspects of Element Call's interface can be controlled through a global API on the `window`:
A few aspects of Element Call's interface can be controlled through a global API on the `window`.
## Picture-in-picture
- `controls.canEnterPip(): boolean` Determines whether it's possible to enter picture-in-picture mode.
- `controls.enablePip(): void` Puts the call interface into picture-in-picture mode. Throws if not in a call.
- `controls.disablePip(): void` Takes the call interface out of picture-in-picture mode, restoring it to its natural display mode. Throws if not in a call.
## Audio devices
On mobile platforms (iOS, Android), web views do not reliably support selecting audio output devices such as the main speaker, earpiece, or headset. To address this limitation, the following functions allow the hosting application (e.g., Element Web, Element X) to manage audio devices via exposed JavaScript interfaces. These functions must be enabled using the URL parameter `controlledAudioDevices` to take effect.
- `controls.setAvailableAudioDevices(devices: { id: string, name: string, forEarpiece?: boolean, isEarpiece?: boolean isSpeaker?: boolean, isExternalHeadset?, boolean; }[]): void` Sets the list of available audio outputs. `forEarpiece` is used on iOS only.
It flags the device that should be used if the user selects earpiece mode. This should be the main stereo loudspeaker of the device.
- `controls.onAudioDeviceSelect: ((id: string) => void) | undefined` Callback called whenever the user or application selects a new audio output.
- `controls.setAudioDevice(id: string): void` Sets the selected audio device in Element Call's menu. This should be used if the OS decides to automatically switch to Bluetooth, for example.
- `controls.setAudioEnabled(enabled: boolean)` Enables/disables all audio output from the application. Output is enabled by default.
- `showNativeAudioDevicePicker: (() => void) | undefined`. Callback called whenever the user presses the output button in the settings menu.
This button is only shown on iOS. (`userAgent.includes("iPhone")`)
- `controls.onAudioPlaybackStarted: ((id: string) => void) | undefined`: This will be called the first time we start
playing audio in the webview. It can be helpful to do device setup on the native app when the webviews audio is ready.

View File

@@ -63,6 +63,7 @@ These parameters are relevant to both [widget](./embedded-standalone.md) and [st
| `lang` | [BCP 47](https://www.rfc-editor.org/info/bcp47) code | No | No | The language the app should use. |
| `password` | | No | No | E2EE password when using a shared secret. (For individual sender keys in embedded mode this is not required.) |
| `perParticipantE2EE` | `true` or `false` | No, defaults to `false` | No, defaults to `false` | Enables per participant encryption with Keys exchanged over encrypted matrix room messages. |
| `controlledAudioDevices` | `true` or `false` | No, defaults to `false` | No, defaults to `false` | Whether the [global JS controls for audio devices](./controls.md#audio-devices) should be enabled, allowing the list of audio devices to be controlled by the app hosting Element Call. |
| `roomId` | [Matrix Room ID](https://spec.matrix.org/v1.12/appendices/#room-ids) | Yes | No | Anything about what room we're pointed to should be from useRoomIdentifier which parses the path and resolves alias with respect to the default server name, however roomId is an exception as we need the room ID in embedded widget mode, and not the room alias (or even the via params because we are not trying to join it). This is also not validated, where it is in `useRoomIdentifier()`. |
| `showControls` | `true` or `false` | No, defaults to `true` | No, defaults to `true` | Displays controls like mute, screen-share, invite, and hangup buttons during a call. |
| `skipLobby` (deprecated: use `intent` instead) | `true` or `false` | No. If `intent` is explicitly `start_call` then defaults to `true`. Otherwise defaults to `false` | No, defaults to `false` | Skips the lobby to join a call directly, can be combined with preload in widget. When `true` the audio and video inputs will be muted by default. (This means there currently is no way to start without muted video if one wants to skip the lobby. Also not in widget mode.) |

View File

@@ -173,6 +173,7 @@
"devices": {
"camera": "Camera",
"camera_numbered": "Camera {{n}}",
"change_device_button": "Change audio device",
"default": "Default",
"default_named": "Default <2>({{name}})</2>",
"earpiece": "Earpiece",

View File

@@ -99,7 +99,7 @@
"i18next-parser": "^9.1.0",
"jsdom": "^26.0.0",
"knip": "^5.27.2",
"livekit-client": "^2.11.3",
"livekit-client": "^2.13.0",
"lodash-es": "^4.17.21",
"loglevel": "^1.9.1",
"matrix-js-sdk": "github:matrix-org/matrix-js-sdk#head=develop",

View File

@@ -124,9 +124,15 @@ export interface UrlParams {
*/
password: string | null;
/**
* Whether we the app should use per participant keys for E2EE.
* Whether the app should use per participant keys for E2EE.
*/
perParticipantE2EE: boolean;
/**
* Whether the global JS controls for audio output devices should be enabled,
* allowing the list of output devices to be controlled by the app hosting
* Element Call.
*/
controlledAudioDevices: boolean;
/**
* Setting this flag skips the lobby and brings you in the call directly.
* In the widget this can be combined with preload to pass the device settings
@@ -173,6 +179,7 @@ export interface UrlParams {
* The Sentry DSN. This is only used in the embedded package of Element Call.
*/
sentryDsn: string | null;
/**
* The Sentry environment. This is only used in the embedded package of Element Call.
*/
@@ -281,6 +288,11 @@ export const getUrlParams = (
fontScale: Number.isNaN(fontScale) ? null : fontScale,
allowIceFallback: parser.getFlagParam("allowIceFallback"),
perParticipantE2EE: parser.getFlagParam("perParticipantE2EE"),
controlledAudioDevices: parser.getFlagParam(
"controlledAudioDevices",
// the deprecated property name
parser.getFlagParam("controlledMediaDevices"),
),
skipLobby: parser.getFlagParam(
"skipLobby",
isWidget && intent === UserIntent.StartNewCall,

View File

@@ -5,16 +5,63 @@ SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { Subject } from "rxjs";
import { BehaviorSubject, Subject } from "rxjs";
export interface Controls {
canEnterPip: () => boolean;
enablePip: () => void;
disablePip: () => void;
canEnterPip(): boolean;
enablePip(): void;
disablePip(): void;
/** @deprecated use setAvailableAudioDevices instead*/
setAvailableOutputDevices(devices: OutputDevice[]): void;
setAvailableAudioDevices(devices: OutputDevice[]): void;
/** @deprecated use setAudioDevice instead*/
setOutputDevice(id: string): void;
setAudioDevice(id: string): void;
/** @deprecated use onAudioDeviceSelect instead*/
onOutputDeviceSelect?: (id: string) => void;
onAudioDeviceSelect?: (id: string) => void;
onAudioPlaybackStarted?: () => void;
/** @deprecated use setAudioEnabled instead*/
setOutputEnabled(enabled: boolean): void;
setAudioEnabled(enabled: boolean): void;
/** @deprecated use showNativeAudioDevicePicker instead*/
showNativeOutputDevicePicker?: () => void;
showNativeAudioDevicePicker?: () => void;
}
export const setPipEnabled$ = new Subject<boolean>();
export interface OutputDevice {
id: string;
name: string;
forEarpiece?: boolean;
isEarpiece?: boolean;
isSpeaker?: boolean;
isExternalHeadset?: boolean;
}
/**
* If pipMode is enabled, EC will render a adapted call view layout.
*/
export const setPipEnabled$ = new Subject<boolean>();
// BehaviorSubject since the client might set this before we have subscribed (GroupCallView still in "loading" state)
// We want the devices that have been set during loading to be available immediately once loaded.
export const availableOutputDevices$ = new BehaviorSubject<OutputDevice[]>([]);
// BehaviorSubject since the client might set this before we have subscribed (GroupCallView still in "loading" state)
// We want the device that has been set during loading to be available immediately once loaded.
export const outputDevice$ = new BehaviorSubject<string | undefined>(undefined);
/**
* This allows the os to mute the call if the user
* presses the volume down button when it is at the minimum volume.
*
* This should also be used to display a darkened overlay screen letting the user know that audio is muted.
*/
export const setAudioEnabled$ = new Subject<boolean>();
let playbackStartedEmitted = false;
export const setPlaybackStarted = (): void => {
if (!playbackStartedEmitted) {
playbackStartedEmitted = true;
window.controls.onAudioPlaybackStarted?.();
}
};
window.controls = {
canEnterPip(): boolean {
return setPipEnabled$.observed;
@@ -27,4 +74,28 @@ window.controls = {
if (!setPipEnabled$.observed) throw new Error("No call is running");
setPipEnabled$.next(false);
},
setAvailableAudioDevices(devices: OutputDevice[]): void {
availableOutputDevices$.next(devices);
},
setAudioDevice(id: string): void {
outputDevice$.next(id);
},
setAudioEnabled(enabled: boolean): void {
if (!setAudioEnabled$.observed)
throw new Error(
"Output controls are disabled. No setAudioEnabled$ observer",
);
setAudioEnabled$.next(enabled);
},
// wrappers for the deprecated controls fields
setOutputEnabled(enabled: boolean): void {
this.setAudioEnabled(enabled);
},
setAvailableOutputDevices(devices: OutputDevice[]): void {
this.setAvailableAudioDevices(devices);
},
setOutputDevice(id: string): void {
this.setAudioDevice(id);
},
};

View File

@@ -7,13 +7,19 @@ Please see LICENSE in the repository root for full details.
import { useEffect, useMemo } from "react";
import { setLocalStorageItem, useLocalStorage } from "../useLocalStorage";
import {
setLocalStorageItemReactive,
useLocalStorage,
} from "../useLocalStorage";
import { type UrlParams, getUrlParams, useUrlParams } from "../UrlParams";
import { E2eeType } from "./e2eeType";
import { useClient } from "../ClientContext";
export function saveKeyForRoom(roomId: string, password: string): void {
setLocalStorageItem(getRoomSharedKeyLocalStorageKey(roomId), password);
setLocalStorageItemReactive(
getRoomSharedKeyLocalStorageKey(roomId),
password,
);
}
const getRoomSharedKeyLocalStorageKey = (roomId: string): string =>
@@ -21,7 +27,7 @@ const getRoomSharedKeyLocalStorageKey = (roomId: string): string =>
const useInternalRoomSharedKey = (roomId: string): string | null => {
const key = getRoomSharedKeyLocalStorageKey(roomId);
const roomSharedKey = useLocalStorage(key)[0];
const [roomSharedKey] = useLocalStorage(key);
return roomSharedKey;
};

View File

@@ -16,15 +16,18 @@ import { type RemoteAudioTrack } from "livekit-client";
import { type ReactNode } from "react";
import { useTracks } from "@livekit/components-react";
import { testAudioContext } from "../useAudioContext.test";
import {
TestAudioConstructor,
testAudioContext,
TestAudioContextConstructor,
} from "../useAudioContext.test";
import * as MediaDevicesContext from "./MediaDevicesContext";
import { MatrixAudioRenderer } from "./MatrixAudioRenderer";
import { mockTrack } from "../utils/test";
export const TestAudioContextConstructor = vi.fn(() => testAudioContext);
beforeEach(() => {
vi.stubGlobal("AudioContext", TestAudioContextConstructor);
vi.stubGlobal("Audio", TestAudioConstructor);
});
afterEach(() => {

View File

@@ -18,7 +18,7 @@ import { logger } from "matrix-js-sdk/lib/logger";
import { useEarpieceAudioConfig } from "./MediaDevicesContext";
import { useReactiveState } from "../useReactiveState";
import * as controls from "../controls";
export interface MatrixAudioRendererProps {
/**
* The list of participants to render audio for.
@@ -204,6 +204,7 @@ function AudioTrackWithAudioNodes({
useContext ? [audioNodes.gain!, audioNodes.pan!] : [],
);
setTrackReady(true);
controls.setPlaybackStarted();
}, [audioContext, audioNodes, setTrackReady, trackReady, trackRef]);
return (

View File

@@ -1,5 +1,5 @@
/*
Copyright 2023, 2024 New Vector Ltd.
Copyright 2023-2025 New Vector Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
@@ -17,9 +17,10 @@ import {
type JSX,
} from "react";
import { createMediaDeviceObserver } from "@livekit/components-core";
import { map, startWith } from "rxjs";
import { useObservableEagerState } from "observable-hooks";
import { combineLatest, distinctUntilChanged, map, startWith } from "rxjs";
import { useObservable, useObservableEagerState } from "observable-hooks";
import { logger } from "matrix-js-sdk/lib/logger";
import { isEqual } from "lodash-es";
import {
useSetting,
@@ -29,7 +30,11 @@ import {
alwaysShowIphoneEarpiece as alwaysShowIphoneEarpieceSetting,
type Setting,
} from "../settings/settings";
import { outputDevice$, availableOutputDevices$ } from "../controls";
import { useUrlParams } from "../UrlParams";
// This hardcoded id is used in EX ios! It can only be changed in coordination with
// the ios swift team.
export const EARPIECE_CONFIG_ID = "earpiece-id";
export type DeviceLabel =
@@ -38,7 +43,7 @@ export type DeviceLabel =
| { type: "earpiece" }
| { type: "default"; name: string | null };
export interface MediaDevice {
export interface MediaDeviceHandle {
/**
* A map from available device IDs to labels.
*/
@@ -59,24 +64,69 @@ export interface MediaDevice {
select: (deviceId: string) => void;
}
export interface MediaDevices {
audioInput: MediaDevice;
audioOutput: MediaDevice;
videoInput: MediaDevice;
interface InputDevices {
audioInput: MediaDeviceHandle;
videoInput: MediaDeviceHandle;
startUsingDeviceNames: () => void;
stopUsingDeviceNames: () => void;
usingNames: boolean;
}
function useMediaDevice(
export interface MediaDevices extends Omit<InputDevices, "usingNames"> {
audioOutput: MediaDeviceHandle;
}
/**
* An observable that represents if we should display the devices menu for iOS.
* This implies the following
* - hide any input devices (they do not work anyhow on ios)
* - Show a button to show the native output picker instead.
* - Only show the earpiece toggle option if the earpiece is available:
* `availableOutputDevices$.includes((d)=>d.forEarpiece)`
*/
export const iosDeviceMenu$ = alwaysShowIphoneEarpieceSetting.value$.pipe(
map((v) => v || navigator.userAgent.includes("iPhone")),
);
function useSelectedId(
available: Map<string, DeviceLabel>,
preferredId: string | undefined,
): string | undefined {
return useMemo(() => {
if (available.size) {
// If the preferred device is available, use it. Or if every available
// device ID is falsy, the browser is probably just being paranoid about
// fingerprinting and we should still try using the preferred device.
// Worst case it is not available and the browser will gracefully fall
// back to some other device for us when requesting the media stream.
// Otherwise, select the first available device.
return (preferredId !== undefined && available.has(preferredId)) ||
(available.size === 1 && available.has(""))
? preferredId
: available.keys().next().value;
}
return undefined;
}, [available, preferredId]);
}
/**
* Hook to get access to a mediaDevice handle for a kind. This allows to list
* the available devices, read and set the selected device.
* @param kind Audio input, output or video output.
* @param setting The setting this handle's selection should be synced with.
* @param usingNames If the hook should query device names for the associated
* list.
* @returns A handle for the chosen kind.
*/
function useMediaDeviceHandle(
kind: MediaDeviceKind,
setting: Setting<string | undefined>,
usingNames: boolean,
): MediaDevice {
// Make sure we don't needlessly reset to a device observer without names,
// once permissions are already given
const [alwaysShowIphoneEarpice] = useSetting(alwaysShowIphoneEarpieceSetting);
): MediaDeviceHandle {
const hasRequestedPermissions = useRef(false);
const requestPermissions = usingNames || hasRequestedPermissions.current;
// Make sure we don't needlessly reset to a device observer without names,
// once permissions are already given
hasRequestedPermissions.current ||= usingNames;
// We use a bare device observer here rather than one of the fancy device
@@ -91,7 +141,13 @@ function useMediaDevice(
kind,
() => logger.error("Error creating MediaDeviceObserver"),
requestPermissions,
).pipe(startWith([])),
// This Observable emits new values whenever the browser fires a
// MediaDevices 'devicechange' event. One would think, innocently, that
// a 'devicechange' event means the devices have changed. But as of the
// time of writing, we are seeing mobile Safari firing spurious
// 'devicechange' events (where no change has actually occurred) when
// we call MediaDevices.getUserMedia. So, filter by deep equality.
).pipe(startWith([]), distinctUntilChanged<MediaDeviceInfo[]>(isEqual)),
[kind, requestPermissions],
);
const available = useObservableEagerState(
@@ -114,52 +170,28 @@ function useMediaDevice(
// recognizes.
// We also create this if we do not have any available devices, so that
// we can use the default or the earpiece.
const showEarpiece =
navigator.userAgent.match("iPhone") || alwaysShowIphoneEarpice;
if (
kind === "audiooutput" &&
!available.has("") &&
!available.has("default") &&
(available.size || showEarpiece)
available.size
)
available = new Map([
["", { type: "default", name: availableRaw[0]?.label || null }],
...available,
]);
if (kind === "audiooutput" && showEarpiece)
// On IPhones we have to create a virtual earpiece device, because
// the earpiece is not available as a device ID.
available = new Map([
...available,
[EARPIECE_CONFIG_ID, { type: "earpiece" }],
]);
// Note: creating virtual default input devices would be another problem
// entirely, because requesting a media stream from deviceId "" won't
// automatically track the default device.
return available;
}),
),
[alwaysShowIphoneEarpice, deviceObserver$, kind],
[deviceObserver$, kind],
),
);
const [preferredId, setPreferredId] = useSetting(setting);
const [asEarpice, setAsEarpiece] = useState(false);
const selectedId = useMemo(() => {
if (available.size) {
// If the preferred device is available, use it. Or if every available
// device ID is falsy, the browser is probably just being paranoid about
// fingerprinting and we should still try using the preferred device.
// Worst case it is not available and the browser will gracefully fall
// back to some other device for us when requesting the media stream.
// Otherwise, select the first available device.
return (preferredId !== undefined && available.has(preferredId)) ||
(available.size === 1 && available.has(""))
? preferredId
: available.keys().next().value;
}
return undefined;
}, [available, preferredId]);
const [preferredId, select] = useSetting(setting);
const selectedId = useSelectedId(available, preferredId);
const selectedGroupId = useObservableEagerState(
useMemo(
@@ -174,37 +206,26 @@ function useMediaDevice(
),
);
const select = useCallback(
(id: string) => {
if (id === EARPIECE_CONFIG_ID) {
setAsEarpiece(true);
} else {
setAsEarpiece(false);
setPreferredId(id);
}
},
[setPreferredId],
);
return useMemo(
() => ({
available,
selectedId,
useAsEarpiece: asEarpice,
useAsEarpiece: false,
selectedGroupId,
select,
}),
[available, selectedId, asEarpice, selectedGroupId, select],
[available, selectedId, selectedGroupId, select],
);
}
export const deviceStub: MediaDevice = {
export const deviceStub: MediaDeviceHandle = {
available: new Map(),
selectedId: undefined,
selectedGroupId: undefined,
select: () => {},
useAsEarpiece: false,
};
export const devicesStub: MediaDevices = {
audioInput: deviceStub,
audioOutput: deviceStub,
@@ -215,26 +236,17 @@ export const devicesStub: MediaDevices = {
export const MediaDevicesContext = createContext<MediaDevices>(devicesStub);
interface Props {
children: JSX.Element;
}
export const MediaDevicesProvider: FC<Props> = ({ children }) => {
function useInputDevices(): InputDevices {
// Counts the number of callers currently using device names.
const [numCallersUsingNames, setNumCallersUsingNames] = useState(0);
const usingNames = numCallersUsingNames > 0;
const audioInput = useMediaDevice(
const audioInput = useMediaDeviceHandle(
"audioinput",
audioInputSetting,
usingNames,
);
const audioOutput = useMediaDevice(
"audiooutput",
audioOutputSetting,
usingNames,
);
const videoInput = useMediaDevice(
const videoInput = useMediaDeviceHandle(
"videoinput",
videoInputSetting,
usingNames,
@@ -249,17 +261,52 @@ export const MediaDevicesProvider: FC<Props> = ({ children }) => {
[setNumCallersUsingNames],
);
return {
audioInput,
videoInput,
startUsingDeviceNames,
stopUsingDeviceNames,
usingNames,
};
}
interface Props {
children: JSX.Element;
}
export const MediaDevicesProvider: FC<Props> = ({ children }) => {
const {
audioInput,
videoInput,
startUsingDeviceNames,
stopUsingDeviceNames,
usingNames,
} = useInputDevices();
const { controlledAudioDevices } = useUrlParams();
const webViewAudioOutput = useMediaDeviceHandle(
"audiooutput",
audioOutputSetting,
usingNames,
);
const controlledAudioOutput = useControlledOutput();
const context: MediaDevices = useMemo(
() => ({
audioInput,
audioOutput,
audioOutput: controlledAudioDevices
? controlledAudioOutput
: webViewAudioOutput,
videoInput,
startUsingDeviceNames,
stopUsingDeviceNames,
}),
[
audioInput,
audioOutput,
controlledAudioDevices,
controlledAudioOutput,
webViewAudioOutput,
videoInput,
startUsingDeviceNames,
stopUsingDeviceNames,
@@ -273,6 +320,80 @@ export const MediaDevicesProvider: FC<Props> = ({ children }) => {
);
};
function useControlledOutput(): MediaDeviceHandle {
const { available } = useObservableEagerState(
useObservable(() => {
const outputDeviceData$ = availableOutputDevices$.pipe(
map((devices) => {
const deviceForEarpiece = devices.find((d) => d.forEarpiece);
const deviceMapTuple: [string, DeviceLabel][] = devices.map(
({ id, name, isEarpiece, isSpeaker /*,isExternalHeadset*/ }) => {
let deviceLabel: DeviceLabel = { type: "name", name };
// if (isExternalHeadset) // Do we want this?
if (isEarpiece) deviceLabel = { type: "earpiece" };
if (isSpeaker) deviceLabel = { type: "default", name };
return [id, deviceLabel];
},
);
return {
devicesMap: new Map<string, DeviceLabel>(deviceMapTuple),
deviceForEarpiece,
};
}),
);
return combineLatest(
[outputDeviceData$, iosDeviceMenu$],
({ devicesMap, deviceForEarpiece }, iosShowEarpiece) => {
let available = devicesMap;
if (iosShowEarpiece && !!deviceForEarpiece) {
available = new Map([
...devicesMap.entries(),
[EARPIECE_CONFIG_ID, { type: "earpiece" }],
]);
}
return { available, deviceForEarpiece };
},
);
}),
);
const [preferredId, setPreferredId] = useSetting(audioOutputSetting);
useEffect(() => {
const subscription = outputDevice$.subscribe((id) => {
if (id) setPreferredId(id);
});
return (): void => subscription.unsubscribe();
}, [setPreferredId]);
const selectedId = useSelectedId(available, preferredId);
const [asEarpiece, setAsEarpiece] = useState(false);
useEffect(() => {
// Let the hosting application know which output device has been selected.
// This information is probably only of interest if the earpiece mode has been
// selected - for example, Element X iOS listens to this to determine whether it
// should enable the proximity sensor.
if (selectedId) {
window.controls.onAudioDeviceSelect?.(selectedId);
// Call deprecated method for backwards compatibility.
window.controls.onOutputDeviceSelect?.(selectedId);
}
setAsEarpiece(selectedId === EARPIECE_CONFIG_ID);
}, [selectedId]);
return useMemo(
() => ({
available: available,
selectedId,
selectedGroupId: undefined,
select: setPreferredId,
useAsEarpiece: asEarpiece,
}),
[available, selectedId, setPreferredId, asEarpiece],
);
}
export const useMediaDevices = (): MediaDevices =>
useContext(MediaDevicesContext);

View File

@@ -25,7 +25,7 @@ import { defaultLiveKitOptions } from "./options";
import { type SFUConfig } from "./openIDSFU";
import { type MuteStates } from "../room/MuteStates";
import {
type MediaDevice,
type MediaDeviceHandle,
type MediaDevices,
useMediaDevices,
} from "./MediaDevicesContext";
@@ -42,6 +42,7 @@ import {
} from "./TrackProcessorContext";
import { useInitial } from "../useInitial";
import { observeTrackReference$ } from "../state/MediaViewModel";
import { useUrlParams } from "../UrlParams";
interface UseLivekitResult {
livekitRoom?: Room;
@@ -54,6 +55,8 @@ export function useLivekit(
sfuConfig: SFUConfig | undefined,
e2eeSystem: EncryptionSystem,
): UseLivekitResult {
const { controlledAudioDevices } = useUrlParams();
const e2eeOptions = useMemo((): E2EEManagerOptions | undefined => {
if (e2eeSystem.kind === E2eeType.NONE) return undefined;
@@ -128,6 +131,7 @@ export function useLivekit(
// @livekit/components-react. JSON.stringify() is used in deps of a
// useEffect() with an argument that references itself, if E2EE is enabled
const room = useMemo(() => {
logger.info("[LivekitRooms] Create LiveKit room with options", roomOptions);
const r = new Room(roomOptions);
r.setE2EEEnabled(e2eeSystem.kind !== E2eeType.NONE).catch((e) => {
logger.error("Failed to set E2EE enabled on room", e);
@@ -303,8 +307,15 @@ export function useLivekit(
useEffect(() => {
// Sync the requested devices with LiveKit's devices
if (room !== undefined && connectionState === ConnectionState.Connected) {
const syncDevice = (kind: MediaDeviceKind, device: MediaDevice): void => {
if (
room !== undefined &&
connectionState === ConnectionState.Connected &&
!controlledAudioDevices
) {
const syncDevice = (
kind: MediaDeviceKind,
device: MediaDeviceHandle,
): void => {
const id = device.selectedId;
// Detect if we're trying to use chrome's default device, in which case
@@ -360,7 +371,7 @@ export function useLivekit(
syncDevice("audiooutput", devices.audioOutput);
syncDevice("videoinput", devices.videoInput);
}
}, [room, devices, connectionState]);
}, [room, devices, connectionState, controlledAudioDevices]);
return {
connState: connectionState,

View File

@@ -29,7 +29,7 @@ window.setLKLogLevel = setLKLogLevel;
initRageshake().catch((e) => {
logger.error("Failed to initialize rageshake", e);
});
setLKLogLevel("warn");
setLKLogLevel("info");
setLKLogExtension((level, msg, context) => {
// we pass a synthetic logger name of "livekit" to the rageshake to make it easier to read
global.mx_rage_logger.log(level, "livekit", msg, context);

View File

@@ -29,6 +29,7 @@ import { useAudioContext } from "../useAudioContext";
import { ActiveCall } from "./InCallView";
import {
flushPromises,
mockEmitter,
mockMatrixRoom,
mockMatrixRoomMember,
mockRtcMembership,
@@ -129,6 +130,7 @@ function createGroupCallView(
getMxcAvatarUrl: () => null,
getCanonicalAlias: () => null,
currentState: {
...mockEmitter(),
getJoinRule: () => JoinRule.Invite,
} as Partial<RoomState> as RoomState,
});

View File

@@ -24,6 +24,7 @@ import {
type MatrixRTCSession,
} from "matrix-js-sdk/lib/matrixrtc";
import { useNavigate } from "react-router-dom";
import { useObservableEagerState } from "observable-hooks";
import type { IWidgetApiRequest } from "matrix-widget-api";
import {
@@ -64,10 +65,10 @@ import { GroupCallErrorBoundary } from "./GroupCallErrorBoundary.tsx";
import {
useNewMembershipManager as useNewMembershipManagerSetting,
useExperimentalToDeviceTransport as useExperimentalToDeviceTransportSetting,
muteAllAudio as muteAllAudioSetting,
useSetting,
} from "../settings/settings";
import { useTypedEventEmitter } from "../useEvents";
import { muteAllAudio$ } from "../state/MuteAllAudioModel.ts";
declare global {
interface Window {
@@ -104,9 +105,9 @@ export const GroupCallView: FC<Props> = ({
const [externalError, setExternalError] = useState<ElementCallError | null>(
null,
);
const [muteAllAudio] = useSetting(muteAllAudioSetting);
const memberships = useMatrixRTCSessionMemberships(rtcSession);
const muteAllAudio = useObservableEagerState(muteAllAudio$);
const leaveSoundContext = useLatest(
useAudioContext({
sounds: callEventAudioSounds,

View File

@@ -96,7 +96,6 @@ import { CallEventAudioRenderer } from "./CallEventAudioRenderer";
import {
debugTileLayout as debugTileLayoutSetting,
useExperimentalToDeviceTransport as useExperimentalToDeviceTransportSetting,
muteAllAudio as muteAllAudioSetting,
developerMode as developerModeSetting,
useSetting,
} from "../settings/settings";
@@ -104,6 +103,8 @@ import { ReactionsReader } from "../reactions/ReactionsReader";
import { ConnectionLostError } from "../utils/errors.ts";
import { useTypedEventEmitter } from "../useEvents.ts";
import { MatrixAudioRenderer } from "../livekit/MatrixAudioRenderer.tsx";
import { muteAllAudio$ } from "../state/MuteAllAudioModel.ts";
import { useMatrixRTCSessionMemberships } from "../useMatrixRTCSessionMemberships.ts";
const canScreenshare = "getDisplayMedia" in (navigator.mediaDevices ?? {});
@@ -235,7 +236,7 @@ export const InCallView: FC<InCallViewProps> = ({
room: livekitRoom,
});
const [muteAllAudio] = useSetting(muteAllAudioSetting);
const muteAllAudio = useObservableEagerState(muteAllAudio$);
// This seems like it might be enough logic to use move it into the call view model?
const [didFallbackToRoomKey, setDidFallbackToRoomKey] = useState(false);
@@ -250,6 +251,7 @@ export const InCallView: FC<InCallViewProps> = ({
useExperimentalToDeviceTransportSetting,
);
const encryptionSystem = useRoomEncryptionSystem(rtcSession.room.roomId);
const memberships = useMatrixRTCSessionMemberships(rtcSession);
const showToDeviceEncryption = useMemo(
() =>
@@ -723,10 +725,7 @@ export const InCallView: FC<InCallViewProps> = ({
</Text>
)
}
<MatrixAudioRenderer
members={rtcSession.memberships}
muted={muteAllAudio}
/>
<MatrixAudioRenderer members={memberships} muted={muteAllAudio} />
{renderContent()}
<CallEventAudioRenderer vm={vm} muted={muteAllAudio} />
<ReactionsAudioRenderer vm={vm} muted={muteAllAudio} />

View File

@@ -53,6 +53,7 @@ import {
useTrackProcessorSync,
} from "../livekit/TrackProcessorContext";
import { usePageTitle } from "../usePageTitle";
import { useLatest } from "../useLatest";
interface Props {
client: MatrixClient;
@@ -159,13 +160,14 @@ export const LobbyView: FC<Props> = ({
],
);
const latestMuteStates = useLatest(muteStates);
const onError = useCallback(
(error: Error) => {
logger.error("Error while creating preview Tracks:", error);
muteStates.audio.setEnabled?.(false);
muteStates.video.setEnabled?.(false);
latestMuteStates.current.audio.setEnabled?.(false);
latestMuteStates.current.video.setEnabled?.(false);
},
[muteStates.audio, muteStates.video],
[latestMuteStates],
);
const tracks = usePreviewTracks(localTrackOptions, onError);

View File

@@ -14,7 +14,7 @@ import userEvent from "@testing-library/user-event";
import { useMuteStates } from "./MuteStates";
import {
type DeviceLabel,
type MediaDevice,
type MediaDeviceHandle,
type MediaDevices,
MediaDevicesContext,
} from "../livekit/MediaDevicesContext";
@@ -73,7 +73,7 @@ const mockCamera: MediaDeviceInfo = {
},
};
function mockDevices(available: Map<string, DeviceLabel>): MediaDevice {
function mockDevices(available: Map<string, DeviceLabel>): MediaDeviceHandle {
return {
available,
selectedId: "",

View File

@@ -16,7 +16,7 @@ import { type IWidgetApiRequest } from "matrix-widget-api";
import { logger } from "matrix-js-sdk/lib/logger";
import {
type MediaDevice,
type MediaDeviceHandle,
useMediaDevices,
} from "../livekit/MediaDevicesContext";
import { useReactiveState } from "../useReactiveState";
@@ -53,14 +53,14 @@ export interface MuteStates {
}
function useMuteState(
device: MediaDevice,
device: MediaDeviceHandle,
enabledByDefault: () => boolean,
): MuteState {
const [enabled, setEnabled] = useReactiveState<boolean | undefined>(
// Determine the default value once devices are actually connected
(prev) =>
prev ?? (device.available.size > 0 ? enabledByDefault() : undefined),
[device],
[device.available.size],
);
return useMemo(
() =>
@@ -70,7 +70,7 @@ function useMuteState(
enabled: enabled ?? false,
setEnabled: setEnabled as Dispatch<SetStateAction<boolean>>,
},
[device, enabled, setEnabled],
[device.available.size, enabled, setEnabled],
);
}

View File

@@ -9,11 +9,13 @@ import {
type MatrixRTCSession,
MatrixRTCSessionEvent,
} from "matrix-js-sdk/lib/matrixrtc";
import { useCallback, useEffect, useState } from "react";
import { useCallback, useEffect, useRef } from "react";
import { deepCompare } from "matrix-js-sdk/lib/utils";
import { logger } from "matrix-js-sdk/lib/logger";
import { type LivekitFocus, isLivekitFocus } from "matrix-js-sdk/lib/matrixrtc";
import { useTypedEventEmitterState } from "../useEvents";
/**
* Gets the currently active (livekit) focus for a MatrixRTC session
* This logic is specific to livekit foci where the whole call must use one
@@ -22,38 +24,27 @@ import { type LivekitFocus, isLivekitFocus } from "matrix-js-sdk/lib/matrixrtc";
export function useActiveLivekitFocus(
rtcSession: MatrixRTCSession,
): LivekitFocus | undefined {
const [activeFocus, setActiveFocus] = useState(() => {
const f = rtcSession.getActiveFocus();
// Only handle foci with type="livekit" for now.
return !!f && isLivekitFocus(f) ? f : undefined;
});
const activeFocus = useTypedEventEmitterState(
rtcSession,
MatrixRTCSessionEvent.MembershipsChanged,
useCallback(() => {
const f = rtcSession.getActiveFocus();
// Only handle foci with type="livekit" for now.
return !!f && isLivekitFocus(f) ? f : undefined;
}, [rtcSession]),
);
const onMembershipsChanged = useCallback(() => {
const newActiveFocus = rtcSession.getActiveFocus();
if (!!newActiveFocus && !isLivekitFocus(newActiveFocus)) return;
if (!deepCompare(activeFocus, newActiveFocus)) {
const prevActiveFocus = useRef(activeFocus);
useEffect(() => {
if (!deepCompare(activeFocus, prevActiveFocus.current)) {
const oldestMembership = rtcSession.getOldestMembership();
logger.warn(
`Got new active focus from membership: ${oldestMembership?.sender}/${oldestMembership?.deviceId}.
Updating focus (focus switch) from ${JSON.stringify(activeFocus)} to ${JSON.stringify(newActiveFocus)}`,
Updated focus (focus switch) from ${JSON.stringify(prevActiveFocus.current)} to ${JSON.stringify(activeFocus)}`,
);
setActiveFocus(newActiveFocus);
prevActiveFocus.current = activeFocus;
}
}, [activeFocus, rtcSession]);
useEffect(() => {
rtcSession.on(
MatrixRTCSessionEvent.MembershipsChanged,
onMembershipsChanged,
);
return (): void => {
rtcSession.off(
MatrixRTCSessionEvent.MembershipsChanged,
onMembershipsChanged,
);
};
});
return activeFocus;
}

View File

@@ -1,18 +1,19 @@
/*
Copyright 2023, 2024 New Vector Ltd.
Copyright 2025 New Vector Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { type Room, RoomEvent } from "matrix-js-sdk";
import { useState } from "react";
import { useCallback } from "react";
import { useTypedEventEmitter } from "../useEvents";
import { useTypedEventEmitterState } from "../useEvents";
export function useRoomName(room: Room): string {
const [, setNumUpdates] = useState(0);
// Whenever the name changes, force an update
useTypedEventEmitter(room, RoomEvent.Name, () => setNumUpdates((n) => n + 1));
return room.name;
return useTypedEventEmitterState(
room,
RoomEvent.Name,
useCallback(() => room.name, [room]),
);
}

View File

@@ -5,10 +5,15 @@ SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { useCallback, useMemo, useState } from "react";
import { type RoomState, RoomStateEvent, type Room } from "matrix-js-sdk";
import { useCallback } from "react";
import {
type RoomState,
RoomStateEvent,
type Room,
RoomEvent,
} from "matrix-js-sdk";
import { useTypedEventEmitter } from "../useEvents";
import { useTypedEventEmitterState } from "../useEvents";
/**
* A React hook for values computed from room state.
@@ -16,14 +21,17 @@ import { useTypedEventEmitter } from "../useEvents";
* @param f A mapping from the current room state to the computed value.
* @returns The computed value.
*/
export const useRoomState = <T>(room: Room, f: (state: RoomState) => T): T => {
const [numUpdates, setNumUpdates] = useState(0);
useTypedEventEmitter(
export function useRoomState<T>(room: Room, f: (state: RoomState) => T): T {
// TODO: matrix-js-sdk says that Room.currentState is deprecated, but it's not
// clear how to reactively track the current state of the room without it
const currentState = useTypedEventEmitterState(
room,
RoomStateEvent.Update,
useCallback(() => setNumUpdates((n) => n + 1), [setNumUpdates]),
RoomEvent.CurrentStateUpdated,
useCallback(() => room.currentState, [room]),
);
// We want any change to the update counter to trigger an update here
// eslint-disable-next-line react-hooks/exhaustive-deps
return useMemo(() => f(room.currentState), [room, f, numUpdates]);
};
return useTypedEventEmitterState(
currentState,
RoomStateEvent.Update,
useCallback(() => f(currentState), [f, currentState]),
);
}

View File

@@ -24,12 +24,12 @@ import { Trans, useTranslation } from "react-i18next";
import {
EARPIECE_CONFIG_ID,
type MediaDevice,
type MediaDeviceHandle,
} from "../livekit/MediaDevicesContext";
import styles from "./DeviceSelection.module.css";
interface Props {
device: MediaDevice;
device: MediaDeviceHandle;
title: string;
numberedLabel: (number: number) => string;
}

View File

@@ -8,8 +8,9 @@ Please see LICENSE in the repository root for full details.
import { type FC, type ReactNode, useState } from "react";
import { useTranslation } from "react-i18next";
import { type MatrixClient } from "matrix-js-sdk";
import { Root as Form, Separator } from "@vector-im/compound-web";
import { Button, Root as Form, Separator } from "@vector-im/compound-web";
import { type Room as LivekitRoom } from "livekit-client";
import { useObservableEagerState } from "observable-hooks";
import { Modal } from "../Modal";
import styles from "./SettingsModal.module.css";
@@ -19,6 +20,7 @@ import { FeedbackSettingsTab } from "./FeedbackSettingsTab";
import {
useMediaDevices,
useMediaDeviceNames,
iosDeviceMenu$,
} from "../livekit/MediaDevicesContext";
import { widget } from "../widget";
import {
@@ -34,6 +36,7 @@ import { useTrackProcessor } from "../livekit/TrackProcessorContext";
import { DeveloperSettingsTab } from "./DeveloperSettingsTab";
import { FieldRow, InputField } from "../input/Input";
import { useSubmitRageshake } from "./submit-rageshake";
import { useUrlParams } from "../UrlParams";
type SettingsTab =
| "audio"
@@ -102,19 +105,42 @@ export const SettingsModal: FC<Props> = ({
const { available: isRageshakeAvailable } = useSubmitRageshake();
// For controlled devices, we will not show the input section:
// Controlled media devices are used on mobile platforms, where input and output are grouped into
// a single device. These are called "headset" or "speaker" (or similar) but contain both input and output.
// On EC, we decided that it is less confusing for the user if they see those options in the output section
// rather than the input section.
const { controlledAudioDevices } = useUrlParams();
// If we are on iOS we will show a button to open the native audio device picker.
const iosDeviceMenu = useObservableEagerState(iosDeviceMenu$);
const audioTab: Tab<SettingsTab> = {
key: "audio",
name: t("common.audio"),
content: (
<>
<Form>
<DeviceSelection
device={devices.audioInput}
title={t("settings.devices.microphone")}
numberedLabel={(n) =>
t("settings.devices.microphone_numbered", { n })
}
/>
{!controlledAudioDevices && (
<DeviceSelection
device={devices.audioInput}
title={t("settings.devices.microphone")}
numberedLabel={(n) =>
t("settings.devices.microphone_numbered", { n })
}
/>
)}
{iosDeviceMenu && (
<Button
onClick={(e): void => {
e.preventDefault();
window.controls.showNativeAudioDevicePicker?.();
// call deprecated method for backwards compatibility.
window.controls.showNativeOutputDevicePicker?.();
}}
>
{t("settings.devices.change_device_button")}
</Button>
)}
<DeviceSelection
device={devices.audioOutput}
title={t("settings.devices.speaker")}

View File

@@ -133,6 +133,6 @@ export const muteAllAudio = new Setting<boolean>("mute-all-audio", false);
export const alwaysShowSelf = new Setting<boolean>("always-show-self", true);
export const alwaysShowIphoneEarpiece = new Setting<boolean>(
"always-show-iphone-earpice",
"always-show-iphone-earpiece",
false,
);

View File

@@ -0,0 +1,36 @@
/*
Copyright 2025 New Vector Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { test, vi } from "vitest";
import { expect } from "vitest";
import { setAudioEnabled$ } from "../controls";
import { muteAllAudio as muteAllAudioSetting } from "../settings/settings";
import { muteAllAudio$ } from "./MuteAllAudioModel";
test("muteAllAudio$", () => {
const valueMock = vi.fn();
const muteAllAudio = muteAllAudio$.subscribe((value) => {
valueMock(value);
});
setAudioEnabled$.next(false);
setAudioEnabled$.next(true);
muteAllAudioSetting.setValue(false);
muteAllAudioSetting.setValue(true);
setAudioEnabled$.next(false);
muteAllAudio.unsubscribe();
expect(valueMock).toHaveBeenCalledTimes(6);
expect(valueMock).toHaveBeenNthCalledWith(1, false); // startWith([false, muteAllAudioSetting.getValue()]);
expect(valueMock).toHaveBeenNthCalledWith(2, true); // setAudioEnabled$.next(false);
expect(valueMock).toHaveBeenNthCalledWith(3, false); // setAudioEnabled$.next(true);
expect(valueMock).toHaveBeenNthCalledWith(4, false); // muteAllAudioSetting.setValue(false);
expect(valueMock).toHaveBeenNthCalledWith(5, true); // muteAllAudioSetting.setValue(true);
expect(valueMock).toHaveBeenNthCalledWith(6, true); // setAudioEnabled$.next(false);
});

View File

@@ -0,0 +1,19 @@
/*
Copyright 2025 New Vector Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { combineLatest, startWith } from "rxjs";
import { setAudioEnabled$ } from "../controls";
import { muteAllAudio as muteAllAudioSetting } from "../settings/settings";
/**
* This can transition into sth more complete: `GroupCallViewModel.ts`
*/
export const muteAllAudio$ = combineLatest(
[setAudioEnabled$.pipe(startWith(true)), muteAllAudioSetting.value$],
(outputEnabled, settingsMute) => !outputEnabled || settingsMute,
);

View File

@@ -9,6 +9,7 @@ import { expect, vi, afterEach, beforeEach, test } from "vitest";
import { type FC } from "react";
import { render } from "@testing-library/react";
import userEvent, { type UserEvent } from "@testing-library/user-event";
import { BrowserRouter } from "react-router-dom";
import { deviceStub, MediaDevicesContext } from "./livekit/MediaDevicesContext";
import { useAudioContext } from "./useAudioContext";
@@ -38,6 +39,13 @@ const TestComponent: FC = () => {
</>
);
};
const TestComponentWrapper: FC = () => {
return (
<BrowserRouter>
<TestComponent />
</BrowserRouter>
);
};
const gainNode = vi.mocked(
{
@@ -79,12 +87,19 @@ export const testAudioContext = {
createGain: vi.fn().mockReturnValue(gainNode),
createStereoPanner: vi.fn().mockReturnValue(panNode),
close: vi.fn().mockResolvedValue(undefined),
createMediaStreamDestination: vi.fn().mockReturnValue({ stream: undefined }),
};
export const TestAudioContextConstructor = vi.fn(() => testAudioContext);
export const testAudioElement = {
setSinkId: vi.fn().mockResolvedValue(null),
};
export const TestAudioConstructor = vi.fn(() => testAudioElement);
let user: UserEvent;
beforeEach(() => {
vi.stubGlobal("AudioContext", TestAudioContextConstructor);
vi.stubGlobal("Audio", TestAudioConstructor);
user = userEvent.setup();
});
@@ -94,18 +109,19 @@ afterEach(() => {
});
test("can play a single sound", async () => {
const { findByText } = render(<TestComponent />);
const { findByText } = render(<TestComponentWrapper />);
await user.click(await findByText("Valid sound"));
expect(testAudioContext.createBufferSource).toHaveBeenCalledOnce();
});
test("will ignore sounds that are not registered", async () => {
const { findByText } = render(<TestComponent />);
const { findByText } = render(<TestComponentWrapper />);
await user.click(await findByText("Invalid sound"));
expect(testAudioContext.createBufferSource).not.toHaveBeenCalled();
});
test("will use the correct device", () => {
testAudioElement.setSinkId.mockClear();
render(
<MediaDevicesContext.Provider
value={{
@@ -122,16 +138,16 @@ test("will use the correct device", () => {
stopUsingDeviceNames: () => {},
}}
>
<TestComponent />
<TestComponentWrapper />
</MediaDevicesContext.Provider>,
);
expect(testAudioContext.createBufferSource).not.toHaveBeenCalled();
expect(testAudioContext.setSinkId).toHaveBeenCalledWith("chosen-device");
expect(testAudioElement.setSinkId).toHaveBeenCalledWith("chosen-device");
});
test("will use the correct volume level", async () => {
soundEffectVolumeSetting.setValue(0.33);
const { findByText } = render(<TestComponent />);
const { findByText } = render(<TestComponentWrapper />);
await user.click(await findByText("Valid sound"));
expect(testAudioContext.gain.gain.setValueAtTime).toHaveBeenCalledWith(
0.33,
@@ -140,7 +156,7 @@ test("will use the correct volume level", async () => {
expect(testAudioContext.pan.pan.setValueAtTime).toHaveBeenCalledWith(0, 0);
});
test("will use the pan if earpice is selected", async () => {
test("will use the pan if earpiece is selected", async () => {
const { findByText } = render(
<MediaDevicesContext.Provider
value={{
@@ -157,7 +173,7 @@ test("will use the pan if earpice is selected", async () => {
stopUsingDeviceNames: () => {},
}}
>
<TestComponent />
<TestComponentWrapper />
</MediaDevicesContext.Provider>,
);
await user.click(await findByText("Valid sound"));

View File

@@ -17,6 +17,9 @@ import {
useMediaDevices,
} from "./livekit/MediaDevicesContext";
import { type PrefetchedSounds } from "./soundUtils";
import { useUrlParams } from "./UrlParams";
import { useInitial } from "./useInitial";
import * as controls from "./controls";
/**
* Play a sound though a given AudioContext. Will take
@@ -41,6 +44,7 @@ async function playSound(
src.buffer = buffer;
src.connect(gain).connect(pan).connect(ctx.destination);
const p = new Promise<void>((r) => src.addEventListener("ended", () => r()));
controls.setPlaybackStarted();
src.start();
return p;
}
@@ -71,19 +75,22 @@ export function useAudioContext<S extends string>(
): UseAudioContext<S> | null {
const [soundEffectVolume] = useSetting(soundEffectVolumeSetting);
const { audioOutput } = useMediaDevices();
const { controlledAudioDevices } = useUrlParams();
const [audioContext, setAudioContext] = useState<AudioContext>();
const [audioBuffers, setAudioBuffers] = useState<Record<S, AudioBuffer>>();
const htmlAudioElement = useInitial((): HTMLAudioElement => new Audio());
useEffect(() => {
const sounds = props.sounds;
if (!sounds) {
if (!sounds || !htmlAudioElement) {
return;
}
const ctx = new AudioContext({
// We want low latency for these effects.
latencyHint: props.latencyHint,
});
htmlAudioElement.srcObject = ctx.createMediaStreamDestination().stream;
// We want to clone the content of our preloaded
// sound buffers into this context. The context may
@@ -106,18 +113,16 @@ export function useAudioContext<S extends string>(
});
setAudioContext(undefined);
};
}, [props.sounds, props.latencyHint]);
}, [props.sounds, props.latencyHint, htmlAudioElement]);
// Update the sink ID whenever we change devices.
useEffect(() => {
if (audioContext && "setSinkId" in audioContext) {
// https://developer.mozilla.org/en-US/docs/Web/API/AudioContext/setSinkId
// @ts-expect-error - setSinkId doesn't exist yet in types, maybe because it's not supported everywhere.
audioContext.setSinkId(audioOutput.selectedId).catch((ex) => {
if (!controlledAudioDevices && audioOutput.selectedId) {
htmlAudioElement.setSinkId(audioOutput.selectedId).catch((ex) => {
logger.warn("Unable to change sink for audio context", ex);
});
}
}, [audioContext, audioOutput.selectedId]);
}, [audioOutput.selectedId, controlledAudioDevices, htmlAudioElement]);
const { pan: earpiecePan, volume: earpieceVolume } = useEarpieceAudioConfig();
// Don't return a function until we're ready.

88
src/useEvents.test.tsx Normal file
View File

@@ -0,0 +1,88 @@
/*
Copyright 2025 New Vector Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { test } from "vitest";
import { render, screen } from "@testing-library/react";
import { type FC, useEffect, useState } from "react";
import userEvent from "@testing-library/user-event";
import { TypedEventEmitter } from "matrix-js-sdk";
import { useTypedEventEmitterState } from "./useEvents";
class TestEmitter extends TypedEventEmitter<"change", { change: () => void }> {
private state = 1;
public readonly getState = (): number => this.state;
public readonly getNegativeState = (): number => -this.state;
public readonly setState = (value: number): void => {
this.state = value;
this.emit("change");
};
}
test("useTypedEventEmitterState reacts to events", async () => {
const user = userEvent.setup();
const emitter = new TestEmitter();
const Test: FC = () => {
const value = useTypedEventEmitterState(
emitter,
"change",
emitter.getState,
);
return (
<>
<button onClick={() => emitter.setState(2)}>Change value</button>
<div>Value is {value}</div>
</>
);
};
render(<Test />);
screen.getByText("Value is 1");
await user.click(screen.getByText("Change value"));
screen.getByText("Value is 2");
});
test("useTypedEventEmitterState reacts to changes made by an effect mounted on the same render", () => {
const emitter = new TestEmitter();
const Test: FC = () => {
useEffect(() => emitter.setState(2), []);
const value = useTypedEventEmitterState(
emitter,
"change",
emitter.getState,
);
return `Value is ${value}`;
};
render(<Test />);
screen.getByText("Value is 2");
});
test("useTypedEventEmitterState reacts to changes in getState", async () => {
const user = userEvent.setup();
const emitter = new TestEmitter();
const Test: FC = () => {
const [fn, setFn] = useState(() => emitter.getState);
const value = useTypedEventEmitterState(emitter, "change", fn);
return (
<>
<button onClick={() => setFn(() => emitter.getNegativeState)}>
Change getState
</button>
<div>Value is {value}</div>
</>
);
};
render(<Test />);
screen.getByText("Value is 1");
await user.click(screen.getByText("Change getState"));
screen.getByText("Value is -1");
});

View File

@@ -5,7 +5,7 @@ SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { useEffect } from "react";
import { useCallback, useEffect, useSyncExternalStore } from "react";
import type {
Listener,
@@ -13,7 +13,9 @@ import type {
TypedEventEmitter,
} from "matrix-js-sdk/lib/models/typed-event-emitter";
// Shortcut for registering a listener on an EventTarget
/**
* Shortcut for registering a listener on an EventTarget.
*/
export function useEventTarget<T extends Event>(
target: EventTarget | null | undefined,
eventType: string,
@@ -33,7 +35,9 @@ export function useEventTarget<T extends Event>(
}, [target, eventType, listener, options]);
}
// Shortcut for registering a listener on a TypedEventEmitter
/**
* Shortcut for registering a listener on a TypedEventEmitter.
*/
export function useTypedEventEmitter<
Events extends string,
Arguments extends ListenerMap<Events>,
@@ -50,3 +54,33 @@ export function useTypedEventEmitter<
};
}, [emitter, eventType, listener]);
}
/**
* Reactively tracks a value which is recalculated whenever the provided event
* emitter emits an event. This is useful for bridging state from matrix-js-sdk
* into React.
*/
export function useTypedEventEmitterState<
Events extends string,
Arguments extends ListenerMap<Events>,
T extends Events,
State,
>(
emitter: TypedEventEmitter<Events, Arguments>,
eventType: T,
getState: () => State,
): State {
const subscribe = useCallback(
(onChange: () => void) => {
emitter.on(eventType, onChange as Listener<Events, Arguments, T>);
return (): void => {
emitter.off(eventType, onChange as Listener<Events, Arguments, T>);
};
},
[emitter, eventType],
);
// See the React docs for useSyncExternalStore; given that we're trying to
// bridge state from an external source into React, using this hook is exactly
// what React recommends.
return useSyncExternalStore(subscribe, getState);
}

View File

@@ -0,0 +1,51 @@
/*
Copyright 2025 New Vector Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { test } from "vitest";
import { render, screen } from "@testing-library/react";
import { type FC, useEffect, useState } from "react";
import userEvent from "@testing-library/user-event";
import {
setLocalStorageItemReactive,
useLocalStorage,
} from "./useLocalStorage";
test("useLocalStorage reacts to changes made by an effect mounted on the same render", () => {
localStorage.clear();
const Test: FC = () => {
useEffect(() => setLocalStorageItemReactive("my-value", "Hello!"), []);
const [myValue] = useLocalStorage("my-value");
return myValue;
};
render(<Test />);
screen.getByText("Hello!");
});
test("useLocalStorage reacts to key changes", async () => {
localStorage.clear();
localStorage.setItem("value-1", "1");
localStorage.setItem("value-2", "2");
const Test: FC = () => {
const [key, setKey] = useState("value-1");
const [value] = useLocalStorage(key);
if (key !== `value-${value}`) throw new Error("Value is out of sync");
return (
<>
<button onClick={() => setKey("value-2")}>Switch keys</button>
<div>Value is: {value}</div>
</>
);
};
const user = userEvent.setup();
render(<Test />);
screen.getByText("Value is: 1");
await user.click(screen.getByRole("button", { name: "Switch keys" }));
screen.getByText("Value is: 2");
});

View File

@@ -5,43 +5,52 @@ SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import EventEmitter from "events";
import { useCallback, useEffect, useState } from "react";
import { useCallback } from "react";
import { TypedEventEmitter } from "matrix-js-sdk";
import { useTypedEventEmitterState } from "./useEvents";
type LocalStorageItem = ReturnType<typeof localStorage.getItem>;
// Bus to notify other useLocalStorage consumers when an item is changed
export const localStorageBus = new EventEmitter();
export const localStorageBus = new TypedEventEmitter<
string,
{ [key: string]: () => void }
>();
// Like useState, but reads from and persists the value to localStorage
export const useLocalStorage = (
/**
* Like useState, but reads from and persists the value to localStorage
* This hook will not update when we write to localStorage.setItem(key, value) directly.
* For the hook to react either use the returned setter or `setLocalStorageItemReactive`.
*/
export function useLocalStorage(
key: string,
): [LocalStorageItem, (value: string) => void] => {
const [value, setValue] = useState<LocalStorageItem>(() =>
localStorage.getItem(key),
): [LocalStorageItem, (value: string) => void] {
const value = useTypedEventEmitterState(
localStorageBus,
key,
useCallback(() => localStorage.getItem(key), [key]),
);
const setValue = useCallback(
(newValue: string) => setLocalStorageItemReactive(key, newValue),
[key],
);
useEffect(() => {
localStorageBus.on(key, setValue);
return (): void => {
localStorageBus.off(key, setValue);
};
}, [key, setValue]);
return [value, setValue];
}
return [
value,
useCallback(
(newValue: string) => {
setValue(newValue);
localStorage.setItem(key, newValue);
localStorageBus.emit(key, newValue);
},
[key, setValue],
),
];
};
export const setLocalStorageItemReactive = (
key: string,
value: string,
): void => {
// Avoid unnecessary updates. Not avoiding them so can cause unexpected state updates across hooks.
// For instance:
// - In call view uses useRoomEncryptionSystem
// - This will set the key again.
// - All other instances of useRoomEncryptionSystem will now do a useMemo update of the e2eeSystem
// - because the dependency `storedPassword = useInternalRoomSharedKey(roomId);` would change.
if (localStorage.getItem(key) === value) return;
export const setLocalStorageItem = (key: string, value: string): void => {
localStorage.setItem(key, value);
localStorageBus.emit(key, value);
localStorageBus.emit(key);
};

View File

@@ -10,33 +10,31 @@ import {
type MatrixRTCSession,
MatrixRTCSessionEvent,
} from "matrix-js-sdk/lib/matrixrtc";
import { useEffect, useState } from "react";
import { TypedEventEmitter } from "matrix-js-sdk";
import { useCallback, useEffect } from "react";
import { useTypedEventEmitterState } from "./useEvents";
const dummySession = new TypedEventEmitter();
export function useMatrixRTCSessionJoinState(
rtcSession: MatrixRTCSession | undefined,
): boolean {
const [, setNumUpdates] = useState(0);
// React doesn't allow you to run a hook conditionally, so we have to plug in
// a dummy event emitter in case there is no rtcSession yet
const isJoined = useTypedEventEmitterState(
rtcSession ?? dummySession,
MatrixRTCSessionEvent.JoinStateChanged,
useCallback(() => rtcSession?.isJoined() ?? false, [rtcSession]),
);
useEffect(() => {
if (rtcSession !== undefined) {
const onJoinStateChanged = (isJoined: boolean): void => {
logger.info(
`Session in room ${rtcSession.room.roomId} changed to ${
isJoined ? "joined" : "left"
}`,
);
setNumUpdates((n) => n + 1); // Force an update
};
rtcSession.on(MatrixRTCSessionEvent.JoinStateChanged, onJoinStateChanged);
logger.info(
`Session in room ${rtcSession?.room.roomId} changed to ${
isJoined ? "joined" : "left"
}`,
);
}, [rtcSession, isJoined]);
return (): void => {
rtcSession.off(
MatrixRTCSessionEvent.JoinStateChanged,
onJoinStateChanged,
);
};
}
}, [rtcSession]);
return rtcSession?.isJoined() ?? false;
return isJoined;
}

View File

@@ -5,39 +5,21 @@ SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { logger } from "matrix-js-sdk/lib/logger";
import {
type CallMembership,
type MatrixRTCSession,
MatrixRTCSessionEvent,
} from "matrix-js-sdk/lib/matrixrtc";
import { useCallback, useEffect, useState } from "react";
import { useCallback } from "react";
import { useTypedEventEmitterState } from "./useEvents";
export function useMatrixRTCSessionMemberships(
rtcSession: MatrixRTCSession,
): CallMembership[] {
const [memberships, setMemberships] = useState(rtcSession.memberships);
const onMembershipsChanged = useCallback(() => {
logger.info(
`Memberships changed for call in room ${rtcSession.room.roomId} (${rtcSession.memberships.length} members)`,
);
setMemberships(rtcSession.memberships);
}, [rtcSession]);
useEffect(() => {
rtcSession.on(
MatrixRTCSessionEvent.MembershipsChanged,
onMembershipsChanged,
);
return (): void => {
rtcSession.off(
MatrixRTCSessionEvent.MembershipsChanged,
onMembershipsChanged,
);
};
}, [rtcSession, onMembershipsChanged]);
return memberships;
return useTypedEventEmitterState(
rtcSession,
MatrixRTCSessionEvent.MembershipsChanged,
useCallback(() => rtcSession.memberships, [rtcSession]),
);
}

View File

@@ -1,13 +1,11 @@
/*
Copyright 2023, 2024 New Vector Ltd.
Copyright 2023-2025 New Vector Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { useCallback, useMemo, useState } from "react";
import { useEventTarget } from "./useEvents";
import { useCallback, useMemo, useSyncExternalStore } from "react";
/**
* React hook that tracks whether the given media query matches.
@@ -15,14 +13,13 @@ import { useEventTarget } from "./useEvents";
export function useMediaQuery(query: string): boolean {
const mediaQuery = useMemo(() => window.matchMedia(query), [query]);
const [numChanges, setNumChanges] = useState(0);
useEventTarget(
mediaQuery,
"change",
useCallback(() => setNumChanges((n) => n + 1), [setNumChanges]),
const subscribe = useCallback(
(onChange: () => void) => {
mediaQuery.addEventListener("change", onChange);
return (): void => mediaQuery.removeEventListener("change", onChange);
},
[mediaQuery],
);
// We want any change to the update counter to trigger an update here
// eslint-disable-next-line react-hooks/exhaustive-deps
return useMemo(() => mediaQuery.matches, [mediaQuery, numChanges]);
const getState = useCallback(() => mediaQuery.matches, [mediaQuery]);
return useSyncExternalStore(subscribe, getState);
}

View File

@@ -108,7 +108,7 @@ interface EmitterMock<T> {
removeListener: () => T;
}
function mockEmitter<T>(): EmitterMock<T> {
export function mockEmitter<T>(): EmitterMock<T> {
return {
on(): T {
return this as T;

View File

@@ -7004,7 +7004,7 @@ __metadata:
i18next-parser: "npm:^9.1.0"
jsdom: "npm:^26.0.0"
knip: "npm:^5.27.2"
livekit-client: "npm:^2.11.3"
livekit-client: "npm:^2.13.0"
lodash-es: "npm:^4.17.21"
loglevel: "npm:^1.9.1"
matrix-js-sdk: "github:matrix-org/matrix-js-sdk#head=develop"
@@ -9386,9 +9386,9 @@ __metadata:
languageName: node
linkType: hard
"livekit-client@npm:^2.11.3":
version: 2.12.0
resolution: "livekit-client@npm:2.12.0"
"livekit-client@npm:^2.13.0":
version: 2.13.0
resolution: "livekit-client@npm:2.13.0"
dependencies:
"@livekit/mutex": "npm:1.1.1"
"@livekit/protocol": "npm:1.38.0"
@@ -9399,7 +9399,7 @@ __metadata:
tslib: "npm:2.8.1"
typed-emitter: "npm:^2.1.0"
webrtc-adapter: "npm:^9.0.1"
checksum: 10c0/8a4657aa6c0f0bc5d1fe77c2cd9603a3b07d4acefa634f1c5151190eed69711e7e599dd09c07915939a418dc8770d87e3529ecf1b029f2a9af7f2172d83acb1c
checksum: 10c0/b5b7b37eea56681e543d754ee5df63651e45a78462d1f0c26334f0aa525761fefc86d5ad85513e9345bbb4d6821fc95dded5f1d7c008da2b558a14905cbd9ac2
languageName: node
linkType: hard
@@ -9612,7 +9612,7 @@ __metadata:
"matrix-js-sdk@github:matrix-org/matrix-js-sdk#head=develop":
version: 37.6.0
resolution: "matrix-js-sdk@https://github.com/matrix-org/matrix-js-sdk.git#commit=93982716951ce2583904bfc26b27d6a86ba17a87"
resolution: "matrix-js-sdk@https://github.com/matrix-org/matrix-js-sdk.git#commit=bf6dc16ad32d47f2c6e167236f4c853ceef01d4f"
dependencies:
"@babel/runtime": "npm:^7.12.5"
"@matrix-org/matrix-sdk-crypto-wasm": "npm:^14.2.0"
@@ -9629,7 +9629,7 @@ __metadata:
sdp-transform: "npm:^2.14.1"
unhomoglyph: "npm:^1.0.6"
uuid: "npm:11"
checksum: 10c0/22a28099d2deaf0ca7f609a5859fe00fbd20c314d3b607a95b4a623a8aa159e428310b1849c77ab7b9875a29fc2d924cddbb6fc83dc4e42a74c4713401dcb0be
checksum: 10c0/2877e7c5b2779200b48f3152bb7b510e58899b1e779e7e6d2bc1a236ed178fa8858f0547b644a2029f48f55dc7cc3954f48e2598c8963d45293c8280ccc23039
languageName: node
linkType: hard