Merge branch 'livekit' into fkwp/change_video_codec

This commit is contained in:
fkwp
2025-08-26 16:18:47 +02:00
41 changed files with 1113 additions and 597 deletions

View File

@@ -143,7 +143,8 @@ via the `org.matrix.msc4143.rtc_foci` key, e.g.:
where the format for MatrixRTC using LiveKit backend is defined in
[MSC4195](https://github.com/hughns/matrix-spec-proposals/blob/hughns/matrixrtc-livekit/proposals/4195-matrixrtc-livekit.md).
In the example above Matrix clients do discover a focus of type `livekit` which
points them to a Matrix LiveKit JWT Auth Service via `livekit_service_url`.
points them to a [MatrixRTC Authorization Service](https://github.com/element-hq/lk-jwt-service)
via `livekit_service_url`.
### Backend Selection
@@ -154,9 +155,9 @@ points them to a Matrix LiveKit JWT Auth Service via `livekit_service_url`.
the **first participant who joined the call** defines via the `foci_preferred`
key in their `org.matrix.msc3401.call.member` which actual MatrixRTC backend
will be used for this call.
- During the actual call join flow, the **LiveKit JWT Auth Service** provides
the client with the **LiveKit SFU WebSocket URL** and an **access JWT token**
in order to exchange media via WebRTC.
- During the actual call join flow, the **[MatrixRTC Authorization Service](https://github.com/element-hq/lk-jwt-service)**
provides the client with the **LiveKit SFU WebSocket URL** and an
**access JWT token** in order to exchange media via WebRTC.
The example below illustrates how backend selection works across **Matrix
federation**, using the setup from sites A, B, and C. It demonstrates backend
@@ -208,7 +209,7 @@ A docker compose file `dev-backend-docker-compose.yml` is provided to start the
whole stack of components which is required for a local development environment:
- Minimum Synapse Setup (servername: `synapse.m.localhost`)
- LiveKit Authorization Service (Note requires Federation API and hence a TLS reverse proxy)
- MatrixRTC Authorization Service (Note requires Federation API and hence a TLS reverse proxy)
- Minimum LiveKit SFU Setup using dev defaults for config
- Redis db for completeness
- Minimum `localhost` Certificate Authority (CA) for Transport Layer Security (TLS)

Binary file not shown.

Before

Width:  |  Height:  |  Size: 86 KiB

After

Width:  |  Height:  |  Size: 91 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 67 KiB

After

Width:  |  Height:  |  Size: 67 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 929 KiB

After

Width:  |  Height:  |  Size: 941 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 886 KiB

After

Width:  |  Height:  |  Size: 908 KiB

View File

@@ -64,7 +64,7 @@ rc_delayed_event_mgmt:
```
As a prerequisite for the
[Matrix LiveKit JWT auth service](https://github.com/element-hq/lk-jwt-service)
[MatrixRTC Authorization Service](https://github.com/element-hq/lk-jwt-service)
make sure that your Synapse server has either a `federation` or `openid`
[listener configured](https://element-hq.github.io/synapse/latest/usage/configuration/config_documentation.html#listeners).
@@ -77,7 +77,7 @@ required for each site deployment.
As depicted above in the `example.com` site deployment, Element Call requires a
[Livekit SFU](https://github.com/livekit/livekit) alongside a
[Matrix Livekit JWT auth service](https://github.com/element-hq/lk-jwt-service)
[MatrixRTC Authorization Service](https://github.com/element-hq/lk-jwt-service)
to implement
[MSC4195: MatrixRTC using LiveKit backend](https://github.com/hughns/matrix-spec-proposals/blob/hughns/matrixrtc-livekit/proposals/4195-matrixrtc-livekit.md).
@@ -89,7 +89,7 @@ the example above, this results in:
| Service | Endpoint | Example |
| -------- | ------- | ------- |
| [Livekit SFU](https://github.com/livekit/livekit) WebSocket signalling connection | `/livekit/sfu` | `matrix-rtc.example.com/livekit/sfu` |
| [Matrix Livekit JWT auth service](https://github.com/element-hq/lk-jwt-service) | `/livekit/jwt` | `matrix-rtc.example.com/livekit/jwt` |
| [MatrixRTC Authorization Service](https://github.com/element-hq/lk-jwt-service) | `/livekit/jwt` | `matrix-rtc.example.com/livekit/jwt` |
Using Nginx, you can achieve this by:
@@ -102,7 +102,7 @@ server {
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
# JWT Service running at port 8080
# MatrixRTC Authorization Service running at port 8080
proxy_pass http://localhost:8080/;
}

View File

@@ -82,7 +82,7 @@
"error": {
"call_is_not_supported": "Volání není podporováno",
"call_not_found": "Volání nebylo nalezeno",
"call_not_found_description": "<0>Zdá se, že tento odkaz nepatří k žádnému existujícímu volání. Zkontrolujte, zda máte správný odkaz, nebo <1>vytvořte nový</1>.</0>",
"call_not_found_description": "<0>Zdá se, že tento odkaz nepatří k žádnému existujícímu hovoru. Zkontrolujte, zda máte správný odkaz, nebo<2> vytvořte nový</2>.</0>",
"connection_lost": "Spojení ztraceno",
"connection_lost_description": "Hovor byl přerušen.",
"e2ee_unsupported": "Nekompatibilní prohlížeč",
@@ -105,6 +105,11 @@
"knock_reject_heading": "Přístup odepřen",
"reason": "Důvod"
},
"handset": {
"overlay_back_button": "Zpět do režimu reproduktoru",
"overlay_description": "Funguje pouze při používání aplikace",
"overlay_title": "Režim sluchátka"
},
"hangup_button_label": "Ukončit hovor",
"header_label": "Domov Element Call",
"header_participants_label": "Účastníci",
@@ -177,6 +182,8 @@
"change_device_button": "Změnit zvukové zařízení",
"default": "Výchozí",
"default_named": "Výchozí <2> ({{name}}) </2>",
"handset": "Sluchátko",
"loudspeaker": "Reproduktor",
"microphone": "Mikrofon",
"microphone_numbered": "Mikrofon {{n}}",
"speaker": "Reproduktor",

View File

@@ -94,6 +94,8 @@
"matrix_rtc_focus_missing": "Serveren er ikke konfigureret til at arbejde med {{brand}}{{domain}}. Kontakt venligst din serveradministrator (domæne:{{domain}}, fejlkode: {{ errorCode }}).",
"open_elsewhere": "Åbnet i en anden fane",
"open_elsewhere_description": "{{brand}} er blevet åbnet i en anden fane. Hvis det ikke lyder rigtigt, kan du prøve at genindlæse siden.",
"room_creation_restricted": "Kunne ikke oprette opkald",
"room_creation_restricted_description": "Oprettelse af opkald er muligvis begrænset til autoriserede brugere. Prøv igen senere, eller kontakt din serveradministrator, hvis problemet fortsætter.",
"unexpected_ec_error": "Der opstod en uventet fejl (<0>Fejlkode:</0> <1> {{ errorCode }}</1>). Kontakt venligst din serveradministrator."
},
"group_call_loader": {
@@ -105,6 +107,11 @@
"knock_reject_heading": "Adgang nægtet",
"reason": "Årsag: {{reason}}"
},
"handset": {
"overlay_back_button": "Tilbage til højttalertilstand",
"overlay_description": "Virker kun, når du bruger appen",
"overlay_title": "Telefon-højtaler"
},
"hangup_button_label": "Afslut opkald",
"header_label": "Element Ring hjem",
"header_participants_label": "Deltagere",
@@ -176,6 +183,7 @@
"change_device_button": "Skift lydenhed",
"default": "Standard",
"default_named": "Standard <2>({{name}})</2>",
"handset": "Telefon",
"loudspeaker": "Højttaler",
"microphone": "Mikrofon",
"microphone_numbered": "Mikrofon {{n}}",

View File

@@ -55,6 +55,7 @@
"profile": "Profil",
"reaction": "Reaktion",
"reactions": "Reaktionen",
"reconnecting": "Verbindung wird wiederhergestellt...",
"settings": "Einstellungen",
"unencrypted": "Nicht verschlüsselt",
"username": "Benutzername",

View File

@@ -55,6 +55,7 @@
"profile": "Profile",
"reaction": "Reaction",
"reactions": "Reactions",
"reconnecting": "Reconnecting…",
"settings": "Settings",
"unencrypted": "Not encrypted",
"username": "Username",

View File

@@ -82,7 +82,7 @@
"error": {
"call_is_not_supported": "Call stöds inte",
"call_not_found": "Samtal hittades inte",
"call_not_found_description": "<0>Den länken verkar inte tillhöra något befintligt samtal. Kontrollera att du har rätt länk, eller <1>skapa en ny</1>.</0>",
"call_not_found_description": "<0>Den länken verkar inte tillhöra något befintligt samtal. Kontrollera att du har rätt länk, eller <2>skapa en ny</2>.</0>",
"connection_lost": "Anslutning förlorad",
"connection_lost_description": "Du kopplades bort från samtalet.",
"e2ee_unsupported": "Inkompatibel webbläsare",
@@ -94,6 +94,8 @@
"matrix_rtc_focus_missing": "Servern är inte konfigurerad för att fungera med {{brand}}. Vänligen kontakta serveradministratören (Domän: {{domain}}, Felkod: {{ errorCode }}).",
"open_elsewhere": "Öppnades i en annan flik",
"open_elsewhere_description": "{{brand}} har öppnats i en annan flik. Om det inte låter rätt, pröva att ladda om sidan.",
"room_creation_restricted": "Misslyckades att skapa samtal",
"room_creation_restricted_description": "Samtalsskapande kan vara begränsat till endast behöriga användare. Försök igen senare eller kontakta serveradministratören om problemet kvarstår.",
"unexpected_ec_error": "Ett oväntat fel inträffade (<0>Felkod:</0> <1>{{ errorCode }}</1>). Kontakta din serveradministratör."
},
"group_call_loader": {
@@ -105,6 +107,11 @@
"knock_reject_heading": "Inte tillåten att gå med",
"reason": "Anledning"
},
"handset": {
"overlay_back_button": "Tillbaka till högtalarläge",
"overlay_description": "Fungerar bara när appen används",
"overlay_title": "Telefonläge"
},
"hangup_button_label": "Avsluta samtal",
"header_label": "Element Call Hem",
"header_participants_label": "Deltagare",
@@ -176,6 +183,8 @@
"change_device_button": "Byt ljudenhet",
"default": "Förval",
"default_named": "Förval <2>({{name}})</2>",
"handset": "Telefonlur",
"loudspeaker": "Högtalare",
"microphone": "Mikrofon",
"microphone_numbered": "Mikrofon {{n}}",
"speaker": "Högtalare",

View File

@@ -24,7 +24,6 @@ import { RegisterPage } from "./auth/RegisterPage";
import { RoomPage } from "./room/RoomPage";
import { ClientProvider } from "./ClientContext";
import { ErrorPage, LoadingPage } from "./FullScreenView";
import { DisconnectedBanner } from "./DisconnectedBanner";
import { Initializer } from "./initializer";
import { widget } from "./widget";
import { useTheme } from "./useTheme";
@@ -86,7 +85,6 @@ export const App: FC<Props> = ({ vm }) => {
<Sentry.ErrorBoundary
fallback={(error) => <ErrorPage error={error} widget={widget} />}
>
<DisconnectedBanner />
<Routes>
<SentryRoute path="/" element={<HomePage />} />
<SentryRoute path="/login" element={<LoginPage />} />

View File

@@ -61,7 +61,11 @@ export const AppBar: FC<Props> = ({ children }) => {
style={{ display: hidden ? "none" : "block" }}
className={styles.bar}
>
<Header>
<Header
// App bar is mainly seen in the call view, which has its own
// 'reconnecting' toast
disconnectedBanner={false}
>
<LeftNav>
<Tooltip label={t("common.back")}>
<IconButton onClick={onBackClick}>

View File

@@ -17,27 +17,38 @@ import Logo from "./icons/Logo.svg?react";
import { Avatar, Size } from "./Avatar";
import { EncryptionLock } from "./room/EncryptionLock";
import { useMediaQuery } from "./useMediaQuery";
import { DisconnectedBanner } from "./DisconnectedBanner";
interface HeaderProps extends HTMLAttributes<HTMLElement> {
ref?: Ref<HTMLElement>;
children: ReactNode;
className?: string;
/**
* Whether the header should display an informational banner whenever the
* client is disconnected from the homeserver.
* @default true
*/
disconnectedBanner?: boolean;
}
export const Header: FC<HeaderProps> = ({
ref,
children,
className,
disconnectedBanner = true,
...rest
}) => {
return (
<header
ref={ref}
className={classNames(styles.header, className)}
{...rest}
>
{children}
</header>
<>
<header
ref={ref}
className={classNames(styles.header, className)}
{...rest}
>
{children}
</header>
{disconnectedBanner && <DisconnectedBanner />}
</>
);
};

View File

@@ -45,6 +45,12 @@ interface Props {
* A supporting icon to display within the toast.
*/
Icon?: ComponentType<SVGAttributes<SVGElement>>;
/**
* Whether the toast should be portaled into the root of the document (rather
* than rendered in-place within the component tree).
* @default true
*/
portal?: boolean;
}
/**
@@ -56,6 +62,7 @@ export const Toast: FC<Props> = ({
autoDismiss,
children,
Icon,
portal = true,
}) => {
const onOpenChange = useCallback(
(open: boolean) => {
@@ -71,29 +78,33 @@ export const Toast: FC<Props> = ({
}
}, [open, autoDismiss, onDismiss]);
const content = (
<>
<DialogOverlay
className={classNames(overlayStyles.bg, overlayStyles.animate)}
/>
<DialogContent aria-describedby={undefined} asChild>
<DialogClose
className={classNames(
overlayStyles.overlay,
overlayStyles.animate,
styles.toast,
)}
>
<DialogTitle asChild>
<Text as="h3" size="sm" weight="semibold">
{children}
</Text>
</DialogTitle>
{Icon && <Icon width={20} height={20} aria-hidden />}
</DialogClose>
</DialogContent>
</>
);
return (
<DialogRoot open={open} onOpenChange={onOpenChange}>
<DialogPortal>
<DialogOverlay
className={classNames(overlayStyles.bg, overlayStyles.animate)}
/>
<DialogContent aria-describedby={undefined} asChild>
<DialogClose
className={classNames(
overlayStyles.overlay,
overlayStyles.animate,
styles.toast,
)}
>
<DialogTitle asChild>
<Text as="h3" size="sm" weight="semibold">
{children}
</Text>
</DialogTitle>
{Icon && <Icon width={20} height={20} aria-hidden />}
</DialogClose>
</DialogContent>
</DialogPortal>
{portal ? <DialogPortal>{content}</DialogPortal> : content}
</DialogRoot>
);
};

View File

@@ -16,7 +16,6 @@ import {
EndCallIcon,
ShareScreenSolidIcon,
SettingsSolidIcon,
SwitchCameraSolidIcon,
} from "@vector-im/compound-design-tokens/assets/web/icons";
import styles from "./Button.module.css";
@@ -67,23 +66,6 @@ export const VideoButton: FC<VideoButtonProps> = ({ muted, ...props }) => {
);
};
export const SwitchCameraButton: FC<ComponentPropsWithoutRef<"button">> = (
props,
) => {
const { t } = useTranslation();
return (
<Tooltip label={t("switch_camera")}>
<CpdButton
iconOnly
Icon={SwitchCameraSolidIcon}
kind="secondary"
{...props}
/>
</Tooltip>
);
};
interface ShareScreenButtonProps extends ComponentPropsWithoutRef<"button"> {
enabled: boolean;
}

View File

@@ -61,7 +61,12 @@ const TileWrapper_ = memo(
useDrag((state) => onDrag?.current!(id, state), {
target: ref,
filterTaps: true,
preventScroll: true,
// Previous designs, which allowed tiles to be dragged and dropped around
// the scrolling grid, required us to set preventScroll to true here. But
// our designs no longer call for this, and meanwhile there's a bug in
// use-gesture that causes filterTaps + preventScroll to break buttons
// within tiles (like the 'switch camera' button) on mobile.
// https://github.com/pmndrs/use-gesture/issues/593
});
return (

View File

@@ -31,6 +31,7 @@ import {
aliceRtcMember,
bobRtcMember,
local,
localRtcMember,
} from "../utils/test-fixtures";
vitest.mock("../useAudioContext");
@@ -66,7 +67,7 @@ beforeEach(() => {
* a noise every time.
*/
test("plays one sound when entering a call", () => {
const { vm, remoteRtcMemberships$ } = getBasicCallViewModelEnvironment([
const { vm, rtcMemberships$ } = getBasicCallViewModelEnvironment([
local,
alice,
]);
@@ -74,56 +75,58 @@ test("plays one sound when entering a call", () => {
// Joining a call usually means remote participants are added later.
act(() => {
remoteRtcMemberships$.next([aliceRtcMember, bobRtcMember]);
rtcMemberships$.next([localRtcMember, aliceRtcMember, bobRtcMember]);
});
expect(playSound).toHaveBeenCalledOnce();
});
test("plays a sound when a user joins", () => {
const { vm, remoteRtcMemberships$ } = getBasicCallViewModelEnvironment([
const { vm, rtcMemberships$ } = getBasicCallViewModelEnvironment([
local,
alice,
]);
render(<CallEventAudioRenderer vm={vm} />);
act(() => {
remoteRtcMemberships$.next([aliceRtcMember, bobRtcMember]);
rtcMemberships$.next([localRtcMember, aliceRtcMember, bobRtcMember]);
});
// Play a sound when joining a call.
expect(playSound).toBeCalledWith("join");
});
test("plays a sound when a user leaves", () => {
const { vm, remoteRtcMemberships$ } = getBasicCallViewModelEnvironment([
const { vm, rtcMemberships$ } = getBasicCallViewModelEnvironment([
local,
alice,
]);
render(<CallEventAudioRenderer vm={vm} />);
act(() => {
remoteRtcMemberships$.next([]);
rtcMemberships$.next([localRtcMember]);
});
expect(playSound).toBeCalledWith("left");
});
test("plays no sound when the participant list is more than the maximum size", () => {
const mockRtcMemberships: CallMembership[] = [];
const mockRtcMemberships: CallMembership[] = [localRtcMember];
for (let i = 0; i < MAX_PARTICIPANT_COUNT_FOR_SOUND; i++) {
mockRtcMemberships.push(
mockRtcMembership(`@user${i}:example.org`, `DEVICE${i}`),
);
}
const { vm, remoteRtcMemberships$ } = getBasicCallViewModelEnvironment(
const { vm, rtcMemberships$ } = getBasicCallViewModelEnvironment(
[local, alice],
mockRtcMemberships,
);
render(<CallEventAudioRenderer vm={vm} />);
expect(playSound).not.toBeCalled();
// Remove the last membership in the array to test the leaving sound
// (The array has length MAX_PARTICIPANT_COUNT_FOR_SOUND + 1)
act(() => {
remoteRtcMemberships$.next(
mockRtcMemberships.slice(0, MAX_PARTICIPANT_COUNT_FOR_SOUND - 1),
rtcMemberships$.next(
mockRtcMemberships.slice(0, MAX_PARTICIPANT_COUNT_FOR_SOUND),
);
});
expect(playSound).toBeCalledWith("left");

View File

@@ -137,11 +137,9 @@ function createGroupCallView(
getJoinRule: () => JoinRule.Invite,
} as Partial<RoomState> as RoomState,
});
const rtcSession = new MockRTCSession(
room,
localRtcMember,
[],
).withMemberships(constant([]));
const rtcSession = new MockRTCSession(room, []).withMemberships(
constant([localRtcMember]),
);
rtcSession.joined = joined;
const muteState = {
audio: { enabled: false },

View File

@@ -452,6 +452,7 @@ export const GroupCallView: FC<Props> = ({
client={client}
matrixInfo={matrixInfo}
rtcSession={rtcSession as MatrixRTCSession}
matrixRoom={room}
participantCount={participantCount}
onLeave={onLeave}
header={header}

View File

@@ -115,7 +115,6 @@ Please see LICENSE in the repository root for full details.
@media (max-width: 340px) {
.invite,
.switchCamera,
.shareScreen {
display: none;
}

View File

@@ -175,6 +175,7 @@ function createInCallView(): RenderResult & {
kind: E2eeType.NONE,
},
}}
matrixRoom={room}
livekitRoom={livekitRoom}
participantCount={0}
onLeave={function (): void {

View File

@@ -7,8 +7,8 @@ Please see LICENSE in the repository root for full details.
import { RoomContext, useLocalParticipant } from "@livekit/components-react";
import { IconButton, Text, Tooltip } from "@vector-im/compound-web";
import { ConnectionState, type Room } from "livekit-client";
import { type MatrixClient } from "matrix-js-sdk";
import { ConnectionState, type Room as LivekitRoom } from "livekit-client";
import { type MatrixClient, type Room as MatrixRoom } from "matrix-js-sdk";
import {
type FC,
type PointerEvent,
@@ -44,7 +44,6 @@ import {
ShareScreenButton,
SettingsButton,
ReactionToggleButton,
SwitchCameraButton,
} from "../button";
import { Header, LeftNav, RightNav, RoomHeaderInfo } from "../Header";
import { type HeaderStyle, useUrlParams } from "../UrlParams";
@@ -94,7 +93,6 @@ import {
useReactionsSender,
} from "../reactions/useReactionsSender";
import { ReactionsAudioRenderer } from "./ReactionAudioRenderer";
import { useSwitchCamera } from "./useSwitchCamera";
import { ReactionsOverlay } from "./ReactionsOverlay";
import { CallEventAudioRenderer } from "./CallEventAudioRenderer";
import {
@@ -113,6 +111,7 @@ import { useMediaDevices } from "../MediaDevicesContext.ts";
import { EarpieceOverlay } from "./EarpieceOverlay.tsx";
import { useAppBarHidden, useAppBarSecondaryButton } from "../AppBar.tsx";
import { useBehavior } from "../useBehavior.ts";
import { Toast } from "../Toast.tsx";
const canScreenshare = "getDisplayMedia" in (navigator.mediaDevices ?? {});
@@ -166,6 +165,7 @@ export const ActiveCall: FC<ActiveCallProps> = (props) => {
const reactionsReader = new ReactionsReader(props.rtcSession);
const vm = new CallViewModel(
props.rtcSession,
props.matrixRoom,
livekitRoom,
mediaDevices,
{
@@ -184,6 +184,7 @@ export const ActiveCall: FC<ActiveCallProps> = (props) => {
}
}, [
props.rtcSession,
props.matrixRoom,
livekitRoom,
mediaDevices,
props.e2eeSystem,
@@ -212,7 +213,8 @@ export interface InCallViewProps {
vm: CallViewModel;
matrixInfo: MatrixInfo;
rtcSession: MatrixRTCSession;
livekitRoom: Room;
matrixRoom: MatrixRoom;
livekitRoom: LivekitRoom;
muteStates: MuteStates;
participantCount: number;
/** Function to call when the user explicitly ends the call */
@@ -228,6 +230,7 @@ export const InCallView: FC<InCallViewProps> = ({
vm,
matrixInfo,
rtcSession,
matrixRoom,
livekitRoom,
muteStates,
participantCount,
@@ -272,7 +275,7 @@ export const InCallView: FC<InCallViewProps> = ({
const [useExperimentalToDeviceTransport] = useSetting(
useExperimentalToDeviceTransportSetting,
);
const encryptionSystem = useRoomEncryptionSystem(rtcSession.room.roomId);
const encryptionSystem = useRoomEncryptionSystem(matrixRoom.roomId);
const memberships = useMatrixRTCSessionMemberships(rtcSession);
const showToDeviceEncryption = useMemo(
@@ -309,6 +312,7 @@ export const InCallView: FC<InCallViewProps> = ({
() => void toggleRaisedHand(),
);
const reconnecting = useBehavior(vm.reconnecting$);
const windowMode = useBehavior(vm.windowMode$);
const layout = useBehavior(vm.layout$);
const tileStoreGeneration = useBehavior(vm.tileStoreGeneration$);
@@ -318,7 +322,6 @@ export const InCallView: FC<InCallViewProps> = ({
const showFooter = useBehavior(vm.showFooter$);
const earpieceMode = useBehavior(vm.earpieceMode$);
const audioOutputSwitcher = useBehavior(vm.audioOutputSwitcher$);
const switchCamera = useSwitchCamera(vm.localVideo$);
useSubscription(vm.autoLeaveWhenOthersLeft$, onLeave);
// Ideally we could detect taps by listening for click events and checking
@@ -502,7 +505,11 @@ export const InCallView: FC<InCallViewProps> = ({
break;
case "standard":
header = (
<Header className={styles.header} ref={headerRef}>
<Header
className={styles.header}
ref={headerRef}
disconnectedBanner={false} // This screen has its own 'reconnecting' toast
>
<LeftNav>
<RoomHeaderInfo
id={matrixInfo.roomId}
@@ -642,7 +649,7 @@ export const InCallView: FC<InCallViewProps> = ({
};
const rageshakeRequestModalProps = useRageshakeRequestModal(
rtcSession.room.roomId,
matrixRoom.roomId,
);
const toggleScreensharing = useCallback(() => {
@@ -676,15 +683,6 @@ export const InCallView: FC<InCallViewProps> = ({
data-testid="incall_videomute"
/>,
);
if (switchCamera !== null)
buttons.push(
<SwitchCameraButton
key="switch_camera"
className={styles.switchCamera}
onClick={switchCamera}
onTouchEnd={onControlsTouchEnd}
/>,
);
if (canScreenshare && !hideScreensharing) {
buttons.push(
<ShareScreenButton
@@ -762,6 +760,9 @@ export const InCallView: FC<InCallViewProps> = ({
</div>
);
// The reconnecting toast cannot be dismissed
const onDismissReconnectingToast = useCallback(() => {}, []);
return (
<div
className={styles.inRoom}
@@ -789,8 +790,15 @@ export const InCallView: FC<InCallViewProps> = ({
{renderContent()}
<CallEventAudioRenderer vm={vm} muted={muteAllAudio} />
<ReactionsAudioRenderer vm={vm} muted={muteAllAudio} />
<Toast
onDismiss={onDismissReconnectingToast}
open={reconnecting}
portal={false}
>
{t("common.reconnecting")}
</Toast>
<EarpieceOverlay
show={earpieceMode}
show={earpieceMode && !reconnecting}
onBackToVideoPressed={audioOutputSwitcher?.switch}
/>
<ReactionsOverlay vm={vm} />
@@ -800,7 +808,7 @@ export const InCallView: FC<InCallViewProps> = ({
<RageshakeRequestModal {...rageshakeRequestModalProps} />
<SettingsModal
client={client}
roomId={rtcSession.room.roomId}
roomId={matrixRoom.roomId}
open={settingsModalOpen}
onDismiss={closeSettings}
tab={settingsTab}

View File

@@ -24,8 +24,7 @@ import {
type LocalVideoTrack,
Track,
} from "livekit-client";
import { useObservable, useObservableEagerState } from "observable-hooks";
import { map } from "rxjs";
import { useObservableEagerState } from "observable-hooks";
import { useNavigate } from "react-router-dom";
import inCallStyles from "./InCallView.module.css";
@@ -38,7 +37,6 @@ import {
EndCallButton,
MicButton,
SettingsButton,
SwitchCameraButton,
VideoButton,
} from "../button/Button";
import { SettingsModal, defaultSettingsTab } from "../settings/SettingsModal";
@@ -47,7 +45,6 @@ import { E2eeType } from "../e2ee/e2eeType";
import { Link } from "../button/Link";
import { useMediaDevices } from "../MediaDevicesContext";
import { useInitial } from "../useInitial";
import { useSwitchCamera as useShowSwitchCamera } from "./useSwitchCamera";
import {
useTrackProcessor,
useTrackProcessorSync,
@@ -195,12 +192,6 @@ export const LobbyView: FC<Props> = ({
}, [devices, videoInputId, videoTrack]);
useTrackProcessorSync(videoTrack);
const showSwitchCamera = useShowSwitchCamera(
useObservable(
(inputs$) => inputs$.pipe(map(([video]) => video)),
[videoTrack],
),
);
// TODO: Unify this component with InCallView, so we can get slick joining
// animations and don't have to feel bad about reusing its CSS
@@ -257,9 +248,6 @@ export const LobbyView: FC<Props> = ({
onClick={onVideoPress}
disabled={muteStates.video.setEnabled === null}
/>
{showSwitchCamera && (
<SwitchCameraButton onClick={showSwitchCamera} />
)}
<SettingsButton onClick={openSettings} />
{!confineToRoom && <EndCallButton onClick={onLeaveClick} />}
</div>

View File

@@ -256,7 +256,7 @@ exports[`InCallView > rendering > renders 1`] = `
>
<button
aria-disabled="false"
aria-labelledby="«r5»"
aria-labelledby="«r8»"
class="_button_vczzf_8 _has-icon_vczzf_57 _icon-only_vczzf_50"
data-kind="primary"
data-size="lg"
@@ -279,7 +279,7 @@ exports[`InCallView > rendering > renders 1`] = `
</button>
<button
aria-disabled="false"
aria-labelledby="«ra»"
aria-labelledby="«rd»"
class="_button_vczzf_8 _has-icon_vczzf_57 _icon-only_vczzf_50"
data-kind="primary"
data-size="lg"
@@ -301,7 +301,7 @@ exports[`InCallView > rendering > renders 1`] = `
</svg>
</button>
<button
aria-labelledby="«rf»"
aria-labelledby="«ri»"
class="_button_vczzf_8 _has-icon_vczzf_57 _icon-only_vczzf_50"
data-kind="secondary"
data-size="lg"
@@ -322,7 +322,7 @@ exports[`InCallView > rendering > renders 1`] = `
</svg>
</button>
<button
aria-labelledby="«rk»"
aria-labelledby="«rn»"
class="_button_vczzf_8 endCall _has-icon_vczzf_57 _icon-only_vczzf_50 _destructive_vczzf_107"
data-kind="primary"
data-size="lg"
@@ -348,7 +348,7 @@ exports[`InCallView > rendering > renders 1`] = `
class="toggle layout"
>
<input
aria-labelledby="«rp»"
aria-labelledby="«rs»"
name="layout"
type="radio"
value="spotlight"
@@ -366,7 +366,7 @@ exports[`InCallView > rendering > renders 1`] = `
/>
</svg>
<input
aria-labelledby="«ru»"
aria-labelledby="«r11»"
checked=""
name="layout"
type="radio"

View File

@@ -1,93 +0,0 @@
/*
Copyright 2024 New Vector Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import {
fromEvent,
map,
merge,
type Observable,
of,
startWith,
switchMap,
} from "rxjs";
import {
facingModeFromLocalTrack,
type LocalVideoTrack,
TrackEvent,
} from "livekit-client";
import { useObservable, useObservableEagerState } from "observable-hooks";
import { logger } from "matrix-js-sdk/lib/logger";
import { useMediaDevices } from "../MediaDevicesContext";
import { platform } from "../Platform";
import { useLatest } from "../useLatest";
/**
* Determines whether the user should be shown a button to switch their camera,
* producing a callback if so.
*/
export function useSwitchCamera(
video$: Observable<LocalVideoTrack | null>,
): (() => void) | null {
const mediaDevices = useMediaDevices();
const setVideoInput = useLatest(mediaDevices.videoInput.select);
// Produce an observable like the input 'video' observable, except make it
// emit whenever the track is muted or the device changes
const videoTrack$: Observable<LocalVideoTrack | null> = useObservable(
(inputs$) =>
inputs$.pipe(
switchMap(([video$]) => video$),
switchMap((video) => {
if (video === null) return of(null);
return merge(
fromEvent(video, TrackEvent.Restarted).pipe(
startWith(null),
map(() => video),
),
fromEvent(video, TrackEvent.Muted).pipe(map(() => null)),
);
}),
),
[video$],
);
const switchCamera$: Observable<(() => void) | null> = useObservable(
(inputs$) =>
platform === "desktop"
? of(null)
: inputs$.pipe(
switchMap(([track$]) => track$),
map((track) => {
if (track === null) return null;
const facingMode = facingModeFromLocalTrack(track).facingMode;
// If the camera isn't front or back-facing, don't provide a switch
// camera shortcut at all
if (facingMode !== "user" && facingMode !== "environment")
return null;
// Restart the track with a camera facing the opposite direction
return (): void =>
void track
.restartTrack({
facingMode: facingMode === "user" ? "environment" : "user",
})
.then(() => {
// Inform the MediaDeviceContext which camera was chosen
const deviceId =
track.mediaStreamTrack.getSettings().deviceId;
if (deviceId !== undefined) setVideoInput.current(deviceId);
})
.catch((e) =>
logger.error("Failed to switch camera", facingMode, e),
);
}),
),
[videoTrack$],
);
return useObservableEagerState(switchCamera$);
}

View File

@@ -6,6 +6,7 @@ Please see LICENSE in the repository root for full details.
*/
import { test, vi, onTestFinished, it } from "vitest";
import EventEmitter from "events";
import {
BehaviorSubject,
combineLatest,
@@ -17,16 +18,18 @@ import {
of,
switchMap,
} from "rxjs";
import { type MatrixClient } from "matrix-js-sdk";
import { ClientEvent, SyncState, type MatrixClient } from "matrix-js-sdk";
import {
ConnectionState,
type LocalParticipant,
type LocalTrackPublication,
type Participant,
ParticipantEvent,
type RemoteParticipant,
} from "livekit-client";
import * as ComponentsCore from "@livekit/components-core";
import {
Status,
type CallMembership,
type MatrixRTCSession,
} from "matrix-js-sdk/lib/matrixrtc";
@@ -94,6 +97,11 @@ vi.mock("rxjs", async (importOriginal) => ({
vi.mock("@livekit/components-core");
const yesNo = {
y: true,
n: false,
};
const daveRtcMember = mockRtcMembership("@dave:example.org", "DDDD");
const carol = local;
@@ -226,33 +234,56 @@ function summarizeLayout$(l$: Observable<Layout>): Observable<LayoutSummary> {
);
}
interface CallViewModelInputs {
remoteParticipants$: Behavior<RemoteParticipant[]>;
rtcMembers$: Behavior<Partial<CallMembership>[]>;
connectionState$: Observable<ECConnectionState>;
speaking: Map<Participant, Observable<boolean>>;
mediaDevices: MediaDevices;
initialSyncState: SyncState;
}
function withCallViewModel(
remoteParticipants$: Behavior<RemoteParticipant[]>,
rtcMembers$: Behavior<Partial<CallMembership>[]>,
connectionState$: Observable<ECConnectionState>,
speaking: Map<Participant, Observable<boolean>>,
mediaDevices: MediaDevices,
{
remoteParticipants$ = constant([]),
rtcMembers$ = constant([localRtcMember]),
connectionState$ = of(ConnectionState.Connected),
speaking = new Map(),
mediaDevices = mockMediaDevices({}),
initialSyncState = SyncState.Syncing,
}: Partial<CallViewModelInputs>,
continuation: (
vm: CallViewModel,
rtcSession: MockRTCSession,
subjects: { raisedHands$: BehaviorSubject<Record<string, RaisedHandInfo>> },
setSyncState: (value: SyncState) => void,
) => void,
options: CallViewModelOptions = {
encryptionSystem: { kind: E2eeType.PER_PARTICIPANT },
autoLeaveWhenOthersLeft: false,
},
): void {
let syncState = initialSyncState;
const setSyncState = (value: SyncState): void => {
const prev = syncState;
syncState = value;
room.client.emit(ClientEvent.Sync, value, prev);
};
const room = mockMatrixRoom({
client: {
getUserId: () => localRtcMember.sender,
getDeviceId: () => localRtcMember.deviceId,
} as Partial<MatrixClient> as MatrixClient,
client: new (class extends EventEmitter {
public getUserId(): string | undefined {
return localRtcMember.sender;
}
public getDeviceId(): string {
return localRtcMember.deviceId;
}
public getSyncState(): SyncState {
return syncState;
}
})() as Partial<MatrixClient> as MatrixClient,
getMember: (userId) => roomMembers.get(userId) ?? null,
});
const rtcSession = new MockRTCSession(
room,
localRtcMember,
[],
).withMemberships(rtcMembers$);
const rtcSession = new MockRTCSession(room, []).withMemberships(rtcMembers$);
const participantsSpy = vi
.spyOn(ComponentsCore, "connectedParticipantsObserver")
.mockReturnValue(remoteParticipants$);
@@ -279,7 +310,7 @@ function withCallViewModel(
.spyOn(ComponentsCore, "roomEventSelector")
.mockImplementation((room, eventType) => of());
const liveKitRoom = mockLivekitRoom(
const livekitRoom = mockLivekitRoom(
{ localParticipant },
{ remoteParticipants$ },
);
@@ -288,7 +319,8 @@ function withCallViewModel(
const vm = new CallViewModel(
rtcSession as unknown as MatrixRTCSession,
liveKitRoom,
room,
livekitRoom,
mediaDevices,
options,
connectionState$,
@@ -304,7 +336,7 @@ function withCallViewModel(
roomEventSelectorSpy!.mockRestore();
});
continuation(vm, { raisedHands$: raisedHands$ });
continuation(vm, rtcSession, { raisedHands$: raisedHands$ }, setSyncState);
}
test("participants are retained during a focus switch", () => {
@@ -317,17 +349,17 @@ test("participants are retained during a focus switch", () => {
const expectedLayoutMarbles = " a";
withCallViewModel(
behavior(participantInputMarbles, {
a: [aliceParticipant, bobParticipant],
b: [],
}),
constant([aliceRtcMember, bobRtcMember]),
behavior(connectionInputMarbles, {
c: ConnectionState.Connected,
s: ECAddonConnectionState.ECSwitchingFocus,
}),
new Map(),
mockMediaDevices({}),
{
remoteParticipants$: behavior(participantInputMarbles, {
a: [aliceParticipant, bobParticipant],
b: [],
}),
rtcMembers$: constant([localRtcMember, aliceRtcMember, bobRtcMember]),
connectionState$: behavior(connectionInputMarbles, {
c: ConnectionState.Connected,
s: ECAddonConnectionState.ECSwitchingFocus,
}),
},
(vm) => {
expectObservable(summarizeLayout$(vm.layout$)).toBe(
expectedLayoutMarbles,
@@ -358,16 +390,15 @@ test("screen sharing activates spotlight layout", () => {
const expectedLayoutMarbles = " abcdaefeg";
const expectedShowSpeakingMarbles = "y----nyny";
withCallViewModel(
behavior(participantInputMarbles, {
a: [aliceParticipant, bobParticipant],
b: [aliceSharingScreen, bobParticipant],
c: [aliceSharingScreen, bobSharingScreen],
d: [aliceParticipant, bobSharingScreen],
}),
constant([aliceRtcMember, bobRtcMember]),
of(ConnectionState.Connected),
new Map(),
mockMediaDevices({}),
{
remoteParticipants$: behavior(participantInputMarbles, {
a: [aliceParticipant, bobParticipant],
b: [aliceSharingScreen, bobParticipant],
c: [aliceSharingScreen, bobSharingScreen],
d: [aliceParticipant, bobSharingScreen],
}),
rtcMembers$: constant([localRtcMember, aliceRtcMember, bobRtcMember]),
},
(vm) => {
schedule(modeInputMarbles, {
s: () => vm.setGridMode("spotlight"),
@@ -419,10 +450,7 @@ test("screen sharing activates spotlight layout", () => {
);
expectObservable(vm.showSpeakingIndicators$).toBe(
expectedShowSpeakingMarbles,
{
y: true,
n: false,
},
yesNo,
);
},
);
@@ -443,24 +471,24 @@ test("participants stay in the same order unless to appear/disappear", () => {
const expectedLayoutMarbles = " a 1999ms b 1999ms a 57999ms c 1999ms a";
withCallViewModel(
constant([aliceParticipant, bobParticipant, daveParticipant]),
constant([aliceRtcMember, bobRtcMember, daveRtcMember]),
of(ConnectionState.Connected),
new Map([
[
{
remoteParticipants$: constant([
aliceParticipant,
behavior(aSpeakingInputMarbles, { y: true, n: false }),
],
[
bobParticipant,
behavior(bSpeakingInputMarbles, { y: true, n: false }),
],
[
daveParticipant,
behavior(dSpeakingInputMarbles, { y: true, n: false }),
],
]),
mockMediaDevices({}),
]),
rtcMembers$: constant([
localRtcMember,
aliceRtcMember,
bobRtcMember,
daveRtcMember,
]),
speaking: new Map([
[aliceParticipant, behavior(aSpeakingInputMarbles, yesNo)],
[bobParticipant, behavior(bSpeakingInputMarbles, yesNo)],
[daveParticipant, behavior(dSpeakingInputMarbles, yesNo)],
]),
},
(vm) => {
schedule(visibilityInputMarbles, {
a: () => {
@@ -510,20 +538,23 @@ test("participants adjust order when space becomes constrained", () => {
const expectedLayoutMarbles = " a-b";
withCallViewModel(
constant([aliceParticipant, bobParticipant, daveParticipant]),
constant([aliceRtcMember, bobRtcMember, daveRtcMember]),
of(ConnectionState.Connected),
new Map([
[
{
remoteParticipants$: constant([
aliceParticipant,
bobParticipant,
behavior(bSpeakingInputMarbles, { y: true, n: false }),
],
[
daveParticipant,
behavior(dSpeakingInputMarbles, { y: true, n: false }),
],
]),
mockMediaDevices({}),
]),
rtcMembers$: constant([
localRtcMember,
aliceRtcMember,
bobRtcMember,
daveRtcMember,
]),
speaking: new Map([
[bobParticipant, behavior(bSpeakingInputMarbles, yesNo)],
[daveParticipant, behavior(dSpeakingInputMarbles, yesNo)],
]),
},
(vm) => {
let setVisibleTiles: ((value: number) => void) | null = null;
vm.layout$.subscribe((layout) => {
@@ -569,24 +600,24 @@ test("spotlight speakers swap places", () => {
const expectedLayoutMarbles = "abcd";
withCallViewModel(
constant([aliceParticipant, bobParticipant, daveParticipant]),
constant([aliceRtcMember, bobRtcMember, daveRtcMember]),
of(ConnectionState.Connected),
new Map([
[
{
remoteParticipants$: constant([
aliceParticipant,
behavior(aSpeakingInputMarbles, { y: true, n: false }),
],
[
bobParticipant,
behavior(bSpeakingInputMarbles, { y: true, n: false }),
],
[
daveParticipant,
behavior(dSpeakingInputMarbles, { y: true, n: false }),
],
]),
mockMediaDevices({}),
]),
rtcMembers$: constant([
localRtcMember,
aliceRtcMember,
bobRtcMember,
daveRtcMember,
]),
speaking: new Map([
[aliceParticipant, behavior(aSpeakingInputMarbles, yesNo)],
[bobParticipant, behavior(bSpeakingInputMarbles, yesNo)],
[daveParticipant, behavior(dSpeakingInputMarbles, yesNo)],
]),
},
(vm) => {
schedule(modeInputMarbles, { s: () => vm.setGridMode("spotlight") });
@@ -628,11 +659,10 @@ test("layout enters picture-in-picture mode when requested", () => {
const expectedLayoutMarbles = " aba";
withCallViewModel(
constant([aliceParticipant, bobParticipant]),
constant([aliceRtcMember, bobRtcMember]),
of(ConnectionState.Connected),
new Map(),
mockMediaDevices({}),
{
remoteParticipants$: constant([aliceParticipant, bobParticipant]),
rtcMembers$: constant([localRtcMember, aliceRtcMember, bobRtcMember]),
},
(vm) => {
schedule(pipControlInputMarbles, {
e: () => window.controls.enablePip(),
@@ -670,11 +700,10 @@ test("spotlight remembers whether it's expanded", () => {
const expectedLayoutMarbles = "abcbada";
withCallViewModel(
constant([aliceParticipant, bobParticipant]),
constant([aliceRtcMember, bobRtcMember]),
of(ConnectionState.Connected),
new Map(),
mockMediaDevices({}),
{
remoteParticipants$: constant([aliceParticipant, bobParticipant]),
rtcMembers$: constant([localRtcMember, aliceRtcMember, bobRtcMember]),
},
(vm) => {
schedule(modeInputMarbles, {
s: () => vm.setGridMode("spotlight"),
@@ -727,23 +756,22 @@ test("participants must have a MatrixRTCSession to be visible", () => {
const expectedLayoutMarbles = "a-bc-b";
withCallViewModel(
behavior(scenarioInputMarbles, {
a: [],
b: [bobParticipant],
c: [aliceParticipant, bobParticipant],
d: [aliceParticipant, daveParticipant, bobParticipant],
e: [aliceParticipant, daveParticipant, bobSharingScreen],
}),
behavior(scenarioInputMarbles, {
a: [],
b: [],
c: [aliceRtcMember],
d: [aliceRtcMember, daveRtcMember],
e: [aliceRtcMember, daveRtcMember],
}),
of(ConnectionState.Connected),
new Map(),
mockMediaDevices({}),
{
remoteParticipants$: behavior(scenarioInputMarbles, {
a: [],
b: [bobParticipant],
c: [aliceParticipant, bobParticipant],
d: [aliceParticipant, daveParticipant, bobParticipant],
e: [aliceParticipant, daveParticipant, bobSharingScreen],
}),
rtcMembers$: behavior(scenarioInputMarbles, {
a: [localRtcMember],
b: [localRtcMember],
c: [localRtcMember, aliceRtcMember],
d: [localRtcMember, aliceRtcMember, daveRtcMember],
e: [localRtcMember, aliceRtcMember, daveRtcMember],
}),
},
(vm) => {
vm.setGridMode("grid");
expectObservable(summarizeLayout$(vm.layout$)).toBe(
@@ -780,15 +808,14 @@ test("shows participants without MatrixRTCSession when enabled in settings", ()
const expectedLayoutMarbles = "abc";
withCallViewModel(
behavior(scenarioInputMarbles, {
a: [],
b: [aliceParticipant],
c: [aliceParticipant, bobParticipant],
}),
constant([]), // No one joins the MatrixRTC session
of(ConnectionState.Connected),
new Map(),
mockMediaDevices({}),
{
remoteParticipants$: behavior(scenarioInputMarbles, {
a: [],
b: [aliceParticipant],
c: [aliceParticipant, bobParticipant],
}),
rtcMembers$: constant([localRtcMember]), // No one else joins the MatrixRTC session
},
(vm) => {
vm.setGridMode("grid");
expectObservable(summarizeLayout$(vm.layout$)).toBe(
@@ -827,16 +854,14 @@ it("should show at least one tile per MatrixRTCSession", () => {
const expectedLayoutMarbles = "abcd";
withCallViewModel(
constant([]),
behavior(scenarioInputMarbles, {
a: [],
b: [aliceRtcMember],
c: [aliceRtcMember, daveRtcMember],
d: [daveRtcMember],
}),
of(ConnectionState.Connected),
new Map(),
mockMediaDevices({}),
{
rtcMembers$: behavior(scenarioInputMarbles, {
a: [localRtcMember],
b: [localRtcMember, aliceRtcMember],
c: [localRtcMember, aliceRtcMember, daveRtcMember],
d: [localRtcMember, daveRtcMember],
}),
},
(vm) => {
vm.setGridMode("grid");
expectObservable(summarizeLayout$(vm.layout$)).toBe(
@@ -875,17 +900,20 @@ test("should disambiguate users with the same displayname", () => {
const expectedLayoutMarbles = "abcde";
withCallViewModel(
constant([]),
behavior(scenarioInputMarbles, {
a: [],
b: [aliceRtcMember],
c: [aliceRtcMember, aliceDoppelgangerRtcMember],
d: [aliceRtcMember, aliceDoppelgangerRtcMember, bobRtcMember],
e: [aliceDoppelgangerRtcMember, bobRtcMember],
}),
of(ConnectionState.Connected),
new Map(),
mockMediaDevices({}),
{
rtcMembers$: behavior(scenarioInputMarbles, {
a: [localRtcMember],
b: [localRtcMember, aliceRtcMember],
c: [localRtcMember, aliceRtcMember, aliceDoppelgangerRtcMember],
d: [
localRtcMember,
aliceRtcMember,
aliceDoppelgangerRtcMember,
bobRtcMember,
],
e: [localRtcMember, aliceDoppelgangerRtcMember, bobRtcMember],
}),
},
(vm) => {
expectObservable(vm.memberDisplaynames$).toBe(expectedLayoutMarbles, {
// Carol has no displayname - So userId is used.
@@ -925,14 +953,12 @@ test("should disambiguate users with invisible characters", () => {
const expectedLayoutMarbles = "ab";
withCallViewModel(
constant([]),
behavior(scenarioInputMarbles, {
a: [],
b: [bobRtcMember, bobZeroWidthSpaceRtcMember],
}),
of(ConnectionState.Connected),
new Map(),
mockMediaDevices({}),
{
rtcMembers$: behavior(scenarioInputMarbles, {
a: [localRtcMember],
b: [localRtcMember, bobRtcMember, bobZeroWidthSpaceRtcMember],
}),
},
(vm) => {
expectObservable(vm.memberDisplaynames$).toBe(expectedLayoutMarbles, {
// Carol has no displayname - So userId is used.
@@ -958,14 +984,12 @@ test("should strip RTL characters from displayname", () => {
const expectedLayoutMarbles = "ab";
withCallViewModel(
constant([]),
behavior(scenarioInputMarbles, {
a: [],
b: [daveRtcMember, daveRTLRtcMember],
}),
of(ConnectionState.Connected),
new Map(),
mockMediaDevices({}),
{
rtcMembers$: behavior(scenarioInputMarbles, {
a: [localRtcMember],
b: [localRtcMember, daveRtcMember, daveRTLRtcMember],
}),
},
(vm) => {
expectObservable(vm.memberDisplaynames$).toBe(expectedLayoutMarbles, {
// Carol has no displayname - So userId is used.
@@ -985,17 +1009,16 @@ test("should strip RTL characters from displayname", () => {
});
it("should rank raised hands above video feeds and below speakers and presenters", () => {
withTestScheduler(({ schedule, expectObservable, behavior }) => {
withTestScheduler(({ schedule, expectObservable }) => {
// There should always be one tile for each MatrixRTCSession
const expectedLayoutMarbles = "ab";
withCallViewModel(
constant([aliceParticipant, bobParticipant]),
constant([aliceRtcMember, bobRtcMember]),
of(ConnectionState.Connected),
new Map(),
mockMediaDevices({}),
(vm, { raisedHands$ }) => {
{
remoteParticipants$: constant([aliceParticipant, bobParticipant]),
rtcMembers$: constant([localRtcMember, aliceRtcMember, bobRtcMember]),
},
(vm, _rtcSession, { raisedHands$ }) => {
schedule("ab", {
a: () => {
// We imagine that only two tiles (the first two) will be visible on screen at a time
@@ -1076,10 +1099,10 @@ function rtcMemberJoinLeave$(
) => Observable<CallMembership[]>,
): Observable<CallMembership[]> {
return hot("a-b-c-d", {
a: [], // Start empty
b: [aliceRtcMember], // Alice joins
c: [aliceRtcMember], // Alice still there
d: [], // Alice leaves
a: [localRtcMember], // Start empty
b: [localRtcMember, aliceRtcMember], // Alice joins
c: [localRtcMember, aliceRtcMember], // Alice still there
d: [localRtcMember], // Alice leaves
});
}
@@ -1087,11 +1110,7 @@ test("allOthersLeft$ emits only when someone joined and then all others left", (
withTestScheduler(({ hot, expectObservable, scope }) => {
// Test scenario 1: No one ever joins - should only emit initial false and never emit again
withCallViewModel(
scope.behavior(nooneEverThere$(hot), []),
scope.behavior(nooneEverThere$(hot), []),
of(ConnectionState.Connected),
new Map(),
mockMediaDevices({}),
{ remoteParticipants$: scope.behavior(nooneEverThere$(hot), []) },
(vm) => {
expectObservable(vm.allOthersLeft$).toBe("n------", { n: false });
},
@@ -1102,11 +1121,10 @@ test("allOthersLeft$ emits only when someone joined and then all others left", (
test("allOthersLeft$ emits true when someone joined and then all others left", () => {
withTestScheduler(({ hot, expectObservable, scope }) => {
withCallViewModel(
scope.behavior(participantJoinLeave$(hot), []),
scope.behavior(rtcMemberJoinLeave$(hot), []),
of(ConnectionState.Connected),
new Map(),
mockMediaDevices({}),
{
remoteParticipants$: scope.behavior(participantJoinLeave$(hot), []),
rtcMembers$: scope.behavior(rtcMemberJoinLeave$(hot), []),
},
(vm) => {
expectObservable(vm.allOthersLeft$).toBe(
"n-----u", // false initially, then at frame 6: true then false emissions in same frame
@@ -1120,11 +1138,10 @@ test("allOthersLeft$ emits true when someone joined and then all others left", (
test("autoLeaveWhenOthersLeft$ emits only when autoLeaveWhenOthersLeft option is enabled", () => {
withTestScheduler(({ hot, expectObservable, scope }) => {
withCallViewModel(
scope.behavior(participantJoinLeave$(hot), []),
scope.behavior(rtcMemberJoinLeave$(hot), []),
of(ConnectionState.Connected),
new Map(),
mockMediaDevices({}),
{
remoteParticipants$: scope.behavior(participantJoinLeave$(hot), []),
rtcMembers$: scope.behavior(rtcMemberJoinLeave$(hot), []),
},
(vm) => {
expectObservable(vm.autoLeaveWhenOthersLeft$).toBe(
"------e", // false initially, then at frame 6: true then false emissions in same frame
@@ -1142,11 +1159,10 @@ test("autoLeaveWhenOthersLeft$ emits only when autoLeaveWhenOthersLeft option is
test("autoLeaveWhenOthersLeft$ never emits autoLeaveWhenOthersLeft option is enabled but no-one is there", () => {
withTestScheduler(({ hot, expectObservable, scope }) => {
withCallViewModel(
scope.behavior(nooneEverThere$(hot), []),
scope.behavior(nooneEverThere$(hot), []),
of(ConnectionState.Connected),
new Map(),
mockMediaDevices({}),
{
remoteParticipants$: scope.behavior(nooneEverThere$(hot), []),
rtcMembers$: scope.behavior(nooneEverThere$(hot), []),
},
(vm) => {
expectObservable(vm.autoLeaveWhenOthersLeft$).toBe("-------");
},
@@ -1161,11 +1177,10 @@ test("autoLeaveWhenOthersLeft$ never emits autoLeaveWhenOthersLeft option is ena
test("autoLeaveWhenOthersLeft$ doesn't emit when autoLeaveWhenOthersLeft option is disabled and all others left", () => {
withTestScheduler(({ hot, expectObservable, scope }) => {
withCallViewModel(
scope.behavior(participantJoinLeave$(hot), []),
scope.behavior(rtcMemberJoinLeave$(hot), []),
of(ConnectionState.Connected),
new Map(),
mockMediaDevices({}),
{
remoteParticipants$: scope.behavior(participantJoinLeave$(hot), []),
rtcMembers$: scope.behavior(rtcMemberJoinLeave$(hot), []),
},
(vm) => {
expectObservable(vm.autoLeaveWhenOthersLeft$).toBe("-------");
},
@@ -1180,27 +1195,26 @@ test("autoLeaveWhenOthersLeft$ doesn't emit when autoLeaveWhenOthersLeft option
test("autoLeaveWhenOthersLeft$ doesn't emits when autoLeaveWhenOthersLeft option is enabled and all others left", () => {
withTestScheduler(({ hot, expectObservable, scope }) => {
withCallViewModel(
scope.behavior(
hot("a-b-c-d", {
a: [], // Alone
b: [aliceParticipant], // Alice joins
c: [aliceParticipant],
d: [], // Local joins with a second device
}),
[], //Alice leaves
),
scope.behavior(
hot("a-b-c-d", {
a: [localRtcMember], // Start empty
b: [localRtcMember, aliceRtcMember], // Alice joins
c: [localRtcMember, aliceRtcMember, localRtcMemberDevice2], // Alice still there
d: [localRtcMember, localRtcMemberDevice2], // The second Alice leaves
}),
[],
),
of(ConnectionState.Connected),
new Map(),
mockMediaDevices({}),
{
remoteParticipants$: scope.behavior(
hot("a-b-c-d", {
a: [], // Alone
b: [aliceParticipant], // Alice joins
c: [aliceParticipant],
d: [], // Local joins with a second device
}),
[], //Alice leaves
),
rtcMembers$: scope.behavior(
hot("a-b-c-d", {
a: [localRtcMember], // Start empty
b: [localRtcMember, aliceRtcMember], // Alice joins
c: [localRtcMember, aliceRtcMember, localRtcMemberDevice2], // Alice still there
d: [localRtcMember, localRtcMemberDevice2], // The second Alice leaves
}),
[],
),
},
(vm) => {
expectObservable(vm.autoLeaveWhenOthersLeft$).toBe("------e", {
e: undefined,
@@ -1234,25 +1248,91 @@ test("audio output changes when toggling earpiece mode", () => {
const expectedEarpieceModeMarbles = "n-yn";
const expectedTargetStateMarbles = " sese";
withCallViewModel({ mediaDevices: devices }, (vm) => {
schedule(toggleInputMarbles, {
a: () => getValue(vm.audioOutputSwitcher$)?.switch(),
});
expectObservable(vm.earpieceMode$).toBe(
expectedEarpieceModeMarbles,
yesNo,
);
expectObservable(
vm.audioOutputSwitcher$.pipe(map((switcher) => switcher?.targetOutput)),
).toBe(expectedTargetStateMarbles, { s: "speaker", e: "earpiece" });
});
});
});
test("media tracks are paused while reconnecting to MatrixRTC", () => {
withTestScheduler(({ schedule, expectObservable }) => {
const trackRunning$ = new BehaviorSubject(true);
const originalPublications = localParticipant.trackPublications;
localParticipant.trackPublications = new Map([
[
"video",
{
track: new (class {
public get isUpstreamPaused(): boolean {
return !trackRunning$.value;
}
public async pauseUpstream(): Promise<void> {
trackRunning$.next(false);
return Promise.resolve();
}
public async resumeUpstream(): Promise<void> {
trackRunning$.next(true);
return Promise.resolve();
}
})(),
} as unknown as LocalTrackPublication,
],
]);
onTestFinished(() => {
localParticipant.trackPublications = originalPublications;
});
// There are three indicators that the client might be disconnected from
// MatrixRTC: whether the sync loop is connected, whether the membership is
// present in local room state, and whether the membership manager thinks
// we've hit the timeout for the delayed leave event. Let's test all
// combinations of these conditions.
const syncingMarbles = " nyny----n--y";
const membershipStatusMarbles = " y---ny-n-yn-y";
const probablyLeftMarbles = " n-----y-ny---n";
const expectedReconnectingMarbles = "n-ynyny------n";
const expectedTrackRunningMarbles = "nynynyn------y";
withCallViewModel(
constant([]),
constant([]),
of(ConnectionState.Connected),
new Map(),
devices,
(vm) => {
schedule(toggleInputMarbles, {
a: () => getValue(vm.audioOutputSwitcher$)?.switch(),
{ initialSyncState: SyncState.Reconnecting },
(vm, rtcSession, _subjects, setSyncState) => {
schedule(syncingMarbles, {
y: () => setSyncState(SyncState.Syncing),
n: () => setSyncState(SyncState.Reconnecting),
});
expectObservable(vm.earpieceMode$).toBe(expectedEarpieceModeMarbles, {
n: false,
y: true,
schedule(membershipStatusMarbles, {
y: () => {
rtcSession.membershipStatus = Status.Connected;
},
n: () => {
rtcSession.membershipStatus = Status.Reconnecting;
},
});
expectObservable(
vm.audioOutputSwitcher$.pipe(
map((switcher) => switcher?.targetOutput),
),
).toBe(expectedTargetStateMarbles, { s: "speaker", e: "earpiece" });
schedule(probablyLeftMarbles, {
y: () => {
rtcSession.probablyLeft = true;
},
n: () => {
rtcSession.probablyLeft = false;
},
});
expectObservable(vm.reconnecting$).toBe(
expectedReconnectingMarbles,
yesNo,
);
expectObservable(trackRunning$).toBe(
expectedTrackRunningMarbles,
yesNo,
);
},
);
});

View File

@@ -11,14 +11,19 @@ import {
observeParticipantMedia,
} from "@livekit/components-core";
import {
ConnectionState,
type Room as LivekitRoom,
type LocalParticipant,
LocalVideoTrack,
ParticipantEvent,
type RemoteParticipant,
Track,
} from "livekit-client";
import { RoomStateEvent, type Room, type RoomMember } from "matrix-js-sdk";
import {
ClientEvent,
RoomStateEvent,
SyncState,
type Room as MatrixRoom,
type RoomMember,
} from "matrix-js-sdk";
import {
BehaviorSubject,
EMPTY,
@@ -50,6 +55,8 @@ import {
type CallMembership,
type MatrixRTCSession,
MatrixRTCSessionEvent,
MembershipManagerEvent,
Status,
} from "matrix-js-sdk/lib/matrixrtc";
import { ViewModel } from "./ViewModel";
@@ -60,12 +67,16 @@ import {
import {
LocalUserMediaViewModel,
type MediaViewModel,
observeTrackReference$,
RemoteUserMediaViewModel,
ScreenShareViewModel,
type UserMediaViewModel,
} from "./MediaViewModel";
import { accumulate, finalizeValue } from "../utils/observable";
import {
accumulate,
and$,
finalizeValue,
pauseWhen,
} from "../utils/observable";
import { ObservableScope } from "./ObservableScope";
import {
duplicateTiles,
@@ -263,6 +274,8 @@ class UserMedia {
participant: LocalParticipant | RemoteParticipant | undefined,
encryptionSystem: EncryptionSystem,
livekitRoom: LivekitRoom,
mediaDevices: MediaDevices,
pretendToBeDisconnected$: Behavior<boolean>,
displayname$: Observable<string>,
handRaised$: Observable<Date | null>,
reaction$: Observable<ReactionOption | null>,
@@ -276,6 +289,7 @@ class UserMedia {
this.participant$ as Behavior<LocalParticipant>,
encryptionSystem,
livekitRoom,
mediaDevices,
this.scope.behavior(displayname$),
this.scope.behavior(handRaised$),
this.scope.behavior(reaction$),
@@ -289,6 +303,7 @@ class UserMedia {
>,
encryptionSystem,
livekitRoom,
pretendToBeDisconnected$,
this.scope.behavior(displayname$),
this.scope.behavior(handRaised$),
this.scope.behavior(reaction$),
@@ -342,7 +357,8 @@ class ScreenShare {
member: RoomMember | undefined,
participant: LocalParticipant | RemoteParticipant,
encryptionSystem: EncryptionSystem,
liveKitRoom: LivekitRoom,
livekitRoom: LivekitRoom,
pretendToBeDisconnected$: Behavior<boolean>,
displayName$: Observable<string>,
) {
this.participant$ = new BehaviorSubject(participant);
@@ -352,7 +368,8 @@ class ScreenShare {
member,
this.participant$.asObservable(),
encryptionSystem,
liveKitRoom,
livekitRoom,
pretendToBeDisconnected$,
this.scope.behavior(displayName$),
participant.isLocal,
);
@@ -368,7 +385,7 @@ type MediaItem = UserMedia | ScreenShare;
function getRoomMemberFromRtcMember(
rtcMember: CallMembership,
room: Room,
room: MatrixRoom,
): { id: string; member: RoomMember | undefined } {
// WARN! This is not exactly the sender but the user defined in the state key.
// This will be available once we change to the new "member as object" format in the MatrixRTC object.
@@ -390,18 +407,79 @@ function getRoomMemberFromRtcMember(
// TODO: Move wayyyy more business logic from the call and lobby views into here
export class CallViewModel extends ViewModel {
public readonly localVideo$ = this.scope.behavior<LocalVideoTrack | null>(
observeTrackReference$(
this.livekitRoom.localParticipant,
Track.Source.Camera,
).pipe(
map((trackRef) => {
const track = trackRef?.publication?.track;
return track instanceof LocalVideoTrack ? track : null;
}),
private readonly userId = this.matrixRoom.client.getUserId();
private readonly matrixConnected$ = this.scope.behavior(
// To consider ourselves connected to MatrixRTC, we check the following:
and$(
// The client is connected to the sync loop
(
fromEvent(this.matrixRoom.client, ClientEvent.Sync) as Observable<
[SyncState]
>
).pipe(
startWith([this.matrixRoom.client.getSyncState()]),
map(([state]) => state === SyncState.Syncing),
),
// Room state observed by session says we're connected
fromEvent(
this.matrixRTCSession,
MembershipManagerEvent.StatusChanged,
).pipe(
startWith(null),
map(() => this.matrixRTCSession.membershipStatus === Status.Connected),
),
// Also watch out for warnings that we've likely hit a timeout and our
// delayed leave event is being sent (this condition is here because it
// provides an earlier warning than the sync loop timeout, and we wouldn't
// see the actual leave event until we reconnect to the sync loop)
fromEvent(
this.matrixRTCSession,
MembershipManagerEvent.ProbablyLeft,
).pipe(
startWith(null),
map(() => this.matrixRTCSession.probablyLeft !== true),
),
),
);
private readonly connected$ = this.scope.behavior(
and$(
this.matrixConnected$,
this.livekitConnectionState$.pipe(
map((state) => state === ConnectionState.Connected),
),
),
);
/**
* Whether we should tell the user that we're reconnecting to the call.
*/
public readonly reconnecting$ = this.scope.behavior(
this.connected$.pipe(
// We are reconnecting if we previously had some successful initial
// connection but are now disconnected
scan(
({ connectedPreviously, reconnecting }, connectedNow) => ({
connectedPreviously: connectedPreviously || connectedNow,
reconnecting: connectedPreviously && !connectedNow,
}),
{ connectedPreviously: false, reconnecting: false },
),
map(({ reconnecting }) => reconnecting),
),
);
/**
* Whether various media/event sources should pretend to be disconnected from
* all network input, even if their connection still technically works.
*/
// We do this when the app is in the 'reconnecting' state, because it might be
// that the LiveKit connection is still functional while the homeserver is
// down, for example, and we want to avoid making people worry that the app is
// in a split-brained state.
private readonly pretendToBeDisconnected$ = this.reconnecting$;
/**
* The raw list of RemoteParticipants as reported by LiveKit
*/
@@ -416,7 +494,7 @@ export class CallViewModel extends ViewModel {
private readonly remoteParticipantHolds$ = this.scope.behavior<
RemoteParticipant[][]
>(
this.connectionState$.pipe(
this.livekitConnectionState$.pipe(
withLatestFrom(this.rawRemoteParticipants$),
mergeMap(([s, ps]) => {
// Whenever we switch focuses, we should retain all the previous
@@ -429,7 +507,7 @@ export class CallViewModel extends ViewModel {
// Wait for time to pass and the connection state to have changed
forkJoin([
timer(POST_FOCUS_PARTICIPANT_UPDATE_DELAY_MS),
this.connectionState$.pipe(
this.livekitConnectionState$.pipe(
filter((s) => s !== ECAddonConnectionState.ECSwitchingFocus),
take(1),
),
@@ -453,74 +531,80 @@ export class CallViewModel extends ViewModel {
/**
* The RemoteParticipants including those that are being "held" on the screen
*/
private readonly remoteParticipants$ = this.scope.behavior<
RemoteParticipant[]
>(
combineLatest(
[this.rawRemoteParticipants$, this.remoteParticipantHolds$],
(raw, holds) => {
const result = [...raw];
const resultIds = new Set(result.map((p) => p.identity));
private readonly remoteParticipants$ = this.scope
.behavior<RemoteParticipant[]>(
combineLatest(
[this.rawRemoteParticipants$, this.remoteParticipantHolds$],
(raw, holds) => {
const result = [...raw];
const resultIds = new Set(result.map((p) => p.identity));
// Incorporate the held participants into the list
for (const hold of holds) {
for (const p of hold) {
if (!resultIds.has(p.identity)) {
result.push(p);
resultIds.add(p.identity);
// Incorporate the held participants into the list
for (const hold of holds) {
for (const p of hold) {
if (!resultIds.has(p.identity)) {
result.push(p);
resultIds.add(p.identity);
}
}
}
}
return result;
},
),
);
private readonly memberships$: Observable<CallMembership[]> = merge(
// Handle call membership changes.
fromEvent(this.matrixRTCSession, MatrixRTCSessionEvent.MembershipsChanged),
// Handle room membership changes (and displayname updates)
fromEvent(this.matrixRTCSession.room, RoomStateEvent.Members),
).pipe(
startWith(this.matrixRTCSession.memberships),
map(() => {
return this.matrixRTCSession.memberships;
}),
);
return result;
},
),
)
.pipe(pauseWhen(this.pretendToBeDisconnected$));
/**
* Displaynames for each member of the call. This will disambiguate
* any displaynames that clashes with another member. Only members
* joined to the call are considered here.
*/
public readonly memberDisplaynames$ = this.memberships$.pipe(
map((memberships) => {
const displaynameMap = new Map<string, string>();
const { room } = this.matrixRTCSession;
// It turns out that doing the disambiguation above is rather expensive on Safari (10x slower
// than on Chrome/Firefox). This means it is important that we multicast the result so that we
// don't do this work more times than we need to. This is achieved by converting to a behavior:
public readonly memberDisplaynames$ = this.scope.behavior(
merge(
// Handle call membership changes.
fromEvent(
this.matrixRTCSession,
MatrixRTCSessionEvent.MembershipsChanged,
),
// Handle room membership changes (and displayname updates)
fromEvent(this.matrixRoom, RoomStateEvent.Members),
).pipe(
startWith(null),
map(() => {
const memberships = this.matrixRTCSession.memberships;
const displaynameMap = new Map<string, string>();
const room = this.matrixRoom;
// We only consider RTC members for disambiguation as they are the only visible members.
for (const rtcMember of memberships) {
const matrixIdentifier = `${rtcMember.sender}:${rtcMember.deviceId}`;
const { member } = getRoomMemberFromRtcMember(rtcMember, room);
if (!member) {
logger.error("Could not find member for media id:", matrixIdentifier);
continue;
// We only consider RTC members for disambiguation as they are the only visible members.
for (const rtcMember of memberships) {
const matrixIdentifier = `${rtcMember.sender}:${rtcMember.deviceId}`;
const { member } = getRoomMemberFromRtcMember(rtcMember, room);
if (!member) {
logger.error(
"Could not find member for media id:",
matrixIdentifier,
);
continue;
}
const disambiguate = shouldDisambiguate(member, memberships, room);
displaynameMap.set(
matrixIdentifier,
calculateDisplayName(member, disambiguate),
);
}
const disambiguate = shouldDisambiguate(member, memberships, room);
displaynameMap.set(
matrixIdentifier,
calculateDisplayName(member, disambiguate),
);
}
return displaynameMap;
}),
// It turns out that doing the disambiguation above is rather expensive on Safari (10x slower
// than on Chrome/Firefox). This means it is important that we multicast the result so that we
// don't do this work more times than we need to. This is achieved by converting to a behavior:
return displaynameMap;
}),
pauseWhen(this.pretendToBeDisconnected$),
),
);
public readonly handsRaised$ = this.scope.behavior(this.handsRaisedSubject$);
public readonly handsRaised$ = this.scope.behavior(
this.handsRaisedSubject$.pipe(pauseWhen(this.pretendToBeDisconnected$)),
);
public readonly reactions$ = this.scope.behavior(
this.reactionsSubject$.pipe(
@@ -532,6 +616,7 @@ export class CallViewModel extends ViewModel {
]),
),
),
pauseWhen(this.pretendToBeDisconnected$),
),
);
@@ -549,7 +634,7 @@ export class CallViewModel extends ViewModel {
fromEvent(
this.matrixRTCSession,
MatrixRTCSessionEvent.MembershipsChanged,
).pipe(startWith(null)),
).pipe(startWith(null), pauseWhen(this.pretendToBeDisconnected$)),
showNonMemberTiles.value$,
]).pipe(
scan(
@@ -565,7 +650,7 @@ export class CallViewModel extends ViewModel {
) => {
const newItems = new Map(
function* (this: CallViewModel): Iterable<[string, MediaItem]> {
const room = this.matrixRTCSession.room;
const room = this.matrixRoom;
// m.rtc.members are the basis for calculating what is visible in the call
for (const rtcMember of this.matrixRTCSession.memberships) {
const { member, id: livekitParticipantId } =
@@ -616,6 +701,8 @@ export class CallViewModel extends ViewModel {
participant,
this.options.encryptionSystem,
this.livekitRoom,
this.mediaDevices,
this.pretendToBeDisconnected$,
this.memberDisplaynames$.pipe(
map((m) => m.get(matrixIdentifier) ?? "[👻]"),
),
@@ -639,6 +726,7 @@ export class CallViewModel extends ViewModel {
participant,
this.options.encryptionSystem,
this.livekitRoom,
this.pretendToBeDisconnected$,
this.memberDisplaynames$.pipe(
map((m) => m.get(matrixIdentifier) ?? "[👻]"),
),
@@ -680,6 +768,8 @@ export class CallViewModel extends ViewModel {
participant,
this.options.encryptionSystem,
this.livekitRoom,
this.mediaDevices,
this.pretendToBeDisconnected$,
this.memberDisplaynames$.pipe(
map(
(m) => m.get(participant.identity) ?? "[👻]",
@@ -783,12 +873,13 @@ export class CallViewModel extends ViewModel {
public readonly allOthersLeft$ = this.matrixUserChanges$.pipe(
map(({ userIds, leftUserIds }) => {
const userId = this.matrixRTCSession.room.client.getUserId();
if (!userId) {
logger.warn("Could access client.getUserId to compute allOthersLeft");
if (!this.userId) {
logger.warn("Could not access user ID to compute allOthersLeft");
return false;
}
return userIds.size === 1 && userIds.has(userId) && leftUserIds.size > 0;
return (
userIds.size === 1 && userIds.has(this.userId) && leftUserIds.size > 0
);
}),
startWith(false),
distinctUntilChanged(),
@@ -900,7 +991,7 @@ export class CallViewModel extends ViewModel {
map((speaker) => (speaker ? [speaker] : [])),
);
}),
distinctUntilChanged(shallowEquals),
distinctUntilChanged<MediaViewModel[]>(shallowEquals),
),
);
@@ -1485,10 +1576,11 @@ export class CallViewModel extends ViewModel {
public constructor(
// A call is permanently tied to a single Matrix room and LiveKit room
private readonly matrixRTCSession: MatrixRTCSession,
private readonly matrixRoom: MatrixRoom,
private readonly livekitRoom: LivekitRoom,
private readonly mediaDevices: MediaDevices,
private readonly options: CallViewModelOptions,
private readonly connectionState$: Observable<ECConnectionState>,
private readonly livekitConnectionState$: Observable<ECConnectionState>,
private readonly handsRaisedSubject$: Observable<
Record<string, RaisedHandInfo>
>,
@@ -1497,5 +1589,51 @@ export class CallViewModel extends ViewModel {
>,
) {
super();
// Pause upstream of all local media tracks when we're disconnected from
// MatrixRTC, because it can be an unpleasant surprise for the app to say
// 'reconnecting' and yet still be transmitting your media to others.
// We use matrixConnected$ rather than reconnecting$ because we want to
// pause tracks during the initial joining sequence too until we're sure
// that our own media is displayed on screen.
this.matrixConnected$.pipe(this.scope.bind()).subscribe((connected) => {
const publications =
this.livekitRoom.localParticipant.trackPublications.values();
if (connected) {
for (const p of publications) {
if (p.track?.isUpstreamPaused === true) {
const kind = p.track.kind;
logger.log(
`Resumming ${kind} track (MatrixRTC connection present)`,
);
p.track
.resumeUpstream()
.catch((e) =>
logger.error(
`Failed to resume ${kind} track after MatrixRTC reconnection`,
e,
),
);
}
}
} else {
for (const p of publications) {
if (p.track?.isUpstreamPaused === false) {
const kind = p.track.kind;
logger.log(
`Pausing ${kind} track (uncertain MatrixRTC connection)`,
);
p.track
.pauseUpstream()
.catch((e) =>
logger.error(
`Failed to pause ${kind} track after entering uncertain MatrixRTC connection`,
e,
),
);
}
}
}
});
}
}

View File

@@ -5,14 +5,40 @@ SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { expect, test, vi } from "vitest";
import { expect, onTestFinished, test, vi } from "vitest";
import {
type LocalTrackPublication,
LocalVideoTrack,
TrackEvent,
} from "livekit-client";
import { waitFor } from "@testing-library/dom";
import {
mockLocalParticipant,
mockMediaDevices,
mockRtcMembership,
withLocalMedia,
withRemoteMedia,
withTestScheduler,
} from "../utils/test";
import { getValue } from "../utils/observable";
import { constant } from "./Behavior";
global.MediaStreamTrack = class {} as unknown as {
new (): MediaStreamTrack;
prototype: MediaStreamTrack;
};
global.MediaStream = class {} as unknown as {
new (): MediaStream;
prototype: MediaStream;
};
const platformMock = vi.hoisted(() => vi.fn(() => "desktop"));
vi.mock("../Platform", () => ({
get platform(): string {
return platformMock();
},
}));
const rtcMembership = mockRtcMembership("@alice:example.org", "AAAA");
@@ -79,17 +105,23 @@ test("toggle fit/contain for a participant's video", async () => {
});
test("local media remembers whether it should always be shown", async () => {
await withLocalMedia(rtcMembership, {}, (vm) =>
withTestScheduler(({ expectObservable, schedule }) => {
schedule("-a|", { a: () => vm.setAlwaysShow(false) });
expectObservable(vm.alwaysShow$).toBe("ab", { a: true, b: false });
}),
await withLocalMedia(
rtcMembership,
{},
mockLocalParticipant({}),
mockMediaDevices({}),
(vm) =>
withTestScheduler(({ expectObservable, schedule }) => {
schedule("-a|", { a: () => vm.setAlwaysShow(false) });
expectObservable(vm.alwaysShow$).toBe("ab", { a: true, b: false });
}),
);
// Next local media should start out *not* always shown
await withLocalMedia(
rtcMembership,
{},
mockLocalParticipant({}),
mockMediaDevices({}),
(vm) =>
withTestScheduler(({ expectObservable, schedule }) => {
schedule("-a|", { a: () => vm.setAlwaysShow(true) });
@@ -97,3 +129,77 @@ test("local media remembers whether it should always be shown", async () => {
}),
);
});
test("switch cameras", async () => {
// Camera switching is only available on mobile
platformMock.mockReturnValue("android");
onTestFinished(() => void platformMock.mockReset());
// Construct a mock video track which knows how to be restarted
const track = new LocalVideoTrack({
getConstraints() {},
addEventListener() {},
removeEventListener() {},
} as unknown as MediaStreamTrack);
let deviceId = "front camera";
const restartTrack = vi.fn(async ({ facingMode }) => {
deviceId = facingMode === "user" ? "front camera" : "back camera";
track.emit(TrackEvent.Restarted);
return Promise.resolve();
});
track.restartTrack = restartTrack;
Object.defineProperty(track, "mediaStreamTrack", {
get() {
return {
label: "Video",
getSettings: (): object => ({
deviceId,
facingMode: deviceId === "front camera" ? "user" : "environment",
}),
};
},
});
const selectVideoInput = vi.fn();
await withLocalMedia(
rtcMembership,
{},
mockLocalParticipant({
getTrackPublication() {
return { track } as unknown as LocalTrackPublication;
},
}),
mockMediaDevices({
videoInput: {
available$: constant(new Map()),
selected$: constant(undefined),
select: selectVideoInput,
},
}),
async (vm) => {
// Switch to back camera
getValue(vm.switchCamera$)!();
expect(restartTrack).toHaveBeenCalledExactlyOnceWith({
facingMode: "environment",
});
await waitFor(() => {
expect(selectVideoInput).toHaveBeenCalledTimes(1);
expect(selectVideoInput).toHaveBeenCalledWith("back camera");
});
expect(deviceId).toBe("back camera");
// Switch to front camera
getValue(vm.switchCamera$)!();
expect(restartTrack).toHaveBeenCalledTimes(2);
expect(restartTrack).toHaveBeenLastCalledWith({ facingMode: "user" });
await waitFor(() => {
expect(selectVideoInput).toHaveBeenCalledTimes(2);
expect(selectVideoInput).toHaveBeenLastCalledWith("front camera");
});
expect(deviceId).toBe("front camera");
},
);
});

View File

@@ -16,6 +16,7 @@ import {
import {
type LocalParticipant,
LocalTrack,
LocalVideoTrack,
type Participant,
ParticipantEvent,
type RemoteParticipant,
@@ -27,6 +28,7 @@ import {
RemoteTrack,
} from "livekit-client";
import { type RoomMember } from "matrix-js-sdk";
import { logger } from "matrix-js-sdk/lib/logger";
import {
BehaviorSubject,
type Observable,
@@ -51,6 +53,8 @@ import { accumulate } from "../utils/observable";
import { type EncryptionSystem } from "../e2ee/sharedKeyManagement";
import { E2eeType } from "../e2ee/e2eeType";
import { type ReactionOption } from "../reactions";
import { platform } from "../Platform";
import { type MediaDevices } from "./MediaDevices";
import { type Behavior } from "./Behavior";
export function observeTrackReference$(
@@ -357,10 +361,7 @@ export type UserMediaViewModel =
* Some participant's user media.
*/
abstract class BaseUserMediaViewModel extends BaseMediaViewModel {
/**
* Whether the participant is speaking.
*/
public readonly speaking$ = this.scope.behavior(
private readonly _speaking$ = this.scope.behavior(
this.participant$.pipe(
switchMap((p) =>
p
@@ -372,15 +373,27 @@ abstract class BaseUserMediaViewModel extends BaseMediaViewModel {
),
),
);
/**
* Whether the participant is speaking.
*/
// Getter backed by a private field so that subclasses can override it
public get speaking$(): Behavior<boolean> {
return this._speaking$;
}
/**
* Whether this participant is sending audio (i.e. is unmuted on their side).
*/
public readonly audioEnabled$: Behavior<boolean>;
private readonly _videoEnabled$: Behavior<boolean>;
/**
* Whether this participant is sending video.
*/
public readonly videoEnabled$: Behavior<boolean>;
// Getter backed by a private field so that subclasses can override it
public get videoEnabled$(): Behavior<boolean> {
return this._videoEnabled$;
}
private readonly _cropVideo$ = new BehaviorSubject(true);
/**
@@ -417,7 +430,7 @@ abstract class BaseUserMediaViewModel extends BaseMediaViewModel {
this.audioEnabled$ = this.scope.behavior(
media$.pipe(map((m) => m?.microphoneTrack?.isMuted === false)),
);
this.videoEnabled$ = this.scope.behavior(
this._videoEnabled$ = this.scope.behavior(
media$.pipe(map((m) => m?.cameraTrack?.isMuted === false)),
);
}
@@ -443,20 +456,38 @@ abstract class BaseUserMediaViewModel extends BaseMediaViewModel {
*/
export class LocalUserMediaViewModel extends BaseUserMediaViewModel {
/**
* Whether the video should be mirrored.
* The local video track as an observable that emits whenever the track
* changes, the camera is switched, or the track is muted.
*/
public readonly mirror$ = this.scope.behavior(
private readonly videoTrack$: Observable<LocalVideoTrack | null> =
this.video$.pipe(
switchMap((v) => {
const track = v?.publication?.track;
if (!(track instanceof LocalTrack)) return of(false);
// Watch for track restarts, because they indicate a camera switch
return fromEvent(track, TrackEvent.Restarted).pipe(
startWith(null),
// Mirror only front-facing cameras (those that face the user)
map(() => facingModeFromLocalTrack(track).facingMode === "user"),
if (!(track instanceof LocalVideoTrack)) return of(null);
return merge(
// Watch for track restarts because they indicate a camera switch.
// This event is also emitted when unmuting the track object.
fromEvent(track, TrackEvent.Restarted).pipe(
startWith(null),
map(() => track),
),
// When the track object is muted, reset it to null.
fromEvent(track, TrackEvent.Muted).pipe(map(() => null)),
);
}),
);
/**
* Whether the video should be mirrored.
*/
public readonly mirror$ = this.scope.behavior(
this.videoTrack$.pipe(
// Mirror only front-facing cameras (those that face the user)
map(
(track) =>
track !== null &&
facingModeFromLocalTrack(track).facingMode === "user",
),
),
);
@@ -467,12 +498,48 @@ export class LocalUserMediaViewModel extends BaseUserMediaViewModel {
public readonly alwaysShow$ = alwaysShowSelf.value$;
public readonly setAlwaysShow = alwaysShowSelf.setValue;
/**
* Callback for switching between the front and back cameras.
*/
public readonly switchCamera$: Behavior<(() => void) | null> =
this.scope.behavior(
platform === "desktop"
? of(null)
: this.videoTrack$.pipe(
map((track) => {
if (track === null) return null;
const facingMode = facingModeFromLocalTrack(track).facingMode;
// If the camera isn't front or back-facing, don't provide a switch
// camera shortcut at all
if (facingMode !== "user" && facingMode !== "environment")
return null;
// Restart the track with a camera facing the opposite direction
return (): void =>
void track
.restartTrack({
facingMode: facingMode === "user" ? "environment" : "user",
})
.then(() => {
// Inform the MediaDevices which camera was chosen
const deviceId =
track.mediaStreamTrack.getSettings().deviceId;
if (deviceId !== undefined)
this.mediaDevices.videoInput.select(deviceId);
})
.catch((e) =>
logger.error("Failed to switch camera", facingMode, e),
);
}),
),
);
public constructor(
id: string,
member: RoomMember | undefined,
participant$: Behavior<LocalParticipant | undefined>,
encryptionSystem: EncryptionSystem,
livekitRoom: LivekitRoom,
private readonly mediaDevices: MediaDevices,
displayName$: Behavior<string>,
handRaised$: Behavior<Date | null>,
reaction$: Behavior<ReactionOption | null>,
@@ -514,6 +581,12 @@ export class LocalUserMediaViewModel extends BaseUserMediaViewModel {
* A remote participant's user media.
*/
export class RemoteUserMediaViewModel extends BaseUserMediaViewModel {
// This private field is used to override the value from the superclass
private __speaking$: Behavior<boolean>;
public get speaking$(): Behavior<boolean> {
return this.__speaking$;
}
private readonly locallyMutedToggle$ = new Subject<void>();
private readonly localVolumeAdjustment$ = new Subject<number>();
private readonly localVolumeCommit$ = new Subject<void>();
@@ -553,6 +626,12 @@ export class RemoteUserMediaViewModel extends BaseUserMediaViewModel {
),
);
// This private field is used to override the value from the superclass
private __videoEnabled$: Behavior<boolean>;
public get videoEnabled$(): Behavior<boolean> {
return this.__videoEnabled$;
}
/**
* Whether this participant's audio is disabled.
*/
@@ -566,6 +645,7 @@ export class RemoteUserMediaViewModel extends BaseUserMediaViewModel {
participant$: Observable<RemoteParticipant | undefined>,
encryptionSystem: EncryptionSystem,
livekitRoom: LivekitRoom,
private readonly pretendToBeDisconnected$: Behavior<boolean>,
displayname$: Behavior<string>,
handRaised$: Behavior<Date | null>,
reaction$: Behavior<ReactionOption | null>,
@@ -581,11 +661,33 @@ export class RemoteUserMediaViewModel extends BaseUserMediaViewModel {
reaction$,
);
this.__speaking$ = this.scope.behavior(
pretendToBeDisconnected$.pipe(
switchMap((disconnected) =>
disconnected ? of(false) : super.speaking$,
),
),
);
this.__videoEnabled$ = this.scope.behavior(
pretendToBeDisconnected$.pipe(
switchMap((disconnected) =>
disconnected ? of(false) : super.videoEnabled$,
),
),
);
// Sync the local volume with LiveKit
combineLatest([
participant$,
this.localVolume$.pipe(this.scope.bind()),
]).subscribe(([p, volume]) => p && p.setVolume(volume));
// The local volume, taking into account whether we're supposed to pretend
// that the audio stream is disconnected (since we don't necessarily want
// that to modify the UI state).
this.pretendToBeDisconnected$.pipe(
switchMap((disconnected) => (disconnected ? of(0) : this.localVolume$)),
this.scope.bind(),
),
]).subscribe(([p, volume]) => p?.setVolume(volume));
}
public toggleLocallyMuted(): void {
@@ -625,12 +727,20 @@ export class RemoteUserMediaViewModel extends BaseUserMediaViewModel {
* Some participant's screen share media.
*/
export class ScreenShareViewModel extends BaseMediaViewModel {
/**
* Whether this screen share's video should be displayed.
*/
public readonly videoEnabled$ = this.scope.behavior(
this.pretendToBeDisconnected$.pipe(map((disconnected) => !disconnected)),
);
public constructor(
id: string,
member: RoomMember | undefined,
participant$: Observable<LocalParticipant | RemoteParticipant>,
encryptionSystem: EncryptionSystem,
livekitRoom: LivekitRoom,
private readonly pretendToBeDisconnected$: Behavior<boolean>,
displayname$: Behavior<string>,
public readonly local: boolean,
) {

View File

@@ -83,3 +83,25 @@ borders don't support gradients */
.volumeSlider {
width: 100%;
}
.tile .switchCamera {
opacity: 1;
background: var(--cpd-color-bg-action-secondary-rest);
border: 1px solid var(--cpd-color-border-interactive-secondary);
}
.tile .switchCamera > svg {
color: var(--cpd-color-icon-primary);
}
@media (hover) {
.tile .switchCamera:hover {
background: var(--cpd-color-bg-subtle-secondary);
border-color: var(--cpd-color-border-interactive-hovered);
}
}
.tile .switchCamera:active {
background: var(--cpd-color-bg-subtle-primary);
border-color: var(--cpd-color-border-interactive-hovered);
}

View File

@@ -28,6 +28,7 @@ import {
UserProfileIcon,
ExpandIcon,
VolumeOffSolidIcon,
SwitchCameraSolidIcon,
} from "@vector-im/compound-design-tokens/assets/web/icons";
import {
ContextMenu,
@@ -65,6 +66,7 @@ interface UserMediaTileProps extends TileProps {
vm: UserMediaViewModel;
mirror: boolean;
locallyMuted: boolean;
primaryButton?: ReactNode;
menuStart?: ReactNode;
menuEnd?: ReactNode;
}
@@ -74,6 +76,7 @@ const UserMediaTile: FC<UserMediaTileProps> = ({
vm,
showSpeakingIndicators,
locallyMuted,
primaryButton,
menuStart,
menuEnd,
className,
@@ -160,20 +163,22 @@ const UserMediaTile: FC<UserMediaTileProps> = ({
}
displayName={displayName}
primaryButton={
<Menu
open={menuOpen}
onOpenChange={setMenuOpen}
title={displayName}
trigger={
<button aria-label={t("common.options")}>
<OverflowHorizontalIcon aria-hidden width={20} height={20} />
</button>
}
side="left"
align="start"
>
{menu}
</Menu>
primaryButton ?? (
<Menu
open={menuOpen}
onOpenChange={setMenuOpen}
title={displayName}
trigger={
<button aria-label={t("common.options")}>
<OverflowHorizontalIcon aria-hidden width={20} height={20} />
</button>
}
side="left"
align="start"
>
{menu}
</Menu>
)
}
raisedHandTime={handRaised ?? undefined}
currentReaction={reaction ?? undefined}
@@ -208,6 +213,8 @@ const LocalUserMediaTile: FC<LocalUserMediaTileProps> = ({
const { t } = useTranslation();
const mirror = useBehavior(vm.mirror$);
const alwaysShow = useBehavior(vm.alwaysShow$);
const switchCamera = useBehavior(vm.switchCamera$);
const latestAlwaysShow = useLatest(alwaysShow);
const onSelectAlwaysShow = useCallback(
(e: Event) => {
@@ -223,6 +230,17 @@ const LocalUserMediaTile: FC<LocalUserMediaTileProps> = ({
vm={vm}
locallyMuted={false}
mirror={mirror}
primaryButton={
switchCamera === null ? undefined : (
<button
className={styles.switchCamera}
aria-label={t("switch_camera")}
onClick={switchCamera}
>
<SwitchCameraSolidIcon aria-hidden width={20} height={20} />
</button>
)
}
menuStart={
<ToggleMenuItem
Icon={VisibilityOnIcon}

View File

@@ -85,6 +85,7 @@ unconditionally select the container so we can use cqmin units */
.nameTag {
grid-area: nameTag;
place-self: end start;
padding: var(--cpd-space-1x);
padding-block: var(--cpd-space-1x);
color: var(--cpd-color-text-primary);
@@ -173,7 +174,7 @@ unconditionally select the container so we can use cqmin units */
}
.fg > button:active {
background: var(--cpd-color-bg-action-primary-pressed) !important;
background: var(--cpd-color-bg-action-primary-pressed);
}
.fg > button[data-state="open"] {

View File

@@ -12,6 +12,8 @@ import userEvent from "@testing-library/user-event";
import { SpotlightTile } from "./SpotlightTile";
import {
mockLocalParticipant,
mockMediaDevices,
mockRtcMembership,
withLocalMedia,
withRemoteMedia,
@@ -39,6 +41,8 @@ test("SpotlightTile is accessible", async () => {
rawDisplayName: "Bob",
getMxcAvatarUrl: () => "mxc://dlskf",
},
mockLocalParticipant({}),
mockMediaDevices({}),
async (vm2) => {
const user = userEvent.setup();
const toggleExpanded = vi.fn();

View File

@@ -54,6 +54,7 @@ interface SpotlightItemBaseProps {
targetWidth: number;
targetHeight: number;
video: TrackReferenceOrPlaceholder | undefined;
videoEnabled: boolean;
member: RoomMember | undefined;
unencryptedWarning: boolean;
encryptionStatus: EncryptionStatus;
@@ -63,7 +64,6 @@ interface SpotlightItemBaseProps {
}
interface SpotlightUserMediaItemBaseProps extends SpotlightItemBaseProps {
videoEnabled: boolean;
videoFit: "contain" | "cover";
}
@@ -90,12 +90,10 @@ const SpotlightUserMediaItem: FC<SpotlightUserMediaItemProps> = ({
vm,
...props
}) => {
const videoEnabled = useBehavior(vm.videoEnabled$);
const cropVideo = useBehavior(vm.cropVideo$);
const baseProps: SpotlightUserMediaItemBaseProps &
RefAttributes<HTMLDivElement> = {
videoEnabled,
videoFit: cropVideo ? "cover" : "contain",
...props,
};
@@ -135,6 +133,7 @@ const SpotlightItem: FC<SpotlightItemProps> = ({
const ref = useMergedRefs(ourRef, theirRef);
const displayName = useBehavior(vm.displayName$);
const video = useBehavior(vm.video$);
const videoEnabled = useBehavior(vm.videoEnabled$);
const unencryptedWarning = useBehavior(vm.unencryptedWarning$);
const encryptionStatus = useBehavior(vm.encryptionStatus$);
@@ -160,6 +159,7 @@ const SpotlightItem: FC<SpotlightItemProps> = ({
targetWidth,
targetHeight,
video,
videoEnabled,
member: vm.member,
unencryptedWarning,
displayName,
@@ -169,7 +169,7 @@ const SpotlightItem: FC<SpotlightItemProps> = ({
};
return vm instanceof ScreenShareViewModel ? (
<MediaView videoEnabled videoFit="contain" mirror={false} {...baseProps} />
<MediaView videoFit="contain" mirror={false} {...baseProps} />
) : (
<SpotlightUserMediaItem vm={vm} {...baseProps} />
);

View File

@@ -0,0 +1,24 @@
/*
Copyright 2025 New Vector Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { test } from "vitest";
import { withTestScheduler } from "./test";
import { pauseWhen } from "./observable";
test("pauseWhen", () => {
withTestScheduler(({ behavior, expectObservable }) => {
const inputMarbles = " abcdefgh-i-jk-";
const pauseMarbles = " n-y--n-yn-y--n";
const outputMarbles = "abc--fgh-i---k";
expectObservable(
behavior(inputMarbles).pipe(
pauseWhen(behavior(pauseMarbles, { y: true, n: false })),
),
).toBe(outputMarbles);
});
});

View File

@@ -7,16 +7,23 @@ Please see LICENSE in the repository root for full details.
import {
type Observable,
audit,
combineLatest,
concat,
defer,
filter,
finalize,
map,
of,
scan,
startWith,
takeWhile,
tap,
withLatestFrom,
} from "rxjs";
import { type Behavior } from "../state/Behavior";
const nothing = Symbol("nothing");
/**
@@ -86,3 +93,27 @@ export function getValue<T>(state$: Observable<T>): T {
if (value === nothing) throw new Error("Not a state Observable");
return value;
}
/**
* Creates an Observable that has a value of true whenever all its inputs are
* true.
*/
export function and$(...inputs: Observable<boolean>[]): Observable<boolean> {
return combineLatest(inputs, (...flags) => flags.every((flag) => flag));
}
/**
* RxJS operator that pauses all changes in the input value whenever a Behavior
* is true. When the Behavior returns to being false, the most recently
* suppressed change is emitted as the most recent value.
*/
export function pauseWhen<T>(pause$: Behavior<boolean>) {
return (value$: Observable<T>): Observable<T> =>
value$.pipe(
withLatestFrom(pause$),
audit(([, pause]) =>
pause ? pause$.pipe(filter((pause) => !pause)) : of(null),
),
map(([value]) => value),
);
}

View File

@@ -14,8 +14,13 @@ import { BehaviorSubject, of } from "rxjs";
import { vitest } from "vitest";
import { type RelationsContainer } from "matrix-js-sdk/lib/models/relations-container";
import EventEmitter from "events";
import {
type RoomMember,
type MatrixClient,
type Room,
SyncState,
} from "matrix-js-sdk";
import type { RoomMember, MatrixClient } from "matrix-js-sdk";
import { E2eeType } from "../e2ee/e2eeType";
import { CallViewModel } from "../state/CallViewModel";
import {
@@ -34,10 +39,11 @@ import { type RaisedHandInfo, type ReactionInfo } from "../reactions";
export function getBasicRTCSession(
members: RoomMember[],
initialRemoteRtcMemberships: CallMembership[] = [aliceRtcMember],
initialRtcMemberships: CallMembership[] = [localRtcMember, aliceRtcMember],
): {
rtcSession: MockRTCSession;
remoteRtcMemberships$: BehaviorSubject<CallMembership[]>;
matrixRoom: Room;
rtcMemberships$: BehaviorSubject<CallMembership[]>;
} {
const matrixRoomId = "!myRoomId:example.com";
const matrixRoomMembers = new Map(members.map((p) => [p.userId, p]));
@@ -51,6 +57,7 @@ export function getBasicRTCSession(
client: {
getUserId: () => localRtcMember.sender,
getDeviceId: () => localRtcMember.deviceId,
getSyncState: () => SyncState.Syncing,
sendEvent: vitest.fn().mockResolvedValue({ event_id: "$fake:event" }),
redactEvent: vitest.fn().mockResolvedValue({ event_id: "$fake:event" }),
decryptEventIfNeeded: vitest.fn().mockResolvedValue(undefined),
@@ -91,52 +98,53 @@ export function getBasicRTCSession(
),
});
const remoteRtcMemberships$ = new BehaviorSubject<CallMembership[]>(
initialRemoteRtcMemberships,
const rtcMemberships$ = new BehaviorSubject<CallMembership[]>(
initialRtcMemberships,
);
const rtcSession = new MockRTCSession(
matrixRoom,
localRtcMember,
).withMemberships(remoteRtcMemberships$);
const rtcSession = new MockRTCSession(matrixRoom).withMemberships(
rtcMemberships$,
);
return {
rtcSession,
remoteRtcMemberships$,
matrixRoom,
rtcMemberships$,
};
}
/**
* Construct a basic CallViewModel to test components that make use of it.
* @param members
* @param initialRemoteRtcMemberships
* @param initialRtcMemberships
* @returns
*/
export function getBasicCallViewModelEnvironment(
members: RoomMember[],
initialRemoteRtcMemberships: CallMembership[] = [aliceRtcMember],
initialRtcMemberships: CallMembership[] = [localRtcMember, aliceRtcMember],
): {
vm: CallViewModel;
remoteRtcMemberships$: BehaviorSubject<CallMembership[]>;
rtcMemberships$: BehaviorSubject<CallMembership[]>;
rtcSession: MockRTCSession;
handRaisedSubject$: BehaviorSubject<Record<string, RaisedHandInfo>>;
reactionsSubject$: BehaviorSubject<Record<string, ReactionInfo>>;
} {
const { rtcSession, remoteRtcMemberships$ } = getBasicRTCSession(
const { rtcSession, matrixRoom, rtcMemberships$ } = getBasicRTCSession(
members,
initialRemoteRtcMemberships,
initialRtcMemberships,
);
const handRaisedSubject$ = new BehaviorSubject({});
const reactionsSubject$ = new BehaviorSubject({});
const remoteParticipants$ = of([aliceParticipant]);
const liveKitRoom = mockLivekitRoom(
const livekitRoom = mockLivekitRoom(
{ localParticipant },
{ remoteParticipants$ },
);
const vm = new CallViewModel(
rtcSession as unknown as MatrixRTCSession,
liveKitRoom,
matrixRoom,
livekitRoom,
mockMediaDevices({}),
{
encryptionSystem: { kind: E2eeType.PER_PARTICIPANT },
@@ -147,7 +155,7 @@ export function getBasicCallViewModelEnvironment(
);
return {
vm,
remoteRtcMemberships$,
rtcMemberships$,
rtcSession,
handRaisedSubject$: handRaisedSubject$,
reactionsSubject$: reactionsSubject$,

View File

@@ -19,8 +19,11 @@ import {
type Focus,
MatrixRTCSessionEvent,
type MatrixRTCSessionEventHandlerMap,
MembershipManagerEvent,
type SessionMembershipData,
Status,
} from "matrix-js-sdk/lib/matrixrtc";
import { type MembershipManagerEventHandlerMap } from "matrix-js-sdk/lib/matrixrtc/IMembershipManager";
import {
type LocalParticipant,
type LocalTrackPublication,
@@ -233,6 +236,7 @@ export function mockLocalParticipant(
): LocalParticipant {
return {
isLocal: true,
trackPublications: new Map(),
getTrackPublication: () =>
({}) as Partial<LocalTrackPublication> as LocalTrackPublication,
...mockEmitter(),
@@ -243,9 +247,10 @@ export function mockLocalParticipant(
export async function withLocalMedia(
localRtcMember: CallMembership,
roomMember: Partial<RoomMember>,
localParticipant: LocalParticipant,
mediaDevices: MediaDevices,
continuation: (vm: LocalUserMediaViewModel) => void | Promise<void>,
): Promise<void> {
const localParticipant = mockLocalParticipant({});
const vm = new LocalUserMediaViewModel(
"local",
mockMatrixRoomMember(localRtcMember, roomMember),
@@ -254,6 +259,7 @@ export async function withLocalMedia(
kind: E2eeType.PER_PARTICIPANT,
},
mockLivekitRoom({ localParticipant }),
mediaDevices,
constant(roomMember.rawDisplayName ?? "nodisplayname"),
constant(null),
constant(null),
@@ -293,6 +299,7 @@ export async function withRemoteMedia(
kind: E2eeType.PER_PARTICIPANT,
},
mockLivekitRoom({}, { remoteParticipants$: of([remoteParticipant]) }),
constant(false),
constant(roomMember.rawDisplayName ?? "nodisplayname"),
constant(null),
constant(null),
@@ -314,8 +321,10 @@ export function mockConfig(config: Partial<ResolvedConfigOptions> = {}): void {
}
export class MockRTCSession extends TypedEventEmitter<
MatrixRTCSessionEvent | RoomAndToDeviceEvents,
MatrixRTCSessionEventHandlerMap & RoomAndToDeviceEventsHandlerMap
MatrixRTCSessionEvent | RoomAndToDeviceEvents | MembershipManagerEvent,
MatrixRTCSessionEventHandlerMap &
RoomAndToDeviceEventsHandlerMap &
MembershipManagerEventHandlerMap
> {
public readonly statistics = {
counters: {},
@@ -325,7 +334,6 @@ export class MockRTCSession extends TypedEventEmitter<
public constructor(
public readonly room: Room,
private localMembership: CallMembership,
public memberships: CallMembership[] = [],
) {
super();
@@ -341,14 +349,37 @@ export class MockRTCSession extends TypedEventEmitter<
): MockRTCSession {
rtcMembers$.subscribe((m) => {
const old = this.memberships;
// always prepend the local participant
const updated = [this.localMembership, ...(m as CallMembership[])];
this.memberships = updated;
this.emit(MatrixRTCSessionEvent.MembershipsChanged, old, updated);
this.memberships = m as CallMembership[];
this.emit(
MatrixRTCSessionEvent.MembershipsChanged,
old,
this.memberships,
);
});
return this;
}
private _membershipStatus = Status.Connected;
public get membershipStatus(): Status {
return this._membershipStatus;
}
public set membershipStatus(value: Status) {
const prev = this._membershipStatus;
this._membershipStatus = value;
if (value !== prev)
this.emit(MembershipManagerEvent.StatusChanged, prev, value);
}
private _probablyLeft = false;
public get probablyLeft(): boolean {
return this._probablyLeft;
}
public set probablyLeft(value: boolean) {
const prev = this._probablyLeft;
this._probablyLeft = value;
if (value !== prev) this.emit(MembershipManagerEvent.ProbablyLeft, value);
}
}
export const mockTrack = (identity: string): TrackReference =>

View File

@@ -120,8 +120,8 @@ export const widget = ((): WidgetHelpers | null => {
const sendState = [
userId, // Legacy call membership events
`_${userId}_${deviceId}`, // Session membership events
`${userId}_${deviceId}`, // The above with no leading underscore, for room versions whose auth rules allow it
`_${userId}_${deviceId}_m.call`, // Session membership events
`${userId}_${deviceId}_m.call`, // The above with no leading underscore, for room versions whose auth rules allow it
].map((stateKey) => ({
eventType: EventType.GroupCallMemberPrefix,
stateKey,

View File

@@ -10046,8 +10046,8 @@ __metadata:
linkType: hard
"livekit-client@npm:^2.13.0":
version: 2.15.5
resolution: "livekit-client@npm:2.15.5"
version: 2.15.6
resolution: "livekit-client@npm:2.15.6"
dependencies:
"@livekit/mutex": "npm:1.1.1"
"@livekit/protocol": "npm:1.39.3"
@@ -10060,7 +10060,7 @@ __metadata:
webrtc-adapter: "npm:^9.0.1"
peerDependencies:
"@types/dom-mediacapture-record": ^1
checksum: 10c0/52a70bdd39d802737ed7c25ae5d06daf9921156c4fc74f918009e86204430b2d200b66c55cefab949be4e5411cbc4d25eac92976f62f96b7226057a5b0706baa
checksum: 10c0/f1ab6cdf2b85647036e9de906734c1394dac497da0bd879a29d0c587c437ada262021478fcef24df99b3489a39d97fe67ab33de0785ed0a63335da2fef577192
languageName: node
linkType: hard
@@ -10279,8 +10279,8 @@ __metadata:
linkType: hard
"matrix-js-sdk@github:matrix-org/matrix-js-sdk#head=develop":
version: 37.12.0
resolution: "matrix-js-sdk@https://github.com/matrix-org/matrix-js-sdk.git#commit=c4e1e0723e3674725f8ed9cda5be7a7efe304e29"
version: 37.13.0
resolution: "matrix-js-sdk@https://github.com/matrix-org/matrix-js-sdk.git#commit=3a33c658bbcb8ce8791ec066db899f2571f5c52f"
dependencies:
"@babel/runtime": "npm:^7.12.5"
"@matrix-org/matrix-sdk-crypto-wasm": "npm:^15.1.0"
@@ -10296,7 +10296,7 @@ __metadata:
sdp-transform: "npm:^2.14.1"
unhomoglyph: "npm:^1.0.6"
uuid: "npm:11"
checksum: 10c0/e8f97b90ded9e46bdd803dde9861e178c76741999c224cbc869ad38695f300702405e5c365e270663b90e2a317b646bbf2b9e9bd7d51ffce36e6d2ca3ab29ed9
checksum: 10c0/1db0d39cfbe4f1c69c8acda0ea7580a4819fc47a7d4bff057382e33e72d9a610f8c03043a6c00bc647dfdc2815aa643c69d25022fb759342a92b77e1841524f1
languageName: node
linkType: hard