Compare commits

...
Author SHA1 Message Date
Robin b27c11c381 Always use OpenID auth flow if available
This removes the notion of an auth flow fallback for now. We will no longer try the CS API when authenticating with transports that include both a `url` and `livekit_service_url`.
2026-10-28 19:16:32 +01:00
Robin c233ac6e70 Improve names of transport shape end-to-end tests 2026-10-28 14:51:27 +01:00
renovate[bot] 6793a4b46e Update ghcr.io/element-hq/element-web:develop Docker digest to ee8ce55 (#4380)
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
2026-10-28 02:32:09 +00:00
Robin 619f5ac3a2 Merge pull request #4375 from element-hq/robin/stats-crash
Fix crash when joining call with connection stats enabled
2026-10-27 18:08:31 +01:00
Robin 5780597b83 Fix crash when joining call with connection stats enabled
As demonstrated by the test. This crashed with "Behavior failed to synchronously emit an initial value."
2026-10-27 16:17:58 +01:00
Robin 76da59be40 Merge pull request #4301 from element-hq/floating-tile
Make drag & drop of floating tiles less janky
2026-10-27 12:16:12 +01:00
Robin c6ad8b2ac7 Use the right comparators 2026-10-27 11:45:35 +01:00
Robin d33a1eb059 Merge branch 'floating-tile' of github.com:element-hq/element-call into floating-tile 2026-10-27 11:04:12 +01:00
Robin 9fcba5a704 Skip redundant tile wrapper updates 2026-10-27 10:44:33 +01:00
Johannes Marbach 96333dbc3a Merge branch 'main' into floating-tile 2026-10-26 13:46:54 +01:00
Robin ed6a0e1341 Merge pull request #4371 from element-hq/ellipsize-title
Ellipsise app bar text on mobile
2026-10-26 13:45:06 +01:00
renovate[bot]andJohannes Marbach 90ccf62d5c Update ghcr.io/element-hq/element-web:develop Docker digest to c9ffe3e (#4356)
* Update ghcr.io/element-hq/element-web:develop Docker digest to 79e47d8

* Adapt to new recovery popup button title

---------

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: Johannes Marbach <n0-0ne+github@mailbox.org>
2026-10-26 12:44:39 +00:00
Robin dde07f6d54 Ellipsise app bar text on mobile
We already had text-overflow: ellipsis set on the title and subtitle elements of the app bar, but due to an annoying quirk of place-items: center, this had no effect. place-items: center stretch seems to give the text elements the right size constraints.
2026-10-26 13:25:13 +01:00
Robin 9ec0cef9c5 Merge pull request #4370 from element-hq/reintroduce-livekit-alias
Reintroduce livekit_alias in published transports
2026-10-26 12:39:14 +01:00
Robin c248ff4280 Don't add livekit_alias to newer transport format 2026-10-26 11:41:45 +01:00
Robin 0f1d0941d0 Reference matrix-rust-sdk PR making livekit_alias optional, too 2026-10-26 11:17:16 +01:00
Robin 7e6e1698c0 Reintroduce livekit_alias in published transports
Some Ruma-based clients still expect it to be present. We should remove it again as soon as https://github.com/ruma/ruma/pull/2642 is widely deployed.
2026-10-26 10:57:49 +01:00
Johannes Marbach de05b82cfe Merge pull request #4369 from element-hq/renovate/npm-storybook-vulnerability
Update dependency storybook to v10.6.2 [SECURITY]
2026-10-26 10:16:21 +01:00
Johannes Marbach b0e985c014 Update related storybook dependencies 2026-10-26 09:57:30 +01:00
Johannes Marbach 33368ab8da Merge pull request #4361 from element-hq/renovate/compound
Update dependency @vector-im/compound-design-tokens to v11.3.1
2026-10-26 09:49:54 +01:00
Johannes Marbach d85a4a1929 Merge pull request #4360 from element-hq/renovate/livekit-client
Update dependency livekit-client to v2.23.0
2026-10-26 09:48:08 +01:00
renovate[bot] a6930c5aa9 Update dependency storybook to v10.6.2 [SECURITY] 2026-10-26 08:31:56 +00:00
Johannes Marbach e83093feb2 Merge pull request #4365 from element-hq/renovate/npm-vite-vulnerability
Update dependency vite to v8.2.4 [SECURITY]
2026-10-26 09:30:23 +01:00
renovate[bot] c50127ab62 Update dependency vite to v8.2.4 [SECURITY] 2026-10-22 17:44:29 +00:00
renovate[bot] c3ae1c516b Update dependency livekit-client to v2.23.0 2026-10-21 14:25:29 +00:00
renovate[bot] b975dce53e Update dependency @vector-im/compound-design-tokens to v11.3.1 2026-10-19 01:30:27 +00:00
Johannes Marbach 8be229a438 Merge pull request #4354 from element-hq/toger5/johennes/rtc-remote-publish
Fall back to publishing on a remote SFU if possible
2026-10-15 16:27:36 +02:00
Timo K. ef7dce7b07 Update url-transport.test.ts 2026-10-15 16:10:02 +02:00
Timo K. 2de5ee2df5 Update test to consider EWs support for cs-api 2026-10-15 12:53:37 +02:00
Timo K. a427206792 Only return null if there is no endpoint available. 2026-10-15 12:42:36 +02:00
Timo K. 1913a099ab bump EW to inlcude the new widget capabilities 2026-10-15 11:28:45 +02:00
TimoandJohannes Marbach 93b893527e Update sdk/src/media-backend/livekit/RemotePublishTransport.ts
Co-authored-by: Johannes Marbach <n0-0ne+github@mailbox.org>
2026-10-15 09:28:40 +02:00
Timo K. 2b26fd472f fmt 2026-10-14 19:03:22 +02:00
Timo K. 281fff0e9f array -> prop|null in discoverLocalTransport 2026-10-14 19:01:14 +02:00
Johannes Marbach bc557dfb5d Revert "Temporarily enable full output to debug the Claude action"
This reverts commit 4827ab5fdc.
2026-10-14 17:55:35 +02:00
Johannes Marbach 4827ab5fdc Temporarily enable full output to debug the Claude action 2026-10-14 17:50:20 +02:00
Timo K. 7af6525f01 fix lints 2026-10-14 16:44:31 +02:00
Timo K. 3b763d87f2 make the transport discovery lk edgecase a EC exclusive workaround
Dont add it to the sdk.
2026-10-14 16:40:27 +02:00
Timo K.andClaude Fable 5.1 8638d4f159 Fall back to publishing on a remote SFU if possible
Port of edf1feae from johannes/rtc-remote-publish onto the SDK split,
where
the MatrixRTC logic lives in sdk/ rather than src/state/CallViewModel/.

Where the original lands:

- getLocalTransport's remote fallback: the LiveKit backend's new
  findRemotePublishTransport (sdk/src/media-backend/livekit/
  RemotePublishTransport.ts), which tries the candidates from the
  js-sdk's
  getRemoteLivekitPublishCandidates oldest first and keeps the first
  whose
  homeserver issues a token with the publish grant.
  discoverLocalTransport
  asks the backend for it after the host's fallback, with a snapshot of
  the
  session's memberships from RTCParticipation, and throws
  MatrixRTCTransportMissingError where that finds nothing too. Transport
  discovery stays backend neutral: the MSC4195 token check is the
  backend's.
- LocalMember's "no delegation on a remote SFU" and the custom
  transport's
  server_name: LivekitBackend.prepareLocalTransport locates the
  transport on
  the homeserver it names (serverNameOfTransport) and skips the
  delegation
  probe where that is not our own.
- SessionBehaviors' membership.getTransportServerName: the same helper
  in
  LivekitBackend.mediaFor.
- decodeSFUJWTPayload, SFUJWTPayload.video.canPublish and the exported
  isCsApiUnavailable: sdk/src/media-backend/livekit/auth/.
- DeveloperSettingsTab, the widget capabilities, the locale string and
  the
  js-sdk pin (9405287c, 44.0.0): applied as is. The error boundary
  snapshots follow the string.

Tests: the LocalTransport cases go to RemotePublishTransport.test.ts and
LocalTransport.test.ts, the LocalMember remote case and the
SessionBehaviors
server-name case to LivekitBackend.test.ts, plus the memberships
snapshot in
RTCParticipation.test.ts.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-14 16:09:11 +02:00
Robin 4441bb5e8c Add snapshot coverage for all call layouts
Snapshotting the layout templates so that they get reviewed as a whole feels a little more straightforward than the existing tests.
2026-09-29 03:20:14 +02:00
Robin f2f49315eb Make drag & drop performant in all layouts
While also giving the logic for drag & drop of a floating tile a common home in the view model layer.
2026-09-28 19:53:30 +02:00
Robin 99256fa01c Make drag & drop of floating tiles less janky
This fixes an issue where upon dragging a floating tile to a different corner of the interface, it would briefly move toward its original position before jerking back in the intended direction.
2026-09-28 19:48:49 +02:00
52 changed files with 1973 additions and 942 deletions
+2 -2
View File
@@ -19,7 +19,7 @@ services:
- ./backend/playwright_homeserver-othersite.yaml:/data/cfg/homeserver.yaml:Z
element-web:
# Pin to a SHA so that upstream cannot break our tests. Renovate handles regular updates.
image: ghcr.io/element-hq/element-web:develop@sha256:e464b2aca73e248195b24b8c6f62cd331100302284d60cc735405807fb2730be
image: ghcr.io/element-hq/element-web:develop@sha256:ee8ce55012dda2c097147859aece32802ce7a9174e40c3711c56f75ccfe1b169
element-web-1:
# Pin to a SHA so that upstream cannot break our tests. Renovate handles regular updates.
image: ghcr.io/element-hq/element-web:develop@sha256:e464b2aca73e248195b24b8c6f62cd331100302284d60cc735405807fb2730be
image: ghcr.io/element-hq/element-web:develop@sha256:ee8ce55012dda2c097147859aece32802ce7a9174e40c3711c56f75ccfe1b169
+1 -1
View File
@@ -106,7 +106,7 @@
"insufficient_capacity_description": "The server has reached its maximum capacity and you cannot join the call at this time. Try again later, or contact your server admin if the problem persists.",
"livekit_connection_error": "Failed to connect to Livekit server",
"livekit_connection_error_description": "An error occurred while connecting to the Livekit server (<1>Reason:</1> <2>{{ reason }}</2>).",
"matrix_rtc_transport_missing": "The server is not configured to work with {{brand}}. Please contact your server admin (Domain: {{domain}}, Error Code: {{ errorCode }}).",
"matrix_rtc_transport_missing": "The server is not configured to work with {{brand}} and no other call participant provides a usable transport. Please contact your server admin (Domain: {{domain}}, Error Code: {{ errorCode }}).",
"membership_manager": "Membership Manager Error",
"membership_manager_description": "The Membership Manager had to shut down. This is caused by many consecutive failed network requests.",
"no_matrix_2_authorization_service": "The authorization service for your media server (SFU) is out of date.",
+1 -1
View File
@@ -107,7 +107,7 @@
"livekit-client": "^2.18.1",
"lodash-es": "^4.17.21",
"loglevel": "^1.9.1",
"matrix-js-sdk": "github:matrix-org/matrix-js-sdk#develop",
"matrix-js-sdk": "github:matrix-org/matrix-js-sdk#3f05dc4d104a3dfc753c5e3549a18c74fab4b076",
"matrix-widget-api": "^1.18.0",
"node-stdlib-browser": "^1.3.1",
"normalize.css": "^8.0.1",
+16 -47
View File
@@ -31,7 +31,11 @@ interface Transport {
*/
const shapes = {
url: { type: "livekit", url: sfuUrl },
both: { type: "livekit", url: sfuUrl, livekit_service_url: serviceUrl },
"url+legacy": {
type: "livekit",
url: sfuUrl,
livekit_service_url: serviceUrl,
},
legacy: null,
} satisfies Record<string, Transport | null>;
type Shape = keyof typeof shapes;
@@ -56,15 +60,15 @@ const cases: {
{
// Publishers keep the legacy participant identity that clients predating
// MSC4195 expect, which only the JWT service assigns.
hostShape: "both",
guestShape: "both",
hostShape: "url+legacy",
guestShape: "url+legacy",
hostFlows: ["service-legacy"],
guestFlows: ["service-legacy"],
},
{
// The MatrixRTC mode has no bearing on the choice.
hostShape: "both",
guestShape: "both",
hostShape: "url+legacy",
guestShape: "url+legacy",
mode: "2_0",
hostFlows: ["service-legacy"],
guestFlows: ["service-legacy"],
@@ -75,6 +79,12 @@ const cases: {
hostFlows: ["cs-api", "service-default"],
guestFlows: ["service-legacy", "cs-api"],
},
{
hostShape: "url",
guestShape: "url+legacy",
hostFlows: ["cs-api", "service-default"],
guestFlows: ["service-legacy", "cs-api"],
},
];
for (const { hostShape, guestShape, mode, hostFlows, guestFlows } of cases) {
@@ -126,47 +136,6 @@ for (const { hostShape, guestShape, mode, hostFlows, guestFlows } of cases) {
});
}
test("Subscriber falls back to the JWT service when the homeserver lacks MSC4195", async ({
browser,
page,
browserName,
}) => {
skipOnFirefox(browserName);
// The host starts the call with both transport shapes.
await advertiseTransport(page, shapes.both);
await page.goto("/");
await SpaHelpers.createCall(page, "Androl", "HelloCall", true);
const inviteLink = await SpaHelpers.getCallInviteLink(page);
// The guest joins but their C-S requests fail.
const guestContext = await browser.newContext({ reducedMotion: "reduce" });
const guestPage = await guestContext.newPage();
await rejectCsApiTokens(guestPage);
const guestRequests = recordAuthRequests(guestPage);
await SpaHelpers.joinCallFromInviteLink(guestPage, inviteLink, "Pevara");
// Wait for the call to connect and render.
await SpaHelpers.expectVideoTilesCount(page, 2);
await SpaHelpers.expectVideoTilesCount(guestPage, 2);
// The guest ran through all flows.
expect(flowsOf(guestRequests)).toEqual(
new Set<Flow>(["cs-api", "service-legacy", "service-default"]),
);
// The C-S token requests failed.
for (const request of guestRequests) {
expect(request.status).toBe(request.flow === "cs-api" ? 404 : 200);
}
// The C-S token requests preceeded the fallback request.
const flows = guestRequests.map((r) => r.flow);
expect(flows.indexOf("service-default")).toBeGreaterThan(
flows.indexOf("cs-api"),
);
});
test("Publisher shows an error when the homeserver lacks MSC4195", async ({
page,
browserName,
@@ -198,7 +167,7 @@ test("Publisher shows an error when the homeserver lacks MSC4195", async ({
const delegationCases: { shape: Shape; via: Delegation["via"] }[] = [
{ shape: "url", via: "homeserver" },
{ shape: "legacy", via: "service" },
{ shape: "both", via: "service" },
{ shape: "url+legacy", via: "service" },
];
for (const { shape, via } of delegationCases) {
+6 -2
View File
@@ -159,7 +159,7 @@ export class TestHelpers {
public static async dismissStartupToasts(page: Page): Promise<void> {
const expectedToasts = [
{ title: "Failed to load service worker", button: "OK" },
{ title: "Back up your chats", button: "Dismiss" },
{ title: "Back up your chats", button: "Not now" },
{ title: "Turn on key storage", button: "Dismiss" },
{ title: "Element does not support this browser", button: "Dismiss" },
];
@@ -178,7 +178,11 @@ export class TestHelpers {
);
if (toastConfig) {
await toast.getByRole("button", { name: toastConfig.button }).click();
// Bounded so that a renamed button leaves the toast in place and the
// next step reports it, instead of hanging until the test timeout.
await toast
.getByRole("button", { name: toastConfig.button })
.click({ timeout: 5000 });
} else {
// Unknown toast. We don't want to act on unknown toasts
break;
+5 -12
View File
@@ -11,13 +11,12 @@ import { widgetTest } from "../fixtures/widget-user.ts";
import { HOST1, TestHelpers } from "./test-helpers.ts";
const sfuUrl = "wss://matrix-rtc.m.localhost/livekit/sfu";
const serviceUrl = "https://matrix-rtc.m.localhost/livekit/jwt";
/** Which endpoint a page asks for its SFU tokens. */
type Flow = "cs-api" | "service-legacy" | "service-default";
widgetTest(
"Call between a url + livekit_service_url transport and a legacy one",
"Call between a url transport and a legacy one",
async ({ addUser, browserName }) => {
test.skip(
browserName === "firefox",
@@ -30,12 +29,10 @@ widgetTest(
addUser("Pevara", HOST1),
]);
// The host's homeserver has started advertising a `url` next to its JWT
// service; the guest's still advertises the JWT service alone.
// Host advertises a url transport, guest still advertises a legacy one
await advertiseTransport(host.page, {
type: "livekit",
url: sfuUrl,
livekit_service_url: serviceUrl,
});
// Element Web answers the widget from a cache it filled at login, so make
// it fetch the transports again now that they are rerouted
@@ -66,13 +63,9 @@ widgetTest(
await TestHelpers.expectVisibleVideoCount(frame, 2);
}
// Element Web does not expose the MSC4195 endpoints to widgets yet, so
// neither side reached the CS API. The host published through the JWT
// service's legacy endpoint and subscribed to the guest through its
// default one; the guest published the same way and fell back to the
// default endpoint for the host's transport.
expect(hostFlows).toEqual(new Set(["service-legacy", "service-default"]));
expect(guestFlows).toEqual(new Set(["service-legacy", "service-default"]));
// Element Web used for Playwright does support MSC4195 for widgets.
expect(hostFlows).toEqual(new Set(["cs-api", "service-default"]));
expect(guestFlows).toEqual(new Set(["service-legacy", "cs-api"]));
},
);
+541 -270
View File
File diff suppressed because it is too large Load Diff
+6 -1
View File
@@ -101,6 +101,8 @@ LiveKit participants.
│ create ───▶│ │ │
│ │ GET rtc/transports (session) ────────▶│ │
│ │◀── first transport of the backend's type │
│ │ (none, LiveKit only: a member's SFU │ │
│ │ that lets us publish, get_token) ─▶│ │
│ │ prepareLocalTransport (backend): │ │
│ │ publisher token: CS API get_token, │ │
│ │ or OpenID token then /sfu/get ─────▶│ │
@@ -556,7 +558,10 @@ export interface MediaBackend {
```
What stays on the client's side: discovery of the local transport from the
homeserver's list (a homeserver endpoint, so not a backend's), the membership
homeserver's list (a homeserver endpoint, so not a backend's) with, for the
LiveKit backend only, a last resort of another member's SFU that lets us
publish (`findRemotePublishLivekitTransport`, wired in by `RTCParticipation` rather
than by the backend interface, since it is an MSC4195 matter), the membership
state machine in `LocalMembership.ts` and its combination with
`local.connectionState$` into `state$`, the choice of delayed leave timings,
mapping a data packet's sender to a member, and the transport registry, which keys on the raw
+9
View File
@@ -0,0 +1,9 @@
lockfileVersion: '9.0'
settings:
autoInstallPeers: false
excludeLinksFromLockfile: false
importers:
.: {}
+61 -6
View File
@@ -8,9 +8,11 @@ Please see LICENSE in the repository root for full details.
import { type MatrixClient, type Room, SyncState } from "matrix-js-sdk";
import { logger } from "matrix-js-sdk/lib/logger";
import {
type CallMembership,
MatrixRTCSessionEvent,
MembershipManagerEvent,
type Transport,
type UnstableLivekitTransport,
} from "matrix-js-sdk/lib/matrixrtc";
import { BehaviorSubject, EMPTY, Subject } from "rxjs";
import { describe, expect, it, type Mock, vi } from "vitest";
@@ -38,13 +40,32 @@ import { constant } from "./reactive/Behavior";
import { ObservableScope } from "./reactive/ObservableScope";
import { createRTCParticipation } from "./RTCParticipation";
import { transportId } from "./matrixrtc/Transports";
import { flushPromises, mockEmitter, MockRTCSession } from "./utils/test";
import {
flushPromises,
mockEmitter,
mockRtcMembership,
MockRTCSession,
} from "./utils/test";
const transport: Transport = {
const transport: UnstableLivekitTransport = {
type: "livekit",
livekit_service_url: "https://lk.example.org",
};
const transportWithAlias: Transport = {
livekit_alias: "!room:example.org",
...transport,
};
const userId = "@alice:example.org";
/** A token with the publish grant, as a homeserver issues it for its own SFU. */
const publisherJWT = [
{},
{ sub: "identity", video: { room: "room", canPublish: true } },
{},
]
.map((d) => global.btoa(JSON.stringify(d)))
.join(".");
const deviceId = "AAAA";
const legacyMemberId = `${userId}:${deviceId}`;
@@ -91,6 +112,13 @@ interface FakeBackend {
function setup(
options: Partial<RTCParticipationOptions> = {},
slotOptions: { matrixRTCMode?: MatrixRTCMode } = {},
{
homeserverTransports = [transport],
memberships = [],
}: {
homeserverTransports?: Transport[];
memberships?: CallMembership[];
} = {},
): {
scope: ObservableScope;
participation: ReturnType<typeof createRTCParticipation>["participation"];
@@ -103,7 +131,7 @@ function setup(
} {
const scope = new ObservableScope();
const room = { roomId: "!room:example.org" } as Room;
const session = new MockRTCSession(room);
const session = new MockRTCSession(room, memberships);
const mockedSession = session.asMockedSession();
const join = vi.spyOn(session, "join");
const client = {
@@ -112,7 +140,8 @@ function setup(
getDeviceId: () => deviceId,
getDomain: () => "example.org",
getSyncState: () => SyncState.Syncing,
_unstable_getRTCTransports: vi.fn().mockResolvedValue([transport]),
_unstable_getRTCTransports: vi.fn().mockResolvedValue(homeserverTransports),
_unstable_getLivekitToken: vi.fn().mockResolvedValue({ jwt: publisherJWT }),
} as unknown as MatrixClient;
const members$ = new BehaviorSubject<RTCMember[]>([]);
@@ -201,7 +230,7 @@ describe("createRTCParticipation", () => {
await flushPromises();
expect(join).toHaveBeenCalledWith(
{ userId, deviceId, memberId: legacyMemberId },
[transport],
[transportWithAlias],
expect.objectContaining({
manageMediaKeys: true,
unstableSendStickyEvents: false,
@@ -220,6 +249,32 @@ describe("createRTCParticipation", () => {
});
});
it("publishes on another member's SFU where the homeserver advertises none", async () => {
const bobOnSfu = mockRtcMembership("@bob:example.org", "BOB", {
fociPreferred: [{ type: "livekit", url: "wss://sfu.example.org" }],
});
const { join, fake } = setup(
{},
{},
{ homeserverTransports: [], memberships: [bobOnSfu] },
);
fake.prepared.resolve({ canDelegateDelayedLeave: false });
await flushPromises();
await flushPromises();
expect(join).toHaveBeenCalledWith(
expect.anything(),
[
{
type: "livekit",
url: "wss://sfu.example.org",
server_name: "example.org",
},
],
expect.anything(),
);
});
it("publishes only while joined", async () => {
const { participation, session, fake, scope } = setup();
const ended = vi.fn();
@@ -271,7 +326,7 @@ describe("createRTCParticipation", () => {
await flushPromises();
expect(join).toHaveBeenCalledWith(
expect.anything(),
[transport],
[transportWithAlias],
expect.objectContaining({ delayedLeaveEventDelayMs: 7_200_000 }),
);
expect(fake.delegateDelayedLeave).not.toHaveBeenCalled();
+43 -14
View File
@@ -11,6 +11,7 @@ import {
type MatrixRTCSession as JsSdkRTCSession,
MatrixRTCSessionEvent,
MembershipManagerEvent,
type Transport,
} from "matrix-js-sdk/lib/matrixrtc";
import { type IMembershipManager } from "matrix-js-sdk/lib/matrixrtc/IMembershipManager";
import { type CallMembershipIdentityParts } from "matrix-js-sdk/lib/matrixrtc/EncryptionManager";
@@ -50,6 +51,7 @@ import {
type MediaKey,
} from "./media-backend/api";
import { createLivekitBackend } from "./media-backend/livekit/LivekitBackend";
import { findRemotePublishLivekitTransport } from "./media-backend/livekit/RemotePublishTransport";
import { createHomeserverConnected$ } from "./matrixrtc/HomeserverConnected";
import { joinJsSdkSession } from "./matrixrtc/joinJsSdkSession";
import {
@@ -130,13 +132,34 @@ export function createRTCParticipation(
logger,
});
// If the backend is LiveKit, we check it for transports as well, as a
// fallback: another member's SFU may let us publish (MSC4195). LiveKit only,
// so it stays out of the backend interface
const transportFromBackends = async (): Promise<Transport | null> =>
backend.transportType === "livekit"
? findRemotePublishLivekitTransport({
client,
// A snapshot: if nobody with a usable SFU is here yet, the user has
// to come back later rather than wait in a half-working lobby
memberships: jsSdkSession.memberships,
ownMembershipIdentity,
roomId: room.roomId,
slotId: jsSdkSession.slotId,
logger,
})
: null;
const preparedTransport$ = from(
discoverLocalTransport(client, backend.transportType, options, logger).then(
async (transport): Promise<PreparedTransport> => ({
transport,
...(await backend.prepareLocalTransport(transport)),
}),
),
discoverLocalTransport(
client,
backend.transportType,
options,
transportFromBackends,
logger,
).then(async (transport): Promise<PreparedTransport> => ({
transport,
...(await backend.prepareLocalTransport(transport)),
})),
);
const localMembership = createLocalMembership$({
@@ -151,14 +174,20 @@ export function createRTCParticipation(
timings.syncDisconnectGracePeriodMs,
),
joinMatrixRTC: (transport, delayedLeave) =>
joinJsSdkSession(jsSdkSession, ownMembershipIdentity, transport, {
encryptMedia: encryptionSystem.kind !== E2eeType.NONE,
matrixRTCMode,
sendNotificationType: options.sendNotificationType,
applicationData: options.applicationData,
timings,
delayedLeave,
}),
joinJsSdkSession(
room.roomId,
jsSdkSession,
ownMembershipIdentity,
transport,
{
encryptMedia: encryptionSystem.kind !== E2eeType.NONE,
matrixRTCMode,
sendNotificationType: options.sendNotificationType,
applicationData: options.applicationData,
timings,
delayedLeave,
},
),
membershipManagerError$: fromEvent(
jsSdkSession,
MatrixRTCSessionEvent.MembershipManagerError,
+36 -1
View File
@@ -20,6 +20,11 @@ const hostTransport: Transport = {
type: "livekit",
livekit_service_url: "https://host.example.org",
};
const bobsTransport: Transport = {
type: "livekit",
url: "wss://sfu.bob.com",
server_name: "bob.com",
};
function client(
advertised: Transport[],
@@ -30,6 +35,12 @@ function client(
};
}
function fromBackends(
transport: Transport | null = null,
): ReturnType<typeof vi.fn<() => Promise<Transport | null>>> {
return vi.fn(async () => Promise.resolve(transport));
}
describe("discoverLocalTransport", () => {
it("takes the transport the host names without asking the homeserver", async () => {
const c = client([homeserverTransport]);
@@ -39,6 +50,7 @@ describe("discoverLocalTransport", () => {
c,
"livekit",
{ transport: hostTransport },
fromBackends(),
logger,
),
).resolves.toBe(hostTransport);
@@ -51,6 +63,7 @@ describe("discoverLocalTransport", () => {
client([homeserverTransport]),
"livekit",
{ fallbackTransport: hostTransport },
fromBackends(),
logger,
),
).resolves.toEqual(homeserverTransport);
@@ -62,13 +75,35 @@ describe("discoverLocalTransport", () => {
client([{ type: "not_livekit" }]),
"livekit",
{ fallbackTransport: hostTransport },
fromBackends(),
logger,
),
).resolves.toEqual(hostTransport);
});
it("prefers a local transport over the backends' without asking them", async () => {
const backends = fromBackends(bobsTransport);
await expect(
discoverLocalTransport(
client([homeserverTransport]),
"livekit",
{},
backends,
logger,
),
).resolves.toEqual(homeserverTransport);
expect(backends).not.toHaveBeenCalled();
});
it("fails naming the homeserver where there is no transport at all", async () => {
const promise = discoverLocalTransport(client([]), "livekit", {}, logger);
const promise = discoverLocalTransport(
client([]),
"livekit",
{},
fromBackends(),
logger,
);
await expect(promise).rejects.toBeInstanceOf(
MatrixRTCTransportMissingError,
+8 -2
View File
@@ -15,7 +15,10 @@ import { TransportAutoDiscovery } from "./TransportAutoDiscovery";
/**
* The transport the local member advertises: the one the host names, else the
* homeserver's first of the type the backend serves, else the host's fallback.
* homeserver's first of the type the backend serves, else the host's fallback,
* else the first the backends can provide, which are asked only then.
*
* @throws {MatrixRTCTransportMissingError} where none of these exists.
*/
export async function discoverLocalTransport(
client: Pick<MatrixClient, "getDomain" | "_unstable_getRTCTransports">,
@@ -24,6 +27,7 @@ export async function discoverLocalTransport(
transport: override,
fallbackTransport,
}: Pick<RTCParticipationOptions, "transport" | "fallbackTransport">,
transportFromBackends: () => Promise<Transport | null>,
logger: Logger,
): Promise<Transport> {
const transport =
@@ -33,7 +37,9 @@ export async function discoverLocalTransport(
transportType,
fallbackTransport,
logger,
}).discoverPreferredTransport());
}).discoverPreferredTransport()) ??
(await transportFromBackends()) ??
null;
if (transport === null)
throw new MatrixRTCTransportMissingError(client.getDomain() ?? "");
return transport;
+35 -6
View File
@@ -6,6 +6,7 @@ Please see LICENSE in the repository root for full details.
*/
import {
type UnstableLivekitTransport,
type MatrixRTCSession,
type Transport,
} from "matrix-js-sdk/lib/matrixrtc";
@@ -15,11 +16,16 @@ import { defaultSessionTimings, MatrixRTCMode } from "../config";
import { ownMemberMock } from "../utils/test";
import { joinJsSdkSession } from "./joinJsSdkSession";
const transport: Transport = {
const transport: UnstableLivekitTransport = {
type: "livekit",
livekit_service_url: "https://lk.example.org",
};
const transportWithAlias: Transport = {
livekit_alias: "!room:example.org",
...transport,
};
/** A session that only records the join; nothing else of it is touched. */
function session(): MatrixRTCSession & {
join: Mock<MatrixRTCSession["join"]>;
@@ -39,7 +45,7 @@ describe("joinJsSdkSession", () => {
keyRotationParticipantLimit: 20,
};
joinJsSdkSession(s, ownMemberMock, transport, {
joinJsSdkSession("!room:example.org", s, ownMemberMock, transport, {
encryptMedia: true,
matrixRTCMode: MatrixRTCMode.Compatibility,
sendNotificationType: "ring",
@@ -51,7 +57,7 @@ describe("joinJsSdkSession", () => {
},
});
expect(s.join).toHaveBeenCalledWith(ownMemberMock, [transport], {
expect(s.join).toHaveBeenCalledWith(ownMemberMock, [transportWithAlias], {
notificationType: "ring",
callIntent: undefined,
applicationData: {},
@@ -72,7 +78,7 @@ describe("joinJsSdkSession", () => {
it("gives up on the network once the delayed leave has probably been sent", () => {
const s = session();
joinJsSdkSession(s, ownMemberMock, transport, {
joinJsSdkSession("!room:example.org", s, ownMemberMock, transport, {
encryptMedia: false,
matrixRTCMode: MatrixRTCMode.Compatibility,
timings: { ...defaultSessionTimings, networkErrorRetryMs: 4_000 },
@@ -81,7 +87,7 @@ describe("joinJsSdkSession", () => {
expect(s.join).toHaveBeenCalledWith(
ownMemberMock,
[transport],
[transportWithAlias],
expect.objectContaining({
manageMediaKeys: false,
maximumNetworkErrorRetryCount: 4,
@@ -92,7 +98,30 @@ describe("joinJsSdkSession", () => {
it("lifts the call intent out of the application data and sends sticky events in Matrix 2.0 mode", () => {
const s = session();
joinJsSdkSession(s, ownMemberMock, transport, {
joinJsSdkSession("!room:example.org", s, ownMemberMock, transport, {
encryptMedia: true,
matrixRTCMode: MatrixRTCMode.Matrix_2_0,
applicationData: { "m.call.intent": "audio", custom: 1 },
timings: defaultSessionTimings,
delayedLeave: defaultSessionTimings.delayedLeave,
});
expect(s.join).toHaveBeenCalledWith(
ownMemberMock,
[transportWithAlias],
expect.objectContaining({
callIntent: "audio",
applicationData: { custom: 1 },
unstableSendStickyEvents: true,
}),
);
});
it("does not add livekit_alias to newer transport format", () => {
const s = session();
const transport = { type: "livekit", url: "wss://lk.example.org" };
joinJsSdkSession("!room:example.org", s, ownMemberMock, transport, {
encryptMedia: true,
matrixRTCMode: MatrixRTCMode.Matrix_2_0,
applicationData: { "m.call.intent": "audio", custom: 1 },
+34 -16
View File
@@ -35,6 +35,7 @@ interface Options {
* @throws If the join could not be started.
*/
export function joinJsSdkSession(
roomId: string,
session: JsSdkRTCSession,
ownMembershipIdentity: CallMembershipIdentityParts,
transport: Transport,
@@ -58,20 +59,37 @@ export function joinJsSdkSession(
timings.syncDisconnectGracePeriodMs,
delayedLeave.delay_ms,
);
session.join(ownMembershipIdentity, [transport], {
notificationType: sendNotificationType,
callIntent,
applicationData: rest,
manageMediaKeys: encryptMedia,
delayedLeaveEventRestartMs: delayedLeave.restart_ms,
delayedLeaveEventDelayMs: delayedLeave.delay_ms,
delayedLeaveEventRestartLocalTimeoutMs: delayedLeave.restart_timeout_ms,
networkErrorRetryMs: timings.networkErrorRetryMs,
makeKeyDelay: timings.waitForKeyRotationMs,
membershipEventExpiryMs: timings.membershipEventExpiryMs,
keyRotationParticipantLimit: timings.keyRotationParticipantLimit,
unstableSendStickyEvents: matrixRTCMode === MatrixRTCMode.Matrix_2_0,
maximumNetworkErrorRetryCount:
Math.ceil(maxWaitMs / timings.networkErrorRetryMs) + 1,
});
session.join(
ownMembershipIdentity,
// TODO: `livekit_alias` should be unused these days, but some Ruma-based
// clients still expect it to be present. Remove from `publishedTransports`
// as soon as https://github.com/ruma/ruma/pull/2642 and
// https://github.com/matrix-org/matrix-rust-sdk/pull/7335 are widely
// deployed.
[
{
...("livekit_service_url" in transport
? { livekit_alias: roomId }
: {}),
...transport,
},
],
{
notificationType: sendNotificationType,
callIntent,
applicationData: rest,
manageMediaKeys: encryptMedia,
delayedLeaveEventRestartMs: delayedLeave.restart_ms,
delayedLeaveEventDelayMs: delayedLeave.delay_ms,
delayedLeaveEventRestartLocalTimeoutMs: delayedLeave.restart_timeout_ms,
networkErrorRetryMs: timings.networkErrorRetryMs,
makeKeyDelay: timings.waitForKeyRotationMs,
membershipEventExpiryMs: timings.membershipEventExpiryMs,
keyRotationParticipantLimit: timings.keyRotationParticipantLimit,
unstableSendStickyEvents: matrixRTCMode === MatrixRTCMode.Matrix_2_0,
maximumNetworkErrorRetryCount:
Math.ceil(maxWaitMs / timings.networkErrorRetryMs) + 1,
},
);
}
@@ -258,6 +258,23 @@ describe("createLivekitBackend", () => {
},
);
it("prepares a remote SFU's transport on its homeserver, where delegation is not possible", async () => {
// Our own homeserver would delegate, so only the SFU being remote can
// explain delegation being skipped
fetchMock.post(homeserverEndpoint, 401);
const remoteTransport: UnstableLivekitTransport = {
type: "livekit",
url: "wss://sfu.sample.com",
server_name: "sample.com",
};
const capabilities = await backend().prepareLocalTransport(remoteTransport);
expect(capabilities).toEqual({ canDelegateDelayedLeave: false });
expect(connections[0].connection.role).toBe("publisher");
expect(connections[0].connection.serverName).toBe("sample.com");
});
it("waits for the publisher connection to hold its token before the join", async () => {
fetchMock.post(homeserverEndpoint, 404);
fetchMock.post(`${exampleServiceUrl}/delegate_delayed_leave`, 404);
@@ -488,6 +505,27 @@ describe("createLivekitBackend", () => {
expect(remote?.connection.transport).toEqual(carlsTransport);
expect(remote?.connection.serverName).toBe("sample.com");
});
it("connects to a remote publisher's transport on the SFU's homeserver", async () => {
const b = await prepared();
// Dave has no SFU of his own and publishes on sample.com's
const remoteTransport: UnstableLivekitTransport = {
type: "livekit",
url: "wss://sfu.sample.com",
server_name: "sample.com",
};
b.mediaFor(testScope(), {
...bob(remoteTransport, ["dave-hashed"]),
userId: "@dave:example.org",
});
await flushPromises();
const remote = connections.find(
(c) => c.connection.role === "subscriber",
);
expect(remote?.connection.transport).toEqual(remoteTransport);
expect(remote?.connection.serverName).toBe("sample.com");
});
});
});
+30 -16
View File
@@ -34,7 +34,7 @@ import {
trackEpoch,
} from "../../reactive/ObservableScope";
import { mapScoped } from "../../utils/mapScoped";
import { serverNameOf } from "../../utils/serverName";
import { serverNameOf, serverNameOfTransport } from "../../utils/serverName";
import {
type BackendConnection,
type MediaBackend,
@@ -146,26 +146,37 @@ export function createLivekitBackend(
logger,
);
const ownServerName = serverNameOf(ownMembershipIdentity.userId);
const prepareLocalTransport = async (
transport: Transport,
): Promise<TransportCapabilities> => {
if (!isUnstableLivekitTransport(transport))
throw new MatrixRTCError(`Not a LiveKit transport: ${transport.type}`);
const locator: TransportLocator = {
const serverName = serverNameOfTransport(
transport,
serverName: serverNameOf(ownMembershipIdentity.userId),
};
// Delegation goes through whichever service authenticates us as a
// publisher, so only that service's support counts
const flow = chooseAuthFlow(transport, "publisher");
const canDelegate =
flow.kind === "openID"
? supportsDelegation(
`${flow.serviceUrl}/delegate_delayed_leave`,
`transport ${flow.serviceUrl}`,
logger,
)
: homeserverSupportsDelegation;
ownMembershipIdentity.userId,
);
const locator: TransportLocator = { transport, serverName };
let canDelegate: Promise<boolean>;
if (serverName !== ownServerName) {
// Our homeserver only takes over leaves for its own SFUs
logger.info(
`Publishing on a remote SFU of ${serverName}, delegation is not possible`,
);
canDelegate = Promise.resolve(false);
} else {
// Check for support using the same auth flow that we intend to use later
const flow = chooseAuthFlow(transport);
canDelegate =
flow.kind === "openID"
? supportsDelegation(
`${flow.serviceUrl}/delegate_delayed_leave`,
`transport ${flow.serviceUrl}`,
logger,
)
: homeserverSupportsDelegation;
}
localTransport$.next(locator);
// The membership is only sent once the publisher connection holds its
// token, so that the LiveKit room exists before anyone subscribes to it:
@@ -204,7 +215,10 @@ export function createLivekitBackend(
map((transport): TransportLocator | undefined => {
const raw = transport?.raw;
return isUnstableLivekitTransport(raw)
? { transport: raw, serverName: serverNameOf(member.userId) }
? {
transport: raw,
serverName: serverNameOfTransport(raw, member.userId),
}
: undefined;
}),
distinctUntilChanged((a, b) => (a && keyFor(a)) === (b && keyFor(b))),
@@ -0,0 +1,172 @@
/*
Copyright 2026 Element Creations Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { MatrixError } from "matrix-js-sdk";
import { logger } from "matrix-js-sdk/lib/logger";
import { type CallMembership } from "matrix-js-sdk/lib/matrixrtc";
import { beforeEach, describe, expect, it, type Mocked, vi } from "vitest";
import { mockRtcMembership, ownMemberMock } from "../../utils/test";
import { type ClientGetTokenParts } from "./auth/csApi";
import { findRemotePublishLivekitTransport } from "./RemotePublishTransport";
/** A JWT whose payload states whether the issuing server lets us publish. */
function jwtWithPublishGrant(canPublish: boolean): string {
return [
{}, // header
{ sub: "identity", video: { room: "room", canPublish } }, // payload
{}, // signature
]
.map((d) => global.btoa(JSON.stringify(d)))
.join(".");
}
const bobSfu = { type: "livekit", url: "wss://sfu.bob.com" };
const carolSfu = { type: "livekit", url: "wss://sfu.carol.com" };
const bobsTransport = { ...bobSfu, server_name: "bob.com" };
const carolsTransport = { ...carolSfu, server_name: "carol.com" };
// Bob is the oldest member, Carol joined after him
const bob = mockRtcMembership("@bob:bob.com", "BOB", {
fociPreferred: [bobSfu],
membership: { created_ts: 100 },
});
const carol = mockRtcMembership("@carol:carol.com", "CAROL", {
fociPreferred: [carolSfu],
membership: { created_ts: 200 },
});
// Bob's other device, and Dave who publishes on Bob's SFU remotely
const bobsLaptop = mockRtcMembership("@bob:bob.com", "LAPTOP", {
fociPreferred: [bobSfu],
membership: { created_ts: 150 },
});
const dave = mockRtcMembership("@dave:dave.com", "DAVE", {
fociPreferred: [bobsTransport],
membership: { created_ts: 160 },
});
const legacyErin = mockRtcMembership("@erin:erin.com", "ERIN", {
fociPreferred: [
{ type: "livekit", livekit_service_url: "https://jwt.erin.com" },
],
membership: { created_ts: 50 },
});
const ourselves = mockRtcMembership(
ownMemberMock.userId,
ownMemberMock.deviceId,
{
fociPreferred: [{ type: "livekit", url: "wss://sfu.example.org" }],
membership: { created_ts: 10 },
},
);
const csApiMissing = new MatrixError(
{ errcode: "M_UNRECOGNIZED", error: "" },
404,
);
describe("findRemotePublishLivekitTransport", () => {
let client: Mocked<ClientGetTokenParts>;
const find = async (
memberships: CallMembership[],
): ReturnType<typeof findRemotePublishLivekitTransport> =>
findRemotePublishLivekitTransport({
client,
memberships,
ownMembershipIdentity: ownMemberMock,
roomId: "!room:example.org",
slotId: "m.call#ROOM",
logger,
});
beforeEach(() => {
client = {
_unstable_getLivekitToken: vi
.fn()
.mockResolvedValue({ jwt: jwtWithPublishGrant(true) }),
};
});
it("publishes on the oldest member's SFU with its server_name", async () => {
await expect(find([carol, bob])).resolves.toEqual(bobsTransport);
expect(client._unstable_getLivekitToken).toHaveBeenCalledTimes(1);
expect(client._unstable_getLivekitToken).toHaveBeenCalledWith({
url: "wss://sfu.bob.com",
server_name: "bob.com",
room_id: "!room:example.org",
slot_id: "m.call#ROOM",
member_id: ownMemberMock.memberId,
});
});
it("asks about an SFU once however many members are on it", async () => {
client._unstable_getLivekitToken.mockResolvedValue({
jwt: jwtWithPublishGrant(false),
});
await expect(find([dave, bobsLaptop, bob])).resolves.toBeNull();
expect(client._unstable_getLivekitToken).toHaveBeenCalledTimes(1);
});
it("skips our own membership", async () => {
await expect(find([ourselves, bob])).resolves.toEqual(bobsTransport);
expect(client._unstable_getLivekitToken).toHaveBeenCalledTimes(1);
});
it("ignores members publishing through a legacy JWT service", async () => {
await expect(find([legacyErin, bob])).resolves.toEqual(bobsTransport);
expect(client._unstable_getLivekitToken).toHaveBeenCalledTimes(1);
});
it("moves on when a homeserver does not let us publish", async () => {
client._unstable_getLivekitToken.mockImplementation(
async ({ server_name }) =>
Promise.resolve({
jwt: jwtWithPublishGrant(server_name === "carol.com"),
}),
);
await expect(find([bob, carol])).resolves.toEqual(carolsTransport);
expect(client._unstable_getLivekitToken).toHaveBeenCalledTimes(2);
});
it("moves on when a homeserver refuses to issue a token", async () => {
client._unstable_getLivekitToken.mockImplementation(
async ({ server_name }) =>
server_name === "bob.com"
? Promise.reject(
new MatrixError(
{ errcode: "M_FORBIDDEN", error: "Not joined" },
403,
),
)
: Promise.resolve({ jwt: jwtWithPublishGrant(true) }),
);
await expect(find([bob, carol])).resolves.toEqual(carolsTransport);
});
it("finds nothing once all candidates are exhausted", async () => {
client._unstable_getLivekitToken.mockResolvedValue({
jwt: jwtWithPublishGrant(false),
});
await expect(find([bob, carol])).resolves.toBeNull();
expect(client._unstable_getLivekitToken).toHaveBeenCalledTimes(2);
});
it("gives up immediately if our homeserver lacks the CS API", async () => {
client._unstable_getLivekitToken.mockRejectedValue(csApiMissing);
await expect(find([bob, carol])).resolves.toBeNull();
expect(client._unstable_getLivekitToken).toHaveBeenCalledTimes(1);
});
it("finds nothing where there are no other members", async () => {
await expect(find([ourselves])).resolves.toBeNull();
expect(client._unstable_getLivekitToken).not.toHaveBeenCalled();
});
});
@@ -0,0 +1,105 @@
/*
Copyright 2026 Element Creations Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { type Logger } from "matrix-js-sdk/lib/logger";
import {
type CallMembership,
isUnstableLivekitTransport,
type Transport,
} from "matrix-js-sdk/lib/matrixrtc";
import { type CallMembershipIdentityParts } from "matrix-js-sdk/lib/matrixrtc/EncryptionManager";
import { isCsApiUnavailable } from "./auth";
import * as csApi from "./auth/csApi";
import { decodeSFUJWTPayload } from "./auth/types";
export interface RemotePublishTransportParams {
client: csApi.ClientGetTokenParts;
/** The members whose SFUs are the candidates. */
memberships: CallMembership[];
ownMembershipIdentity: CallMembershipIdentityParts;
roomId: string;
slotId: string;
logger: Logger;
}
/**
* The SFU of another member that its homeserver lets us publish on, as the
* transport to advertise, or null where none does. Candidates are tried
* oldest membership first, each SFU once, and each is checked by asking for a
* publisher token through our homeserver (MSC4195): a homeserver may restrict
* remote publishers, which shows as a token without the publish grant.
*/
export async function findRemotePublishLivekitTransport({
client,
memberships,
ownMembershipIdentity: own,
roomId,
slotId,
logger,
}: RemotePublishTransportParams): Promise<Transport | null> {
logger.info(
"No local transport available, looking for another member's SFU to publish on",
);
const tried = new Set<string>();
const byAge = [...memberships].sort((a, b) => a.createdTs() - b.createdTs());
for (const membership of byAge) {
if (
membership.userId === own.userId &&
membership.deviceId === own.deviceId
)
continue;
for (const transport of membership.transports) {
// Publishing remotely goes through the client-server API, which a JWT
// service cannot stand in for, so only a `url` qualifies
if (!isUnstableLivekitTransport(transport) || !("url" in transport))
continue;
const { url } = transport;
const serverName = membership.getTransportServerName(transport);
const key = JSON.stringify([url, serverName]);
if (tried.has(key)) continue;
tried.add(key);
try {
const { jwt } = await csApi.getSFUConfig({
client,
membership: own,
url,
serverName,
roomId,
slotId,
});
try {
// The SFU treats an absent canPublish grant as publishing allowed.
if (decodeSFUJWTPayload(jwt).video.canPublish !== false) {
logger.info(`Publishing on remote SFU ${url} of ${serverName}`);
return { type: "livekit", url, server_name: serverName };
}
logger.info(
`${serverName} does not let us publish on ${url}, skipping`,
);
continue;
} catch (e) {
logger.info(
`${serverName}: failed to decode SFUJWT payload: ${e}, skipping`,
);
continue;
}
} catch (e) {
if (isCsApiUnavailable(e)) {
// Without the endpoint no further candidate can work
logger.warn(
`Cannot use the MSC4195 get_token endpoint (${e}), so no remote SFU can be used`,
);
return null;
}
logger.warn(`Failed to get a token for ${url} of ${serverName}:`, e);
continue;
}
}
}
return null;
}
@@ -7,7 +7,7 @@ Please see LICENSE in the repository root for full details.
import { afterEach, describe, expect, it, type Mocked, vi } from "vitest";
import fetchMock from "fetch-mock";
import { ConnectionError, MatrixError } from "matrix-js-sdk";
import { MatrixError } from "matrix-js-sdk";
import { logger } from "matrix-js-sdk/lib/logger";
import { type UnstableLivekitTransport } from "matrix-js-sdk/lib/matrixrtc";
import { type CallMembershipIdentityParts } from "matrix-js-sdk/lib/matrixrtc/EncryptionManager";
@@ -144,26 +144,8 @@ describe("getSFUConfig", () => {
expect(jwtServiceCalls()).toEqual(["/get_token"]);
});
it("uses CS API when subscribing with url + livekit_service_url transport", async () => {
it("uses OpenID when subscribing with url + livekit_service_url transport", async () => {
mockJWTService();
expect(
await getSFUConfig({
...defaultParams,
transport: urlAndServiceUrlTransport,
}),
).toEqual(
expect.objectContaining({
jwt: csApiJWTToken,
livekitAlias: "via CS API",
}),
);
expect(client._unstable_getLivekitToken).toHaveBeenCalled();
expect(jwtServiceCalls()).toEqual([]);
});
it("falls back to OpenID with url + livekit_service_url transport", async () => {
mockJWTService();
client._unstable_getLivekitToken.mockRejectedValueOnce(unsupportedError);
expect(
await getSFUConfig({
...defaultParams,
@@ -175,25 +157,7 @@ describe("getSFUConfig", () => {
livekitAlias: "via default endpoint",
}),
);
expect(client._unstable_getLivekitToken).toHaveBeenCalled();
expect(jwtServiceCalls()).toEqual(["/get_token"]);
});
it("falls back to OpenID when the widget host refuses the CS API", async () => {
mockJWTService();
// In widget mode a host without MSC4533 rejects the request before it
// reaches the homeserver, with a plain error rather than a Matrix error
client._unstable_getLivekitToken.mockRejectedValueOnce(
new Error(
"The rtc_livekit_get_token action is not supported by the client.",
),
);
expect(
await getSFUConfig({
...defaultParams,
transport: urlAndServiceUrlTransport,
}),
).toEqual(expect.objectContaining({ jwt: defaultJWTToken }));
expect(client._unstable_getLivekitToken).not.toHaveBeenCalled();
expect(jwtServiceCalls()).toEqual(["/get_token"]);
});
@@ -222,7 +186,7 @@ describe("getSFUConfig", () => {
expect(jwtServiceCalls()).toEqual([]);
});
it("does not fall back to OpenID when publishing", async () => {
it("does not fall back to OpenID when publishing with url transport", async () => {
mockJWTService();
client._unstable_getLivekitToken.mockRejectedValueOnce(unsupportedError);
await expect(
@@ -255,33 +219,6 @@ describe("getSFUConfig", () => {
expect(client._unstable_getLivekitToken).not.toHaveBeenCalled();
expect(jwtServiceCalls()).toEqual(["/sfu/get"]);
});
it("does not fall back to OpenID on connection error", async () => {
vi.useFakeTimers();
try {
mockJWTService();
const client_ = {
...client,
_unstable_getLivekitToken: vi
.fn()
.mockRejectedValue(new ConnectionError("No connection")),
};
await Promise.all([
expect(
getSFUConfig({
...defaultParams,
client: client_,
transport: urlAndServiceUrlTransport,
}),
).rejects.toThrow(expect.any(ConnectionError)),
vi.runAllTimersAsync(), // Allow request to be retried
]);
expect(client_._unstable_getLivekitToken).toHaveBeenCalled();
expect(jwtServiceCalls()).toEqual([]);
} finally {
vi.useRealTimers();
}
});
});
describe("delegateDelayedLeave", () => {
@@ -362,46 +299,21 @@ describe("delegateDelayedLeave", () => {
describe("chooseAuthFlow", () => {
it.each([
[
"url",
"publisher",
urlTransport,
{ kind: "csApi", url: urlTransport.url },
],
[
"url",
"subscriber",
urlTransport,
{ kind: "csApi", url: urlTransport.url },
],
["url", urlTransport, { kind: "csApi", url: urlTransport.url }],
[
"livekit_service_url",
"publisher",
serviceUrlTransport,
{ kind: "openID", serviceUrl },
],
[
"livekit_service_url",
"subscriber",
serviceUrlTransport,
{ kind: "openID", serviceUrl },
],
[
"url + livekit_service_url",
"publisher",
urlAndServiceUrlTransport,
{ kind: "openID", serviceUrl },
],
[
"url + livekit_service_url",
"subscriber",
urlAndServiceUrlTransport,
{ kind: "csApi", url: urlTransport.url },
],
] as const)(
"chooses the flow for a %s transport as %s",
(_shape, role, transport, flow) => {
expect(chooseAuthFlow(transport, role)).toEqual(flow);
(_shape, transport, flow) => {
expect(chooseAuthFlow(transport)).toEqual(flow);
},
);
});
+40 -54
View File
@@ -48,7 +48,7 @@ function isEndpointUnsupported(e: unknown): boolean {
* (MSC4533), which the widget API reports as a plain error. A connection
* error is neither: the endpoint may well exist.
*/
function isCsApiUnavailable(e: unknown): boolean {
export function isCsApiUnavailable(e: unknown): boolean {
if (isEndpointUnsupported(e)) return true;
return !(e instanceof MatrixError) && !(e instanceof ConnectionError);
}
@@ -65,7 +65,10 @@ export interface GetSFUConfigParams {
logger: Logger;
}
/** Authenticates with a transport, through whichever flow suits it and our role. */
/**
* Gets an {@link SFUConfig} appropriate for connecting to a given transport,
* using whichever auth flow is appropriate.
*/
export async function getSFUConfig({
client,
membership,
@@ -76,48 +79,36 @@ export async function getSFUConfig({
role,
logger,
}: GetSFUConfigParams): Promise<SFUConfig> {
const viaOpenID = async (serviceUrl: string): Promise<SFUConfig> =>
await openID.getSFUConfig({
const flow = chooseAuthFlow(transport);
if (flow.kind === "openID") {
return await openID.getSFUConfig({
client,
membership,
serviceUrl,
serviceUrl: flow.serviceUrl,
roomId,
slotId,
role,
logger,
});
const flow = chooseAuthFlow(transport, role);
if (flow.kind === "openID") return await viaOpenID(flow.serviceUrl);
try {
return await csApi.getSFUConfig({
client,
membership,
url: flow.url,
serverName,
roomId,
slotId,
});
} catch (e) {
// A publisher never falls back: a homeserver that advertises a `url` but
// does not implement MSC4195 is misconfigured, and the admin should hear
// about it before `livekit_service_url` is deprecated
const mayFallBack =
role === "subscriber" && "livekit_service_url" in transport;
if (mayFallBack && isCsApiUnavailable(e)) {
logger.warn(
`Cannot use the MSC4195 get_token endpoint (${e}), falling back to the OpenID flow with service ${transport.livekit_service_url}`,
);
return await viaOpenID(transport.livekit_service_url);
} else {
try {
return await csApi.getSFUConfig({
client,
membership,
url: flow.url,
serverName,
roomId,
slotId,
});
} catch (e) {
if (isEndpointUnsupported(e)) {
logger.error(
`Homeserver does not support the MSC4195 get_token endpoint, cannot get token for transport ${flow.url}`,
);
throw new NoMatrix2AuthorizationService(e as Error);
}
throw e;
}
if (isEndpointUnsupported(e)) {
logger.error(
`Homeserver does not support the MSC4195 get_token endpoint, cannot get token for transport ${flow.url}`,
);
throw new NoMatrix2AuthorizationService(e as Error);
}
throw e;
}
}
@@ -149,7 +140,7 @@ export async function delegateDelayedLeave({
delayTimeoutMs,
logger,
}: DelegateDelayedLeaveParams): Promise<void> {
const flow = chooseAuthFlow(transport, "publisher");
const flow = chooseAuthFlow(transport);
if (flow.kind === "csApi") {
try {
await csApi.delegateDelayedLeave({
@@ -190,23 +181,18 @@ export type AuthFlow =
| { kind: "openID"; serviceUrl: string };
/**
* Chooses between the MSC4195 client-server API and the OpenID flow against
* the transport's JWT service.
*
* Only the JWT service's `/sfu/get` endpoint assigns the legacy participant
* identity that clients predating MSC4195 expect from anyone advertising a
* membership, so a publisher keeps using the JWT service wherever the
* transport still offers one, even next to a `url`. A subscriber may use the
* client-server API regardless, as nobody matches its identity.
* Chooses which auth flow to use for a given transport. For now this returns a
* single auth flow, with no provision for a fallback behavior.
*/
export function chooseAuthFlow(
transport: UnstableLivekitTransport,
role: ConnectionRole,
): AuthFlow {
if ("url" in transport) {
const useCsApi =
!("livekit_service_url" in transport) || role === "subscriber";
if (useCsApi) return { kind: "csApi", url: transport.url };
}
export function chooseAuthFlow(transport: UnstableLivekitTransport): AuthFlow {
if ("url" in transport && !("livekit_service_url" in transport))
return { kind: "csApi", url: transport.url };
// All other transports use the OpenID flow, even if they contain a `url`.
// This is necessary since the auth flow determines the participant identity
// format, and we still have older clients out in the wild which would not yet
// know to look for a hashed participant identity in case `url` is present in
// someone else's transport.
// TODO: Once users have updated their clients, choose the CS API flow if
// available, keeping OpenID as a fallback.
return { kind: "openID", serviceUrl: transport.livekit_service_url };
}
+10 -3
View File
@@ -34,11 +34,13 @@ export interface SFUConfig {
}
/** The parts of the JWT payload the backend reads. */
interface SFUJWTPayload {
export interface SFUJWTPayload {
/** The subject: the participant identity the token carries. */
sub: string;
video: {
room: string;
/** Whether the token lets us publish; a homeserver may deny this to remote publishers. */
canPublish?: boolean;
};
}
@@ -47,8 +49,7 @@ export function extractFullConfigFromToken(sfuConfig: {
url: string;
jwt: string;
}): SFUConfig {
const [, payloadStr] = sfuConfig.jwt.split(".");
const payload = JSON.parse(global.atob(payloadStr)) as SFUJWTPayload;
const payload = decodeSFUJWTPayload(sfuConfig.jwt);
return {
jwt: sfuConfig.jwt,
url: sfuConfig.url,
@@ -56,3 +57,9 @@ export function extractFullConfigFromToken(sfuConfig: {
livekitIdentity: payload.sub,
};
}
/** Decodes the payload of an SFU JWT, without verifying its signature. */
export function decodeSFUJWTPayload(jwt: string): SFUJWTPayload {
const [, payloadStr] = jwt.split(".");
return JSON.parse(global.atob(payloadStr)) as SFUJWTPayload;
}
+16
View File
@@ -5,7 +5,23 @@ SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { type Transport } from "matrix-js-sdk/lib/matrixrtc";
/** The homeserver a Matrix user id belongs to: everything after the first colon. */
export function serverNameOf(userId: string): string {
return userId.replace(/^.*?:/, "");
}
/**
* The homeserver that operates a transport: the one the transport names, else
* the one of the member advertising it. A member publishing on another
* homeserver's SFU (MSC4195) names that homeserver in `server_name`.
*/
export function serverNameOfTransport(
transport: Transport,
userId: string,
): string {
return typeof transport.server_name === "string"
? transport.server_name
: serverNameOf(userId);
}
+2 -1
View File
@@ -7,6 +7,7 @@
padding-top: env(safe-area-inset-top);
opacity: 1;
transition: opacity 0.15s;
text-align: center;
}
/* Pseudo-element for the gradient background */
@@ -52,7 +53,7 @@
". . ."
"primaryButton title secondaryButton"
". . .";
place-items: center;
place-items: center stretch;
column-gap: var(--cpd-space-2x);
}
+1 -6
View File
@@ -6,12 +6,11 @@ Please see LICENSE in the repository root for full details.
*/
import { describe, expect, it, vi } from "vitest";
import { BehaviorSubject } from "rxjs";
import { testScope, mockMuteStates, mockMediaDevices } from "../utils/test";
import { constant } from "../state/Behavior";
import type { CallViewModel } from "../state/CallViewModel/CallViewModel";
import type { Alignment, Layout } from "../state/layout-types";
import type { Layout } from "../state/layout-types";
import type { SpotlightTileViewModel } from "../state/TileViewModel";
import type { DeviceLabel } from "../state/MediaDevices";
import { createCallFooterViewModel } from "./CallFooterViewModel";
@@ -65,10 +64,6 @@ function buildMinimalCallViewModel(layout: Layout): CallViewModel {
const gridLayout: Layout = {
type: "grid",
grid: [],
spotlightAlignment$: new BehaviorSubject<Alignment>({
inline: "end",
block: "end",
}),
setVisibleTiles: (_: number) => {},
};
+100 -150
View File
@@ -5,28 +5,25 @@ SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { act, render, screen } from "@testing-library/react";
import { BehaviorSubject } from "rxjs";
import { describe, expect, it, vi } from "vitest";
import { render } from "@testing-library/react";
import { expect, test, vi } from "vitest";
import { arrangeTiles, type Bounds } from "./CallLayout";
import { type CallLayout } from "./CallLayout";
import { makeGridLayout } from "./GridLayout";
import { makeSpotlightLandscapeLayout } from "./SpotlightLandscapeLayout";
import { makeSpotlightPortraitLayout } from "./SpotlightPortraitLayout";
import { type DragCallback, type LayoutProps } from "./Grid";
import { type LayoutProps } from "./Grid";
import { constant } from "../state/Behavior";
import {
type Alignment,
type GridLayout,
type SpotlightLandscapeLayout,
type SpotlightPortraitLayout,
} from "../state/layout-types";
import { type Alignment } from "../state/layout-types";
import { type UserMediaViewModel } from "../state/media/UserMediaViewModel";
import {
GridTileViewModel,
SpotlightTileViewModel,
type TileViewModel,
} from "../state/TileViewModel";
import { makeOneOnOneMobileLayout } from "./OneOnOneMobileLayout";
import { makeSpotlightExpandedLayout } from "./SpotlightExpandedLayout";
import { makeOneOnOneDesktopLayout } from "./OneOnOneDesktopLayout";
// Grid supplies these hooks to the layouts it renders; on their own the
// layouts have no grid to report to
@@ -36,152 +33,105 @@ vi.mock("./Grid", async (importOriginal) => ({
useVisibleTiles: (): void => {},
}));
describe("grid layout", () => {
it("sizes the tiles from the minimum bounds", () => {
const minBounds$ = new BehaviorSubject<Bounds>({ width: 800, height: 600 });
const { scrolling: Scrolling } = makeGridLayout({ minBounds$ });
render(<Scrolling model={gridModel()} Slot={Slot} />);
const layer = screen.getByTestId("slot-0").parentElement!;
expect(layer.style.width).toBe("800px");
expect(layer.style.getPropertyValue("--width")).toBe(
`${Math.floor(arrangeTiles(800, 600, 3).tileWidth)}px`,
);
const minBounds$ = constant({ width: 800, height: 800 });
act(() => minBounds$.next({ width: 400, height: 300 }));
expect(layer.style.width).toBe("400px");
expect(layer.style.getPropertyValue("--width")).toBe(
`${Math.floor(arrangeTiles(400, 300, 3).tileWidth)}px`,
);
});
it("moves the spotlight only when a drag reaches another corner", () => {
const model = gridModel();
const { fixed: Fixed } = makeGridLayout({ minBounds$: bounds });
render(<Fixed model={model} Slot={Slot} />);
const alignments: Alignment[] = [];
model.spotlightAlignment$.subscribe((a) => alignments.push(a));
const drag = drags.get("spotlight")!;
// Still in the bottom right corner
act(() => drag({ x: 0, y: 0, xRatio: 0.9, yRatio: 0.9 }));
// Over to the bottom left
act(() => drag({ x: 0, y: 0, xRatio: 0.1, yRatio: 0.9 }));
// And a little further, still bottom left
act(() => drag({ x: 0, y: 0, xRatio: 0.2, yRatio: 0.8 }));
// Up to the top left
act(() => drag({ x: 0, y: 0, xRatio: 0.2, yRatio: 0.1 }));
expect(alignments).toEqual([
{ inline: "end", block: "end" },
{ inline: "start", block: "end" },
{ inline: "start", block: "start" },
]);
const slot = screen.getByTestId("slot-spotlight");
expect(slot.getAttribute("data-inline-alignment")).toBe("start");
expect(slot.getAttribute("data-block-alignment")).toBe("start");
});
});
describe("spotlight landscape layout", () => {
it("puts the spotlight in the fixed layer and the grid in the scrolling one", () => {
const { fixed: Fixed, scrolling: Scrolling } = makeSpotlightLandscapeLayout(
{ minBounds$: bounds },
);
const model = landscapeModel();
const { unmount } = render(<Fixed model={model} Slot={Slot} />);
expect(screen.getByTestId("slot-spotlight")).toBeInTheDocument();
unmount();
render(<Scrolling model={model} Slot={Slot} />);
expect(slotIds()).toEqual(model.grid.map((tile) => tile.id));
});
});
describe("spotlight portrait layout", () => {
it("sizes the grid tiles from the minimum width", () => {
const minBounds$ = new BehaviorSubject<Bounds>({ width: 300, height: 700 });
const { scrolling: Scrolling } = makeSpotlightPortraitLayout({
minBounds$,
});
const model = portraitModel();
render(<Scrolling model={model} Slot={Slot} />);
expect(slotIds()).toEqual(model.grid.map((tile) => tile.id));
const layer = screen.getByTestId(`slot-${model.grid[0].id}`).parentElement!
.parentElement!;
expect(layer.style.getPropertyValue("--grid-tile-width")).toBe(
`${Math.floor(arrangeTiles(300, 300, 3).tileWidth)}px`,
);
act(() => minBounds$.next({ width: 500, height: 700 }));
expect(layer.style.getPropertyValue("--grid-tile-width")).toBe(
`${Math.floor(arrangeTiles(500, 500, 3).tileWidth)}px`,
);
});
});
const bounds = constant<Bounds>({ width: 800, height: 600 });
// The layouts only pass tile view models through to their slots
const media = {} as UserMediaViewModel;
const spotlight = new SpotlightTileViewModel(
constant([]),
constant(false),
constant("solid"),
const spotlight = {
vm: new SpotlightTileViewModel(
constant([]),
constant(false),
constant("solid"),
),
alignment$: constant<Alignment>({ inline: "end", block: "end" }),
onDrag: () => {},
};
const grid = Array.from(
{ length: 3 },
() => new GridTileViewModel(constant({} as UserMediaViewModel)),
);
const tiles = (): GridTileViewModel[] =>
Array.from({ length: 3 }, () => new GridTileViewModel(constant(media)));
const pip = {
alignment$: constant<Alignment>({ inline: "end", block: "end" }),
onDrag: () => {},
vm: { id: "pip" } as GridTileViewModel,
size$: constant("sm" as const),
};
function gridModel(): GridLayout {
return {
type: "grid",
spotlight,
grid: tiles(),
spotlightAlignment$: new BehaviorSubject<Alignment>({
inline: "end",
block: "end",
}),
setVisibleTiles: () => {},
};
}
function landscapeModel(): SpotlightLandscapeLayout {
return {
type: "spotlight-landscape",
spotlight,
grid: tiles(),
setVisibleTiles: () => {},
};
}
function portraitModel(): SpotlightPortraitLayout {
return {
type: "spotlight-portrait",
spotlight,
grid: tiles(),
setVisibleTiles: () => {},
};
}
// Slots record their drag callback so a test can drag them, and show the id
// of the tile they hold
const drags = new Map<string, DragCallback | undefined>();
const Slot: LayoutProps<unknown, TileViewModel, HTMLDivElement>["Slot"] = ({
id,
model,
onDrag,
children,
...props
}) => {
drags.set(id, onDrag);
return (
<div data-testid={`slot-${id}`} {...props}>
{model instanceof GridTileViewModel ? model.id : "spotlight"}
</div>
);
};
}) => (
<div data-testid={`slot-${id}`} {...props}>
{children}
</div>
);
function slotIds(): string[] {
return screen
.getAllByTestId(/^slot-/)
.map((slot) => slot.textContent ?? "")
.filter((id) => id !== "spotlight");
function snapshotLayout<Model>(layout: CallLayout<Model>, model: Model): void {
const {
foreground,
scrolling: Scrolling,
fixed: Fixed,
} = layout({ minBounds$ });
const scrollingLayer = <Scrolling model={model} Slot={Slot} />;
const fixedLayer = <Fixed model={model} Slot={Slot} />;
const { container } = render(
foreground === "fixed" ? (
<>
{scrollingLayer}
{fixedLayer}
</>
) : (
<>
{fixedLayer}
{scrollingLayer}
</>
),
);
expect(container).toMatchSnapshot();
}
test("grid layout", () =>
snapshotLayout(makeGridLayout, {
type: "grid",
spotlight,
grid,
setVisibleTiles: () => {},
}));
test("spotlight landscape layout", () =>
snapshotLayout(makeSpotlightLandscapeLayout, {
type: "spotlight-landscape",
spotlight,
grid,
setVisibleTiles: () => {},
}));
test("spotlight portrait layout", () =>
snapshotLayout(makeSpotlightPortraitLayout, {
type: "spotlight-portrait",
spotlight,
grid,
setVisibleTiles: () => {},
}));
test("spotlight expanded layout", () =>
snapshotLayout(makeSpotlightExpandedLayout, {
type: "spotlight-expanded",
spotlight: spotlight.vm,
pip,
}));
test("one-on-one desktop layout", () =>
snapshotLayout(makeOneOnOneDesktopLayout, {
type: "one-on-one-desktop",
spotlight: { id: "spotlight" } as GridTileViewModel,
pip,
}));
test("one-on-one mobile layout", () =>
snapshotLayout(makeOneOnOneMobileLayout, {
type: "one-on-one-mobile",
spotlight: spotlight.vm,
pip,
}));
+12 -29
View File
@@ -40,6 +40,7 @@ import { useMergedRefs } from "../useMergedRefs";
import { TileWrapper } from "./TileWrapper";
import { usePrefersReducedMotion } from "../usePrefersReducedMotion";
import { useInitial } from "../useInitial";
import { type DragCallback } from "../state/layout-types";
const MAX_ANIMATED_TILES = 50; // Capped for performance reasons
@@ -83,7 +84,10 @@ interface DragState {
cursorY: number;
}
interface SlotProps<Model> extends Omit<ComponentProps<"div">, "onDrag"> {
export interface SlotProps<Model> extends Omit<
ComponentProps<"div">,
"onDrag"
> {
id: string;
model: Model;
onDrag?: DragCallback;
@@ -181,27 +185,6 @@ export interface TileProps<Model, R extends HTMLElement> {
model: Model;
}
interface Drag {
/**
* The X coordinate of the dragged tile in grid space.
*/
x: number;
/**
* The Y coordinate of the dragged tile in grid space.
*/
y: number;
/**
* The X coordinate of the dragged tile, as a scalar of the grid width.
*/
xRatio: number;
/**
* The Y coordinate of the dragged tile, as a scalar of the grid height.
*/
yRatio: number;
}
export type DragCallback = (drag: Drag) => void;
interface LayoutMemoProps<
LayoutModel,
TileModel,
@@ -464,13 +447,13 @@ export function Grid<
)
.catch(logger.error);
if (endOfGesture)
callback({
x: tileX,
y: tileY,
xRatio: tileX / (gridBounds.width - tile.width),
yRatio: tileY / (gridBounds.height - tile.height),
});
callback({
x: tileX,
y: tileY,
xRatio: tileX / (gridBounds.width - tile.width),
yRatio: tileY / (gridBounds.height - tile.height),
endOfGesture,
});
};
// Callback for useDrag. We could call useDrag here, but the default
+30 -25
View File
@@ -8,15 +8,21 @@ Please see LICENSE in the repository root for full details.
import {
type CSSProperties,
type ReactNode,
useCallback,
useMemo,
type FC,
type ComponentType,
} from "react";
import { type GridLayout as GridLayoutModel } from "../state/layout-types.ts";
import styles from "./GridLayout.module.css";
import { type CallLayout, arrangeTiles } from "./CallLayout";
import { type DragCallback, useUpdateLayout, useVisibleTiles } from "./Grid";
import { type SlotProps, useUpdateLayout, useVisibleTiles } from "./Grid";
import { useBehavior } from "../useBehavior";
import {
type SpotlightTileViewModel,
type TileViewModel,
} from "../state/TileViewModel.ts";
import { type FloatingTile } from "../state/FloatingTile.ts";
interface GridCSSProperties extends CSSProperties {
"--gap": string;
@@ -24,6 +30,27 @@ interface GridCSSProperties extends CSSProperties {
"--height": string;
}
interface SpotlightSlotProps {
model: FloatingTile & { vm: SpotlightTileViewModel };
Slot: ComponentType<SlotProps<TileViewModel>>;
}
const SpotlightSlot: FC<SpotlightSlotProps> = ({ model, Slot }) => {
useUpdateLayout();
const alignment = useBehavior(model.alignment$);
return (
<Slot
className={styles.slot}
id="spotlight"
model={model.vm}
onDrag={model.onDrag}
data-block-alignment={alignment.block}
data-inline-alignment={alignment.inline}
/>
);
};
/**
* An implementation of the "grid" layout, in which all participants are shown
* together in a scrolling grid.
@@ -37,32 +64,10 @@ export const makeGridLayout: CallLayout<GridLayoutModel> = ({
// lives
fixed: function GridLayoutFixed({ ref, model, Slot }): ReactNode {
useUpdateLayout();
const alignment = useBehavior(model.spotlightAlignment$);
const onDragSpotlight: DragCallback = useCallback(
({ xRatio, yRatio }) => {
const block = yRatio < 0.5 ? "start" : "end";
const inline = xRatio < 0.5 ? "start" : "end";
// A drag reports the same alignment on every move; only a change is
// worth a re-render
const current = model.spotlightAlignment$.value;
if (current.block !== block || current.inline !== inline)
model.spotlightAlignment$.next({ block, inline });
},
[model.spotlightAlignment$],
);
return (
<div ref={ref} className={styles.fixed}>
{model.spotlight && (
<Slot
className={styles.slot}
id="spotlight"
model={model.spotlight}
onDrag={onDragSpotlight}
data-block-alignment={alignment.block}
data-inline-alignment={alignment.inline}
/>
<SpotlightSlot model={model.spotlight} Slot={Slot} />
)}
</div>
);
+6 -15
View File
@@ -6,13 +6,13 @@ SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { type ReactNode, useCallback, useMemo } from "react";
import { type ReactNode, useMemo } from "react";
import classNames from "classnames";
import { type OneOnOneDesktopLayout as OneOnOneDesktopLayoutModel } from "../state/layout-types.ts";
import { type CallLayout, arrangeTiles } from "./CallLayout";
import styles from "./OneOnOneDesktopLayout.module.css";
import { type DragCallback, useUpdateLayout } from "./Grid";
import { useUpdateLayout } from "./Grid";
import { useBehavior } from "../useBehavior";
/**
@@ -37,21 +37,12 @@ export const makeOneOnOneDesktopLayout: CallLayout<
}): ReactNode {
useUpdateLayout();
const { width, height } = useBehavior(minBounds$);
const pipAlignment = useBehavior(model.pipAlignment$);
const pipAlignment = useBehavior(model.pip.alignment$);
const { tileWidth, tileHeight } = useMemo(
() => arrangeTiles(width, height, 1),
[width, height],
);
const onDragLocalTile: DragCallback = useCallback(
({ xRatio, yRatio }) =>
model.pipAlignment$.next({
block: yRatio < 0.5 ? "start" : "end",
inline: xRatio < 0.5 ? "start" : "end",
}),
[model.pipAlignment$],
);
return (
<div ref={ref} className={styles.layer}>
<Slot
@@ -62,9 +53,9 @@ export const makeOneOnOneDesktopLayout: CallLayout<
>
<Slot
className={classNames(styles.slot, styles.local)}
id={model.pip.id}
model={model.pip}
onDrag={onDragLocalTile}
id={model.pip.vm.id}
model={model.pip.vm}
onDrag={model.pip.onDrag}
data-block-alignment={pipAlignment.block}
data-inline-alignment={pipAlignment.inline}
/>
+32 -24
View File
@@ -6,14 +6,43 @@ SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { type ReactNode, useCallback } from "react";
import { type ComponentType, type FC, type ReactNode } from "react";
import classNames from "classnames";
import { type OneOnOneMobileLayout as OneOnOneMobileLayoutModel } from "../state/layout-types.ts";
import { type CallLayout } from "./CallLayout";
import styles from "./OneOnOneMobileLayout.module.css";
import { type DragCallback, useUpdateLayout } from "./Grid";
import { type SlotProps, useUpdateLayout } from "./Grid";
import { useBehavior } from "../useBehavior";
import { type FloatingTile } from "../state/FloatingTile.ts";
import {
type GridTileViewModel,
type TileViewModel,
} from "../state/TileViewModel.ts";
import { type Behavior } from "../state/Behavior.ts";
interface PipSlotProps {
model: FloatingTile & { vm: GridTileViewModel; size$: Behavior<"sm" | "lg"> };
Slot: ComponentType<SlotProps<TileViewModel>>;
}
const PipSlot: FC<PipSlotProps> = ({ model, Slot }) => {
useUpdateLayout();
const size = useBehavior(model.size$);
const alignment = useBehavior(model.alignment$);
return (
<Slot
className={classNames(styles.pip)}
id={model.vm.id}
model={model.vm}
onDrag={model.onDrag}
data-size={size}
data-block-alignment={alignment.block}
data-inline-alignment={alignment.inline}
/>
);
};
/**
* An implementation of the "one-on-one" layout for mobile platforms, in which
@@ -44,30 +73,9 @@ export const makeOneOnOneMobileLayout: CallLayout<
Slot,
}): ReactNode {
useUpdateLayout();
const pipSize = useBehavior(model.pipSize$);
const pipAlignment = useBehavior(model.pipAlignment$);
const onDragLocalTile: DragCallback = useCallback(
({ xRatio, yRatio }) =>
model.pipAlignment$.next({
block: yRatio < 0.5 ? "start" : "end",
inline: xRatio < 0.5 ? "start" : "end",
}),
[model.pipAlignment$],
);
return (
<div ref={ref} className={styles.layer}>
{model.pip && (
<Slot
className={classNames(styles.pip)}
id={model.pip.id}
model={model.pip}
onDrag={onDragLocalTile}
data-size={pipSize}
data-block-alignment={pipAlignment.block}
data-inline-alignment={pipAlignment.inline}
/>
)}
{model.pip && <PipSlot model={model.pip} Slot={Slot} />}
</div>
);
},
+29 -24
View File
@@ -5,13 +5,39 @@ SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { type ReactNode, useCallback } from "react";
import { type ReactNode, type ComponentType, type FC } from "react";
import { type SpotlightExpandedLayout as SpotlightExpandedLayoutModel } from "../state/layout-types.ts";
import { type CallLayout } from "./CallLayout";
import { type DragCallback, useUpdateLayout } from "./Grid";
import { type SlotProps, useUpdateLayout } from "./Grid";
import styles from "./SpotlightExpandedLayout.module.css";
import { useBehavior } from "../useBehavior";
import { type FloatingTile } from "../state/FloatingTile.ts";
import {
type GridTileViewModel,
type TileViewModel,
} from "../state/TileViewModel.ts";
interface PipSlotProps {
model: FloatingTile & { vm: GridTileViewModel };
Slot: ComponentType<SlotProps<TileViewModel>>;
}
const PipSlot: FC<PipSlotProps> = ({ model, Slot }) => {
useUpdateLayout();
const alignment = useBehavior(model.alignment$);
return (
<Slot
className={styles.pip}
id={model.vm.id}
model={model.vm}
onDrag={model.onDrag}
data-block-alignment={alignment.block}
data-inline-alignment={alignment.inline}
/>
);
};
/**
* An implementation of the "expanded spotlight" layout, in which the spotlight
@@ -28,7 +54,6 @@ export const makeSpotlightExpandedLayout: CallLayout<
Slot,
}): ReactNode {
useUpdateLayout();
return (
<div ref={ref} className={styles.layer}>
<Slot
@@ -46,29 +71,9 @@ export const makeSpotlightExpandedLayout: CallLayout<
Slot,
}): ReactNode {
useUpdateLayout();
const pipAlignment = useBehavior(model.pipAlignment$);
const onDragPip: DragCallback = useCallback(
({ xRatio, yRatio }) =>
model.pipAlignment$.next({
block: yRatio < 0.5 ? "start" : "end",
inline: xRatio < 0.5 ? "start" : "end",
}),
[model.pipAlignment$],
);
return (
<div ref={ref} className={styles.layer}>
{model.pip && (
<Slot
className={styles.pip}
id={model.pip.id}
model={model.pip}
onDrag={onDragPip}
data-block-alignment={pipAlignment.block}
data-inline-alignment={pipAlignment.inline}
/>
)}
{model.pip && <PipSlot model={model.pip} Slot={Slot} />}
</div>
);
},
+2 -2
View File
@@ -40,7 +40,7 @@ export const makeSpotlightLandscapeLayout: CallLayout<
<Slot
className={styles.slot}
id="spotlight"
model={model.spotlight}
model={model.spotlight.vm}
/>
</div>
<div className={styles.grid} />
@@ -59,7 +59,7 @@ export const makeSpotlightLandscapeLayout: CallLayout<
return (
<div ref={ref} className={styles.layer}>
<SpotlightSlot media$={model.spotlight.media$} />
<SpotlightSlot media$={model.spotlight.vm.media$} />
<div className={styles.grid}>
{model.grid.map((m) => (
<Slot key={m.id} className={styles.slot} id={m.id} model={m} />
+2 -2
View File
@@ -43,7 +43,7 @@ export const makeSpotlightPortraitLayout: CallLayout<
<Slot
className={styles.slot}
id="spotlight"
model={model.spotlight}
model={model.spotlight.vm}
/>
</div>
</div>
@@ -65,7 +65,7 @@ export const makeSpotlightPortraitLayout: CallLayout<
width,
model.grid.length,
);
const withIndicators = useBehavior(model.spotlight.media$).length > 1;
const withIndicators = useBehavior(model.spotlight.vm.media$).length > 1;
return (
<div
@@ -0,0 +1,186 @@
// Vitest Snapshot v1, https://vitest.dev/guide/snapshot.html
exports[`grid layout 1`] = `
<div>
<div
class="_scrolling_e8555d"
style="width: 800px; --gap: 20px; --width: 370px; --height: 277px;"
>
<div
class="_slot_e8555d"
data-testid="slot-0"
/>
<div
class="_slot_e8555d"
data-testid="slot-1"
/>
<div
class="_slot_e8555d"
data-testid="slot-2"
/>
</div>
<div
class="_fixed_e8555d"
>
<div
class="_slot_e8555d"
data-block-alignment="end"
data-inline-alignment="end"
data-testid="slot-spotlight"
/>
</div>
</div>
`;
exports[`one-on-one desktop layout 1`] = `
<div>
<div
class="_layer_185815"
>
<div
class="_container_185815"
data-testid="slot-spotlight"
style="width: 760px; height: 570px;"
>
<div
class="_slot_185815 _local_185815"
data-block-alignment="end"
data-inline-alignment="end"
data-testid="slot-pip"
/>
</div>
</div>
<div />
</div>
`;
exports[`one-on-one mobile layout 1`] = `
<div>
<div
class="_layer_c81c30"
>
<div
class="_spotlight_c81c30"
data-testid="slot-spotlight"
/>
</div>
<div
class="_layer_c81c30"
>
<div
class="_pip_c81c30"
data-block-alignment="end"
data-inline-alignment="end"
data-size="sm"
data-testid="slot-pip"
/>
</div>
</div>
`;
exports[`spotlight expanded layout 1`] = `
<div>
<div
class="_layer_e70191"
>
<div
class="_spotlight_e70191"
data-testid="slot-spotlight"
/>
</div>
<div
class="_layer_e70191"
>
<div
class="_pip_e70191"
data-block-alignment="end"
data-inline-alignment="end"
data-testid="slot-pip"
/>
</div>
</div>
`;
exports[`spotlight landscape layout 1`] = `
<div>
<div
class="_layer_29809c"
>
<div
class="_spotlight_29809c"
>
<div
class="_slot_29809c"
data-testid="slot-spotlight"
/>
</div>
<div
class="_grid_29809c"
/>
</div>
<div
class="_layer_29809c"
>
<div
class="_spotlight_29809c"
/>
<div
class="_grid_29809c"
>
<div
class="_slot_29809c"
data-testid="slot-0"
/>
<div
class="_slot_29809c"
data-testid="slot-1"
/>
<div
class="_slot_29809c"
data-testid="slot-2"
/>
</div>
</div>
</div>
`;
exports[`spotlight portrait layout 1`] = `
<div>
<div
class="_layer_065bbc"
style="--grid-gap: 20px; --grid-tile-width: 370px; --grid-tile-height: 277px;"
>
<div
class="_spotlight_065bbc"
/>
<div
class="_grid_065bbc"
>
<div
class="_slot_065bbc"
data-testid="slot-0"
/>
<div
class="_slot_065bbc"
data-testid="slot-1"
/>
<div
class="_slot_065bbc"
data-testid="slot-2"
/>
</div>
</div>
<div
class="_layer_065bbc"
>
<div
class="_spotlight_065bbc"
>
<div
class="_slot_065bbc"
data-testid="slot-spotlight"
/>
</div>
</div>
</div>
`;
@@ -1228,7 +1228,7 @@ exports[`should have a close button when the host can dismiss us 1`] = `
Call is not supported
</h1>
<p>
The server is not configured to work with Element Call. Please contact your server admin (Domain: example.com, Error Code: MISSING_MATRIX_RTC_TRANSPORT).
The server is not configured to work with Element Call and no other call participant provides a usable transport. Please contact your server admin (Domain: example.com, Error Code: MISSING_MATRIX_RTC_TRANSPORT).
</p>
<button
class="_button_1nw83_8"
@@ -1381,7 +1381,7 @@ exports[`should render the error page with link back to home 1`] = `
Call is not supported
</h1>
<p>
The server is not configured to work with Element Call. Please contact your server admin (Domain: example.com, Error Code: MISSING_MATRIX_RTC_TRANSPORT).
The server is not configured to work with Element Call and no other call participant provides a usable transport. Please contact your server admin (Domain: example.com, Error Code: MISSING_MATRIX_RTC_TRANSPORT).
</p>
<button
class="_button_1nw83_8"
@@ -1534,7 +1534,7 @@ exports[`should report correct error for 'Call is not supported' 1`] = `
Call is not supported
</h1>
<p>
The server is not configured to work with Element Call. Please contact your server admin (Domain: example.com, Error Code: MISSING_MATRIX_RTC_TRANSPORT).
The server is not configured to work with Element Call and no other call participant provides a usable transport. Please contact your server admin (Domain: example.com, Error Code: MISSING_MATRIX_RTC_TRANSPORT).
</p>
<button
class="_button_1nw83_8"
+1 -1
View File
@@ -517,7 +517,7 @@ export const DeveloperSettingsTab: FC<Props> = ({
client,
membership: { userId, deviceId, memberId: "" },
transport,
serverName,
serverName: transport.server_name ?? serverName,
roomId,
slotId: "org.example.fake-slot#room",
role: "publisher",
+10 -10
View File
@@ -155,7 +155,7 @@ function summarizeLayout$(l$: Observable<Layout>): Observable<LayoutSummary> {
case "grid":
return combineLatest(
[
l.spotlight?.media$ ?? constant(undefined),
l.spotlight?.vm.media$ ?? constant(undefined),
...l.grid.map((vm) => vm.media$),
],
(spotlight, ...grid) => ({
@@ -167,7 +167,7 @@ function summarizeLayout$(l$: Observable<Layout>): Observable<LayoutSummary> {
case "spotlight-landscape":
case "spotlight-portrait":
return combineLatest(
[l.spotlight.media$, ...l.grid.map((vm) => vm.media$)],
[l.spotlight.vm.media$, ...l.grid.map((vm) => vm.media$)],
(spotlight, ...grid) => ({
type: l.type,
spotlight: spotlight.map((vm) => vm.id),
@@ -176,7 +176,7 @@ function summarizeLayout$(l$: Observable<Layout>): Observable<LayoutSummary> {
);
case "spotlight-expanded":
return combineLatest(
[l.spotlight.media$, l.pip?.media$ ?? constant(undefined)],
[l.spotlight.media$, l.pip?.vm.media$ ?? constant(undefined)],
(spotlight, pip) => ({
type: l.type,
spotlight: spotlight.map((vm) => vm.id),
@@ -185,7 +185,7 @@ function summarizeLayout$(l$: Observable<Layout>): Observable<LayoutSummary> {
);
case "one-on-one-desktop":
return combineLatest(
[l.spotlight.media$, l.pip.media$],
[l.spotlight.media$, l.pip.vm.media$],
(spotlight, pip) => ({
type: l.type,
spotlight: spotlight.id,
@@ -196,8 +196,8 @@ function summarizeLayout$(l$: Observable<Layout>): Observable<LayoutSummary> {
return combineLatest(
[
l.spotlight.media$,
l.pip?.media$ ?? constant(undefined),
l.pipSize$,
l.pip?.vm.media$ ?? constant(undefined),
l.pip?.size$ ?? constant(undefined),
],
(spotlight, pip, pipSize) => ({
type: l.type,
@@ -506,7 +506,7 @@ describe.each(modes)("CallViewModel (%s mode)", (mode) => {
type: "one-on-one-mobile",
spotlight: [`${aliceId}:0`],
pip: undefined,
pipSize: "lg",
pipSize: undefined,
},
b: {
type: "one-on-one-mobile",
@@ -544,7 +544,7 @@ describe.each(modes)("CallViewModel (%s mode)", (mode) => {
type: "one-on-one-mobile",
spotlight: [`${aliceId}:0`],
pip: undefined,
pipSize: "lg",
pipSize: undefined,
},
});
// It wouldn't be clear whether Alice or Bob is the remote video tile,
@@ -592,7 +592,7 @@ describe.each(modes)("CallViewModel (%s mode)", (mode) => {
type: "one-on-one-mobile",
spotlight: [`${aliceId}:0`],
pip: undefined,
pipSize: "sm",
pipSize: undefined,
},
b: {
type: "spotlight-landscape",
@@ -791,7 +791,7 @@ describe.each(modes)("CallViewModel (%s mode)", (mode) => {
type: "one-on-one-mobile",
spotlight: [`${aliceId}:0`],
pip: undefined,
pipSize: "lg",
pipSize: undefined,
},
b: {
// In a larger window, expect the one-on-one desktop layout
+32 -32
View File
@@ -78,14 +78,13 @@ import {
type ReactionInfo,
type ReactionOption,
} from "../../reactions";
import { shallowEquals as shallowArrayEquals } from "../../utils/array";
import { shallowEquals as shallowObjectEquals } from "../../utils/object";
import { shallowEquals as arrayShallowEquals } from "../../utils/array";
import { type MediaDevice, type MediaDevices } from "../MediaDevices";
import { type Handler, type MuteStates } from "../MuteStates";
import { HeaderStyle, type UrlParams } from "../../UrlParams";
import { type HostBridge, nullHostBridge } from "../../HostBridge";
import {
type Alignment,
layoutShallowEquals,
type Layout,
type LayoutMedia,
type WindowMode,
@@ -126,6 +125,7 @@ import {
type RingingMediaViewModel,
} from "../media/RingingMediaViewModel.ts";
import { type GridTileViewModel } from "../TileViewModel.ts";
import { createFloatingTile } from "../FloatingTile.ts";
/** How a call captures and encodes the tracks it publishes. */
export interface CaptureSettings {
@@ -830,7 +830,7 @@ export function createCallViewModel$(
bins.sort(([, bin1], [, bin2]) => bin1 - bin2).map(([m]) => m),
);
}),
distinctUntilChanged(shallowArrayEquals),
distinctUntilChanged(arrayShallowEquals),
),
);
@@ -862,13 +862,18 @@ export function createCallViewModel$(
[ringingMedia$, screenShares$, spotlightSpeaker$, localUserMediaForPip$],
(ringing, screenShares, speaker, localPip) =>
computeSpotlight({ ringing, screenShares, speaker, localPip }),
).pipe(distinctUntilChanged(shallowObjectEquals)),
).pipe(
distinctUntilChanged(
(a, b) =>
arrayShallowEquals(a.spotlight, b.spotlight) && a.pip === b.pip,
),
),
);
const spotlight$ = scope.behavior<MediaViewModel[]>(
spotlightAndPip$.pipe(
map(({ spotlight }) => spotlight),
distinctUntilChanged<MediaViewModel[]>(shallowArrayEquals),
distinctUntilChanged<MediaViewModel[]>(arrayShallowEquals),
),
);
@@ -1029,7 +1034,7 @@ export function createCallViewModel$(
desktop: platform === "desktop",
}),
),
distinctUntilChanged(shallowObjectEquals<LayoutMedia>),
distinctUntilChanged(layoutShallowEquals),
),
);
@@ -1221,32 +1226,28 @@ export function createCallViewModel$(
);
/**
* The alignment of the floating spotlight tile, if present.
* State of the floating spotlight tile, if present.
*/
const spotlightAlignment$ = new BehaviorSubject<Alignment>({
const floatingSpotlightTile = createFloatingTile(scope, {
inline: "end",
block: "end",
});
/**
* The size of the small picture-in-picture tile, if present, when in portrait.
* State of the small picture-in-picture tile, if present, when in landscape.
*/
const portraitPipSize$ = scope.behavior(
showFooter$.pipe(map((showFooter) => (showFooter ? "lg" : "sm"))),
);
/**
* The alignment of the small picture-in-picture tile, if present, when in portrait.
*/
const portraitPipAlignment$ = new BehaviorSubject<Alignment>({
inline: "end",
block: "end",
});
/**
* The alignment of the small picture-in-picture tile, if present, when in landscape.
*/
const landscapePipAlignment$ = new BehaviorSubject<Alignment>({
const landscapePipTile = createFloatingTile(scope, {
inline: "end",
block: "start",
});
/**
* State of the small picture-in-picture tile, if present, when in portrait.
*/
const portraitPipTile = {
...createFloatingTile(scope, { inline: "end", block: "end" }),
size$: scope.behavior(
showFooter$.pipe(map((showFooter) => (showFooter ? "lg" : "sm"))),
),
};
// There is a cyclical dependency here: the layout algorithms want to know
// which tiles are on screen, but to know which tiles are on screen we have to
@@ -1276,7 +1277,7 @@ export function createCallViewModel$(
case "spotlight-portrait":
[layout, newTiles] = gridLikeLayout(
media,
spotlightAlignment$,
floatingSpotlightTile,
visibleTiles,
setVisibleTiles,
prevTiles,
@@ -1286,26 +1287,25 @@ export function createCallViewModel$(
case "spotlight-expanded":
[layout, newTiles] = spotlightExpandedLayout(
media,
landscapePipAlignment$,
landscapePipTile,
prevTiles,
);
break;
case "one-on-one-desktop":
[layout, newTiles] = oneOnOneDesktopLayout(
media,
landscapePipAlignment$,
landscapePipTile,
prevTiles,
);
pip = layout.pip;
pip = layout.pip.vm;
break;
case "one-on-one-mobile":
[layout, newTiles] = oneOnOneMobileLayout(
media,
portraitPipSize$,
portraitPipAlignment$,
portraitPipTile,
prevTiles,
);
pip = layout.pip;
pip = layout.pip?.vm;
break;
case "pip":
[layout, newTiles] = pipLayout(media, prevTiles);
@@ -1330,7 +1330,7 @@ export function createCallViewModel$(
layoutInternals$.pipe(
map(({ layout }) => layout),
// Drop redundant layout updates before they would hit React.
distinctUntilChanged(shallowObjectEquals<Layout>),
distinctUntilChanged(layoutShallowEquals),
),
);
+38
View File
@@ -0,0 +1,38 @@
/*
Copyright 2026 Element Creations Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { test } from "vitest";
import { withTestScheduler } from "../utils/test";
import { createFloatingTile } from "./FloatingTile";
test("FloatingTile changes alignment on drag", () =>
withTestScheduler(({ scope, expectObservable, schedule }) => {
// Tile starts in the top left
const { alignment$, onDrag } = createFloatingTile(scope, {
inline: "start",
block: "start",
});
// Drag to all four corners, and then repeatedly into the top left.
schedule("-↗↘↙↖↖", {
"↗": () =>
onDrag({ x: 100, y: 0, xRatio: 1, yRatio: 0, endOfGesture: false }),
"↘": () =>
onDrag({ x: 100, y: 100, xRatio: 1, yRatio: 1, endOfGesture: false }),
"↙": () =>
onDrag({ x: 0, y: 100, xRatio: 0, yRatio: 1, endOfGesture: false }),
"↖": () =>
onDrag({ x: 0, y: 0, xRatio: 0, yRatio: 0, endOfGesture: false }),
});
// Alignment should change accordingly. Notably, the alignment should *not*
// update when repeatedly dragged into the same corner, for performance.
expectObservable(alignment$).toBe("↖↗↘↙↖-", {
"↗": { inline: "end", block: "start" },
"↘": { inline: "end", block: "end" },
"↙": { inline: "start", block: "end" },
"↖": { inline: "start", block: "start" },
});
}));
+43
View File
@@ -0,0 +1,43 @@
/*
Copyright 2026 Element Creations Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { distinctUntilChanged, map, startWith, Subject } from "rxjs";
import { type Behavior } from "./Behavior";
import { type DragCallback, type Alignment, type Drag } from "./layout-types";
import { type ObservableScope } from "./ObservableScope";
/**
* Represents a floating tile which may be dragged to different corners of the
* call interface.
*/
export interface FloatingTile {
alignment$: Behavior<Alignment>;
onDrag: DragCallback;
}
export function createFloatingTile(
scope: ObservableScope,
defaultAlignment: Alignment,
): FloatingTile {
const dragEvents$ = new Subject<Drag>();
return {
alignment$: scope.behavior(
dragEvents$.pipe(
map<Drag, Alignment>(({ xRatio, yRatio }) => ({
inline: xRatio < 0.5 ? "start" : "end",
block: yRatio < 0.5 ? "start" : "end",
})),
startWith(defaultAlignment),
distinctUntilChanged(
(a1, a2) => a1.inline === a2.inline && a1.block === a2.block,
),
),
),
onDrag: (drag) => dragEvents$.next(drag),
};
}
+7 -10
View File
@@ -5,14 +5,9 @@ SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { type BehaviorSubject } from "rxjs";
import {
type Alignment,
type Layout,
type LayoutMedia,
} from "./layout-types.ts";
import { type Layout, type LayoutMedia } from "./layout-types.ts";
import { type TileStore } from "./TileStore";
import { type FloatingTile } from "./FloatingTile.ts";
export type GridLikeLayoutType =
| "grid"
@@ -25,7 +20,7 @@ export type GridLikeLayoutType =
*/
export function gridLikeLayout(
media: LayoutMedia & { type: GridLikeLayoutType },
spotlightAlignment$: BehaviorSubject<Alignment>,
floatingSpotlightTile: FloatingTile,
visibleTiles: number,
setVisibleTiles: (value: number) => void,
prevTiles: TileStore,
@@ -42,9 +37,11 @@ export function gridLikeLayout(
return [
{
type: media.type,
spotlight: tiles.spotlightTile,
spotlight: tiles.spotlightTile && {
vm: tiles.spotlightTile,
...floatingSpotlightTile,
},
grid: tiles.gridTiles,
spotlightAlignment$,
setVisibleTiles,
} as Layout & { type: GridLikeLayoutType },
tiles,
+3 -6
View File
@@ -6,21 +6,19 @@ SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { type BehaviorSubject } from "rxjs";
import {
type Alignment,
type OneOnOneDesktopLayout,
type OneOnOneDesktopLayoutMedia,
} from "./layout-types";
import { type TileStore } from "./TileStore";
import { type FloatingTile } from "./FloatingTile";
/**
* Produces a one-on-one desktop layout with the given media.
*/
export function oneOnOneDesktopLayout(
media: OneOnOneDesktopLayoutMedia,
pipAlignment$: BehaviorSubject<Alignment>,
pipTile: FloatingTile,
prevTiles: TileStore,
): [OneOnOneDesktopLayout, TileStore] {
const update = prevTiles.from(2);
@@ -32,8 +30,7 @@ export function oneOnOneDesktopLayout(
{
type: media.type,
spotlight: tiles.gridTilesByMedia.get(media.spotlight)!,
pip: tiles.gridTilesByMedia.get(media.pip)!,
pipAlignment$,
pip: { vm: tiles.gridTilesByMedia.get(media.pip)!, ...pipTile },
},
tiles,
];
+4 -8
View File
@@ -6,23 +6,20 @@ SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { type BehaviorSubject } from "rxjs";
import {
type Alignment,
type OneOnOneMobileLayout,
type OneOnOneMobileLayoutMedia,
} from "./layout-types";
import { type TileStore } from "./TileStore";
import { type Behavior } from "./Behavior";
import { type FloatingTile } from "./FloatingTile";
/**
* Produces a one-on-one mobile layout with the given media.
*/
export function oneOnOneMobileLayout(
media: OneOnOneMobileLayoutMedia,
pipSize$: Behavior<"sm" | "lg">,
pipAlignment$: BehaviorSubject<Alignment>,
pipTile: FloatingTile & { size$: Behavior<"sm" | "lg"> },
prevTiles: TileStore,
): [OneOnOneMobileLayout, TileStore] {
const update = prevTiles.from(media.pip === undefined ? 0 : 1);
@@ -30,13 +27,12 @@ export function oneOnOneMobileLayout(
if (media.pip !== undefined) update.registerGridTile(media.pip);
const tiles = update.build();
const pipVm = media.pip && tiles.gridTilesByMedia.get(media.pip);
return [
{
type: media.type,
spotlight: tiles.spotlightTile!,
pip: media.pip && tiles.gridTilesByMedia.get(media.pip),
pipSize$,
pipAlignment$,
pip: pipVm && { vm: pipVm, ...pipTile },
},
tiles,
];
+4 -6
View File
@@ -5,21 +5,19 @@ SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { type BehaviorSubject } from "rxjs";
import {
type Alignment,
type SpotlightExpandedLayout,
type SpotlightExpandedLayoutMedia,
} from "./layout-types";
import { type TileStore } from "./TileStore";
import { type FloatingTile } from "./FloatingTile";
/**
* Produces an expanded spotlight layout with the given media.
*/
export function spotlightExpandedLayout(
media: SpotlightExpandedLayoutMedia,
pipAlignment$: BehaviorSubject<Alignment>,
pipTile: FloatingTile,
prevTiles: TileStore,
): [SpotlightExpandedLayout, TileStore] {
const update = prevTiles.from(1);
@@ -27,12 +25,12 @@ export function spotlightExpandedLayout(
if (media.pip !== undefined) update.registerPipTile(media.pip);
const tiles = update.build();
const pipVm = tiles.gridTiles.at(0);
return [
{
type: media.type,
spotlight: tiles.spotlightTile!,
pip: tiles.gridTiles.at(0),
pipAlignment$,
pip: pipVm && { vm: pipVm, ...pipTile },
},
tiles,
];
+82
View File
@@ -0,0 +1,82 @@
/*
Copyright 2026 Element Creations Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { test, expect } from "vitest";
import { layoutShallowEquals, type Layout } from "./layout-types";
import {
type SpotlightTileViewModel,
type GridTileViewModel,
} from "./TileViewModel";
import { type FloatingTile } from "./FloatingTile";
const spotlightTile = {} as unknown as SpotlightTileViewModel;
const gridTile = {} as unknown as GridTileViewModel;
const pipTile = {} as unknown as FloatingTile & { vm: GridTileViewModel };
const spotlightExpanded: Layout = {
type: "spotlight-expanded",
spotlight: spotlightTile,
};
const spotlightPortrait: Layout = {
type: "spotlight-portrait",
spotlight: { vm: spotlightTile },
grid: [gridTile],
setVisibleTiles: () => {},
};
test("layoutShallowEquals considers a layout to be equal to its shallow clone", () =>
expect(layoutShallowEquals(spotlightExpanded, { ...spotlightExpanded })).toBe(
true,
));
test("layoutShallowEquals detects a missing key", () => {
expect(
layoutShallowEquals(spotlightExpanded, {
...spotlightExpanded,
pip: pipTile,
}),
).toBe(false);
expect(
layoutShallowEquals(
{ ...spotlightExpanded, pip: pipTile },
spotlightExpanded,
),
).toBe(false);
});
test("layoutShallowEquals considers tile wrappers with equal contents to be equal", () =>
expect(
layoutShallowEquals(spotlightPortrait, {
...spotlightPortrait,
spotlight: { vm: spotlightTile },
}),
).toBe(true));
test("layoutShallowEquals detects tile wrappers with different contents", () =>
expect(
layoutShallowEquals(spotlightPortrait, {
...spotlightPortrait,
spotlight: { vm: {} as unknown as SpotlightTileViewModel },
}),
).toBe(false));
test("layoutShallowEquals considers grid arrays with equal contents to be equal", () =>
expect(
layoutShallowEquals(spotlightPortrait, {
...spotlightPortrait,
grid: [...spotlightPortrait.grid],
}),
).toBe(true));
test("layoutShallowEquals detects grid arrays with different contents", () =>
expect(
layoutShallowEquals(spotlightPortrait, {
...spotlightPortrait,
grid: [...spotlightPortrait.grid, gridTile],
}),
).toBe(false));
+65 -13
View File
@@ -5,8 +5,6 @@ SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { type BehaviorSubject } from "rxjs";
import { type LocalUserMediaViewModel } from "./media/LocalUserMediaViewModel.ts";
import { type MediaViewModel } from "./media/MediaViewModel.ts";
import { type RingingMediaViewModel } from "./media/RingingMediaViewModel.ts";
@@ -16,6 +14,9 @@ import {
type SpotlightTileViewModel,
} from "./TileViewModel.ts";
import { type Behavior } from "./Behavior.ts";
import { type FloatingTile } from "./FloatingTile.ts";
import { shallowEquals } from "../utils/object.ts";
import { shallowEquals as arrayShallowEquals } from "../utils/array.ts";
/**
* The general shape of the space the call is drawn in. Called a window because
@@ -86,24 +87,48 @@ export interface Alignment {
block: "start" | "end";
}
export interface Drag {
/**
* The X offset of the dragged tile relative to its container.
*/
x: number;
/**
* The Y offset of the dragged tile relative to its container.
*/
y: number;
/**
* The X coordinate of the dragged tile, as a scalar of the container's width.
*/
xRatio: number;
/**
* The Y coordinate of the dragged tile, as a scalar of the container's height.
*/
yRatio: number;
/**
* Whether this event ends the drag gesture.
*/
endOfGesture: boolean;
}
export type DragCallback = (drag: Drag) => void;
export interface GridLayout {
type: "grid";
spotlight?: SpotlightTileViewModel;
spotlight?: FloatingTile & { vm: SpotlightTileViewModel };
grid: GridTileViewModel[];
spotlightAlignment$: BehaviorSubject<Alignment>;
setVisibleTiles: (value: number) => void;
}
export interface SpotlightLandscapeLayout {
type: "spotlight-landscape";
spotlight: SpotlightTileViewModel;
spotlight: { vm: SpotlightTileViewModel };
grid: GridTileViewModel[];
setVisibleTiles: (value: number) => void;
}
export interface SpotlightPortraitLayout {
type: "spotlight-portrait";
spotlight: SpotlightTileViewModel;
spotlight: { vm: SpotlightTileViewModel };
grid: GridTileViewModel[];
setVisibleTiles: (value: number) => void;
}
@@ -111,23 +136,19 @@ export interface SpotlightPortraitLayout {
export interface SpotlightExpandedLayout {
type: "spotlight-expanded";
spotlight: SpotlightTileViewModel;
pip?: GridTileViewModel;
pipAlignment$: BehaviorSubject<Alignment>;
pip?: FloatingTile & { vm: GridTileViewModel };
}
export interface OneOnOneDesktopLayout {
type: "one-on-one-desktop";
spotlight: GridTileViewModel;
pip: GridTileViewModel;
pipAlignment$: BehaviorSubject<Alignment>;
pip: FloatingTile & { vm: GridTileViewModel };
}
export interface OneOnOneMobileLayout {
type: "one-on-one-mobile";
spotlight: SpotlightTileViewModel;
pip?: GridTileViewModel;
pipSize$: Behavior<"sm" | "lg">;
pipAlignment$: BehaviorSubject<Alignment>;
pip?: FloatingTile & { vm: GridTileViewModel; size$: Behavior<"sm" | "lg"> };
}
export interface PipLayout {
@@ -147,3 +168,34 @@ export type Layout =
| OneOnOneDesktopLayout
| OneOnOneMobileLayout
| PipLayout;
/**
* Tests whether the top-level properties, tile wrapper properties, and array
* elements of layout `a` are equal to those of layout `b`. Useful for deduping
* redundant layout updates.
*/
export function layoutShallowEquals<T extends Layout | LayoutMedia>(
a: T,
b: T,
): boolean {
return shallowEquals<T>(a, b, (aValue, bValue) => {
// Special case for tile wrapper objects so we can detect when the
// spotlight or PiP tile is essentially the same
if (
typeof aValue === "object" &&
aValue !== null &&
"vm" in aValue &&
typeof bValue === "object" &&
bValue !== null &&
"vm" in bValue
)
return shallowEquals(aValue, bValue);
// Special case for arrays so we can detect when the grid tile arrays are
// essentially the same
if (Array.isArray(aValue) && Array.isArray(bValue))
return arrayShallowEquals(aValue, bValue);
return aValue === bValue;
});
}
+24 -11
View File
@@ -235,22 +235,35 @@ test("audio and video follow the tracks' mute state", () => {
test("user media polls stream stats only while the setting is on", () => {
onTestFinished(() => showConnectionStats.setValue(false));
withTestScheduler(({ cold, expectObservable, schedule }) => {
const stats = { type: "inbound-rtp" } as RTCInboundRtpStreamStats;
const vm = mockRemoteMedia(rtcMembership, {}, [
mockAudioTrack({
stats$: () => cold("-s", { s: stats }),
}),
mockAudioTrack({ stats$: () => cold("-s", { s: { ...stats } }) }),
]);
schedule("-a-b", {
a() {
showConnectionStats.setValue(true);
},
b() {
showConnectionStats.setValue(false);
},
schedule("-t-f-t", {
t: () => showConnectionStats.setValue(true),
f: () => showConnectionStats.setValue(false),
});
expectObservable(vm.audioStreamStats$).toBe("u-su", {
expectObservable(vm.audioStreamStats$).toBe("u-s---s", {
u: undefined,
s: stats,
});
});
});
test("stream stats can be enabled before creating media", () => {
showConnectionStats.setValue(true);
onTestFinished(() => showConnectionStats.setValue(false));
withTestScheduler(({ cold, expectObservable }) => {
const stats = { type: "inbound-rtp" } as RTCInboundRtpStreamStats;
const vm = mockRemoteMedia(rtcMembership, {}, [
mockAudioTrack({ stats$: () => cold("-s", { s: stats }) }),
]);
expectObservable(vm.audioStreamStats$).toBe("us", {
u: undefined,
s: stats,
});
+4 -3
View File
@@ -16,7 +16,9 @@ import {
BehaviorSubject,
combineLatest,
map,
NEVER,
of,
startWith,
Subject,
switchMap,
} from "rxjs";
@@ -105,10 +107,9 @@ export function createBaseUserMedia(
showConnectionStats.value$,
]).pipe(
switchMap(([track, show]) =>
track !== undefined && show
? track.stats$(statsIntervalMs)
: of(undefined),
track !== undefined && show ? track.stats$(statsIntervalMs) : NEVER,
),
startWith<MediaStreamStats>(undefined),
),
);
+19 -4
View File
@@ -7,6 +7,7 @@ Please see LICENSE in the repository root for full details.
import { test, expect } from "vitest";
import { shallowEquals } from "./object";
import { shallowEquals as arrayShallowEquals } from "./array";
interface Thing {
type: string;
@@ -29,12 +30,26 @@ test("shallowEquals detects a missing key", () => {
test("shallowEquals detects a differing value", () =>
expect(shallowEquals(withArray, { ...withArray, value: 2 })).toBe(false));
test("shallowEquals considers arrays with equal contents to be equal", () =>
const comparator = (aValue: unknown, bValue: unknown): boolean => {
if (Array.isArray(aValue) && Array.isArray(bValue))
return arrayShallowEquals(aValue, bValue);
return aValue === bValue;
};
test("shallowEquals with comparator considers arrays with equal contents to be equal", () =>
expect(
shallowEquals(withArray, { ...withArray, items: [...withArray.items] }),
shallowEquals(
withArray,
{ ...withArray, items: [...withArray.items] },
comparator,
),
).toBe(true));
test("shallowEquals detects arrays with different contents", () =>
test("shallowEquals with comparator detects arrays with different contents", () =>
expect(
shallowEquals(withArray, { ...withArray, items: [...withArray.items, {}] }),
shallowEquals(
withArray,
{ ...withArray, items: [...withArray.items, {}] },
comparator,
),
).toBe(false));
+6 -8
View File
@@ -5,17 +5,17 @@ SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { shallowEquals as arrayShallowEquals } from "./array.ts";
/**
* Determine whether two objects are equal by shallow comparison of their
* top-level properties. Array-valued properties are themselves compared
* shallowly, so two objects holding distinct arrays with the same elements are
* considered equal.
* top-level properties, using the given `comparator`.
*/
export function shallowEquals<T extends object>(
a: Readonly<T>,
b: Readonly<T>,
comparator: (aValue: unknown, bValue: unknown) => boolean = (
aValue,
bValue,
) => aValue === bValue,
): boolean {
// If a and b have the same number of keys and every key in a is also in b,
// then they have the same keys.
@@ -29,9 +29,7 @@ export function shallowEquals<T extends object>(
// Now check that they have the same values.
const aValue = (a as any)[key];
const bValue = (b as any)[key];
if (Array.isArray(aValue) && Array.isArray(bValue)) {
if (!arrayShallowEquals(aValue, bValue)) return false;
} else if (aValue !== bValue) return false;
if (!comparator(aValue, bValue)) return false;
}
return true;
+2
View File
@@ -111,6 +111,8 @@ describe("widget", () => {
receiveToDevice: sendRecvToDevice,
turnServers: false,
rtcTransports: true,
rtcLivekitGetToken: true,
rtcLivekitDelegateDelayedLeave: true,
sendDelayedEvents: true,
updateDelayedEvents: true,
sendSticky: true,
+2
View File
@@ -175,6 +175,8 @@ export const initializeWidget = (
receiveToDevice: sendRecvToDevice,
turnServers: false,
rtcTransports: true,
rtcLivekitGetToken: true,
rtcLivekitDelegateDelayedLeave: true,
sendDelayedEvents: true,
updateDelayedEvents: true,
sendSticky: true,