Compare commits

...
Author SHA1 Message Date
Johannes Marbach 80ade28eed Stabilise m.call.intent 2026-11-06 15:50:19 +01:00
Johannes Marbach 1ae57e9647 Merge pull request #4420 from element-hq/johannes/mark-release-announcements-viewed
Mark known release announcements as viewed
2026-11-06 13:50:54 +01:00
Timo 4b05ab503e Merge pull request #4416 from element-hq/toger5/fix-flickering-device-id-mismatch
fix camera restart due to device id mismatch
2026-11-06 13:45:04 +01:00
Johannes Marbach ea977cecd9 Mark known release announcements as viewed 2026-11-06 11:31:01 +01:00
Johannes Marbach aadbda39b1 Merge pull request #4405 from element-hq/renovate/npm-minimatch-vulnerability
Update dependency minimatch to v11 [SECURITY]
2026-11-06 11:12:09 +01:00
fkwp 53c1e4c30f Merge pull request #4289 from element-hq/spec/background-effects/9-remove-own
Remove a background of one's own
2026-11-06 09:58:42 +00:00
renovate[bot] 6b56c7e683 Update dependency minimatch to v11 [SECURITY] 2026-11-06 09:35:24 +00:00
fkwpandClaude Opus 5.5 fb4cbc72e2 Remove a background of one's own
- A cross on the tile's top-right corner removes an added background,
  critical-coloured at rest and named by its tooltip, as design draws it.
  The one in force offers none, since taking it away would leave nothing
  chosen; the view model refuses it too.
- The cross sits beside the item rather than in it, which would take the
  press first and choose the tile, and isn't a control of its own, so the
  menu keeps one keyboard model: Delete or Backspace on the tile does the
  same, and the tile says so with aria-keyshortcuts.
- Removed from IndexedDB and its picture released. End to end on
  Chromium and Firefox, a background removed by Delete is still gone
  after a reload.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-06 10:33:22 +01:00
fkwp bac9811829 Merge pull request #4288 from element-hq/spec/background-effects/8-own-backgrounds
Add a background of one's own, kept on this device
2026-11-06 10:33:17 +01:00
renovate[bot]andJohannes Marbach a2cfc2af93 Update ghcr.io/element-hq/element-web:develop Docker digest to 49104e3 (#4415)
* Update ghcr.io/element-hq/element-web:develop Docker digest to e063472

* Fix tests

* Dismiss further toasts

* Close even more toasts

* Yet more toast fixes

* Toast fixes all over

---------

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: Johannes Marbach <n0-0ne+github@mailbox.org>
2026-11-06 07:24:53 +00:00
Valere Fedronic dcb2b9e8f6 Merge pull request #4418 from element-hq/valere/bug_fix_no_config_uncaught_error
Fix: Uncaught exception in downloadConfig
2026-11-05 16:30:40 +01:00
fkwpandClaude Opus 5.5 373b1e0fc0 Guard what can be added as a background
- Five of one's own at most, so the grid is whole: at the limit the add
  tile goes, and it comes back once one is removed. The store holds the
  limit too.
- An added image is refused if it isn't an image, is animated, or can't
  be decoded; reduced to 1920px on its longest edge; and laid on an opaque
  ground, since a transparent pixel would be a hole in the background.
- Animation is read from the file, as a still and an animated WebP share
  a type. Chromium has no selected track until the decoder's tracks are
  ready; without waiting for them an animated GIF got through.
- A refusal says why, in place of the notice, and is scrolled into view.
  It can be dismissed, is gone once the menu closes, and the next shows
  even if it reads the same. What was in force stays.
- End to end on Chromium and Firefox, from what the device then keeps: an
  animated GIF is refused with its message, a 4000x3000 image is kept at
  1920x1440, and a half transparent one is kept opaque. All three fail
  with the preparation bypassed.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-05 16:20:33 +01:00
fkwpandClaude Opus 5.5 1bc372b587 Add a background of one's own, kept on this device
- An add tile after the pictures opens the file picker, and the image
  chosen joins the backgrounds on offer without being put on; it is in
  force once chosen, like a shipped one. The tile is a command rather
  than a choice, so a menu item beside the radios, and it isn't offered
  where effects can't run.
- Kept in IndexedDB on this device and never sent anywhere, behind a
  small interface, so the unit tests use an in-memory store; the
  IndexedDB store itself is tested against fake-indexeddb, a new dev
  dependency.
- The picker takes the focus, which the menu reads as a click elsewhere,
  so the menu holds itself open until a file is chosen or the picker is
  cancelled.
- A chosen image the device no longer keeps, as after its storage is
  cleared, falls back to no effect once the device has said what it
  keeps.
- Switching is checked between any two kinds of effect. End to end on
  Chromium and Firefox: an added image shows in the preview once chosen;
  once it is in force, no request carries the file or the copy kept,
  which a deliberate upload fails; chosen, it is in force again after a
  reload, gone once storage is cleared, and can be added again.
- Kept as given here; what can be added is checked in the commit after.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-05 16:20:33 +01:00
fkwp 82ba51ea6a Merge pull request #4287 from element-hq/spec/background-effects/7-hold-choice
Hold the chosen effect across calls, and forget one that can't run
2026-11-05 16:20:27 +01:00
Valere e9cffb60dd Fix: Uncaught exception in downloadConfig 2026-11-05 16:08:41 +01:00
fkwpandClaude Opus 5.5 741dab6dbb Hold the chosen effect across calls, and forget one that can't run
- A remembered effect this browser can't run is cleared rather than kept,
  as the menu already showed no effect in force there.
- A camera turned off and on again with an effect chosen gets the same
  pipeline as its track arrives, neither rebuilt nor switched again.
- End to end, on Chromium and Firefox: the effect chosen before a call is
  still chosen, and worn, in the next call after leaving; and a second
  tab still shuts the first down with the existing "Opened in another
  tab" error while an effect runs.
- Switching camera and sharing a screen are checked by hand: they need
  two cameras and a screen, which the fake devices don't give yet.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-05 15:30:25 +01:00
Timo K. 11e143c8ab fix camera restart due to device id mismatch 2026-11-05 13:21:42 +01:00
54 changed files with 1998 additions and 274 deletions
+5 -3
View File
@@ -26,8 +26,7 @@
"features": {
"feature_element_call_video_rooms": true,
"feature_video_rooms": true,
"feature_group_calls": true,
"feature_release_announcement": false
"feature_group_calls": true
},
"default_federate": true,
"default_theme": "light",
@@ -40,7 +39,10 @@
},
"setting_defaults": {
"breadcrumbs": true,
"feature_group_calls": true
"feature_group_calls": true,
"releaseAnnouncementData": {
"room_list_section": true
}
},
"jitsi": {
"preferred_domain": "meet.element.io"
+5 -3
View File
@@ -26,8 +26,7 @@
"features": {
"feature_element_call_video_rooms": true,
"feature_video_rooms": true,
"feature_group_calls": true,
"feature_release_announcement": false
"feature_group_calls": true
},
"default_federate": true,
"default_theme": "light",
@@ -40,7 +39,10 @@
},
"setting_defaults": {
"breadcrumbs": true,
"feature_group_calls": true
"feature_group_calls": true,
"releaseAnnouncementData": {
"room_list_section": true
}
},
"jitsi": {
"preferred_domain": "meet.element.io"
+2 -2
View File
@@ -19,7 +19,7 @@ services:
- ./backend/playwright_homeserver-othersite.yaml:/data/cfg/homeserver.yaml:Z
element-web:
# Pin to a SHA so that upstream cannot break our tests. Renovate handles regular updates.
image: ghcr.io/element-hq/element-web:develop@sha256:a79801206c628a912d57fa6f355d5c5ab9c6c40c2d1b6208d4192678fb6e80c7
image: ghcr.io/element-hq/element-web:develop@sha256:e063472b6bdf6d0bfa1d43dfa4ef7effee1efd2c9684daee665d1bedfe746a47
element-web-1:
# Pin to a SHA so that upstream cannot break our tests. Renovate handles regular updates.
image: ghcr.io/element-hq/element-web:develop@sha256:a79801206c628a912d57fa6f355d5c5ab9c6c40c2d1b6208d4192678fb6e80c7
image: ghcr.io/element-hq/element-web:develop@sha256:e063472b6bdf6d0bfa1d43dfa4ef7effee1efd2c9684daee665d1bedfe746a47
+2
View File
@@ -48,6 +48,8 @@ Test what a user observes: the peer sees the change, it survives a reconnect, it
right after a reload. Reuse `playwright/spa-helpers.ts`,
`playwright/widget/test-helpers.ts`, `playwright/fixtures/`.
To mark new release announcements as viewed, add them in `backend/ew.test*.config.json`.
```sh
pnpm test # unit + storybook
pnpm backend # Synapse + LiveKit, required for e2e
+5
View File
@@ -3,6 +3,7 @@
"user_menu": "User menu"
},
"action": {
"add_background_image": "Add image",
"close": "Close",
"copy_link": "Copy link",
"edit": "Edit",
@@ -98,6 +99,10 @@
},
"disconnected_banner": "Connectivity to the server has been lost.",
"error": {
"background_animated": "Animated images cannot be used as a background",
"background_not_an_image": "That file is not a supported image",
"background_not_kept": "That background could not be saved",
"background_undecodable": "That image could not be opened",
"call_is_not_supported": "Call is not supported",
"call_not_found": "Call not found",
"call_not_found_description": "<0>That link doesn't appear to belong to any existing call. Check that you have the right link, or <2>create a new one</2>.</0>",
+2 -1
View File
@@ -97,6 +97,7 @@
"copy-to-clipboard": "^3.3.3",
"eslint-plugin-element-call": "link:eslint",
"eslint-plugin-storybook": "^10.3.6",
"fake-indexeddb": "^6.2.5",
"fetch-mock": "11.1.5",
"i18next": "^26.0.0",
"i18next-browser-languagedetector": "^8.0.0",
@@ -106,7 +107,7 @@
"livekit-client": "^2.18.1",
"lodash-es": "^4.17.21",
"loglevel": "^1.9.1",
"matrix-js-sdk": "github:matrix-org/matrix-js-sdk#a9ef09fbf2f1644c8663d0aeaf6a921cd16e7b01",
"matrix-js-sdk": "github:matrix-org/matrix-js-sdk#4cb91f3155c31f982cbb7375ca34a14fbecc7848",
"matrix-widget-api": "^1.18.0",
"node-stdlib-browser": "^1.3.1",
"normalize.css": "^8.0.1",
+380
View File
@@ -151,6 +151,386 @@ test.describe("background effects section", () => {
});
});
test.describe("the chosen effect", () => {
test("chosen effect survives leaving and rejoining", async ({ page }) => {
test.setTimeout(120_000);
await page.goto("/");
await SpaHelpers.createCall(page, "Returning user", "Effect remembered");
const { camera, picture } = await chooseAndWear(page, "Background 1");
await page.getByTestId("lobby_joinCall").click();
await page.getByTestId("incall_leave").click();
await expect(page.getByRole("heading")).toContainText(
"your call has ended",
);
await page.goto("/");
await page.getByTestId("home_callName").fill("Effect remembered again");
await page.getByTestId("home_go").click();
await expect(page.getByTestId("lobby_joinCall")).toBeVisible();
await page.getByRole("button", { name: "Camera", exact: true }).click();
const tile = page
.getByRole("group", { name: "Background effects" })
.getByRole("menuitemradio", { name: "Background 1" });
await expect(tile).toHaveAttribute("aria-checked", "true");
const preview = page.locator("video").first();
await expect
.poll(async () => distance(await averageColour(preview), picture), {
timeout: 60_000,
})
.toBeLessThan(distance(camera, picture) / 2);
});
test("second session still shows the existing error", async ({ page }) => {
test.setTimeout(120_000);
await page.goto("/");
await SpaHelpers.createCall(page, "Two tabs", "Effect in two tabs");
await chooseAndWear(page, "Background 1");
const second = await page.context().newPage();
await second.goto("/");
await expect(
page.getByRole("heading", { name: "Opened in another tab" }),
).toBeVisible();
});
});
/** Chooses an effect in the lobby, and waits until the preview wears it. */
async function chooseAndWear(
page: Page,
name: string,
): Promise<{ camera: Colour; picture: Colour }> {
const preview = page.locator("video").first();
const camera = await cameraColour(preview);
await page.getByRole("button", { name: "Camera", exact: true }).click();
const tile = page.getByRole("menuitemradio", { name });
await tile.click();
const picture = await averageColour(tile.locator("img"));
await expect
.poll(async () => distance(await averageColour(preview), picture), {
timeout: 60_000,
})
.toBeLessThan(distance(camera, picture) / 2);
await page.keyboard.press("Escape");
return { camera, picture };
}
test.describe("a background of one's own", () => {
test("adds an image from the device", async ({ page }) => {
test.setTimeout(120_000);
await page.goto("/");
await SpaHelpers.createCall(page, "Adding user", "Own background");
const preview = page.locator("video").first();
const camera = await cameraColour(preview);
const red = await redImage(page);
await addImage(page, red);
const mine = page
.getByRole("group", { name: "Background effects" })
.getByRole("menuitemradio", { name: "Background 3" });
await expect(mine).toBeVisible();
await expect(mine).toHaveAttribute("aria-checked", "false");
await mine.click();
await expect
.poll(async () => distance(await averageColour(preview), RED), {
timeout: 60_000,
})
.toBeLessThan(distance(camera, RED) / 2);
});
test("adding an image makes no upload", async ({ page }) => {
test.setTimeout(120_000);
await page.goto("/");
await SpaHelpers.createCall(page, "Private user", "Own background kept");
const preview = page.locator("video").first();
const camera = await cameraColour(preview);
const red = await redImage(page);
const sent: { request: string; body: Buffer | null }[] = [];
page.on("request", (request) =>
sent.push({
request: `${request.method()} ${request.url()}`,
body: request.postDataBuffer(),
}),
);
await addImage(page, red);
await page
.getByRole("group", { name: "Background effects" })
.getByRole("menuitemradio", { name: "Background 3" })
.click();
// Kept, chosen and in force: anything that would send it has had its turn.
await expect
.poll(async () => distance(await averageColour(preview), RED), {
timeout: 60_000,
})
.toBeLessThan(distance(camera, RED) / 2);
// The file as given, and the copy kept, which is encoded anew.
const samples = [red, await keptBytes(page)].map((bytes) =>
bytes.subarray(bytes.length - 64),
);
const carriesIt = sent.filter(
({ request, body }) =>
request.includes("/_matrix/media/") ||
samples.some((sample) => body?.includes(sample)),
);
expect(carriesIt.map(({ request }) => request)).toEqual([]);
});
test("added image survives rejoining and is gone after storage is cleared", async ({
page,
}) => {
test.setTimeout(120_000);
await page.goto("/");
await SpaHelpers.createCall(page, "Keeping user", "Own background again");
const preview = page.locator("video").first();
const camera = await cameraColour(preview);
const red = await redImage(page);
await addImage(page, red);
const mine = page
.getByRole("group", { name: "Background effects" })
.getByRole("menuitemradio", { name: "Background 3" });
await mine.click();
await page.keyboard.press("Escape");
await page.reload();
await expect(page.getByTestId("lobby_joinCall")).toBeVisible();
await page.getByRole("button", { name: "Camera", exact: true }).click();
await expect(mine).toHaveAttribute("aria-checked", "true");
await page.keyboard.press("Escape");
await expect
.poll(async () => distance(await averageColour(preview), RED), {
timeout: 60_000,
})
.toBeLessThan(distance(camera, RED) / 2);
await page.evaluate(
async () =>
new Promise<void>((resolve, reject) => {
const request = indexedDB.deleteDatabase(
"element-call-background-images",
);
request.onsuccess = (): void => resolve();
request.onerror = (): void => reject(request.error);
request.onblocked = (): void => resolve();
}),
);
await page.reload();
await expect(page.getByTestId("lobby_joinCall")).toBeVisible();
await page.getByRole("button", { name: "Camera", exact: true }).click();
await expect(
page.getByRole("group", { name: "Background effects" }),
).toBeVisible();
await expect(mine).toHaveCount(0);
await page.keyboard.press("Escape");
await addImage(page, red);
await expect(mine).toBeVisible();
});
test("a removed image stays removed", async ({ page }) => {
test.setTimeout(120_000);
await page.goto("/");
await SpaHelpers.createCall(
page,
"Removing user",
"Own background removed",
);
await addImage(page, await redImage(page));
const mine = page
.getByRole("group", { name: "Background effects" })
.getByRole("menuitemradio", { name: "Background 3" });
await expect(mine).toHaveAttribute("aria-keyshortcuts", "Delete");
await mine.focus();
await page.keyboard.press("Delete");
await expect(mine).toHaveCount(0);
await page.reload();
await expect(page.getByTestId("lobby_joinCall")).toBeVisible();
await page.getByRole("button", { name: "Camera", exact: true }).click();
await expect(
page.getByRole("group", { name: "Background effects" }),
).toBeVisible();
await expect(mine).toHaveCount(0);
expect(await keptImages(page)).toEqual([]);
});
});
test.describe("what can be added", () => {
test("refuses an animated image", async ({ page }) => {
test.setTimeout(120_000);
await page.goto("/");
await SpaHelpers.createCall(page, "Refused user", "Animated refused");
await addImage(page, ANIMATED_GIF, {
name: "moving.gif",
type: "image/gif",
});
await expect(
page.getByText("Animated images cannot be used as a background"),
).toBeVisible();
const section = page.getByRole("group", { name: "Background effects" });
await expect(
section.getByRole("menuitemradio", { name: "Background 3" }),
).toHaveCount(0);
await expect(
section.getByRole("menuitemradio", { name: "None" }),
).toHaveAttribute("aria-checked", "true");
expect(await keptImages(page)).toEqual([]);
});
test("reduces an oversized image", async ({ page }) => {
test.setTimeout(120_000);
await page.goto("/");
await SpaHelpers.createCall(page, "Large user", "Oversized reduced");
await addImage(page, await pngImage(page, { width: 4000, height: 3000 }));
await expect(
page
.getByRole("group", { name: "Background effects" })
.getByRole("menuitemradio", { name: "Background 3" }),
).toBeVisible();
await expect(page.getByRole("alert")).toHaveCount(0);
const [kept] = await keptImages(page);
expect(kept).toMatchObject({ width: 1920, height: 1440 });
});
test("lays every image on an opaque ground", async ({ page }) => {
test.setTimeout(120_000);
await page.goto("/");
await SpaHelpers.createCall(page, "Logo user", "Transparent made opaque");
await addImage(
page,
await pngImage(page, { width: 320, height: 180, transparent: true }),
);
await expect(
page
.getByRole("group", { name: "Background effects" })
.getByRole("menuitemradio", { name: "Background 3" }),
).toBeVisible();
const [kept] = await keptImages(page);
expect(kept.minAlpha).toBe(255);
});
});
/** Two frames of one pixel, red then blue. */
const ANIMATED_GIF = Buffer.from(
"R0lGODlhAQABAPEAAP8AAAAA/wAAAAAAACH/C05FVFNDQVBFMi4wAwEAAAAh+QQACgAAACwAAAAAAQABAAACAkQBACH5BAAKAAAALAAAAAABAAEAAAICTAEAOw==",
"base64",
);
/** A red PNG of the given size; transparent leaves its right half empty. */
async function pngImage(
page: Page,
size: { width: number; height: number; transparent?: boolean },
): Promise<Buffer> {
const dataUrl = await page.evaluate(({ width, height, transparent }) => {
const canvas = document.createElement("canvas");
canvas.width = width;
canvas.height = height;
const context = canvas.getContext("2d")!;
context.fillStyle = "rgb(255, 0, 0)";
context.fillRect(0, 0, transparent ? width / 2 : width, height);
return canvas.toDataURL("image/png");
}, size);
return Buffer.from(dataUrl.split(",")[1], "base64");
}
/** The images the device keeps, as it keeps them. */
async function keptImages(
page: Page,
): Promise<{ width: number; height: number; minAlpha: number }[]> {
return page.evaluate(async () => {
const db = await new Promise<IDBDatabase>((resolve, reject) => {
const request = indexedDB.open("element-call-background-images");
request.onsuccess = (): void => resolve(request.result);
request.onerror = (): void => reject(request.error);
});
if (!db.objectStoreNames.contains("backgrounds")) return [];
const kept = await new Promise<{ image: Blob }[]>((resolve, reject) => {
const request = db
.transaction("backgrounds")
.objectStore("backgrounds")
.getAll();
request.onsuccess = (): void => resolve(request.result);
request.onerror = (): void => reject(request.error);
});
db.close();
return Promise.all(
kept.map(async ({ image }) => {
const bitmap = await createImageBitmap(image);
const canvas = new OffscreenCanvas(bitmap.width, bitmap.height);
const context = canvas.getContext("2d")!;
context.drawImage(bitmap, 0, 0);
const { data } = context.getImageData(
0,
0,
bitmap.width,
bitmap.height,
);
let minAlpha = 255;
for (let i = 3; i < data.length; i += 4)
minAlpha = Math.min(minAlpha, data[i]);
return { width: bitmap.width, height: bitmap.height, minAlpha };
}),
);
});
}
/** The bytes of the one image the device keeps. */
async function keptBytes(page: Page): Promise<Buffer> {
const base64 = await page.evaluate(
async () =>
new Promise<string>((resolve, reject) => {
const open = indexedDB.open("element-call-background-images");
open.onerror = (): void => reject(open.error);
open.onsuccess = (): void => {
const all = open.result
.transaction("backgrounds")
.objectStore("backgrounds")
.getAll();
all.onerror = (): void => reject(all.error);
all.onsuccess = (): void => {
const reader = new FileReader();
reader.onload = (): void =>
resolve((reader.result as string).split(",")[1]);
reader.readAsDataURL((all.result[0] as { image: Blob }).image);
};
};
}),
);
return Buffer.from(base64, "base64");
}
const RED: Colour = [255, 0, 0];
/** A solid red PNG, unlike the camera and the shipped pictures. */
async function redImage(page: Page): Promise<Buffer> {
const dataUrl = await page.evaluate(() => {
const canvas = document.createElement("canvas");
canvas.width = 320;
canvas.height = 180;
const context = canvas.getContext("2d")!;
context.fillStyle = "rgb(255, 0, 0)";
context.fillRect(0, 0, canvas.width, canvas.height);
return canvas.toDataURL("image/png");
});
return Buffer.from(dataUrl.split(",")[1], "base64");
}
/** Adds an image from the camera menu's add tile, leaving the menu open. */
async function addImage(
page: Page,
image: Buffer,
{ name, type } = { name: "mine.png", type: "image/png" },
): Promise<void> {
if (!(await page.getByRole("menu").isVisible()))
await page.getByRole("button", { name: "Camera", exact: true }).click();
const chooser = page.waitForEvent("filechooser");
await page.getByRole("menuitem", { name: "Add image" }).click();
await (await chooser).setFiles({ name, mimeType: type, buffer: image });
}
test.describe("joining with a background chosen", () => {
test("publishes no frame of the room", async ({ browser }) => {
// Two first builds of the pipeline, one of them held back.
+4 -64
View File
@@ -29,47 +29,10 @@ export interface MyFixtures {
addUser: (username: string, host: string) => Promise<UserBaseFixture>;
}
// Minimal config.json for the local element-web instance
const CONFIG_JSON = {
default_server_config: {
"m.homeserver": {
base_url: "https://synapse.m.localhost",
server_name: "synapse.m.localhost",
},
},
element_call: {
participant_limit: 8,
brand: "Element Call",
},
// The default language is set here for test consistency
setting_defaults: {
language: "en-GB",
feature_group_calls: true,
},
// the location tests want a map style url.
map_style_url:
"https://api.maptiler.com/maps/streets/style.json?key=fU3vlMsMn4Jb6dnEIFsx",
features: {
// We don't want to go through the feature announcement during the e2e test
feature_release_announcement: false,
feature_element_call_video_rooms: true,
feature_video_rooms: true,
feature_group_calls: true,
},
};
export const widgetTest = test.extend<MyFixtures>({
// allow per-test override: `widgetTest.use({ callType: "dm" })`
callType: ["room", { option: true }],
asWidget: async ({ browser, context, callType }, pUse) => {
await context.route(`http://localhost:8081/config.json*`, async (route) => {
await route.fulfill({ json: CONFIG_JSON });
});
asWidget: async ({ browser, callType }, pUse) => {
const brooksDisplayName = `brooks_${Date.now()}`;
const whistlerDisplayName = `whistler_${Date.now()}`;
@@ -109,15 +72,7 @@ export const widgetTest = test.extend<MyFixtures>({
await TestHelpers.dismissInviteUnknownUserModal(ewPage1);
// Accept the invite
await TestHelpers.closeReleaseAnnouncement(
ewPage2,
"Introducing Sections",
);
await TestHelpers.expandAllSections(ewPage2);
await expect(
TestHelpers.roomListItem(ewPage2, "Welcome Room"),
).toBeVisible();
await TestHelpers.roomListItem(ewPage2, "Welcome Room").click();
await TestHelpers.openRoomInvite(ewPage2, "Welcome Room");
await ewPage2.getByRole("button", { name: "Accept" }).click();
await expect(
ewPage2
@@ -125,17 +80,7 @@ export const widgetTest = test.extend<MyFixtures>({
.getByRole("heading", { name: "Welcome Room" }),
).toBeVisible();
} else if (callType === "dm") {
await TestHelpers.closeReleaseAnnouncement(
ewPage1,
"Introducing Sections",
);
await ewPage1
.getByRole("navigation", { name: "Room list" })
.getByRole("button", { name: "New", exact: true })
.click();
await ewPage1.getByRole("menuitem", { name: "Start chat" }).click();
await TestHelpers.chooseFromRoomListNewMenu(ewPage1, "Start chat");
await ewPage1.getByRole("textbox", { name: "Search" }).click();
await ewPage1.getByRole("textbox", { name: "Search" }).fill(whistlerMxId);
await ewPage1.getByRole("button", { name: "Go" }).click();
@@ -157,12 +102,7 @@ export const widgetTest = test.extend<MyFixtures>({
).toBeVisible();
// Accept the DM invite from brooks
await TestHelpers.closeReleaseAnnouncement(
ewPage2,
"Introducing Sections",
);
await TestHelpers.expandAllSections(ewPage2);
await TestHelpers.roomListItem(ewPage2, brooksDisplayName).click();
await TestHelpers.openRoomInvite(ewPage2, brooksDisplayName);
await ewPage2.getByRole("button", { name: "Start chatting" }).click();
}
+72 -36
View File
@@ -153,29 +153,40 @@ export class TestHelpers {
return { page, clientHandle, mxId };
}
// Dismisses any toasts that appear on startup, such as "Failed to load service worker" or "Back up your chats".
// Toast can be stacked, and only the top one can be dismiss, so just look at what is on top and
// dismiss (if part of expected toats)
// Dismisses the toasts a fresh device gets after sign-in.
//
// Toasts stack and only the top one can be dismissed, so look
// at what is on top each time.
public static async dismissStartupToasts(page: Page): Promise<void> {
const expectedToasts = [
const expectedToasts: {
title: string;
button: string;
confirm?: string;
}[] = [
{ title: "Failed to load service worker", button: "OK" },
{ title: "Verify this device", button: "Later" },
{ title: "Back up your chats", button: "Not now" },
{ title: "Turn on key storage", button: "Dismiss" },
{
title: "Turn on key storage",
button: "Dismiss",
confirm: "Yes, dismiss",
},
{ title: "Notifications", button: "Dismiss" },
{ title: "Element does not support this browser", button: "Dismiss" },
];
const toast = page.locator(".mx_Toast_toast");
const toastTitle = toast.locator(".mx_Toast_title h2");
// eslint-disable-next-line no-constant-condition
while (true) {
try {
await toast.waitFor({ state: "visible", timeout: 700 });
const title = await toast.locator(".mx_Toast_title h2").textContent();
const title = await toastTitle.textContent();
if (!title) break;
// Find the matching toast config
const toastConfig = expectedToasts.find((t) =>
title?.includes(t.title),
);
const toastConfig = expectedToasts.find((t) => title.includes(t.title));
if (toastConfig) {
// Bounded so that a renamed button leaves the toast in place and the
@@ -183,6 +194,16 @@ export class TestHelpers {
await toast
.getByRole("button", { name: toastConfig.button })
.click({ timeout: 5000 });
if (toastConfig.confirm) {
const dialog = page.getByRole("dialog");
await dialog
.getByRole("button", { name: toastConfig.confirm })
.click({ timeout: 5000 });
await expect(dialog).toBeHidden({ timeout: 5000 });
}
// The same element shows the next toast a moment after the click, so
// reading it straight away acts on the dismissed one a second time
await expect(toastTitle).not.toHaveText(title, { timeout: 5000 });
} else {
// Unknown toast. We don't want to act on unknown toasts
break;
@@ -194,20 +215,6 @@ export class TestHelpers {
}
}
public static async closeReleaseAnnouncement(
page: Page,
name: string,
): Promise<void> {
try {
await page
.getByRole("dialog", { name })
.getByRole("button", { name: "OK" })
.click({ timeout: 2000 });
} catch {
// Announcement not shown; nothing to do
}
}
public static async expandAllSections(page: Page): Promise<void> {
try {
await page
@@ -223,16 +230,7 @@ export class TestHelpers {
page: Page,
andInvite: string[] = [],
): Promise<void> {
await TestHelpers.closeReleaseAnnouncement(page, "Introducing Sections");
await page
.getByRole("navigation", { name: "Room list" })
.getByRole("button", { name: "New", exact: true })
.click();
await page.getByRole("menuitem", { name: "New Room" }).click({
timeout: 5000,
});
await TestHelpers.chooseFromRoomListNewMenu(page, "New Room");
await page.getByRole("textbox", { name: "Name" }).fill(name);
await page.getByRole("button", { name: "Create room" }).click();
await expect(page.getByText("You created this room.")).toBeVisible({
@@ -259,6 +257,24 @@ export class TestHelpers {
}
}
/**
* Opens the room list's "New" menu and picks the given item.
*
* Driven by keyboard, as a toast arriving late sits over the button and
* swallows clicks.
*/
public static async chooseFromRoomListNewMenu(
page: Page,
itemName: string,
): Promise<void> {
await TestHelpers.dismissStartupToasts(page);
await page
.getByRole("navigation", { name: "Room list" })
.getByRole("button", { name: "New", exact: true })
.press("Enter");
await page.getByRole("menuitem", { name: itemName }).press("Enter");
}
/**
* Accepts a room invite using the room name.
* Locatest the invite in the room list.
@@ -268,9 +284,7 @@ export class TestHelpers {
roomName: string,
page: Page,
): Promise<void> {
await TestHelpers.closeReleaseAnnouncement(page, "Introducing Sections");
await TestHelpers.expandAllSections(page);
await TestHelpers.roomListItem(page, roomName).click({ timeout: 10000 });
await TestHelpers.openRoomInvite(page, roomName);
await page.getByRole("button", { name: "Accept" }).click({
timeout: 5000,
});
@@ -376,6 +390,28 @@ export class TestHelpers {
return page.locator(`[aria-label^="Open room ${roomName}"]`);
}
/**
* Opens a pending invite from the room list by its room name.
*
* Element Web collapses the Invites section every time it appears, so it is
* expanded here once the invite has arrived.
*/
public static async openRoomInvite(
page: Page,
roomName: string,
): Promise<void> {
const invitesSection = page.getByRole("button", {
name: "Toggle Invites section",
});
await expect(invitesSection).toBeVisible({ timeout: 10000 });
// Toasts sit over the top of the room list, where this header is.
await TestHelpers.dismissStartupToasts(page);
if ((await invitesSection.getAttribute("aria-expanded")) !== "true") {
await invitesSection.click();
}
await TestHelpers.roomListItem(page, roomName).click({ timeout: 10000 });
}
/**
* Switches to a room in the room list by its name.
* @param page - The EW page
+41 -26
View File
@@ -8,7 +8,7 @@ overrides:
'@livekit/components-core>rxjs': ^7.8.1
'@livekit/track-processors>@mediapipe/tasks-vision': ^0.10.18
tar: ^7.5.11
minimatch: ^10.2.3
minimatch: ^11.0.0
glob: ^10.5.0
qs: ^6.15.2
js-yaml: ^4.2.0
@@ -160,6 +160,9 @@ importers:
eslint-plugin-storybook:
specifier: ^10.3.6
version: 10.6.2(eslint@8.57.1(supports-color@7.2.0))(supports-color@7.2.0)(typescript@5.9.3)
fake-indexeddb:
specifier: ^6.2.5
version: 6.2.5
fetch-mock:
specifier: 11.1.5
version: 11.1.5(node-fetch@2.7.0)
@@ -188,8 +191,8 @@ importers:
specifier: ^1.9.1
version: 1.9.2
matrix-js-sdk:
specifier: github:matrix-org/matrix-js-sdk#a9ef09fbf2f1644c8663d0aeaf6a921cd16e7b01
version: https://codeload.github.com/matrix-org/matrix-js-sdk/tar.gz/a9ef09fbf2f1644c8663d0aeaf6a921cd16e7b01
specifier: github:matrix-org/matrix-js-sdk#4cb91f3155c31f982cbb7375ca34a14fbecc7848
version: https://codeload.github.com/matrix-org/matrix-js-sdk/tar.gz/4cb91f3155c31f982cbb7375ca34a14fbecc7848
matrix-widget-api:
specifier: ^1.18.0
version: 1.20.0
@@ -1142,6 +1145,9 @@ packages:
resolution: {integrity: sha512-O8jcjabXaleOG9DQ0+ARXWZBTfnP4WNAqzuiJK7ll44AmxGKv/J2M4TPjxjY3znBCfvBXFzucm1twdyFybFqEA==}
engines: {node: '>=12'}
'@isaacs/nonrecursive-glob-match@1.0.3':
resolution: {integrity: sha512-SI7kyKHq4XXDRozw+r9GlRZkia1ZkFrpLW98r2IC2Bpr8YkFGE3fvjzmxCG0x78A++69HmDNrDvSMYBZmHGpKA==}
'@joshwooding/vite-plugin-react-docgen-typescript@0.7.0':
resolution: {integrity: sha512-qvsTEwEFefhdirGOPnu9Wp6ChfIwy2dBCRuETU3uE+4cC+PFoxMSiiEhxk4lOluA34eARHA0OxqsEUYDqRMgeQ==}
peerDependencies:
@@ -3527,8 +3533,8 @@ packages:
boolbase@1.0.0:
resolution: {integrity: sha512-JZOSA7Mo9sNGB8+UjSgzdLtokWAky1zbztM3WRLCbZ70/3cTANmQmOdR7y2g+J0e2WXywy1yS468tY+IruqEww==}
brace-expansion@5.0.9:
resolution: {integrity: sha512-ScQ4IuvIEF1TMlP7Zt+vjJ//9zlPb2SDcxWxM3bk8s6t6GGdJ7KO1dCcTidOPJKePW30LE/2cT7wCyPho9/Wxg==}
brace-expansion@5.0.12:
resolution: {integrity: sha512-YovQ3rzhaLMIrDjNDMkNS01tea93qhEhG5xy8f6+R0l+dw3Ki+5sCoIoI942iuLZTHWogWktgwVDhU09iNEimQ==}
engines: {node: 20 || >=22}
braces@3.0.3:
@@ -4048,6 +4054,10 @@ packages:
resolution: {integrity: sha512-KfYbmpRm0VbLjEvVa9yGwCi9GI34xvi7A/HXYWQO65CSD2u3MczUJSuwXKFIxlGsgBQizV9q5J9NHj4VG0n+pA==}
engines: {node: '>=12.0.0'}
fake-indexeddb@6.2.5:
resolution: {integrity: sha512-CGnyrvbhPlWYMngksqrSSUT1BAVP49dZocrHuK0SvtR0D5TMs5wP0o3j7jexDJW01KSadjBp1M/71o/KR3nD1w==}
engines: {node: '>=18'}
fast-deep-equal@3.1.3:
resolution: {integrity: sha512-f3qQ9oQy9j2AhBe/H9VC91wLmKBCCU/gDOnKNAYG5hswO7BLKj09Hc5HYNz9cGI++xlpDCIgDaitVs03ATR84Q==}
@@ -4724,8 +4734,8 @@ packages:
matrix-events-sdk@0.0.1:
resolution: {integrity: sha512-1QEOsXO+bhyCroIe2/A5OwaxHvBm7EsSQ46DEDn8RBIfQwN5HWBpFvyWWR4QY0KHPPnnJdI99wgRiAl7Ad5qaA==}
matrix-js-sdk@https://codeload.github.com/matrix-org/matrix-js-sdk/tar.gz/a9ef09fbf2f1644c8663d0aeaf6a921cd16e7b01:
resolution: {gitHosted: true, integrity: sha512-ZAhn/zB44MgrXqUzi9ffSFcmZY3+2nx7bEgfEJR7chYDFyT/XmlmfCJx7Ar4rOTAYZJRVnMxrEwiEHGDapUi0Q==, tarball: https://codeload.github.com/matrix-org/matrix-js-sdk/tar.gz/a9ef09fbf2f1644c8663d0aeaf6a921cd16e7b01}
matrix-js-sdk@https://codeload.github.com/matrix-org/matrix-js-sdk/tar.gz/4cb91f3155c31f982cbb7375ca34a14fbecc7848:
resolution: {gitHosted: true, integrity: sha512-3OCLpmzR0zO9TsTdUcxhN8dhy8ZpoZqGzbAQFESaDU60nJXrJRUKiKJAB61KzimVdy2xNscLl4M4NGdqX3aNuw==, tarball: https://codeload.github.com/matrix-org/matrix-js-sdk/tar.gz/4cb91f3155c31f982cbb7375ca34a14fbecc7848}
version: 45.0.0
engines: {node: '>=22.0.0'}
@@ -4761,9 +4771,9 @@ packages:
minimalistic-crypto-utils@1.0.1:
resolution: {integrity: sha512-JIYlbt6g8i5jKfJ3xz7rF0LXmv2TkDxBLUkiBeZ7bAx4GnnNMr8xFpGnOxn6GhTEHx3SjRrZEoU+j04prX1ktg==}
minimatch@10.2.6:
resolution: {integrity: sha512-vpLQEs+VLCr1nU0BXS07maYoFwlDAH0gngQuuttxIwutDFEMHq2blX+8vpgxDdK3J1PwjCJiep77OitTZ4Ll1A==}
engines: {node: 18 || 20 || >=22}
minimatch@11.0.4:
resolution: {integrity: sha512-/UJ1dEoVQTjPNDh6SGlUo+ueLMmoHTas4q+8yaAR/5t6nLAIiBDJtoYUh20IYleix7ehiQkgczb84PE3uzHkxg==}
engines: {node: 20 || >=22}
minimist@1.2.8:
resolution: {integrity: sha512-2yyAR8qBkN3YuheJanUpWC5U3bb5osDywNB8RzDVlDwDHbocAJveqqj1u8+SVD7jkWT4yvsHCpWqqWqAxb0zCA==}
@@ -4936,8 +4946,8 @@ packages:
resolution: {integrity: sha512-LaNjtRWUBY++zB5nE/NwcaoMylSPk+S+ZHNB1TzdbMJMny6dynpAGt7X/tl/QYq3TIeE6nxHppbo2LGymrG5Pw==}
engines: {node: '>=10'}
p-retry@8.0.1:
resolution: {integrity: sha512-NAigyu8hfvJe7MsS18mnc4is0MZtau3QMdBklaJKK23oBQb6occO3UPM3vHmTckW8KhfyjMZaUOqdTFEYliHgg==}
p-retry@9.0.0:
resolution: {integrity: sha512-HuEats8gYn/UUYa5uyveyQGv75mQmwdFtuMgnZlS12dj/7bdnJSO6keaTjgcZUhvkANTj5EO3lACUmV3xmgj0Q==}
engines: {node: '>=22'}
p-try@2.2.0:
@@ -6895,7 +6905,7 @@ snapshots:
ignore: 5.3.2
import-fresh: 3.3.1
js-yaml: 4.3.1
minimatch: 10.2.6
minimatch: 11.0.4
strip-json-comments: 3.1.1
transitivePeerDependencies:
- supports-color
@@ -6974,7 +6984,7 @@ snapshots:
dependencies:
'@humanwhocodes/object-schema': 2.0.3
debug: 4.4.3(supports-color@7.2.0)
minimatch: 10.2.6
minimatch: 11.0.4
transitivePeerDependencies:
- supports-color
@@ -7110,6 +7120,8 @@ snapshots:
wrap-ansi: 8.1.0
wrap-ansi-cjs: wrap-ansi@7.0.0
'@isaacs/nonrecursive-glob-match@1.0.3': {}
'@joshwooding/vite-plugin-react-docgen-typescript@0.7.0(typescript@5.9.3)(vite@8.2.4(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(sass@1.103.1)(terser@5.46.1)(yaml@2.9.0))':
dependencies:
glob: 10.5.0
@@ -8769,7 +8781,7 @@ snapshots:
'@typescript-eslint/types': 8.69.0
'@typescript-eslint/visitor-keys': 8.69.0
debug: 4.4.3(supports-color@7.2.0)
minimatch: 10.2.6
minimatch: 11.0.4
semver: 7.8.5
tinyglobby: 0.2.17
ts-api-utils: 2.5.0(typescript@5.9.3)
@@ -8784,7 +8796,7 @@ snapshots:
'@typescript-eslint/types': 8.72.0
'@typescript-eslint/visitor-keys': 8.72.0
debug: 4.4.3(supports-color@7.2.0)
minimatch: 10.2.6
minimatch: 11.0.4
semver: 7.8.5
tinyglobby: 0.2.17
ts-api-utils: 2.5.0(typescript@5.9.3)
@@ -9109,7 +9121,7 @@ snapshots:
boolbase@1.0.0: {}
brace-expansion@5.0.9:
brace-expansion@5.0.12:
dependencies:
balanced-match: 4.0.4
@@ -9654,7 +9666,7 @@ snapshots:
json-stable-stringify-without-jsonify: 1.0.1
levn: 0.4.1
lodash.merge: 4.6.2
minimatch: 10.2.6
minimatch: 11.0.4
natural-compare: 1.4.0
optionator: 0.9.4
strip-ansi: 6.0.1
@@ -9712,6 +9724,8 @@ snapshots:
expect-type@1.4.0: {}
fake-indexeddb@6.2.5: {}
fast-deep-equal@3.1.3: {}
fast-glob@3.3.3:
@@ -9772,7 +9786,7 @@ snapshots:
filelist@1.0.6:
dependencies:
minimatch: 10.2.6
minimatch: 11.0.4
fill-range@7.1.1:
dependencies:
@@ -9872,7 +9886,7 @@ snapshots:
dependencies:
foreground-child: 3.3.1
jackspeak: 3.4.3
minimatch: 10.2.6
minimatch: 11.0.4
minipass: 7.1.3
package-json-from-dist: 1.0.1
path-scurry: 1.11.1
@@ -9997,7 +10011,7 @@ snapshots:
jiti: 2.7.0
jsonc-parser: 3.3.1
magic-string: 1.2.3
minimatch: 10.2.6
minimatch: 11.0.4
ora: 9.4.1
react: 19.2.8
react-i18next: 17.0.13(i18next@26.4.1(typescript@5.9.3))(react-dom@19.2.8(react@19.2.8))(react@19.2.8)(typescript@5.9.3)
@@ -10395,7 +10409,7 @@ snapshots:
matrix-events-sdk@0.0.1: {}
matrix-js-sdk@https://codeload.github.com/matrix-org/matrix-js-sdk/tar.gz/a9ef09fbf2f1644c8663d0aeaf6a921cd16e7b01:
matrix-js-sdk@https://codeload.github.com/matrix-org/matrix-js-sdk/tar.gz/4cb91f3155c31f982cbb7375ca34a14fbecc7848:
dependencies:
'@babel/runtime': 8.0.8
'@matrix-org/matrix-sdk-crypto-wasm': 18.9.0
@@ -10405,7 +10419,7 @@ snapshots:
loglevel: 1.9.2
matrix-events-sdk: 0.0.1
matrix-widget-api: 1.20.0
p-retry: 8.0.1
p-retry: 9.0.0
sdp-transform: 3.0.0
unhomoglyph: 1.0.6
@@ -10440,9 +10454,10 @@ snapshots:
minimalistic-crypto-utils@1.0.1: {}
minimatch@10.2.6:
minimatch@11.0.4:
dependencies:
brace-expansion: 5.0.9
'@isaacs/nonrecursive-glob-match': 1.0.3
brace-expansion: 5.0.12
minimist@1.2.8: {}
@@ -10726,7 +10741,7 @@ snapshots:
dependencies:
p-limit: 3.1.0
p-retry@8.0.1:
p-retry@9.0.0:
dependencies:
is-network-error: 1.3.2
+1 -1
View File
@@ -27,7 +27,7 @@ overrides:
# Security fixes:
# - https://github.com/advisories/GHSA-7r86-cg39-jmmj
# - https://github.com/advisories/GHSA-23c5-xmqv-rm74
"minimatch": "^10.2.3"
"minimatch": "^11.0.0"
# Security fix: https://github.com/element-hq/element-call/security/dependabot/109
"glob": "^10.5.0"
# Security fixes:
+1
View File
@@ -184,6 +184,7 @@ export function createRTCParticipation(
encryptMedia: encryptionSystem.kind !== E2eeType.NONE,
matrixRTCMode,
sendNotificationType: options.sendNotificationType,
callIntent: options.callIntent,
applicationData: options.applicationData,
timings,
delayedLeave,
+8 -4
View File
@@ -12,6 +12,7 @@ Please see LICENSE in the repository root for full details.
*/
import {
type RTCCallIntent,
type RTCNotificationType,
type Transport,
} from "matrix-js-sdk/lib/matrixrtc";
@@ -103,11 +104,14 @@ export interface RTCParticipationOptions {
* to a notification (ringing, timeouts, declines) is the application's job.
*/
sendNotificationType?: RTCNotificationType;
/**
* The kind of call the local member intends to take part in: `"m.audio"`
* or `"m.video"`. Carried by the membership as a field of its own.
*/
callIntent?: RTCCallIntent;
/**
* Whatever the application wants to say about itself in the membership,
* under namespaced keys. Opaque to the SDK, apart from `m.call.intent`
* (`"audio"` or `"video"`), which the membership format carries as a
* field of its own.
* under namespaced keys. Opaque to the SDK.
*/
applicationData?: Record<string, unknown>;
/** Timings the host has configured; the defaults otherwise. */
@@ -287,7 +291,7 @@ export interface RTCMember {
memberId: string;
/** Which transport this member is on; undefined when the membership has none. */
transport$: Behavior<TransportMetadata | undefined>;
/** What the member's application says about itself, e.g. `m.call.intent`. */
/** What the member's application says about itself, e.g. `intent`. */
applicationData$: Behavior<Record<string, unknown>>;
displayName$: Behavior<string>;
+8 -6
View File
@@ -60,7 +60,7 @@ describe("joinJsSdkSession", () => {
expect(s.join).toHaveBeenCalledWith(ownMemberMock, [transportWithAlias], {
notificationType: "ring",
callIntent: undefined,
applicationData: {},
applicationData: undefined,
manageMediaKeys: true,
delayedLeaveEventRestartMs: 300_000,
delayedLeaveEventDelayMs: 3_600_000,
@@ -95,13 +95,14 @@ describe("joinJsSdkSession", () => {
);
});
it("lifts the call intent out of the application data and sends sticky events in Matrix 2.0 mode", () => {
it("passes the call intent and application data through and sends sticky events in Matrix 2.0 mode", () => {
const s = session();
joinJsSdkSession("!room:example.org", s, ownMemberMock, transport, {
encryptMedia: true,
matrixRTCMode: MatrixRTCMode.Matrix_2_0,
applicationData: { "m.call.intent": "audio", custom: 1 },
callIntent: "m.audio",
applicationData: { custom: 1 },
timings: defaultSessionTimings,
delayedLeave: defaultSessionTimings.delayedLeave,
});
@@ -110,7 +111,7 @@ describe("joinJsSdkSession", () => {
ownMemberMock,
[transportWithAlias],
expect.objectContaining({
callIntent: "audio",
callIntent: "m.audio",
applicationData: { custom: 1 },
unstableSendStickyEvents: true,
}),
@@ -124,7 +125,8 @@ describe("joinJsSdkSession", () => {
joinJsSdkSession("!room:example.org", s, ownMemberMock, transport, {
encryptMedia: true,
matrixRTCMode: MatrixRTCMode.Matrix_2_0,
applicationData: { "m.call.intent": "audio", custom: 1 },
callIntent: "m.audio",
applicationData: { custom: 1 },
timings: defaultSessionTimings,
delayedLeave: defaultSessionTimings.delayedLeave,
});
@@ -133,7 +135,7 @@ describe("joinJsSdkSession", () => {
ownMemberMock,
[transport],
expect.objectContaining({
callIntent: "audio",
callIntent: "m.audio",
applicationData: { custom: 1 },
unstableSendStickyEvents: true,
}),
+3 -5
View File
@@ -23,6 +23,7 @@ interface Options {
encryptMedia: boolean;
matrixRTCMode: MatrixRTCMode;
sendNotificationType?: RTCNotificationType;
callIntent?: RTCCallIntent;
applicationData?: Record<string, unknown>;
timings: SessionTimings;
/** The delayed leave timings to use: delegated to the SFU or not. */
@@ -43,15 +44,12 @@ export function joinJsSdkSession(
encryptMedia,
matrixRTCMode,
sendNotificationType,
callIntent,
applicationData,
timings,
delayedLeave,
}: Options,
): void {
// The membership format carries the intent as a field of its own
const { "m.call.intent": intent, ...rest } = applicationData ?? {};
const callIntent =
typeof intent === "string" ? (intent as RTCCallIntent) : undefined;
// Give up on the network as soon as either the sync has been down for the
// grace period or the delayed leave has probably been sent, whichever is
// sooner
@@ -78,7 +76,7 @@ export function joinJsSdkSession(
{
notificationType: sendNotificationType,
callIntent,
applicationData: rest,
applicationData,
manageMediaKeys: encryptMedia,
delayedLeaveEventRestartMs: delayedLeave.restart_ms,
delayedLeaveEventDelayMs: delayedLeave.delay_ms,
@@ -225,6 +225,51 @@ describe("createLivekitMediaTrack", () => {
);
});
// LiveKit's media stream track is the processed one while a processor runs,
// and that one names no device; the camera behind it does
it.each([
{ deviceId: "cam-1", switches: false },
{ deviceId: "cam-2", switches: true },
])(
"leaves a processed camera alone unless its device changes: $deviceId",
async ({ deviceId, switches }) => {
vi.stubGlobal("MediaStreamTrack", class {});
onTestFinished(() => {
vi.unstubAllGlobals();
});
const camera = Object.assign(Object.create(LocalVideoTrack.prototype), {
source: Track.Source.Camera,
_mediaStreamTrack: {
readyState: "live",
label: "",
getSettings: () => ({ deviceId: "cam-1" }),
},
// What LiveKit shows as the track, with no device of its own
processor: {
processedTrack: { label: "", getSettings: () => ({}) },
},
}) as LocalVideoTrack;
const switchActiveDevice = vi.fn().mockResolvedValue(true);
const { participant, publication, room } = fakes({
track: camera as unknown as FakeTrack,
});
const track = createLocalLivekitMediaTrack(
scope,
Object.assign(participant, {
isLocal: true,
}) as unknown as LocalParticipant,
publication as unknown as TrackPublication,
Object.assign(room, { switchActiveDevice }),
vi.fn(),
);
await track.setDevice(deviceId);
expect(switchActiveDevice.mock.calls).toEqual(
switches ? [["videoinput", deviceId, false]] : [],
);
},
);
// LiveKit stops a camera that is turned off; the processor is built when it
// is turned on again
it.each([
@@ -199,11 +199,13 @@ export function createLocalLivekitMediaTrack(
throw new MatrixRTCError(`A ${base.source} track has no device`);
// LiveKit decides whether to restart by the constraints it keeps, which
// a browser that does not echo the device id back (Firefox) leaves
// without one; a track already on the device is left alone here
// without one; a track already on the device is left alone here. Asked
// of the camera track itself: with a processor on, LiveKit's media
// stream track is the processed one, which names no device
const current = publication.track;
if (
current instanceof LocalTrack &&
current.mediaStreamTrack.getSettings().deviceId === deviceId
(await current.getDeviceId(false)) === deviceId
)
return;
await room.switchActiveDevice(
+10 -8
View File
@@ -18,6 +18,8 @@ Please see LICENSE in the repository root for full details.
*/
import {
RTC_CALL_INTENT_AUDIO,
RTC_CALL_INTENT_VIDEO,
type RTCCallIntent,
type RTCNotificationType,
} from "matrix-js-sdk/lib/matrixrtc";
@@ -179,40 +181,40 @@ export function configurationForIntent(intent: UserIntent): UrlConfiguration {
switch (intent) {
case UserIntent.StartNewCall:
preset.skipLobby = false;
preset.callIntent = "video";
preset.callIntent = RTC_CALL_INTENT_VIDEO;
break;
case UserIntent.JoinExistingCall:
// On desktop this will be overridden based on which button was used to join the call
preset.skipLobby = false;
preset.callIntent = "video";
preset.callIntent = RTC_CALL_INTENT_VIDEO;
break;
case UserIntent.StartNewCallVoice:
preset.skipLobby = false;
preset.callIntent = "audio";
preset.callIntent = RTC_CALL_INTENT_AUDIO;
break;
case UserIntent.JoinExistingCallVoice:
// On desktop this will be overridden based on which button was used to join the call
preset.skipLobby = false;
preset.callIntent = "audio";
preset.callIntent = RTC_CALL_INTENT_AUDIO;
break;
case UserIntent.StartNewCallDMVoice:
preset.callIntent = "audio";
preset.callIntent = RTC_CALL_INTENT_AUDIO;
// Fall through
case UserIntent.StartNewCallDM:
preset.skipLobby = true;
preset.sendNotificationType = "ring";
preset.autoLeaveWhenOthersLeft = true;
preset.waitForCallPickup = true;
preset.callIntent = preset.callIntent ?? "video";
preset.callIntent = preset.callIntent ?? RTC_CALL_INTENT_VIDEO;
break;
case UserIntent.JoinExistingCallDMVoice:
preset.callIntent = "audio";
preset.callIntent = RTC_CALL_INTENT_AUDIO;
// Fall through
case UserIntent.JoinExistingCallDM:
// On desktop this will be overridden based on which button was used to join the call
preset.skipLobby = true;
preset.autoLeaveWhenOthersLeft = true;
preset.callIntent = preset.callIntent ?? "video";
preset.callIntent = preset.callIntent ?? RTC_CALL_INTENT_VIDEO;
break;
// Non widget usecase defaults
default:
+10 -6
View File
@@ -21,6 +21,10 @@ import {
UserIntent,
} from "../src/UrlParams";
import { mockConfig } from "./utils/test";
import {
RTC_CALL_INTENT_AUDIO,
RTC_CALL_INTENT_VIDEO,
} from "matrix-js-sdk/lib/matrixrtc";
const ROOM_NAME = "roomNameHere";
const ROOM_ID = "!d45f138fsd";
@@ -275,7 +279,7 @@ describe("UrlParams", () => {
).toMatchObject({
...startNewCallDefaults("desktop"),
skipLobby: false,
callIntent: "video",
callIntent: RTC_CALL_INTENT_VIDEO,
});
});
@@ -322,7 +326,7 @@ describe("UrlParams", () => {
).toMatchObject({
...startNewCallDefaults("desktop"),
skipLobby: false,
callIntent: "audio",
callIntent: RTC_CALL_INTENT_AUDIO,
});
});
@@ -333,7 +337,7 @@ describe("UrlParams", () => {
),
).toMatchObject({
...joinExistingCallDefaults("desktop"),
callIntent: "audio",
callIntent: RTC_CALL_INTENT_AUDIO,
});
});
@@ -348,7 +352,7 @@ describe("UrlParams", () => {
sendNotificationType: "ring",
autoLeaveWhenOthersLeft: true,
waitForCallPickup: true,
callIntent: "audio",
callIntent: RTC_CALL_INTENT_AUDIO,
});
});
@@ -363,7 +367,7 @@ describe("UrlParams", () => {
skipLobby: true,
autoLeaveWhenOthersLeft: true,
waitForCallPickup: false,
callIntent: "video",
callIntent: RTC_CALL_INTENT_VIDEO,
});
});
@@ -377,7 +381,7 @@ describe("UrlParams", () => {
skipLobby: true,
autoLeaveWhenOthersLeft: true,
waitForCallPickup: false,
callIntent: "audio",
callIntent: RTC_CALL_INTENT_AUDIO,
});
});
});
@@ -90,3 +90,42 @@ Please see LICENSE in the repository root for full details.
text-align: center;
overflow-wrap: anywhere;
}
.removable {
position: relative;
display: flex;
min-inline-size: 0;
}
.removable > .tile {
flex: 1;
}
/* Clear of the target for choosing the tile. */
.remove {
position: absolute;
inset-block-start: 0;
inset-inline-end: 0;
translate: 20% -20%;
display: none;
align-items: center;
justify-content: center;
inline-size: 28px;
block-size: 28px;
border-radius: 50%;
background: var(--cpd-color-bg-canvas-default);
color: var(--cpd-color-icon-critical-primary);
box-shadow: 0 1.2px 2.4px 0 rgb(0 0 0 / 15%);
cursor: pointer;
}
.remove:hover {
background: var(--cpd-color-bg-critical-subtle-hovered);
box-shadow: 0 0 0 var(--cpd-border-width-1)
var(--cpd-color-border-critical-subtle);
}
.removable:hover .remove,
.removable:focus-within .remove {
display: flex;
}
+82 -9
View File
@@ -5,14 +5,16 @@ SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { type FC, type ReactElement } from "react";
import { type FC, type KeyboardEvent, type ReactElement } from "react";
import { useTranslation } from "react-i18next";
import { VisuallyHidden } from "@radix-ui/react-visually-hidden";
import { InlineSpinner, MenuItem } from "@vector-im/compound-web";
import { InlineSpinner, MenuItem, Tooltip } from "@vector-im/compound-web";
import {
BlockIcon,
BlurIcon,
CheckCircleSolidIcon,
CloseIcon,
PlusIcon,
} from "@vector-im/compound-design-tokens/assets/web/icons";
import styles from "./BackgroundEffectGrid.module.css";
@@ -27,6 +29,8 @@ export interface BackgroundEffectGridItem {
kind: "none" | "blur" | "image";
/** For kind "image". */
imageUrl?: string;
/** Whether it is the user's to remove. */
removable?: boolean;
}
interface Props {
@@ -38,6 +42,12 @@ interface Props {
settling?: boolean;
/** Id of the text saying what the user should know before choosing. */
describedBy?: string;
/** Offers to add a background of one's own. Omit to leave that tile out. */
onAdd?: () => void;
addLabel?: string;
/** Removes a removable one, never the one in force. Omit to offer none. */
onRemove?: (id: string) => void;
removeLabel?: string;
}
/**
@@ -51,6 +61,10 @@ export const BackgroundEffectGrid: FC<Props> = ({
onSelect,
settling = false,
describedBy,
onAdd,
addLabel,
onRemove,
removeLabel,
}) => {
const { t } = useTranslation();
const choose = (id: string): void => {
@@ -66,7 +80,30 @@ export const BackgroundEffectGrid: FC<Props> = ({
: t("background_effects.numbered", { n: ++pictures });
const checked = effect.id === selected;
const disabled = onSelect === undefined && effect.kind !== "none";
return (
const remove =
effect.removable && !checked && onRemove !== undefined
? (): void => onRemove(effect.id)
: undefined;
const removeProps =
remove === undefined
? {}
: {
"aria-keyshortcuts": "Delete",
onKeyDown: (e: KeyboardEvent): void => {
if (e.key !== "Delete" && e.key !== "Backspace") return;
e.preventDefault();
remove();
},
};
const content = (
<TileContent
effect={effect}
label={label}
checked={checked}
settling={checked && settling}
/>
);
const tile = (
<MenuItem
key={effect.id}
role="menuitemradio"
@@ -88,21 +125,57 @@ export const BackgroundEffectGrid: FC<Props> = ({
choose(effect.id);
}
}
{...removeProps}
>
<TileContent
effect={effect}
label={label}
checked={checked}
settling={checked && settling}
/>
{content}
</MenuItem>
);
if (remove === undefined) return tile;
// Beside the item, not in it: inside, the item takes the press first and
// the tile is chosen instead.
return (
<div key={effect.id} className={styles.removable}>
{tile}
<Tooltip label={removeLabel ?? ""}>
{/* Not a control of its own, so the menu keeps one keyboard model:
the keyboard reaches the same action through Delete. */}
<span aria-hidden className={styles.remove} onClick={remove}>
<CloseIcon width={20} height={20} />
</span>
</Tooltip>
</div>
);
});
// A command rather than a choice, so an item beside the radios.
const addContent = (
<>
<span aria-hidden className={styles.swatch}>
<PlusIcon width={24} height={24} />
</span>
<VisuallyHidden>{addLabel}</VisuallyHidden>
</>
);
const addTile =
onAdd === undefined ? null : (
<MenuItem
label={null}
hideChevron
className={styles.tile}
onSelect={(e) => {
e.preventDefault();
onAdd();
}}
>
{addContent}
</MenuItem>
);
return (
// A group of its own, as the section's can't carry the description.
<div role="group" aria-describedby={describedBy} className={styles.grid}>
{tiles}
{addTile}
</div>
);
};
+6 -2
View File
@@ -127,16 +127,20 @@ export const Default: Story = {
selectBackgroundEffect: fn(),
backgroundEffect: "blur",
backgroundEffects: [
{ id: "none", kind: "none" },
{ id: "blur", kind: "blur" },
{ id: "none", kind: "none", removable: false },
{ id: "blur", kind: "blur", removable: false },
...shippedBackgrounds.map((background) => ({
id: `image:${background.id}`,
kind: "image" as const,
imageUrl: background.imagePath,
removable: false,
})),
],
backgroundEffectNotice: undefined,
backgroundEffectSettling: false,
backgroundImageRefusal: undefined,
addBackgroundImage: fn(),
removeBackgroundEffect: fn(),
hangup: fn(),
buttonSize: "lg",
showFooter: true,
+31
View File
@@ -33,6 +33,11 @@ import { useBehavior } from "../useBehavior";
import { type LayoutSwitchViewModel } from "../state/LayoutSwitchViewModel";
import { LayoutSwitch } from "../room/LayoutSwitch";
import { type BackgroundEffectGridItem } from "./BackgroundEffectGrid";
import { type UnusableReason } from "../tracks/backgroundImages";
export interface BackgroundImageRefusal {
reason: UnusableReason | "not-kept";
}
export interface AudioOutputSwitcher {
targetOutput: string;
@@ -60,6 +65,10 @@ export interface FooterActions {
toggleVideo: (() => void) | undefined;
/** Undefined where background effects can't be chosen. */
selectBackgroundEffect: ((id: string) => void) | undefined;
/** Undefined where no background can be added. */
addBackgroundImage: ((file: File) => void) | undefined;
/** Removes an added background by its option id; never the one in force. */
removeBackgroundEffect: ((id: string) => void) | undefined;
toggleScreenSharing: (() => void) | undefined;
/** Also controls if the settings button is visible */
openSettings: (() => void) | undefined;
@@ -81,6 +90,8 @@ export interface FooterState {
backgroundEffectNotice: "unavailable" | "slow" | undefined;
/** Whether the first effect chosen is still being prepared. */
backgroundEffectSettling: boolean;
/** Why the last file offered couldn't be kept, if it couldn't. */
backgroundImageRefusal: BackgroundImageRefusal | undefined;
showFooter: boolean;
/* This is needed for WindowMode = "flat" */
@@ -168,6 +179,23 @@ export const CallFooter: FC<FooterProps> = ({
const selectBackgroundEffect = useBehavior(vm.selectBackgroundEffect$);
const backgroundEffectNotice = useBehavior(vm.backgroundEffectNotice$);
const backgroundEffectSettling = useBehavior(vm.backgroundEffectSettling$);
const addBackgroundImage = useBehavior(vm.addBackgroundImage$);
const removeBackgroundEffect = useBehavior(vm.removeBackgroundEffect$);
const refusal = useBehavior(vm.backgroundImageRefusal$);
const refusalMessage = useMemo(() => {
switch (refusal?.reason) {
case undefined:
return undefined;
case "not-an-image":
return { text: t("error.background_not_an_image") };
case "animated":
return { text: t("error.background_animated") };
case "undecodable":
return { text: t("error.background_undecodable") };
case "not-kept":
return { text: t("error.background_not_kept") };
}
}, [refusal, t]);
const backgroundEffects = useBehavior(vm.backgroundEffects$);
const buttonSize = useBehavior(vm.buttonSize$);
const showLogo = useBehavior(vm.showLogo$);
@@ -234,6 +262,9 @@ export const CallFooter: FC<FooterProps> = ({
selectedBackgroundEffect={backgroundEffect}
onSelectBackgroundEffect={selectBackgroundEffect}
backgroundEffectSettling={backgroundEffectSettling}
onAddBackgroundImage={addBackgroundImage}
onRemoveBackgroundEffect={removeBackgroundEffect}
backgroundImageRefusal={refusalMessage}
backgroundEffectNotice={
backgroundEffectNotice === "unavailable"
? t("background_effects.unavailable")
+101 -4
View File
@@ -8,7 +8,12 @@ Please see LICENSE in the repository root for full details.
import { afterEach, describe, expect, it, vi } from "vitest";
import { BehaviorSubject } from "rxjs";
import { testScope, mockMuteStates, mockMediaDevices } from "../utils/test";
import {
flushPromises,
testScope,
mockMuteStates,
mockMediaDevices,
} from "../utils/test";
import { constant } from "../state/Behavior";
import type { CallViewModel } from "../state/CallViewModel/CallViewModel";
import type { Layout } from "../state/layout-types";
@@ -21,6 +26,13 @@ import {
import { HeaderStyle } from "../UrlParams";
import { backgroundEffect as backgroundEffectSetting } from "../settings/settings";
import { shippedBackgrounds } from "../tracks/backgroundEffectCatalog";
import type * as BackgroundImages from "../tracks/backgroundImages";
import {
type AddedBackground,
addedBackgrounds,
maxAddedBackgrounds,
UnusableImage,
} from "../tracks/backgroundImages";
const platformMock = vi.hoisted(() => vi.fn(() => "desktop"));
vi.mock("../Platform", () => ({
@@ -37,6 +49,22 @@ vi.mock("@livekit/track-processors", () => ({
supportsModernBackgroundProcessors: (): boolean => modernRouteMock(),
}));
const store = vi.hoisted(() => ({
add: vi.fn<(file: Blob) => Promise<string>>(),
remove: vi.fn<(id: string) => Promise<void>>(),
}));
vi.mock("../tracks/backgroundImages", async (original) => {
const { BehaviorSubject } = await import("rxjs");
return {
...(await original<typeof BackgroundImages>()),
addedBackgrounds: {
added$: new BehaviorSubject<AddedBackground[] | undefined>([]),
add: store.add,
remove: store.remove,
},
};
});
const outputSelectionMock = vi.hoisted(() => vi.fn(() => true));
vi.mock("livekit-client", () => ({
supportsAudioOutputSelection: (): boolean => outputSelectionMock(),
@@ -263,6 +291,7 @@ describe("createCallFooterViewModel", () => {
it("offers nothing where the browser itself cannot run them", () => {
sdkSupportMock.mockReturnValue(false);
expect(lobbyOn("desktop").selectBackgroundEffect$.value).toBeUndefined();
expect(lobbyOn("desktop").addBackgroundImage$.value).toBeUndefined();
});
it("puts no effect in force where the browser cannot run them", () => {
@@ -293,17 +322,29 @@ describe("createCallFooterViewModel", () => {
expect(lobbyOn("desktop").backgroundEffectNotice$.value).toBeUndefined();
});
it("offers every effect in order", () => {
it("offers every effect in order, the added ones removable", () => {
sdkSupportMock.mockReturnValue(true);
const added$ = addedBackgrounds.added$ as BehaviorSubject<
AddedBackground[] | undefined
>;
added$.next([{ id: "mine", url: "blob:mine" }]);
expect(lobbyOn("desktop").backgroundEffects$.value).toEqual([
{ id: "none", kind: "none" },
{ id: "blur", kind: "blur" },
{ id: "none", kind: "none", removable: false },
{ id: "blur", kind: "blur", removable: false },
...shippedBackgrounds.map((background) => ({
id: `image:${background.id}`,
kind: "image",
imageUrl: background.imagePath,
removable: false,
})),
{
id: "added:mine",
kind: "image",
imageUrl: "blob:mine",
removable: true,
},
]);
added$.next([]);
});
it("says they are unavailable where they cannot be chosen", () => {
@@ -330,6 +371,62 @@ describe("createCallFooterViewModel", () => {
expect(vm.backgroundEffectSettling$.value).toBe(false);
});
it("adding is unavailable at the limit", () => {
sdkSupportMock.mockReturnValue(true);
const added$ = addedBackgrounds.added$ as BehaviorSubject<
AddedBackground[] | undefined
>;
const vm = lobbyOn("desktop");
expect(vm.addBackgroundImage$.value).toBeDefined();
added$.next(
Array.from({ length: maxAddedBackgrounds }, (_, i) => ({
id: `${i}`,
url: `blob:${i}`,
})),
);
expect(vm.addBackgroundImage$.value).toBeUndefined();
added$.next(added$.value!.slice(1));
expect(vm.addBackgroundImage$.value).toBeDefined();
added$.next([]);
});
it("says why each refused file was refused", async () => {
sdkSupportMock.mockReturnValue(true);
const vm = lobbyOn("desktop");
store.add.mockRejectedValue(new UnusableImage("animated"));
vm.addBackgroundImage$.value!(new File(["x"], "a.gif"));
await flushPromises();
const first = vm.backgroundImageRefusal$.value;
expect(first).toEqual({ reason: "animated" });
// The same again is a refusal of its own, so it is shown again.
vm.addBackgroundImage$.value!(new File(["x"], "a.gif"));
await flushPromises();
expect(vm.backgroundImageRefusal$.value).toEqual({ reason: "animated" });
expect(vm.backgroundImageRefusal$.value).not.toBe(first);
store.add.mockRejectedValue(new Error("quota"));
vm.addBackgroundImage$.value!(new File(["x"], "b.png"));
await flushPromises();
expect(vm.backgroundImageRefusal$.value).toEqual({ reason: "not-kept" });
store.add.mockResolvedValue("kept");
vm.addBackgroundImage$.value!(new File(["x"], "c.png"));
await flushPromises();
expect(vm.backgroundImageRefusal$.value).toBeUndefined();
});
it("never removes the background in force", () => {
sdkSupportMock.mockReturnValue(true);
store.remove.mockResolvedValue();
const vm = lobbyOn("desktop");
backgroundEffectSetting.setValue("added:worn");
vm.removeBackgroundEffect$.value!("added:worn");
expect(store.remove).not.toHaveBeenCalled();
vm.removeBackgroundEffect$.value!("added:spare");
expect(store.remove).toHaveBeenCalledWith("spare");
});
it("availability is the same before and during a call", () => {
for (const supported of [true, false]) {
sdkSupportMock.mockReturnValue(supported);
+71 -5
View File
@@ -6,6 +6,7 @@ Please see LICENSE in the repository root for full details.
*/
import {
BehaviorSubject,
combineLatest,
distinctUntilChanged,
map,
@@ -17,12 +18,19 @@ import {
supportsBackgroundProcessors,
supportsModernBackgroundProcessors,
} from "@livekit/track-processors";
import { logger } from "matrix-js-sdk/lib/logger";
import {
parseEffect,
serializeEffect,
shippedBackgrounds,
} from "../tracks/backgroundEffectCatalog";
import {
type AddedBackground,
addedBackgrounds,
maxAddedBackgrounds,
UnusableImage,
} from "../tracks/backgroundImages";
import { type CallViewModel } from "../state/CallViewModel/CallViewModel";
import { type MenuOptions } from "./MediaMuteAndSwitchButton";
@@ -37,7 +45,7 @@ import { type MuteStates } from "../state/MuteStates";
import { createStaticViewModel, type ViewModel } from "../state/ViewModel";
import { HeaderStyle } from "../UrlParams";
import { platform } from "../Platform";
import { type FooterSnapshot } from "./CallFooter";
import { type BackgroundImageRefusal, type FooterSnapshot } from "./CallFooter";
import { type BackgroundEffectGridItem } from "./BackgroundEffectGrid";
/**
@@ -96,6 +104,9 @@ function buildDeviceBehaviors(
| "backgroundEffects$"
| "backgroundEffectNotice$"
| "backgroundEffectSettling$"
| "addBackgroundImage$"
| "backgroundImageRefusal$"
| "removeBackgroundEffect$"
> {
const options$ = (
available$: Behavior<Map<string, MenuOptions["label"]>>,
@@ -117,6 +128,10 @@ function buildDeviceBehaviors(
const offered$ = disableSwitcher$.pipe(
map((switcherDisabled) => !switcherDisabled && supported),
);
// A new object for each refusal, so the same one twice is shown twice.
const refusal$ = new BehaviorSubject<BackgroundImageRefusal | undefined>(
undefined,
);
return {
audioOptions$: scope.behavior(options$(mediaDevices.audioInput.available$)),
selectedAudio$: scope.behavior(
@@ -158,7 +173,11 @@ function buildDeviceBehaviors(
),
),
),
backgroundEffects$: constant(backgroundEffectChoices()),
backgroundEffects$: scope.behavior(
addedBackgrounds.added$.pipe(
map((added) => backgroundEffectChoices(added ?? [])),
),
),
backgroundEffectNotice$: scope.behavior(
offered$.pipe(
map((offered) =>
@@ -170,6 +189,44 @@ function buildDeviceBehaviors(
backgroundEffectSettling$.pipe(distinctUntilChanged()),
false,
),
// Kept and offered, not put on: that waits for the user to choose it.
addBackgroundImage$: scope.behavior(
combineLatest([offered$, addedBackgrounds.added$]).pipe(
map(([offered, added]) =>
offered && (added?.length ?? 0) < maxAddedBackgrounds
? (file: File): void => {
refusal$.next(undefined);
addedBackgrounds.add(file).catch((e) => {
logger.warn("Could not keep that background", e);
refusal$.next({
reason: e instanceof UnusableImage ? e.reason : "not-kept",
});
});
}
: undefined,
),
),
),
backgroundImageRefusal$: refusal$,
// Never the one in force: taking it away would leave nothing chosen.
removeBackgroundEffect$: scope.behavior(
offered$.pipe(
map((offered) =>
offered
? (optionId: string): void => {
const effect = parseEffect(optionId);
if (effect.kind !== "added") return;
if (backgroundEffectSetting.getValue() === optionId) return;
addedBackgrounds
.remove(effect.id)
.catch((e) =>
logger.warn("Could not remove that background", e),
);
}
: undefined,
),
),
),
};
}
@@ -337,14 +394,23 @@ export function createLobbyFooterViewModel(
};
}
function backgroundEffectChoices(): BackgroundEffectGridItem[] {
function backgroundEffectChoices(
added: AddedBackground[],
): BackgroundEffectGridItem[] {
return [
{ id: serializeEffect({ kind: "none" }), kind: "none" },
{ id: serializeEffect({ kind: "blur" }), kind: "blur" },
{ id: serializeEffect({ kind: "none" }), kind: "none", removable: false },
{ id: serializeEffect({ kind: "blur" }), kind: "blur", removable: false },
...shippedBackgrounds.map((background) => ({
id: serializeEffect({ kind: "shipped", id: background.id }),
kind: "image" as const,
imageUrl: background.imagePath,
removable: false,
})),
...added.map((background) => ({
id: serializeEffect({ kind: "added", id: background.id }),
kind: "image" as const,
imageUrl: background.url,
removable: true,
})),
];
}
@@ -93,3 +93,7 @@ Please see LICENSE in the repository root for full details.
flex-shrink: 0;
color: var(--cpd-color-icon-secondary);
}
.refusal {
margin: var(--cpd-space-2x) var(--cpd-space-4x) 0;
}
@@ -198,6 +198,7 @@ export const VideoUnmute: Story = {
backgroundEffects: effects,
selectedBackgroundEffect: "none",
onSelectBackgroundEffect: fn(),
onAddBackgroundImage: fn(),
},
};
@@ -241,6 +242,11 @@ export const BackgroundEffects: Story = {
true,
);
const add = within(section).getByRole("menuitem", { name: "Add image" });
await expect(
tiles[3].compareDocumentPosition(add) & Node.DOCUMENT_POSITION_FOLLOWING,
).toBeTruthy();
await userEvent.click(menu.getByRole("menuitemradio", { name: "Blur" }));
await expect(args.onSelectBackgroundEffect).toHaveBeenCalledWith("blur");
await expect(menu.getByRole("menu")).toBeVisible();
@@ -341,6 +347,160 @@ export const BackgroundEffectsSettling: Story = {
},
};
/** A file that can't be used: said where it was chosen, and nothing changes. */
export const BackgroundImageRefused: Story = {
args: {
...VideoUnmute.args,
selectedBackgroundEffect: "blur",
backgroundEffectNotice:
"Background effects run slowly on this platform, which may cause your video to stutter.",
backgroundImageRefusal: { text: "That file is not a supported image" },
},
decorators: [
(Story): JSX.Element => (
<WithACallArea blockSize={300}>
<Story />
</WithACallArea>
),
],
play: async ({ canvasElement }) => {
const canvas = within(canvasElement);
await userEvent.click(canvas.getByRole("button", { name: "Camera" }));
const menu = within(document.body);
const message = await menu.findByText("That file is not a supported image");
const list = document.body.querySelector<HTMLElement>(
`.${styles.deviceList}`,
)!;
await expect(list.scrollHeight).toBeGreaterThan(list.clientHeight);
const scrollport = list.getBoundingClientRect();
const shown = message.getBoundingClientRect();
await expect(shown.bottom).toBeLessThanOrEqual(scrollport.bottom + 1);
await expect(shown.top).toBeGreaterThanOrEqual(scrollport.top - 1);
await expect(menu.queryByText(/run slowly/)).toBeNull();
await expect(
menu.getByRole("menuitemradio", { name: "Blur" }),
).toHaveAttribute("aria-checked", "true");
},
};
/** A flat picture, for backgrounds of one's own. */
function swatch(colour: string): string {
return `data:image/svg+xml,${encodeURIComponent(
`<svg xmlns="http://www.w3.org/2000/svg" width="16" height="9"><rect width="16" height="9" fill="${colour}"/></svg>`,
)}`;
}
/** One's own backgrounds are removable, never the one in force. */
export const AddedBackgroundsCanBeRemoved: Story = {
args: {
...VideoUnmute.args,
backgroundEffects: [
...effects,
{
id: "added:one",
kind: "image",
imageUrl: swatch("#c2410c"),
removable: true,
},
{
id: "added:two",
kind: "image",
imageUrl: swatch("#1d4ed8"),
removable: true,
},
],
selectedBackgroundEffect: "added:one",
onRemoveBackgroundEffect: fn(),
},
play: async ({ args, canvasElement }) => {
const canvas = within(canvasElement);
await userEvent.click(canvas.getByRole("button", { name: "Camera" }));
const body = within(document.body);
const inForce = await body.findByRole("menuitemradio", {
name: "Background 3",
});
await expect(inForce).not.toHaveAttribute("aria-keyshortcuts");
const other = body.getByRole("menuitemradio", { name: "Background 4" });
await expect(other).toHaveAttribute("aria-keyshortcuts", "Delete");
other.focus();
await userEvent.keyboard("{Delete}");
await expect(args.onRemoveBackgroundEffect).toHaveBeenCalledWith(
"added:two",
);
const wrapper = other.parentElement!;
await userEvent.hover(wrapper);
const cross = wrapper.querySelector<HTMLElement>(
"[aria-hidden]:last-child",
)!;
const probe = document.createElement("span");
probe.style.color = "var(--cpd-color-icon-critical-primary)";
cross.append(probe);
const critical = getComputedStyle(probe).color;
probe.remove();
await expect(getComputedStyle(cross).color).toBe(critical);
const tile = other.getBoundingClientRect();
const box = cross.getBoundingClientRect();
await expect(box.right).toBeGreaterThan(tile.right);
await expect(box.top).toBeLessThan(tile.top);
await userEvent.hover(cross);
await waitFor(async () =>
expect(
[...document.body.querySelectorAll("div")].some(
(d) => d.textContent === "Remove" && d.offsetParent !== null,
),
).toBe(true),
);
},
};
/** Removing by pointer while the selection really changes. */
export const RemovingWithLiveSelection: Story = {
args: AddedBackgroundsCanBeRemoved.args,
render: function WithLiveSelection(args): JSX.Element {
const [selected, setSelected] = useState("none");
const [offered, setOffered] = useState(args.backgroundEffects ?? []);
return (
<MediaMuteAndSwitchButton
{...args}
backgroundEffects={offered}
selectedBackgroundEffect={selected}
onSelectBackgroundEffect={setSelected}
onRemoveBackgroundEffect={(id): void =>
setOffered((current) => current.filter((o) => o.id !== id))
}
/>
);
},
play: async ({ canvasElement }) => {
const canvas = within(canvasElement);
await userEvent.click(canvas.getByRole("button", { name: "Camera" }));
const body = within(document.body);
const tile = await body.findByRole("menuitemradio", {
name: "Background 4",
});
await userEvent.hover(tile);
const cross = tile.parentElement!.querySelector<HTMLElement>(
"[aria-hidden]:last-child",
)!;
await expect(cross).toBeVisible();
await userEvent.click(cross);
await waitFor(async () =>
expect(
body.queryByRole("menuitemradio", { name: "Background 4" }),
).toBeNull(),
);
await expect(
body.getByRole("menuitemradio", { name: "None" }),
).toHaveAttribute("aria-checked", "true");
},
};
/** In a short call the effects scroll into view with the list. */
export const BackgroundEffectsScrollWhenTheyDoNotFit: Story = {
args: {
@@ -5,7 +5,7 @@ SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { afterEach, describe, expect, test, vi } from "vitest";
import { afterEach, describe, expect, onTestFinished, test, vi } from "vitest";
import { axe } from "vitest-axe";
import {
act,
@@ -367,6 +367,120 @@ describe("MediaMuteAndSwitchButton", () => {
expect(onSelectBackgroundEffect).not.toHaveBeenCalled();
});
test("added images are chosen like shipped ones", async () => {
const user = userEvent.setup();
const onSelectBackgroundEffect = vi.fn();
const { getByRole } = renderComponent(
<MediaMuteAndSwitchButton
iconsAndLabels="video"
enabled={true}
options={[{ label: { type: "name", name: "Camera 1" }, id: "cam1" }]}
selectedOption="cam1"
backgroundEffects={[
...effects,
{
id: "added:mine",
kind: "image",
imageUrl: "",
},
]}
selectedBackgroundEffect="none"
onSelectBackgroundEffect={onSelectBackgroundEffect}
/>,
);
await user.click(getByRole("button", { name: "Camera" }));
const section = screen.getByRole("group", { name: "Background effects" });
await user.click(
within(section).getByRole("menuitemradio", { name: "Background 3" }),
);
expect(onSelectBackgroundEffect).toHaveBeenCalledWith("added:mine");
});
test("adds the file chosen from the add tile, and stays open", async () => {
const user = userEvent.setup();
const onAddBackgroundImage = vi.fn();
const pick = vi
.spyOn(HTMLInputElement.prototype, "click")
.mockImplementation(() => {});
try {
const { getByRole, container } = renderComponent(
<MediaMuteAndSwitchButton
iconsAndLabels="video"
enabled={true}
options={[{ label: { type: "name", name: "Camera 1" }, id: "cam1" }]}
selectedOption="cam1"
backgroundEffects={effects}
selectedBackgroundEffect="none"
onSelectBackgroundEffect={vi.fn()}
onAddBackgroundImage={onAddBackgroundImage}
/>,
);
await user.click(getByRole("button", { name: "Camera" }));
await user.click(screen.getByRole("menuitem", { name: "Add image" }));
expect(pick).toHaveBeenCalled();
// The picker takes the focus, which would otherwise close the menu.
await user.keyboard("{Escape}");
expect(screen.getByRole("menu")).toBeInTheDocument();
const file = new File(["x"], "mine.png", { type: "image/png" });
// What the picker hands back; the open menu blocks pointer events.
fireEvent.change(
container.querySelector<HTMLInputElement>("input[type=file]")!,
{ target: { files: [file] } },
);
expect(onAddBackgroundImage).toHaveBeenCalledWith(file);
screen.getByRole("menuitemradio", { name: "None", checked: true });
} finally {
pick.mockRestore();
}
});
test("a refusal can be dismissed and does not outlive the menu", async () => {
const user = userEvent.setup();
// jsdom lays nothing out, so has nothing to scroll.
const scroll = Element.prototype.scrollIntoView;
Element.prototype.scrollIntoView = vi.fn();
onTestFinished(() => {
Element.prototype.scrollIntoView = scroll;
});
const menu = (refusal?: { text: string }): JSX.Element =>
withProviders(
<MediaMuteAndSwitchButton
iconsAndLabels="video"
enabled={true}
options={[{ label: { type: "name", name: "Camera 1" }, id: "cam1" }]}
selectedOption="cam1"
backgroundEffects={effects}
selectedBackgroundEffect="blur"
onSelectBackgroundEffect={vi.fn()}
onAddBackgroundImage={vi.fn()}
backgroundImageRefusal={refusal}
/>,
);
const { rerender, getByRole } = render(menu());
await user.click(getByRole("button", { name: "Camera" }));
rerender(menu({ text: "That file is not a supported image" }));
expect(
screen.getByText("That file is not a supported image"),
).toBeVisible();
screen.getByRole("menuitemradio", { name: "Blur", checked: true });
await user.click(screen.getByRole("button", { name: "Close" }));
expect(screen.queryByText("That file is not a supported image")).toBeNull();
// The next refusal is shown, even if it says the same.
rerender(menu({ text: "That file is not a supported image" }));
expect(
screen.getByText("That file is not a supported image"),
).toBeVisible();
await user.keyboard("{Escape}");
await user.click(getByRole("button", { name: "Camera" }));
expect(screen.queryByText("That file is not a supported image")).toBeNull();
});
test("offers the background effects in a phone's drawer", async () => {
const userAgent = vi
.spyOn(navigator, "userAgent", "get")
+90 -8
View File
@@ -8,6 +8,7 @@ Please see LICENSE in the repository root for full details.
import {
useCallback,
useId,
useRef,
useState,
type CSSProperties,
type FC,
@@ -15,6 +16,7 @@ import {
type ReactElement,
} from "react";
import {
Alert,
Button,
Menu,
MenuItem,
@@ -79,6 +81,12 @@ export interface MediaMuteAndSwitchButtonProps {
backgroundEffectNotice?: string;
/** Whether the first effect chosen is still being prepared. */
backgroundEffectSettling?: boolean;
/** Called with the file chosen from the add tile. Omit to leave it out. */
onAddBackgroundImage?: (file: File) => void;
/** Removes a removable effect, never the one in force. */
onRemoveBackgroundEffect?: (id: string) => void;
/** Why the last file offered couldn't be used; a new object each time. */
backgroundImageRefusal?: { text: string };
/**
* For any toggle and option this method will be called.
* So toggles need to be implemented by listening here and setting the right toggle item to `enabled`
@@ -113,6 +121,9 @@ export const MediaMuteAndSwitchButton: FC<MediaMuteAndSwitchButtonProps> = ({
onSelectBackgroundEffect,
backgroundEffectNotice,
backgroundEffectSettling,
onAddBackgroundImage,
onRemoveBackgroundEffect,
backgroundImageRefusal,
onSelect,
}) => {
// Requested but not yet selected. Keyed by kind too, since Chrome uses
@@ -122,11 +133,42 @@ export const MediaMuteAndSwitchButton: FC<MediaMuteAndSwitchButtonProps> = ({
id: string;
} | null>(null);
const [menuOpen, setMenuOpen] = useState(false);
const onOpenChange = useCallback((open: boolean): void => {
setMenuOpen(open);
// Drop a request that never arrived.
if (!open) setPlannedSelection(null);
// The file picker takes the focus, which the menu reads as a click
// elsewhere; it is held open until the picker is done.
const choosingFile = useRef(false);
const fileInput = useRef<HTMLInputElement | null>(null);
// A refusal is about the attempt that caused it: shown until dismissed or
// until the menu closes, and again for the next one.
const [seenRefusal, setSeenRefusal] = useState<object>();
const refusal =
backgroundImageRefusal !== seenRefusal ? backgroundImageRefusal : undefined;
const onOpenChange = useCallback(
(open: boolean): void => {
if (!open && choosingFile.current) return;
setMenuOpen(open);
// Drop a request that never arrived.
if (!open) setPlannedSelection(null);
if (!open) setSeenRefusal(backgroundImageRefusal);
},
[backgroundImageRefusal],
);
// Scrolled to as it appears, as the list may be scrolled away from it.
const scrollIntoView = useCallback((element: HTMLElement | null): void => {
element?.scrollIntoView({ block: "nearest" });
}, []);
const watchFileInput = useCallback(
(input: HTMLInputElement | null): (() => void) | undefined => {
fileInput.current = input;
if (input === null) return;
// Dismissing the picker fires cancel, which React doesn't type.
const done = (): void => {
choosingFile.current = false;
};
input.addEventListener("cancel", done);
return (): void => input.removeEventListener("cancel", done);
},
[],
);
const isBusy = busy ?? false;
const { t } = useTranslation();
const devices = useMediaDevices();
@@ -326,6 +368,21 @@ export const MediaMuteAndSwitchButton: FC<MediaMuteAndSwitchButtonProps> = ({
>
{/* The mute button lives inside */}
{button}
{onAddBackgroundImage !== undefined && (
<input
ref={watchFileInput}
type="file"
accept="image/*"
hidden
onChange={(e) => {
const file = e.target.files?.[0];
// Cleared, so the same file can be chosen twice in a row.
e.target.value = "";
choosingFile.current = false;
if (file) onAddBackgroundImage(file);
}}
/>
)}
<Menu
className={styles.menu}
// Named for screen readers only: each section has its own heading.
@@ -394,14 +451,39 @@ export const MediaMuteAndSwitchButton: FC<MediaMuteAndSwitchButtonProps> = ({
describedBy={
backgroundEffectNotice === undefined ? undefined : noticeId
}
onAdd={
onAddBackgroundImage === undefined
? undefined
: (): void => {
choosingFile.current = true;
fileInput.current?.click();
}
}
addLabel={t("action.add_background_image")}
onRemove={onRemoveBackgroundEffect}
removeLabel={t("action.remove")}
/>
{/* In the list, so the menu grows no taller for it. */}
{backgroundEffectNotice !== undefined && (
<div id={noticeId} role="none" className={styles.notice}>
<InfoIcon width={20} height={20} />
<span>{backgroundEffectNotice}</span>
{refusal !== undefined && (
<div
ref={scrollIntoView}
role="none"
className={styles.refusal}
>
<Alert
type="critical"
title={refusal.text}
onClose={(): void => setSeenRefusal(refusal)}
/>
</div>
)}
{refusal === undefined &&
backgroundEffectNotice !== undefined && (
<div id={noticeId} role="none" className={styles.notice}>
<InfoIcon width={20} height={20} />
<span>{backgroundEffectNotice}</span>
</div>
)}
</MenuScrollSection>
)}
</MenuScrollArea>
+13
View File
@@ -95,6 +95,19 @@ describe("Config.initWith", () => {
expect(Config.get().ssla).toBe("https://example.invalid/ssla");
});
it("falls back to the defaults when fetching config.json rejects", async () => {
// A rejected promise is what a webview gives for a missing local file.
vi.spyOn(globalThis, "fetch").mockRejectedValue(
new TypeError("Load failed"),
);
vi.spyOn(logger, "info").mockImplementation(() => {});
Reflect.set(Config, "internalInstance", undefined);
await expect(Config.init()).resolves.toBeUndefined();
expect(Config.get().media_quality).toEqual(DEFAULT_CONFIG.media_quality);
});
it("replaces a config initialized earlier", () => {
Config.initWith({ ssla: "https://first.invalid/ssla" });
Config.initWith({ ssla: "https://second.invalid/ssla" });
+12 -5
View File
@@ -140,13 +140,20 @@ export function validateConfig(config: ConfigOptions): ConfigOptions {
}
async function downloadConfig(fetchTarget: string): Promise<ConfigOptions> {
const response = await fetch(fetchTarget);
let response: Response;
try {
response = await fetch(fetchTarget);
} catch (e) {
// The embedded package ships no config.json, and some webviews (Element X
// iOS) reject the fetch of a missing local file rather than answering 404.
logger.info(`No config at ${fetchTarget}, using defaults`, e);
// resolveConfig merges in the defaults.
return {};
}
if (isFailure(response)) {
// Lack of a config isn't an error, we should just use the defaults.
// Also treat a blank config as no config, assuming the status code is 0, because we don't get 404s from file:
// URIs so this is the only way we can not fail if the file doesn't exist when loading from a file:// URI.
return DEFAULT_CONFIG;
// resolveConfig merges in the defaults.
return {};
}
return response.json();
+1 -3
View File
@@ -825,9 +825,7 @@ export function rtcParticipationOptions(
const session = config.matrix_rtc_session;
return {
sendNotificationType: urlParams.sendNotificationType,
applicationData: urlParams.callIntent
? { "m.call.intent": urlParams.callIntent }
: undefined,
callIntent: urlParams.callIntent,
timings: {
syncDisconnectGracePeriodMs: config.sync_disconnect_grace_period_ms,
networkErrorRetryMs: session.network_error_retry_ms,
+2 -1
View File
@@ -8,6 +8,7 @@ Please see LICENSE in the repository root for full details.
import { expect, type MockedFunction, test, vi } from "vitest";
import { act, render } from "@testing-library/react";
import { BehaviorSubject } from "rxjs";
import { RTC_CALL_INTENT_AUDIO } from "matrix-js-sdk/lib/matrixrtc";
import { useAudioContext } from "../useAudioContext";
import { createRingingMedia } from "../state/media/RingingMediaViewModel";
@@ -39,7 +40,7 @@ test("ringtone plays on loop while ringing", () => {
userId: alice.userId,
displayName$: constant("Alice"),
mxcAvatarUrl$: constant(undefined),
intent: "audio",
intent: RTC_CALL_INTENT_AUDIO,
pickupState$,
});
+18 -12
View File
@@ -7,7 +7,11 @@ Please see LICENSE in the repository root for full details.
import { it, vi, expect, beforeEach, afterEach, describe } from "vitest";
import { firstValueFrom, of, Subject, take, toArray } from "rxjs";
import { type RTCCallIntent } from "matrix-js-sdk/lib/matrixrtc";
import {
RTC_CALL_INTENT_AUDIO,
RTC_CALL_INTENT_VIDEO,
type RTCCallIntent,
} from "matrix-js-sdk/lib/matrixrtc";
import { AndroidControlledAudioOutput } from "./AndroidControlledAudioOutput.ts";
import type { Controls, OutputDevice } from "../controls";
@@ -117,7 +121,7 @@ describe("Default selection", () => {
const controlledAudioOutput = new AndroidControlledAudioOutput(
of(BASE_DEVICE_LIST),
testScope,
"video",
RTC_CALL_INTENT_VIDEO,
mockControls,
);
@@ -142,7 +146,7 @@ describe("Default selection", () => {
const controlledAudioOutput = new AndroidControlledAudioOutput(
of(BASE_DEVICE_LIST),
testScope,
"audio",
RTC_CALL_INTENT_AUDIO,
mockControls,
);
@@ -163,7 +167,7 @@ describe("Default selection", () => {
});
});
["audio", "video"].forEach((callIntent) => {
[RTC_CALL_INTENT_AUDIO, RTC_CALL_INTENT_VIDEO].forEach((callIntent) => {
it(`Default to BT headset for ${callIntent} calls if present`, async () => {
const controlledAudioOutput = new AndroidControlledAudioOutput(
of(BT_HEADSET_BASE_DEVICE_LIST),
@@ -190,7 +194,7 @@ describe("Default selection", () => {
});
});
["audio", "video"].forEach((callIntent) => {
[RTC_CALL_INTENT_AUDIO, RTC_CALL_INTENT_VIDEO].forEach((callIntent) => {
it(`Default to wired headset for ${callIntent} calls if present`, async () => {
const controlledAudioOutput = new AndroidControlledAudioOutput(
of(WIRED_HEADSET_BASE_DEVICE_LIST),
@@ -300,7 +304,7 @@ describe("Test select a device", () => {
b: BT_HEADSET_BASE_DEVICE_LIST,
}),
testScope,
"audio",
RTC_CALL_INTENT_AUDIO,
mockControls,
);
@@ -340,7 +344,7 @@ describe("Test select a device", () => {
c: BASE_DEVICE_LIST,
}),
testScope,
"audio",
RTC_CALL_INTENT_AUDIO,
mockControls,
);
@@ -393,7 +397,7 @@ describe("Available device changes", () => {
});
it("When a BT headset is added, control should switch to use it", () => {
createAudioControlledOutput("video");
createAudioControlledOutput(RTC_CALL_INTENT_VIDEO);
// Emit the base device list, the speaker should be selected
availableSource$.next(BASE_DEVICE_LIST);
@@ -419,7 +423,9 @@ describe("Available device changes", () => {
// Android does not set `isExternalHeadset` to true for wired headphones, so we can't test this case.'
it.skip("When a wired headset is added, control should switch to use it", async () => {
const controlledAudioOutput = createAudioControlledOutput("video");
const controlledAudioOutput = createAudioControlledOutput(
RTC_CALL_INTENT_VIDEO,
);
// Emit the base device list, the speaker should be selected
availableSource$.next(BASE_DEVICE_LIST);
@@ -446,7 +452,7 @@ describe("Available device changes", () => {
});
it("When the active bt headset is removed on audio call, control should switch to earpiece", () => {
createAudioControlledOutput("audio");
createAudioControlledOutput(RTC_CALL_INTENT_AUDIO);
// Emit the BT headset device list, the BT headset should be selected
availableSource$.next(BT_HEADSET_BASE_DEVICE_LIST);
@@ -472,7 +478,7 @@ describe("Available device changes", () => {
});
it("When the active bt headset is removed on video call, control should switch to speaker", () => {
createAudioControlledOutput("video");
createAudioControlledOutput(RTC_CALL_INTENT_VIDEO);
availableSource$.next(BT_HEADSET_BASE_DEVICE_LIST);
@@ -497,7 +503,7 @@ describe("Available device changes", () => {
});
it("Do not repeatidly set the same device", () => {
createAudioControlledOutput("video");
createAudioControlledOutput(RTC_CALL_INTENT_VIDEO);
availableSource$.next(BT_HEADSET_BASE_DEVICE_LIST);
availableSource$.next(BT_HEADSET_BASE_DEVICE_LIST);
+6 -3
View File
@@ -22,7 +22,10 @@ import {
type SelectedAudioOutputDevice,
} from "./MediaDevices.ts";
import type { ObservableScope } from "./ObservableScope.ts";
import type { RTCCallIntent } from "matrix-js-sdk/lib/matrixrtc";
import {
RTC_CALL_INTENT_AUDIO,
type RTCCallIntent,
} from "matrix-js-sdk/lib/matrixrtc";
import { type Controls, type OutputDevice } from "../controls.ts";
import { type Behavior } from "./Behavior.ts";
@@ -102,7 +105,7 @@ export class AndroidControlledAudioOutput implements MediaDevice<
* @constructor
* @param controlledDevices$ - The list of available output devices coming from the hosting application, ordered by preference order (most preferred first).
* @param scope - The ObservableScope to create the Behaviors in.
* @param initialIntent - The initial call intent (e.g. "audio" or "video") that can be used to determine the default audio routing (e.g. default to earpiece for audio calls and speaker for video calls).
* @param initialIntent - The initial call intent (e.g. "m.audio" or "m.video") that can be used to determine the default audio routing (e.g. default to earpiece for audio calls and speaker for video calls).
* @param controls - The controls provided by the hosting application to control the audio routing and notify of user actions.
*/
public constructor(
@@ -316,7 +319,7 @@ export class AndroidControlledAudioOutput implements MediaDevice<
this.logger.debug(
`Android routing logic intent: ${this.initialIntent} finding best default...`,
);
if (this.initialIntent === "audio") {
if (this.initialIntent === RTC_CALL_INTENT_AUDIO) {
const systemProposed = available[0];
// If no headset is connected, android will route to the speaker by default,
// but for a voice call we want to route to the earpiece instead,
+68
View File
@@ -15,6 +15,7 @@ import {
type BackgroundEffectsOptions,
type BackgroundEffectsState,
} from "./BackgroundEffects";
import { type AddedBackground } from "../tracks/backgroundImages";
import { shippedBackgrounds } from "../tracks/backgroundEffectCatalog";
import { flushPromises, testScope, withTestScheduler } from "../utils/test";
@@ -69,6 +70,8 @@ describe("the pipeline's state", () => {
const effects = createBackgroundEffects(testScope(), {
supported: true,
effect$: behavior(effect, { n: "none", b: "blur" }),
setEffect: vi.fn(),
added$: new BehaviorSubject<AddedBackground[] | undefined>([]),
pipeline: fakePipeline().pipeline,
transformer,
});
@@ -94,6 +97,8 @@ describe("the pipeline's state", () => {
describe("background effects", () => {
let effect$: BehaviorSubject<string>;
let added$: BehaviorSubject<AddedBackground[] | undefined>;
let setEffect: (raw: string) => void;
let fake: ReturnType<typeof fakePipeline>;
function build(
@@ -102,6 +107,8 @@ describe("background effects", () => {
return createBackgroundEffects(testScope(), {
supported: true,
effect$,
setEffect,
added$,
pipeline: fake.pipeline,
transformer: { onFirstFrame: undefined },
...options,
@@ -116,6 +123,8 @@ describe("background effects", () => {
beforeEach(() => {
effect$ = new BehaviorSubject("none");
added$ = new BehaviorSubject<AddedBackground[] | undefined>(undefined);
setEffect = vi.fn((raw: string) => effect$.next(raw));
fake = fakePipeline();
});
@@ -169,4 +178,63 @@ describe("background effects", () => {
{ mode: "disabled" },
]);
});
it("clears a remembered effect when it cannot be honoured", () => {
effect$.next(`image:${shippedBackgrounds[0].id}`);
const effects = build({ supported: false });
expect(effect$.value).toBe("none");
expect(effects.state$.value.processor).toBeUndefined();
});
it("puts an added background on as its picture", async () => {
added$.next([{ id: "mine", url: "blob:mine" }]);
build();
await choose("added:mine");
expect(fake.switches).toEqual([
{ mode: "virtual-background", imagePath: "blob:mine" },
]);
});
it("switches between any two effects in place", async () => {
added$.next([{ id: "mine", url: "blob:mine" }]);
const shipped = shippedBackgrounds[0];
const options: Record<string, unknown> = {
none: { mode: "disabled" },
blur: { mode: "background-blur", blurRadius: 15 },
[`image:${shipped.id}`]: {
mode: "virtual-background",
imagePath: shipped.imagePath,
},
"added:mine": { mode: "virtual-background", imagePath: "blob:mine" },
};
// Each kind to each other kind, and back.
const order = [
"blur",
`image:${shipped.id}`,
"added:mine",
"blur",
"added:mine",
`image:${shipped.id}`,
"none",
"added:mine",
"none",
"blur",
];
const effects = build();
await choose(order[0]);
const pipeline = effects.state$.value.processor;
for (const raw of order.slice(1)) await choose(raw);
expect(effects.state$.value.processor).toBe(pipeline);
expect(fake.switches).toEqual(order.map((raw) => options[raw]));
});
it("forgets an added background the device no longer keeps", () => {
effect$.next("added:gone");
build();
// Not before the device has said what it keeps.
expect(effect$.value).toBe("added:gone");
added$.next([]);
expect(effect$.value).toBe("none");
});
});
+49 -10
View File
@@ -27,9 +27,11 @@ import { type Behavior } from "./Behavior";
import { type ObservableScope } from "./ObservableScope";
import {
type BackgroundEffect,
type EffectId,
imagePathFor,
parseEffect,
} from "../tracks/backgroundEffectCatalog";
import { type AddedBackground } from "../tracks/backgroundImages";
const blurRadius = 15;
@@ -45,6 +47,10 @@ export interface BackgroundEffectsOptions {
supported: boolean;
/** The effect chosen, as the setting stores it. */
effect$: Behavior<string>;
/** Stores a choice, to forget one that can't be honoured. */
setEffect: (id: EffectId) => void;
/** The backgrounds the device keeps, undefined until it has said. */
added$: Behavior<AddedBackground[] | undefined>;
/** The background processor pipeline to be switched between effects. */
pipeline: BackgroundProcessorWrapper;
/** Tells, once, that a frame carrying an effect has been drawn. */
@@ -59,7 +65,14 @@ export interface BackgroundEffects {
/** Switches the pipeline as the choice changes, for as long as the scope lasts. */
export function createBackgroundEffects(
scope: ObservableScope,
{ supported, effect$, pipeline, transformer }: BackgroundEffectsOptions,
{
supported,
effect$,
setEffect,
added$,
pipeline,
transformer,
}: BackgroundEffectsOptions,
): BackgroundEffects {
// Wrapped once, so the call and the preview see one processor throughout.
const processor = fromLivekitProcessor(pipeline);
@@ -69,6 +82,19 @@ export function createBackgroundEffects(
distinctUntilChanged(),
);
combineLatest([choice$, added$])
.pipe(scope.bind())
.subscribe(([effect, added]) => {
if (effect.kind === "none") return;
if (
!supported ||
(effect.kind === "added" &&
added !== undefined &&
!added.some(({ id }) => id === effect.id))
)
setEffect("none");
});
const drewAFrame$ = scope.behavior(
new Observable<boolean>((subscriber) => {
subscriber.next(false);
@@ -101,9 +127,16 @@ export function createBackgroundEffects(
const switchOptions$ = scope.behavior<
SwitchBackgroundProcessorOptions | undefined
>(
combineLatest([state$, choice$]).pipe(
map(([{ processor }, effect]) =>
processor === undefined ? undefined : switchOptionsFor(effect),
combineLatest([state$, choice$, added$]).pipe(
map(([{ processor }, effect, added]) =>
processor === undefined
? undefined
: switchOptionsFor(
effect,
effect.kind === "added"
? added?.find(({ id }) => id === effect.id)?.url
: undefined,
),
),
distinctUntilChanged(deepCompare),
),
@@ -121,18 +154,24 @@ export function createBackgroundEffects(
return { state$ };
}
/** `addedUrl` is the picture of the added background chosen, if still kept. */
function switchOptionsFor(
effect: BackgroundEffect,
addedUrl: string | undefined,
): SwitchBackgroundProcessorOptions {
const withImage = (
imagePath: string | undefined,
): SwitchBackgroundProcessorOptions =>
imagePath
? { mode: "virtual-background", imagePath }
: { mode: "disabled" };
switch (effect.kind) {
case "blur":
return { mode: "background-blur", blurRadius };
case "shipped": {
const imagePath = imagePathFor(effect.id);
return imagePath
? { mode: "virtual-background", imagePath }
: { mode: "disabled" };
}
case "shipped":
return withImage(imagePathFor(effect.id));
case "added":
return withImage(addedUrl);
default:
return { mode: "disabled" };
}
@@ -12,7 +12,11 @@ import {
MatrixEvent,
type Room,
} from "matrix-js-sdk";
import { type RTCCallIntent } from "matrix-js-sdk/lib/matrixrtc";
import {
RTC_CALL_INTENT_AUDIO,
RTC_CALL_INTENT_VIDEO,
type RTCCallIntent,
} from "matrix-js-sdk/lib/matrixrtc";
import { map, mergeMap, NEVER, type Observable, startWith } from "rxjs";
import { withTestScheduler } from "../../utils/test";
@@ -33,10 +37,12 @@ function mockRingEvent(
eventId: string,
lifetimeMs: number | undefined,
sender = local.userId,
legacyIntent?: string,
): CallNotificationWrapper {
return {
event_id: eventId,
...(lifetimeMs === undefined ? {} : { lifetime: lifetimeMs }),
...(legacyIntent === undefined ? {} : { "m.call.intent": legacyIntent }),
notification_type: "ring",
sender,
} as unknown as CallNotificationWrapper;
@@ -128,7 +134,25 @@ test("ring attempt times out after nobody joins", () => {
});
expectObservable(summarizeRingAttempts$(ringAttempts$)).toBe("-a 29ms A", {
a: { intent: "audio", recipient: alice.userId },
a: { intent: RTC_CALL_INTENT_AUDIO, recipient: alice.userId },
A: { outcome: "timeout" },
});
});
});
test("ring attempt maps the legacy intent on the notification to its stable value", () => {
withTestScheduler(({ scope, expectObservable, hot }) => {
const { ringAttempts$ } = createCallNotificationLifecycle$({
scope,
...defaultProps,
memberUserIds$: constant<string[]>([]),
sentCallNotification$: hot("-a", {
a: mockRingEvent("$notif1", 30, local.userId, "video"),
}),
});
expectObservable(summarizeRingAttempts$(ringAttempts$)).toBe("-a 29ms A", {
a: { intent: RTC_CALL_INTENT_VIDEO, recipient: alice.userId },
A: { outcome: "timeout" },
});
});
@@ -148,7 +172,7 @@ test("ring attempt is accepted once recipient joins", () => {
});
expectObservable(summarizeRingAttempts$(ringAttempts$)).toBe("-aA", {
a: { intent: "audio", recipient: alice.userId },
a: { intent: RTC_CALL_INTENT_AUDIO, recipient: alice.userId },
A: { outcome: "accept" },
});
});
@@ -166,7 +190,7 @@ test("ring attempt is immediately accepted if recipient is already joined", () =
});
expectObservable(summarizeRingAttempts$(ringAttempts$)).toBe("-(aA)", {
a: { intent: "audio", recipient: alice.userId },
a: { intent: RTC_CALL_INTENT_AUDIO, recipient: alice.userId },
A: { outcome: "accept" },
});
});
@@ -201,7 +225,7 @@ test("ring attempt can be declined", () => {
});
expectObservable(summarizeRingAttempts$(ringAttempts$)).toBe("-aA", {
a: { intent: "audio", recipient: alice.userId },
a: { intent: RTC_CALL_INTENT_AUDIO, recipient: alice.userId },
A: { outcome: "decline" },
});
});
@@ -236,7 +260,7 @@ test("ring attempt times out if recipient declines too late", () => {
});
expectObservable(summarizeRingAttempts$(ringAttempts$)).toBe("-a 29ms A", {
a: { intent: "audio", recipient: alice.userId },
a: { intent: RTC_CALL_INTENT_AUDIO, recipient: alice.userId },
A: { outcome: "timeout" },
});
});
@@ -271,7 +295,7 @@ test("decline event relating to wrong event is ignored (times out)", () => {
});
expectObservable(summarizeRingAttempts$(ringAttempts$)).toBe("-a 29ms A", {
a: { intent: "audio", recipient: alice.userId },
a: { intent: RTC_CALL_INTENT_AUDIO, recipient: alice.userId },
A: { outcome: "timeout" },
});
});
@@ -306,7 +330,7 @@ test("decline event from wrong sender is ignored (times out)", () => {
});
expectObservable(summarizeRingAttempts$(ringAttempts$)).toBe("-a 29ms A", {
a: { intent: "audio", recipient: alice.userId },
a: { intent: RTC_CALL_INTENT_AUDIO, recipient: alice.userId },
A: { outcome: "timeout" },
});
});
@@ -9,7 +9,9 @@ import {
type IRTCNotificationContent,
type MatrixRTCSession,
MatrixRTCSessionEvent,
RTC_CALL_INTENT_AUDIO,
type RTCCallIntent,
toStableCallIntent,
} from "matrix-js-sdk/lib/matrixrtc";
import {
filter,
@@ -155,7 +157,10 @@ export function createCallNotificationLifecycle$({
);
return of({
intent: notificationEvent["m.call.intent"] ?? "audio",
// The notification still carries the legacy intent values
intent:
toStableCallIntent(notificationEvent["m.call.intent"]) ??
RTC_CALL_INTENT_AUDIO,
recipient,
outcome$: race(timeout$, accept$, decline$).pipe(
take(1),
@@ -21,6 +21,7 @@ import {
type CallMembership,
type IRTCNotificationContent,
MatrixRTCSessionEvent,
RTC_CALL_INTENT_AUDIO,
} from "matrix-js-sdk/lib/matrixrtc";
import { deepCompare } from "matrix-js-sdk/lib/utils";
@@ -1557,7 +1558,7 @@ describe.each(modes)("CallViewModel (%s mode)", (mode) => {
b: expect.objectContaining({
type: "ringing",
userId: alice.userId,
intent: "audio",
intent: RTC_CALL_INTENT_AUDIO,
}),
});
// Layout should show placeholder media for the participant we're
@@ -1603,7 +1604,7 @@ describe.each(modes)("CallViewModel (%s mode)", (mode) => {
b: expect.objectContaining({
type: "ringing",
userId: alice.userId,
intent: "audio",
intent: RTC_CALL_INTENT_AUDIO,
}),
});
// Layout should show placeholder media for the participant we're
+9 -5
View File
@@ -21,6 +21,10 @@ import {
EARPIECE_CONFIG_ID,
IOSControlledAudioOutput,
} from "./IOSControlledAudioOutput";
import {
RTC_CALL_INTENT_AUDIO,
RTC_CALL_INTENT_VIDEO,
} from "matrix-js-sdk/lib/matrixrtc";
// `vi.mock` calls are hoisted above all imports, so the static imports below
// already see these mocks. Force the iOS platform so that the virtual earpiece
@@ -81,7 +85,7 @@ describe("Default selection", () => {
const output = new IOSControlledAudioOutput(
constant(false),
testScope,
"audio",
RTC_CALL_INTENT_AUDIO,
);
const selected = latestSelection(output);
@@ -100,7 +104,7 @@ describe("Default selection", () => {
const output = new IOSControlledAudioOutput(
constant(false),
testScope,
"video",
RTC_CALL_INTENT_VIDEO,
);
const selected = latestSelection(output);
@@ -113,7 +117,7 @@ describe("Default selection", () => {
const output = new IOSControlledAudioOutput(
constant(false),
testScope,
"audio",
RTC_CALL_INTENT_AUDIO,
);
const selected = latestSelection(output);
@@ -130,7 +134,7 @@ describe("Explicit selection", () => {
const output = new IOSControlledAudioOutput(
constant(false),
testScope,
"audio",
RTC_CALL_INTENT_AUDIO,
);
const selected = latestSelection(output);
@@ -150,7 +154,7 @@ describe("Explicit selection", () => {
const output = new IOSControlledAudioOutput(
constant(false),
testScope,
"audio",
RTC_CALL_INTENT_AUDIO,
);
const selected = latestSelection(output);
+6 -3
View File
@@ -8,7 +8,10 @@ Please see LICENSE in the repository root for full details.
import { logger as rootLogger } from "matrix-js-sdk/lib/logger";
import { combineLatest, merge, startWith, Subject, tap } from "rxjs";
import type { RTCCallIntent } from "matrix-js-sdk/lib/matrixrtc";
import {
RTC_CALL_INTENT_AUDIO,
type RTCCallIntent,
} from "matrix-js-sdk/lib/matrixrtc";
import {
availableOutputDevices$ as controlledAvailableOutputDevices$,
outputDevice$ as controlledOutputSelection$,
@@ -111,7 +114,7 @@ export class IOSControlledAudioOutput implements MediaDevice<
* Chooses the default output device when no explicit selection (from the user
* or the hosting application) has been made yet.
*
* For voice calls (`initialIntent === "audio"`) we want to start on the
* For voice calls (`initialIntent === "m.audio"`) we want to start on the
* earpiece rather than the speaker, like a regular phone call. We only
* override when the device that would otherwise be the default is the
* speaker: if the host already routed to a headset (e.g. Bluetooth) — which
@@ -122,7 +125,7 @@ export class IOSControlledAudioOutput implements MediaDevice<
available: Map<string, AudioOutputDeviceLabel>,
): string | undefined {
const firstId = available.keys().next().value;
if (this.initialIntent === "audio") {
if (this.initialIntent === RTC_CALL_INTENT_AUDIO) {
const firstLabel =
firstId !== undefined ? available.get(firstId) : undefined;
if (firstLabel?.type === "speaker") {
+6 -2
View File
@@ -35,6 +35,10 @@ import { AudioOutput, MediaDevices } from "./MediaDevices";
import { AndroidControlledAudioOutput } from "./AndroidControlledAudioOutput";
import { IOSControlledAudioOutput } from "./IOSControlledAudioOutput";
import { ObservableScope } from "./ObservableScope";
import {
RTC_CALL_INTENT_AUDIO,
RTC_CALL_INTENT_VIDEO,
} from "matrix-js-sdk/lib/matrixrtc";
// Which audio output implementation is used is decided by what the app hosting
// Element Call told it, rather than being discovered from the environment.
@@ -52,7 +56,7 @@ describe("MediaDevices audio output", () => {
const devices = new MediaDevices(new ObservableScope(), {
controlledAudioDevices: true,
callIntent: "audio",
callIntent: RTC_CALL_INTENT_AUDIO,
});
expect(devices.audioOutput).toBeInstanceOf(AndroidControlledAudioOutput);
@@ -63,7 +67,7 @@ describe("MediaDevices audio output", () => {
const devices = new MediaDevices(new ObservableScope(), {
controlledAudioDevices: true,
callIntent: "video",
callIntent: RTC_CALL_INTENT_VIDEO,
});
expect(devices.audioOutput).toBeInstanceOf(IOSControlledAudioOutput);
+15 -11
View File
@@ -6,7 +6,11 @@ Please see LICENSE in the repository root for full details.
*/
import { test, expect } from "vitest";
import { type RTCCallIntent } from "matrix-js-sdk/lib/matrixrtc";
import {
RTC_CALL_INTENT_AUDIO,
RTC_CALL_INTENT_VIDEO,
type RTCCallIntent,
} from "matrix-js-sdk/lib/matrixrtc";
import { calculateInitialMuteState } from "./initialMuteState";
@@ -14,10 +18,10 @@ test.each<{
callIntent: RTCCallIntent;
allowJoinUnmutedViaIntent: boolean;
}>([
{ callIntent: "audio", allowJoinUnmutedViaIntent: false },
{ callIntent: "audio", allowJoinUnmutedViaIntent: true },
{ callIntent: "video", allowJoinUnmutedViaIntent: false },
{ callIntent: "video", allowJoinUnmutedViaIntent: true },
{ callIntent: RTC_CALL_INTENT_AUDIO, allowJoinUnmutedViaIntent: false },
{ callIntent: RTC_CALL_INTENT_AUDIO, allowJoinUnmutedViaIntent: true },
{ callIntent: RTC_CALL_INTENT_VIDEO, allowJoinUnmutedViaIntent: false },
{ callIntent: RTC_CALL_INTENT_VIDEO, allowJoinUnmutedViaIntent: true },
{ callIntent: "unknown", allowJoinUnmutedViaIntent: false },
{ callIntent: "unknown", allowJoinUnmutedViaIntent: true },
])(
@@ -29,15 +33,15 @@ test.each<{
allowJoinUnmutedViaIntent,
);
expect(audioEnabled).toBe(true);
expect(videoEnabled).toBe(callIntent !== "audio");
expect(videoEnabled).toBe(callIntent !== RTC_CALL_INTENT_AUDIO);
},
);
test.each<{
callIntent: RTCCallIntent;
}>([
{ callIntent: "audio" },
{ callIntent: "video" },
{ callIntent: RTC_CALL_INTENT_AUDIO },
{ callIntent: RTC_CALL_INTENT_VIDEO },
{ callIntent: "unknown" },
])(
"Should always mute on start if skipping lobby and the host does not vouch for the intent (callIntent: $callIntent)",
@@ -55,8 +59,8 @@ test.each<{
test.each<{
callIntent: RTCCallIntent;
}>([
{ callIntent: "audio" },
{ callIntent: "video" },
{ callIntent: RTC_CALL_INTENT_AUDIO },
{ callIntent: RTC_CALL_INTENT_VIDEO },
{ callIntent: "unknown" },
])(
"Can start unmuted if skipping lobby and the host vouches for the intent (callIntent: $callIntent)",
@@ -67,6 +71,6 @@ test.each<{
true,
);
expect(audioEnabled).toBe(true);
expect(videoEnabled).toBe(callIntent !== "audio");
expect(videoEnabled).toBe(callIntent !== RTC_CALL_INTENT_AUDIO);
},
);
+6 -3
View File
@@ -6,7 +6,10 @@ Please see LICENSE in the repository root for full details.
*/
import { logger } from "matrix-js-sdk/lib/logger";
import { type RTCCallIntent } from "matrix-js-sdk/lib/matrixrtc";
import {
RTC_CALL_INTENT_AUDIO,
type RTCCallIntent,
} from "matrix-js-sdk/lib/matrixrtc";
/**
* Calculates the initial mute state for media devices based on configuration.
@@ -40,10 +43,10 @@ export function calculateInitialMuteState(
// A host that vouches for the intent is a trusted environment, so it allows
// unmuted by default. Same for when showing a lobby, as users can adjust
// their settings there. Additionally, if the call intent is "audio", we
// their settings there. Additionally, if the call intent is "m.audio", we
// disable video by default.
return {
audioEnabled: true,
videoEnabled: callIntent != "audio",
videoEnabled: callIntent != RTC_CALL_INTENT_AUDIO,
};
}
+5 -2
View File
@@ -8,7 +8,10 @@ Please see LICENSE in the repository root for full details.
import { test, expect } from "vitest";
import { act, render, screen } from "@testing-library/react";
import { axe } from "vitest-axe";
import { type MatrixRTCSession } from "matrix-js-sdk/lib/matrixrtc";
import {
RTC_CALL_INTENT_AUDIO,
type MatrixRTCSession,
} from "matrix-js-sdk/lib/matrixrtc";
import { BehaviorSubject } from "rxjs";
import { GridTile } from "./GridTile";
@@ -122,7 +125,7 @@ test("GridTile displays ringing media", async () => {
const vm = createRingingMedia({
pickupState$,
id: "test",
intent: "audio",
intent: RTC_CALL_INTENT_AUDIO,
userId: "@alice:example.org",
displayName$: constant("Alice"),
mxcAvatarUrl$: constant(undefined),
+2 -1
View File
@@ -12,6 +12,7 @@ import {
EndCallIcon,
} from "@vector-im/compound-design-tokens/assets/web/icons";
import { useTranslation } from "react-i18next";
import { RTC_CALL_INTENT_VIDEO } from "matrix-js-sdk/lib/matrixrtc";
import { type RingingMediaViewModel } from "../state/media/RingingMediaViewModel";
import { useBehavior } from "../useBehavior";
@@ -25,7 +26,7 @@ export const RingingStatus: FC<Props> = ({ vm }) => {
const pickupState = useBehavior(vm.pickupState$);
const Icon =
pickupState === "ringing"
? vm.intent === "video"
? vm.intent === RTC_CALL_INTENT_VIDEO
? VideoCallSolidIcon
: VoiceCallSolidIcon
: EndCallIcon;
+2 -1
View File
@@ -11,6 +11,7 @@ import { axe } from "vitest-axe";
import userEvent from "@testing-library/user-event";
import { TooltipProvider } from "@vector-im/compound-web";
import { BehaviorSubject } from "rxjs";
import { RTC_CALL_INTENT_AUDIO } from "matrix-js-sdk/lib/matrixrtc";
import { SpotlightTile } from "./SpotlightTile";
import {
@@ -176,7 +177,7 @@ test("SpotlightTile displays ringing media", async () => {
const vm = createRingingMedia({
pickupState$,
id: "test",
intent: "audio",
intent: RTC_CALL_INTENT_AUDIO,
userId: "@alice:example.org",
displayName$: constant("Alice"),
mxcAvatarUrl$: constant(undefined),
+12 -1
View File
@@ -19,8 +19,10 @@ import {
type BackgroundEffectsState,
} from "../state/BackgroundEffects";
import { backgroundEffect } from "../settings/settings";
import { type AddedBackground } from "./backgroundImages";
import { flushPromises, testScope } from "../utils/test";
const sdkSupportMock = vi.hoisted(() => vi.fn(() => true));
const pipelines = vi.hoisted(() => ({
built: 0,
destroyed: 0,
@@ -41,10 +43,18 @@ vi.mock("@livekit/track-processors", () => ({
return Promise.resolve();
});
}),
supportsBackgroundProcessors: (): boolean => true,
supportsBackgroundProcessors: (): boolean => sdkSupportMock(),
}));
const platformMock = vi.hoisted(() => ({ platform: "desktop" }));
vi.mock("../Platform", () => platformMock);
vi.mock("./backgroundImages", async () => {
const { BehaviorSubject } = await import("rxjs");
return {
addedBackgrounds: {
added$: new BehaviorSubject<AddedBackground[] | undefined>(undefined),
},
};
});
vi.mock("./BackgroundEffectTransformer", () => ({
BackgroundEffectTransformer: vi.fn(),
}));
@@ -77,6 +87,7 @@ describe("BackgroundEffectsProvider", () => {
pipelines.destroyed = 0;
pipelines.switches = [];
platformMock.platform = "desktop";
sdkSupportMock.mockReturnValue(true);
backgroundEffect.setValue("none");
});
afterEach(() => backgroundEffect.setValue("none"));
+3
View File
@@ -13,6 +13,7 @@ import { createContext, type FC, type JSX, use } from "react";
import { backgroundEffect as backgroundEffectSetting } from "../settings/settings";
import { BackgroundEffectTransformer } from "./BackgroundEffectTransformer";
import { addedBackgrounds } from "./backgroundImages";
import { type Behavior } from "../state/Behavior";
import { type ObservableScope } from "../state/ObservableScope";
import {
@@ -53,6 +54,8 @@ export function createAppBackgroundEffects(
return createBackgroundEffects(scope, {
supported: supportsBackgroundProcessors(),
effect$: backgroundEffectSetting.value$,
setEffect: backgroundEffectSetting.setValue,
added$: addedBackgrounds.added$,
pipeline: new BackgroundProcessorWrapper(transformer, "background-effect"),
transformer,
});
@@ -15,6 +15,7 @@ import {
serializeEffect,
shippedBackgrounds,
} from "./backgroundEffectCatalog";
import { maxAddedBackgrounds } from "./backgroundImages";
describe("the chosen background effect", () => {
test("round-trips through its stored form", () => {
@@ -22,6 +23,7 @@ describe("the chosen background effect", () => {
{ kind: "none" },
{ kind: "blur" },
...shippedBackgrounds.map(({ id }) => ({ kind: "shipped" as const, id })),
{ kind: "added", id: "d3b07384" },
];
for (const effect of effects)
expect(parseEffect(serializeEffect(effect))).toEqual(effect);
@@ -30,6 +32,7 @@ describe("the chosen background effect", () => {
test("falls back to no effect when the stored form names nothing we ship", () => {
expect(parseEffect("image:gone")).toEqual({ kind: "none" });
expect(parseEffect("")).toEqual({ kind: "none" });
expect(parseEffect("added:")).toEqual({ kind: "none" });
});
test("reads a stored value that is not a string as no effect", () => {
@@ -37,6 +40,14 @@ describe("the chosen background effect", () => {
expect(parseEffect(raw)).toEqual({ kind: "none" });
});
test("fills the grid at the limit and one short of it", () => {
// No effect, blur and the shipped ones, then the added ones.
const fixed = 2 + shippedBackgrounds.length;
expect((fixed + maxAddedBackgrounds) % 3).toBe(0);
// One short, the add tile takes the last place.
expect((fixed + maxAddedBackgrounds - 1 + 1) % 3).toBe(0);
});
test("gives every shipped background an image to draw", () => {
for (const { id } of shippedBackgrounds)
expect(imagePathFor(id)).toBeTruthy();
+12 -4
View File
@@ -22,16 +22,18 @@ export const shippedBackgrounds: ShippedBackground[] = [
export type BackgroundEffect =
| { kind: "none" }
| { kind: "blur" }
| { kind: "shipped"; id: string };
| { kind: "shipped"; id: string }
| { kind: "added"; id: string };
/** An effect as the setting stores it, and as the menu names its option. */
export type EffectId = "none" | "blur" | `image:${string}`;
export type EffectId = "none" | "blur" | `image:${string}` | `added:${string}`;
export const noEffect: BackgroundEffect = { kind: "none" };
/**
* Reads a stored effect; one we no longer ship reads as no effect. Stored by
* hand or by another build, the value need not even be a string.
* Reads a stored effect; one we no longer ship reads as no effect. Whether an
* added one is still kept only the device knows. Stored by hand or by another
* build, the value need not even be a string.
*/
export function parseEffect(raw: unknown): BackgroundEffect {
if (typeof raw !== "string") return noEffect;
@@ -41,6 +43,10 @@ export function parseEffect(raw: unknown): BackgroundEffect {
if (shippedBackgrounds.some((b) => b.id === id))
return { kind: "shipped", id };
}
if (raw.startsWith("added:")) {
const id = raw.slice("added:".length);
if (id) return { kind: "added", id };
}
return noEffect;
}
@@ -50,6 +56,8 @@ export function serializeEffect(effect: BackgroundEffect): EffectId {
return "blur";
case "shipped":
return `image:${effect.id}`;
case "added":
return `added:${effect.id}`;
default:
return "none";
}
+172
View File
@@ -0,0 +1,172 @@
/*
Copyright 2026 Element Creations Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { beforeEach, describe, expect, it, vi } from "vitest";
import { IDBFactory } from "fake-indexeddb";
import {
AddedBackgrounds,
type BackgroundImageStorage,
IndexedDBImageStorage,
type KeptImage,
maxAddedBackgrounds,
prepareImage,
UnusableImage,
} from "./backgroundImages";
import { flushPromises } from "../utils/test";
describe("AddedBackgrounds", () => {
beforeEach(() => {
let n = 0;
vi.spyOn(URL, "createObjectURL").mockImplementation(() => `blob:${++n}`);
});
it("offers what the device keeps, oldest first", async () => {
const image = new Blob(["x"], { type: "image/png" });
const store = new AddedBackgrounds(
memoryStorage([
{ id: "later", image, addedAt: 2 },
{ id: "earlier", image, addedAt: 1 },
]),
asGiven,
);
expect(store.added$.value).toBeUndefined();
await flushPromises();
expect(store.added$.value?.map(({ id }) => id)).toEqual([
"earlier",
"later",
]);
});
it("keeps an added image and offers it after the others", async () => {
const storage = memoryStorage();
const store = new AddedBackgrounds(storage, asGiven);
await flushPromises();
const id = await store.add(new Blob(["x"], { type: "image/png" }));
expect(store.added$.value?.map((a) => a.id)).toEqual([id]);
expect((await storage.list()).map((k) => k.id)).toEqual([id]);
});
it("keeps the prepared image, not the file", async () => {
const storage = memoryStorage();
const prepared = new Blob(["y"], { type: "image/webp" });
const store = new AddedBackgrounds(storage, async () =>
Promise.resolve(prepared),
);
await store.add(new Blob(["x"], { type: "image/png" }));
expect((await storage.list())[0].image).toBe(prepared);
});
it("keeps nothing of a file it refuses", async () => {
const storage = memoryStorage();
const store = new AddedBackgrounds(storage, async () =>
Promise.reject(new UnusableImage("animated")),
);
await flushPromises();
await expect(store.add(new Blob(["x"]))).rejects.toThrow(UnusableImage);
expect(await storage.list()).toEqual([]);
expect(store.added$.value).toEqual([]);
});
it("keeps no more than the grid holds", async () => {
const storage = memoryStorage();
const store = new AddedBackgrounds(storage, asGiven);
for (let i = 0; i < maxAddedBackgrounds; i++)
await store.add(new Blob([`${i}`]));
await expect(store.add(new Blob(["more"]))).rejects.toThrow(RangeError);
expect(await storage.list()).toHaveLength(maxAddedBackgrounds);
});
it("forgets a removed image, and lets its picture go", async () => {
const revoke = vi
.spyOn(URL, "revokeObjectURL")
.mockImplementation(() => {});
const storage = memoryStorage();
const store = new AddedBackgrounds(storage, asGiven);
await flushPromises();
const first = await store.add(new Blob(["a"]));
const second = await store.add(new Blob(["b"]));
const url = store.added$.value![0].url;
await store.remove(first);
expect(store.added$.value?.map((a) => a.id)).toEqual([second]);
expect((await storage.list()).map((k) => k.id)).toEqual([second]);
expect(revoke).toHaveBeenCalledWith(url);
});
it("offers nothing where the browser keeps nothing", async () => {
const store = new AddedBackgrounds(null);
await flushPromises();
expect(store.added$.value).toEqual([]);
await expect(store.add(new Blob(["x"]))).rejects.toThrow();
});
it("offers none, rather than waiting, where the device can't be read", async () => {
const unreadable: BackgroundImageStorage = {
...memoryStorage(),
list: async () => Promise.reject(new Error("storage cleared")),
};
const store = new AddedBackgrounds(unreadable);
await flushPromises();
expect(store.added$.value).toEqual([]);
});
});
describe("prepareImage", () => {
// Refused before anything is decoded, which only a real browser could do.
it("refuses a file that isn't an image", async () => {
await expect(
prepareImage(new Blob(["x"], { type: "text/plain" })),
).rejects.toEqual(new UnusableImage("not-an-image"));
});
it("refuses an animated type where the browser can't count frames", async () => {
expect(globalThis.ImageDecoder).toBeUndefined();
await expect(
prepareImage(new Blob(["GIF89a"], { type: "image/gif" })),
).rejects.toEqual(new UnusableImage("animated"));
});
});
describe("IndexedDBImageStorage", () => {
it("keeps images across sessions until they are deleted", async () => {
const indexedDB = new IDBFactory();
const image = new Blob(["x"], { type: "image/webp" });
const first = new IndexedDBImageStorage(indexedDB);
await first.put({ id: "a", image, addedAt: 1 });
await first.put({ id: "b", image, addedAt: 2 });
// A session of its own, as after a reload.
const second = new IndexedDBImageStorage(indexedDB);
expect(
(await second.list()).map(({ id, addedAt }) => [id, addedAt]),
).toEqual([
["a", 1],
["b", 2],
]);
await second.delete("a");
expect((await first.list()).map(({ id }) => id)).toEqual(["b"]);
});
});
// Decoding needs a real browser; the end-to-end checks cover it.
const asGiven = async (file: Blob): Promise<Blob> => Promise.resolve(file);
function memoryStorage(kept: KeptImage[] = []): BackgroundImageStorage {
return {
list: async (): Promise<KeptImage[]> => Promise.resolve([...kept]),
put: async (image): Promise<void> => {
kept.push(image);
return Promise.resolve();
},
delete: async (id): Promise<void> => {
const index = kept.findIndex((k) => k.id === id);
if (index !== -1) kept.splice(index, 1);
return Promise.resolve();
},
};
}
+220
View File
@@ -0,0 +1,220 @@
/*
Copyright 2026 Element Creations Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { BehaviorSubject } from "rxjs";
import { logger } from "matrix-js-sdk/lib/logger";
import { type Behavior } from "../state/Behavior";
export const maxAddedBackgrounds = 5;
/** The longest edge kept; the pipeline scales every image to the camera. */
export const maxStoredEdge = 1920;
/** Why a file can't be used as a background. */
export type UnusableReason = "not-an-image" | "animated" | "undecodable";
export class UnusableImage extends Error {
public constructor(public readonly reason: UnusableReason) {
super(reason);
this.name = "UnusableImage";
}
}
/** A background the user added, as the menu and the pipeline use it. */
export interface AddedBackground {
id: string;
url: string;
}
/** A background the user added, as it is kept on the device. */
export interface KeptImage {
id: string;
image: Blob;
addedAt: number;
}
/** Where added backgrounds are kept. */
export interface BackgroundImageStorage {
list(): Promise<KeptImage[]>;
put(image: KeptImage): Promise<void>;
delete(id: string): Promise<void>;
}
/** The backgrounds this device keeps, which never leave it. */
export class AddedBackgrounds {
private readonly subject$ = new BehaviorSubject<
AddedBackground[] | undefined
>(undefined);
/** Undefined until what the device keeps has been read. */
public readonly added$: Behavior<AddedBackground[] | undefined> =
this.subject$;
public constructor(
private readonly storage: BackgroundImageStorage | null,
private readonly prepare: (file: Blob) => Promise<Blob> = prepareImage,
) {
this.read().catch((e) => {
logger.warn("Could not read added backgrounds", e);
this.subject$.next([]);
});
}
private async read(): Promise<void> {
const kept = (await this.storage?.list()) ?? [];
kept.sort((a, b) => a.addedAt - b.addedAt);
this.subject$.next(
kept.map(({ id, image }) => ({ id, url: URL.createObjectURL(image) })),
);
}
/**
* Keeps a file as a background, and answers with its id. Throws
* {@link UnusableImage} for a file that can't be used, and a RangeError
* once maxAddedBackgrounds are kept.
*/
public async add(file: Blob): Promise<string> {
if (this.storage === null)
throw new Error("This browser keeps no backgrounds");
const image = await this.prepare(file);
if ((await this.storage.list()).length >= maxAddedBackgrounds)
throw new RangeError(
`${maxAddedBackgrounds} backgrounds are kept already`,
);
const kept: KeptImage = {
id: crypto.randomUUID(),
image,
addedAt: Date.now(),
};
await this.storage.put(kept);
this.subject$.next([
...(this.subject$.value ?? []),
{ id: kept.id, url: URL.createObjectURL(image) },
]);
return kept.id;
}
public async remove(id: string): Promise<void> {
await this.storage?.delete(id);
const current = this.subject$.value ?? [];
const gone = current.find((a) => a.id === id);
if (gone) URL.revokeObjectURL(gone.url);
this.subject$.next(current.filter((a) => a.id !== id));
}
}
/**
* Refuses what can't be used, reduces what is larger than is kept, and lays
* every image on an opaque ground: a background has to cover what is behind
* it, so a transparent pixel would be a hole in it.
*/
export async function prepareImage(file: Blob): Promise<Blob> {
if (!file.type.startsWith("image/")) throw new UnusableImage("not-an-image");
if (await isAnimated(file)) throw new UnusableImage("animated");
let bitmap: ImageBitmap;
try {
bitmap = await createImageBitmap(file);
} catch (e) {
logger.debug("Could not decode the chosen file", e);
throw new UnusableImage("undecodable");
}
try {
const scale = Math.min(
1,
maxStoredEdge / Math.max(bitmap.width, bitmap.height),
);
const canvas = new OffscreenCanvas(
Math.round(bitmap.width * scale),
Math.round(bitmap.height * scale),
);
const context = canvas.getContext("2d");
if (!context) throw new UnusableImage("undecodable");
context.fillStyle = "black";
context.fillRect(0, 0, canvas.width, canvas.height);
context.drawImage(bitmap, 0, 0, canvas.width, canvas.height);
return await canvas.convertToBlob({ type: "image/webp", quality: 0.9 });
} finally {
bitmap.close();
}
}
// Read from the file, not its type: a still and an animated WebP share one.
async function isAnimated(file: Blob): Promise<boolean> {
if (typeof ImageDecoder === "undefined") return /gif|apng/.test(file.type);
try {
const decoder = new ImageDecoder({
data: await file.arrayBuffer(),
type: file.type,
});
// Some browsers have no selected track until the tracks are ready.
await decoder.tracks.ready;
await decoder.completed;
return (decoder.tracks.selectedTrack?.frameCount ?? 1) > 1;
} catch (e) {
logger.debug("Could not read the frame count, judging by type", e);
return /gif|apng/.test(file.type);
}
}
const DB_NAME = "element-call-background-images";
const STORE = "backgrounds";
export class IndexedDBImageStorage implements BackgroundImageStorage {
private db?: Promise<IDBDatabase>;
public constructor(private readonly indexedDB: IDBFactory) {}
private async open(): Promise<IDBDatabase> {
return (this.db ??= new Promise((resolve, reject) => {
const request = this.indexedDB.open(DB_NAME, 1);
request.onupgradeneeded = (): void => {
request.result.createObjectStore(STORE, { keyPath: "id" });
};
request.onsuccess = (): void => resolve(request.result);
request.onerror = (): void => reject(request.error);
}));
}
private async run<T>(
mode: IDBTransactionMode,
body: (store: IDBObjectStore) => IDBRequest<T>,
): Promise<T> {
const db = await this.open();
return new Promise((resolve, reject) => {
const request = body(db.transaction(STORE, mode).objectStore(STORE));
request.onsuccess = (): void => resolve(request.result);
request.onerror = (): void => reject(request.error);
});
}
public async list(): Promise<KeptImage[]> {
return this.run<KeptImage[]>("readonly", (s) => s.getAll());
}
public async put(image: KeptImage): Promise<void> {
await this.run("readwrite", (s) => s.put(image));
}
public async delete(id: string): Promise<void> {
await this.run("readwrite", (s) => s.delete(id));
}
}
function openStorage(): BackgroundImageStorage | null {
// Merely reading indexedDB throws where it is disabled.
try {
return globalThis.indexedDB
? new IndexedDBImageStorage(globalThis.indexedDB)
: null;
} catch (e) {
logger.warn("Could not reach IndexedDB", e);
return null;
}
}
export const addedBackgrounds = new AddedBackgrounds(openStorage());