Compare commits

...
Author SHA1 Message Date
fkwpandClaude Opus 5.5 2704821f7d Offer background effects on any platform whose browser can run them
- The desktop rule from #4059 goes: now that the menu says where effects
  run slowly, using them there is the user's choice. A phone on the fast
  route held 99% of its frame rate and was refused; a desktop on the slow
  one held 73% and was allowed.
- Asks @livekit/track-processors directly; the wrapper module goes.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-26 18:19:45 +01:00
fkwpandClaude Opus 5.5 f950ff75da Show the wait for the first background effect on its tile
- The first effect chosen in a session shows a spinner on its own tile,
  in place of the tick, until a frame carrying it is drawn. Later changes
  show nothing, having nothing to prepare.
- Ended by the frame, not the promises, which resolve while the segmenter
  is still being built. The transformer reports the first frame that went
  through the segmenter; one passed through untouched doesn't count.
- The pipeline publishes the wait with its state, and both footer view
  models take it as an input.
- The lobby tests' provider mock gains the observable hook, built once: a
  fresh observable per render rebuilt the footer on every render.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-26 18:19:45 +01:00
fkwpandClaude Opus 5.5 18debf6489 Say where background effects run slowly
- Where the browser has no MediaStreamTrackProcessor, effects draw each
  frame through a canvas: slower, and the first build stalls the page.
  They stay offered and choosable there, with a notice under the grid
  that says so and describes the group.
- Decided in the view model, beside whether effects can run at all, so
  the lobby and the call give the same answer.
- The notice stays within the menu's width however long the sentence,
  which the story checks.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-26 18:19:45 +01:00
fkwpandClaude Opus 5.5 e3229390ef Offer background effects as a grid in the camera menu
- The camera menu's blur toggle becomes a Background effects section
  under the cameras: no effect, blur and the two shipped backgrounds as
  one radio choice of tiles, three across. It scrolls with the device
  list, so it stays reachable in a short call, and choosing keeps the
  menu open. The lobby gets it too, as both render the same footer.
- The footer's view model gives the options, the notice and the actions;
  the footer only names them.
- Each tile is Compound's MenuItem as a menuitemradio, styled as a tile.
  Compound makes it an item of the dropdown, so the arrow keys walk the
  tiles with the device rows, and takes its tap in the phone drawer. A
  tile that can't be chosen has no select handler, so the keyboard passes
  over it.
- Where the browser can't run effects the tiles are shown, disabled and
  explained by a notice the group is described by. No effect stays
  choosable and shows as in force. The notice is in the scrolling list,
  so a short call doesn't push the menu out of it.
- Section headings are raised, or the positioned tiles scroll over the
  one they sit under; raised within the list only, so the menu's frame
  stays over them. Tiles clip with overflow: clip; with hidden they are
  scroll containers, which focus scrolls into view without their border.
- End to end on Chromium and Firefox: a shipped background chosen in the
  lobby shows in the preview, and None after Blur draws nothing, counted
  in WebGL draws (Blur: about 1400 a second on Chromium, 2300 on
  Firefox).
- The Quick Audio Menu's test of the camera menu keeps its name and now
  finds blur among the effects rather than as a toggle.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-26 18:19:44 +01:00
fkwpandClaude Opus 5.5 7a1eaf6ca1 Choose a background effect, not only blur on or off
- A catalogue of what can be chosen: no effect, blur, and two shipped
  backgrounds, the app's own gradients flattened onto its canvas colour.
  Their ids are what a saved choice stores, so they are named for the
  pictures.
- One setting for the choice, typed as the ids an effect can have and
  defaulting to blur for anyone who had it on. A stored value we no
  longer ship, or one that isn't even a string, reads as no effect: it
  comes from storage another build or a hand may have written.
- The pipeline follows it, switching in place to a picture as readily as
  to blur, and the camera menu's blur toggle reads and writes it.
- So does Settings' blur checkbox: a picture reads as blur off, and
  checking it replaces the picture with blur. The old on/off setting is
  only read, to carry blur over. First tests for the Settings modal,
  which fail with the checkbox on the old setting.
- The transformer's own init loads a picture chosen before it was built,
  as the base init does and it had not: the first effect always is, and
  the picture was drawn black.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-26 18:19:44 +01:00
fkwpandClaude Opus 5.5 461100ea30 Update compound-design-tokens to 11.1.0
- Brings design's blur icon, which the background effects grid uses.
- Of the icons the release redrew, guest and zoom-in, Element Call uses
  neither, and none of the icons it removed are used here.

Rebased onto main, which already locks 11.3.1: this now only raises the
range to ^11.1.0, so the blur icon is required rather than incidental.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-26 18:19:44 +01:00
fkwpandClaude Opus 5.5 8da79f1c72 Create the background effects with the app's view model
- AppViewModel creates them, next to its media devices, as review asked;
  BackgroundEffectsProvider takes them as a prop rather than building
  them in an effect.
- The component build, which has no AppViewModel of its own, now makes one
  in the effect that made its media devices, so both builds get their
  state the same way.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-26 18:19:44 +01:00
fkwpandClaude Opus 5.5 935a202e57 Switch the background effect with ObservableScope.reconcile
- The pipeline's switches run through reconcile, which already runs them
  one at a time and skips those overtaken, in place of a queue of our own.
- A failed switch is logged and the next still runs, now tested.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-26 18:19:44 +01:00
fkwpandClaude Opus 5.5 00df27b464 Make the background effects state an interface built by a function
- BackgroundEffects is an interface, built by createBackgroundEffects,
  following the convention for state holders, as review asked.
- Same behaviour: the body moves from the constructor unchanged.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-26 18:19:44 +01:00
fkwpandClaude Opus 5.5 56854e0fca Name the background effect pieces after background effects
- One name, as review asked: TrackProcessorContext becomes
  BackgroundEffectsContext, with BackgroundEffectsProvider,
  BackgroundEffectsState, useBackgroundEffects,
  useBackgroundEffectsState$ and useSyncBackgroundEffects.
- The state's Behavior is backgroundEffectsState$ wherever it is passed,
  in place of trackProcessorState$, trackerProcessorState$ and
  processorState$.
- Renames only: nothing changes in behaviour.

Rebased onto main: the context lives in src/tracks, main's
fromLivekitProcessor keeps its name, and the Publisher, ConnectionFactory
and their tests this renamed in are gone, as main moved them into the SDK.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-26 18:19:44 +01:00
fkwpandClaude Opus 5.5 4e4b309fc7 Keep background effects to desktops until the menu says what they cost
- The one support check asks for a desktop again, as #4059 had the
  pipeline do: menu and pipeline still agree, so a phone is neither
  offered blur nor refused it.
- Lifting it moves to the PR that tells the user where effects run
  slowly, so it is their choice to make.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-26 18:19:44 +01:00
fkwpandClaude Opus 5.5 c26cce8f0c Tidy the background effect pipeline after review
- The Vite transform wrapper takes Vite's own types, with no cast.
- The pipeline's options are compared with deepCompare, not as JSON.
- Renames `attached` to `enable` and shortens the pipeline's doc comment.
- Drops a comment that blamed an ended track on the pipeline's queue.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-26 18:19:44 +01:00
fkwpandClaude Opus 5.5 2290346232 Ask one question about whether background effects can run
- The menu asked the browser; the pipeline asked the browser and for a
  desktop. On a phone whose browser can run them, blur was offered and
  then refused: the toggle moved and the video did not change.
- Both now ask one function, and it asks the browser alone. The platform
  was the wrong measure: a phone on the fast route held 99% of its frame
  rate and was refused, a desktop on the slow one held 73% and was
  allowed.
- The in-call device switcher is still withheld on a phone; that is the
  device menu's own rule and is unchanged.
- The provider tests run as a phone, and fail with the desktop test back.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-26 18:19:44 +01:00
fkwpandClaude Opus 5.5 ef68e1347e Attach background effects through one switchable pipeline
- One pipeline, shared by the pre-join preview and the call, attached the
  first time an effect is wanted and never detached after; turning blur
  off switches it to a disabled mode in place. Main dropped the processor
  whenever blur went off, and a pipeline taken off the camera is destroyed
  and re-primed, so each time blur came back one unprocessed frame went
  out. Built once, the priming frame is spent once.
- Still driven by the existing blur setting: nothing a user sees changes.
- Its logic is BackgroundEffects in src/state/, which the provider only
  builds, so the tests drive it without rendering anything.
- Disabled drops the blur radius too. The SDK's disabled mode keeps it,
  and with a radius every frame is still segmented and thrown away; with
  neither a radius nor a picture, frames pass untouched.
- Switches run one at a time and skip any overtaken, so an earlier,
  slower switch can't land after a later choice.
- Builds, rebuilds and teardowns of the pipeline run one at a time
  (OneStepPipeline). The preview's and the call's tracks attach and stop
  it independently, and the SDK's wrapper holds one track's streams: two
  builds at once overwrite each other's, and a stop during a build does
  nothing.
- Every camera track starts with the pipeline on: the room's capture
  defaults follow it, and a camera turned off before the first effect is
  replaced on turning it on, since a stopped track can't take one. One
  attached after publishing lets the room through while it is built.
- BlurBackgroundTransformer becomes BackgroundEffectTransformer: it also
  sits disabled, and its init applies that state as the SDK's would.

Rebased onto main, where src/livekit moved to src/tracks and publishing
moved into the SDK:
- Publisher.ts is gone. Starting each camera track with the pipeline is now
  the processor on the camera publish request, at the join and on turning
  the camera on (CallViewModel). Replacing a camera turned off before the
  first effect is done in the camera's mute handler, which unpublishes and
  publishes it anew; its tests moved to CallViewModel.test.ts. The SDK's
  publisher, starting the microphone and camera in one call, takes the
  requests' options into the room's capture defaults, since a camera turned
  on before the start would otherwise come up without the processor.
- The SDK gives one VideoProcessor one LiveKit adapter, and setProcessor
  leaves a track alone that already runs it, so a track made with the
  pipeline isn't torn down when the call sets it again.
- BackgroundEffects wraps the pipeline once with fromLivekitProcessor, so the
  call and preview see one VideoProcessor throughout.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-26 18:19:44 +01:00
fkwpandClaude Opus 5.5 d3d93444af Let background effects run under the dev server
- nodePolyfills rewrites any module that mentions process, and MediaPipe's
  WASM loader does. The loader is a classic script that sets
  self.ModuleFactory; as a module it sets nothing, so under pnpm dev the
  segmenter failed with "ModuleFactory not set". Blur on main fails the
  same way there. Production is unaffected: the loader ships as a static
  asset.
- The plugin's transform now skips that loader. Checked with main's own
  BlurBackgroundTransformer: it initialises with this and fails without.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-26 15:54:07 +01:00
fkwpandClaude Opus 5.5 9e6f2e89a8 Size and scroll the device menus with Compound's menu
- Compound 10.3.0 bounds a menu by the space it has and scrolls what
  doesn't fit, within a boundary the app passes. The menus pass the call
  area, 16px in from its sides, rather than measuring its width.
- The 296px width is the menu's own style. Compound gives that style only
  to the dropdown, so menuIsDrawer, which copied its phone rule, goes.
- Compound draws the menu's frame over its content, so nothing that
  scrolls can hide it in Safari: the list, headings and meter no longer
  keep a border width clear of it, and the checks that they did go.
- The device list is Compound's scroll area, so it fades out at the
  foot with Compound's chevron while there is more below.
- The microphone menu's meter sticks at that foot, with rows scrolling
  under it as before, so the list has no fade above it: Compound's is a
  mask fixed to the list's edge, while a bounce at the top carries the
  meter with the rows. The band under the meter is filled, or rows would
  show there. The edge fades from where the chevron starts, so a bounce
  fades the meter out before it reaches the chevron. A row reached by
  keyboard stays above the meter.
- The list's height is still its share of the call.
- The component harness leaves the host's stylesheets unscoped, Compound's
  among them, as a host does: scoped, they never reached the menu, which is
  portalled out of Element Call's root.
- Its specs check only Element Call's own document styles on the host
  page, and find the device list within Compound's scroll area.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-26 15:48:52 +01:00
fkwpandClaude Opus 5.5 7a48b09c4c Hold both device menus at 296px, framed in Safari too
- The camera and microphone menus were as wide as their longest device
  name, so a long one widened the menu over more of the call: 340px for a
  long speaker name. Both are now 296px, as design sets them, and a long
  name wraps.
- Narrower only where the call area is, so the menu still fits inside it:
  268px in a 300px call. Taken from the observation of the call area that
  already bounds the list's height.
- Held only for the dropdown. On Android and iOS Compound renders the menu
  as a drawer as wide as the screen; menuIsDrawer follows its rule.
- Once the list is long enough to scroll, Safari paints it above the
  outline the menu is framed with, and the frame vanished down both sides
  of it: 384px of it on the many-devices microphone menu, in WebKit. The
  list now keeps a border width clear of the frame, so the headings and
  the meter inside it no longer need their own.
- Guarded with a long speaker name, which fails at 340px on main's sizing;
  a unit test run as a phone, which fails with the width held there; and
  the many-devices story's inset, which fails on the old stylesheet, since
  the stories run in Chromium, which never showed the Safari bug.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-26 15:48:07 +01:00
Robin ed6a0e1341 Merge pull request #4371 from element-hq/ellipsize-title
Ellipsise app bar text on mobile
2026-10-26 13:45:06 +01:00
renovate[bot]andJohannes Marbach 90ccf62d5c Update ghcr.io/element-hq/element-web:develop Docker digest to c9ffe3e (#4356)
* Update ghcr.io/element-hq/element-web:develop Docker digest to 79e47d8

* Adapt to new recovery popup button title

---------

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: Johannes Marbach <n0-0ne+github@mailbox.org>
2026-10-26 12:44:39 +00:00
Robin 9ec0cef9c5 Merge pull request #4370 from element-hq/reintroduce-livekit-alias
Reintroduce livekit_alias in published transports
2026-10-26 12:39:14 +01:00
Robin c248ff4280 Don't add livekit_alias to newer transport format 2026-10-26 11:41:45 +01:00
Robin 0f1d0941d0 Reference matrix-rust-sdk PR making livekit_alias optional, too 2026-10-26 11:17:16 +01:00
Robin 7e6e1698c0 Reintroduce livekit_alias in published transports
Some Ruma-based clients still expect it to be present. We should remove it again as soon as https://github.com/ruma/ruma/pull/2642 is widely deployed.
2026-10-26 10:57:49 +01:00
61 changed files with 2736 additions and 477 deletions
@@ -109,4 +109,12 @@ describe("scopeStylesToRoot", () => {
".a .b_abc12:hover { border: 0 }",
);
});
it("leaves an excluded stylesheet as it is", async () => {
const css = "body { margin: 0 } .menu { color: red }";
const result = await postcss([
scopeStylesToRoot({ exclude: [/host\.css$/] }),
]).process(css, { from: "/app/host.css" });
expect(result.css).toBe(css);
});
});
+9 -3
View File
@@ -60,12 +60,18 @@ const IS_ROOT = `:where(${ROOT_SELECTOR})`;
* tokens, the component styles, the fonts — is not in this build at all: the
* host supplies it, unscoped, along with its copy of Compound.
*/
export function scopeStylesToRoot(): Plugin {
interface Options {
/** Stylesheets left as they are: a host's own, which it doesn't scope. */
exclude?: RegExp[];
}
export function scopeStylesToRoot({ exclude = [] }: Options = {}): Plugin {
return {
postcssPlugin: "element-call-scope-styles-to-root",
Once(root) {
const isModule =
root.source?.input.file?.endsWith(".module.css") ?? false;
const file = root.source?.input.file ?? "";
if (exclude.some((pattern) => pattern.test(file))) return;
const isModule = file.endsWith(".module.css");
root.walkRules((rule) => {
if (isRelative(rule)) return;
rule.selector = (isModule ? scopeBare : scopeAll).processSync(
+11 -12
View File
@@ -72,9 +72,9 @@ import {
useUrlParams,
} from "../src/UrlParams";
import { MediaDevicesContext } from "../src/MediaDevicesContext";
import { MediaDevices } from "../src/state/MediaDevices";
import { AppViewModel } from "../src/state/AppViewModel";
import { ObservableScope } from "../src/state/ObservableScope";
import { ProcessorProvider } from "../src/tracks/TrackProcessorContext";
import { BackgroundEffectsProvider } from "../src/tracks/BackgroundEffectsContext";
import { Config } from "../src/config/Config";
import { type ConfigOptions } from "../src/config/ConfigOptions";
import { i18n } from "../src/utils/i18n";
@@ -194,14 +194,12 @@ export const ElementCall: FC<ElementCallProps> = ({
// observers running for the rest of the page's life. Null until then, which
// is one render.
const { controlledAudioDevices, callIntent } = params;
const [mediaDevices, setMediaDevices] = useState<MediaDevices | null>(null);
const [vm, setVm] = useState<AppViewModel | null>(null);
useEffect(() => {
const scope = new ObservableScope();
setMediaDevices(
new MediaDevices(scope, { controlledAudioDevices, callIntent }),
);
setVm(new AppViewModel(scope, { controlledAudioDevices, callIntent }));
return (): void => {
setMediaDevices(null);
setVm(null);
scope.end();
};
}, [controlledAudioDevices, callIntent]);
@@ -222,8 +220,7 @@ export const ElementCall: FC<ElementCallProps> = ({
// is told that Element Call has loaded, as the widget tells its client once
// its own initialisation is over. Once per mount, however often the pieces
// are later swapped out.
const ready =
container !== null && rtcSession !== null && mediaDevices !== null;
const ready = container !== null && rtcSession !== null && vm !== null;
const announcedLoaded = useRef(false);
useEffect(() => {
if (!ready || announcedLoaded.current) return;
@@ -254,8 +251,10 @@ export const ElementCall: FC<ElementCallProps> = ({
<Decoration>
<TooltipProvider>
<ClientProvider client={client}>
<MediaDevicesContext value={mediaDevices}>
<ProcessorProvider>
<MediaDevicesContext value={vm.mediaDevices}>
<BackgroundEffectsProvider
effects={vm.backgroundEffects}
>
<CallView
client={client}
rtcSession={rtcSession}
@@ -264,7 +263,7 @@ export const ElementCall: FC<ElementCallProps> = ({
preload={params.preload}
skipLobby={params.skipLobby}
/>
</ProcessorProvider>
</BackgroundEffectsProvider>
</MediaDevicesContext>
</ClientProvider>
</TooltipProvider>
+2 -2
View File
@@ -19,7 +19,7 @@ services:
- ./backend/playwright_homeserver-othersite.yaml:/data/cfg/homeserver.yaml:Z
element-web:
# Pin to a SHA so that upstream cannot break our tests. Renovate handles regular updates.
image: ghcr.io/element-hq/element-web:develop@sha256:b2a5a30afb4f583a7560448b005d8332bee29672f15826e4a995b393058f0291
image: ghcr.io/element-hq/element-web:develop@sha256:79e47d8d5a93848403cb83042a405d8ed233a7727c499fa53b4b39a80f090be4
element-web-1:
# Pin to a SHA so that upstream cannot break our tests. Renovate handles regular updates.
image: ghcr.io/element-hq/element-web:develop@sha256:b2a5a30afb4f583a7560448b005d8332bee29672f15826e4a995b393058f0291
image: ghcr.io/element-hq/element-web:develop@sha256:79e47d8d5a93848403cb83042a405d8ed233a7727c499fa53b4b39a80f090be4
+8 -1
View File
@@ -3,7 +3,6 @@
"user_menu": "User menu"
},
"action": {
"blur_background": "Blur background",
"close": "Close",
"copy_link": "Copy link",
"edit": "Edit",
@@ -23,6 +22,13 @@
"upload_file": "Upload file"
},
"analytics_notice": "By participating in this beta, you consent to the collection of anonymous data, which we use to improve the product. You can find more information about which data we track in our <2>Privacy Policy</2> and our <6>Cookie Policy</6>.",
"background_effects": {
"blur": "Blur",
"none": "None",
"numbered": "Background {{n}}",
"slow": "Background effects run slowly on this platform, which may cause your video to stutter.",
"unavailable": "Background effects are not supported on this platform."
},
"call_ended_view": {
"create_account_button": "Create account",
"create_account_prompt": "<0>Why not finish by setting up a password to keep your account?</0><1>You'll be able to keep your name and set an avatar for use on future calls</1>",
@@ -212,6 +218,7 @@
"auto_gain_control_label": "Automatic gain control",
"background_blur_header": "Background",
"background_blur_label": "Blur the background of the video",
"background_effects_header": "Background effects",
"bitrate_label": "Bitrate",
"blur_not_supported_by_browser": "(Background blur is not supported by this device.)",
"camera_header": "Camera quality",
+1 -1
View File
@@ -87,7 +87,7 @@
"@types/sdp-transform": "^2.4.5",
"@typescript-eslint/utils": "^8.61.0",
"@use-gesture/react": "^10.2.11",
"@vector-im/compound-design-tokens": "^11.0.0",
"@vector-im/compound-design-tokens": "^11.1.0",
"@vector-im/compound-web": "^10.4.0",
"@vitejs/plugin-react": "^6.0.2",
"@vitest/browser-playwright": "^4.1.5",
+126
View File
@@ -0,0 +1,126 @@
/*
Copyright 2026 Element Creations Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { expect, test, type Locator, type Page } from "@playwright/test";
import { SpaHelpers } from "./spa-helpers.ts";
// Background effects need WebGL2, and headless Firefox on a CI runner has none,
// so it rightly offers none of them.
test.skip(
({ browserName }) => browserName === "firefox",
"Background effects need WebGL2, which headless Firefox on CI does not have",
);
type Colour = [number, number, number];
test.describe("background effects", () => {
test("pre-join preview shows the chosen effect", async ({ page }) => {
test.slow();
await page.goto("/");
await SpaHelpers.createCall(page, "Effect user", "Background effects");
const preview = page.locator("video").first();
const camera = await averageColour(preview);
await page.getByRole("button", { name: "Camera", exact: true }).click();
const tile = page
.getByRole("group", { name: "Background effects" })
.getByRole("menuitemradio", { name: "Background 1" });
await tile.click();
await expect(tile).toHaveAttribute("aria-checked", "true");
// The fake camera shows no person, so the whole frame is background: the
// preview takes on the picture's colours and leaves the camera's.
const picture = await averageColour(tile.locator("img"));
// Long enough for the first effect of a session to be built.
await expect
.poll(async () => distance(await averageColour(preview), picture), {
timeout: 60_000,
})
.toBeLessThan(distance(camera, picture) / 2);
});
});
test.describe("no effect", () => {
test("draws nothing, even after blurring", async ({ page }) => {
test.setTimeout(120_000);
await countDraws(page);
await page.goto("/");
await SpaHelpers.createCall(page, "Idle user", "No effect after blur");
await page.getByRole("button", { name: "Camera", exact: true }).click();
const effects = page.getByRole("group", { name: "Background effects" });
await effects.getByRole("menuitemradio", { name: "Blur" }).click();
// Long enough for the first effect of a session to be built.
await expect
.poll(async () => drawsPerSecond(page), { timeout: 60_000 })
.toBeGreaterThan(0);
const blurred = await drawsPerSecond(page);
await effects.getByRole("menuitemradio", { name: "None" }).click();
// A frame already on its way may still be drawn.
await expect.poll(async () => drawsPerSecond(page)).toBe(0);
test.info().annotations.push({
type: "WebGL draws per second",
description: `blurred ${blurred}, none 0`,
});
});
});
/** Counts every WebGL draw on the page, the segmenter's and the effect's. */
async function countDraws(page: Page): Promise<void> {
await page.addInitScript(() => {
const counter = { draws: 0 };
(window as unknown as { webglDraws: typeof counter }).webglDraws = counter;
for (const name of ["drawArrays", "drawElements"] as const) {
const draw = WebGL2RenderingContext.prototype[name] as (
...args: unknown[]
) => void;
WebGL2RenderingContext.prototype[name] = function (
this: WebGL2RenderingContext,
...args: unknown[]
): void {
counter.draws += 1;
draw.apply(this, args);
} as never;
}
});
}
/** WebGL draws over the next second. */
async function drawsPerSecond(page: Page): Promise<number> {
return page.evaluate(async () => {
const counter = (window as unknown as { webglDraws: { draws: number } })
.webglDraws;
const before = counter.draws;
await new Promise((resolve) => setTimeout(resolve, 1000));
return counter.draws - before;
});
}
/** The mean colour of a video's current frame or an image, drawn small. */
async function averageColour(element: Locator): Promise<Colour> {
return element.evaluate(
async (source: HTMLVideoElement | HTMLImageElement) => {
if (source instanceof HTMLImageElement) await source.decode();
const canvas = document.createElement("canvas");
canvas.width = canvas.height = 32;
const context = canvas.getContext("2d")!;
context.drawImage(source, 0, 0, 32, 32);
const { data } = context.getImageData(0, 0, 32, 32);
const sum = [0, 0, 0];
for (let i = 0; i < data.length; i += 4)
for (let c = 0; c < 3; c++) sum[c] += data[i + c];
const pixels = data.length / 4;
return sum.map((s) => s / pixels) as Colour;
},
);
}
function distance(a: Colour, b: Colour): number {
return Math.hypot(a[0] - b[0], a[1] - b[1], a[2] - b[2]);
}
+9 -15
View File
@@ -125,25 +125,19 @@ test("leaves the host's own page unstyled", async ({ page }) => {
timeout: 60_000,
});
// Element Call's stylesheet is written for a page of its own: normalize.css
// gives `html` a line height, Compound gives `body` its font and feature
// settings, and the design tokens live on `:root`. None of that may reach the
// host's document — the harness sets none of these itself, so anything other
// than the browser's defaults here came from us.
// Compound's document styles reach the host, but it loads them itself.
const host = await page.evaluate(() => {
const html = getComputedStyle(document.documentElement);
const body = getComputedStyle(document.body);
const sub = document.body.appendChild(document.createElement("sub"));
const verticalAlign = getComputedStyle(sub).verticalAlign;
sub.remove();
return {
lineHeight: html.lineHeight,
fontFeatureSettings: body.fontFeatureSettings,
token: html.getPropertyValue("--cpd-color-text-primary"),
verticalAlign,
token: getComputedStyle(document.documentElement).getPropertyValue(
"--font-scale",
),
};
});
expect(host).toEqual({
lineHeight: "normal",
fontFeatureSettings: "normal",
token: "",
});
expect(host).toEqual({ verticalAlign: "sub", token: "" });
// While inside the container, the same rules do apply
const root = panes.first().locator("[data-element-call-root]");
+6 -2
View File
@@ -159,7 +159,7 @@ export class TestHelpers {
public static async dismissStartupToasts(page: Page): Promise<void> {
const expectedToasts = [
{ title: "Failed to load service worker", button: "OK" },
{ title: "Back up your chats", button: "Dismiss" },
{ title: "Back up your chats", button: "Not now" },
{ title: "Turn on key storage", button: "Dismiss" },
{ title: "Element does not support this browser", button: "Dismiss" },
];
@@ -178,7 +178,11 @@ export class TestHelpers {
);
if (toastConfig) {
await toast.getByRole("button", { name: toastConfig.button }).click();
// Bounded so that a renamed button leaves the toast in place and the
// next step reports it, instead of hanging until the test timeout.
await toast
.getByRole("button", { name: toastConfig.button })
.click({ timeout: 5000 });
} else {
// Unknown toast. We don't want to act on unknown toasts
break;
+1 -1
View File
@@ -131,7 +131,7 @@ importers:
specifier: ^10.2.11
version: 10.3.1(react@19.2.8)
'@vector-im/compound-design-tokens':
specifier: ^11.0.0
specifier: ^11.1.0
version: 11.3.1(@types/react@19.2.18)(react@19.2.8)
'@vector-im/compound-web':
specifier: ^10.4.0
+9
View File
@@ -0,0 +1,9 @@
lockfileVersion: '9.0'
settings:
autoInstallPeers: false
excludeLinksFromLockfile: false
importers:
.: {}
+10 -3
View File
@@ -12,6 +12,7 @@ import {
MatrixRTCSessionEvent,
MembershipManagerEvent,
type Transport,
type UnstableLivekitTransport,
} from "matrix-js-sdk/lib/matrixrtc";
import { BehaviorSubject, EMPTY, Subject } from "rxjs";
import { describe, expect, it, type Mock, vi } from "vitest";
@@ -46,10 +47,16 @@ import {
MockRTCSession,
} from "./utils/test";
const transport: Transport = {
const transport: UnstableLivekitTransport = {
type: "livekit",
livekit_service_url: "https://lk.example.org",
};
const transportWithAlias: Transport = {
livekit_alias: "!room:example.org",
...transport,
};
const userId = "@alice:example.org";
/** A token with the publish grant, as a homeserver issues it for its own SFU. */
const publisherJWT = [
@@ -223,7 +230,7 @@ describe("createRTCParticipation", () => {
await flushPromises();
expect(join).toHaveBeenCalledWith(
{ userId, deviceId, memberId: legacyMemberId },
[transport],
[transportWithAlias],
expect.objectContaining({
manageMediaKeys: true,
unstableSendStickyEvents: false,
@@ -319,7 +326,7 @@ describe("createRTCParticipation", () => {
await flushPromises();
expect(join).toHaveBeenCalledWith(
expect.anything(),
[transport],
[transportWithAlias],
expect.objectContaining({ delayedLeaveEventDelayMs: 7_200_000 }),
);
expect(fake.delegateDelayedLeave).not.toHaveBeenCalled();
+14 -8
View File
@@ -174,14 +174,20 @@ export function createRTCParticipation(
timings.syncDisconnectGracePeriodMs,
),
joinMatrixRTC: (transport, delayedLeave) =>
joinJsSdkSession(jsSdkSession, ownMembershipIdentity, transport, {
encryptMedia: encryptionSystem.kind !== E2eeType.NONE,
matrixRTCMode,
sendNotificationType: options.sendNotificationType,
applicationData: options.applicationData,
timings,
delayedLeave,
}),
joinJsSdkSession(
room.roomId,
jsSdkSession,
ownMembershipIdentity,
transport,
{
encryptMedia: encryptionSystem.kind !== E2eeType.NONE,
matrixRTCMode,
sendNotificationType: options.sendNotificationType,
applicationData: options.applicationData,
timings,
delayedLeave,
},
),
membershipManagerError$: fromEvent(
jsSdkSession,
MatrixRTCSessionEvent.MembershipManagerError,
+35 -6
View File
@@ -6,6 +6,7 @@ Please see LICENSE in the repository root for full details.
*/
import {
type UnstableLivekitTransport,
type MatrixRTCSession,
type Transport,
} from "matrix-js-sdk/lib/matrixrtc";
@@ -15,11 +16,16 @@ import { defaultSessionTimings, MatrixRTCMode } from "../config";
import { ownMemberMock } from "../utils/test";
import { joinJsSdkSession } from "./joinJsSdkSession";
const transport: Transport = {
const transport: UnstableLivekitTransport = {
type: "livekit",
livekit_service_url: "https://lk.example.org",
};
const transportWithAlias: Transport = {
livekit_alias: "!room:example.org",
...transport,
};
/** A session that only records the join; nothing else of it is touched. */
function session(): MatrixRTCSession & {
join: Mock<MatrixRTCSession["join"]>;
@@ -39,7 +45,7 @@ describe("joinJsSdkSession", () => {
keyRotationParticipantLimit: 20,
};
joinJsSdkSession(s, ownMemberMock, transport, {
joinJsSdkSession("!room:example.org", s, ownMemberMock, transport, {
encryptMedia: true,
matrixRTCMode: MatrixRTCMode.Compatibility,
sendNotificationType: "ring",
@@ -51,7 +57,7 @@ describe("joinJsSdkSession", () => {
},
});
expect(s.join).toHaveBeenCalledWith(ownMemberMock, [transport], {
expect(s.join).toHaveBeenCalledWith(ownMemberMock, [transportWithAlias], {
notificationType: "ring",
callIntent: undefined,
applicationData: {},
@@ -72,7 +78,7 @@ describe("joinJsSdkSession", () => {
it("gives up on the network once the delayed leave has probably been sent", () => {
const s = session();
joinJsSdkSession(s, ownMemberMock, transport, {
joinJsSdkSession("!room:example.org", s, ownMemberMock, transport, {
encryptMedia: false,
matrixRTCMode: MatrixRTCMode.Compatibility,
timings: { ...defaultSessionTimings, networkErrorRetryMs: 4_000 },
@@ -81,7 +87,7 @@ describe("joinJsSdkSession", () => {
expect(s.join).toHaveBeenCalledWith(
ownMemberMock,
[transport],
[transportWithAlias],
expect.objectContaining({
manageMediaKeys: false,
maximumNetworkErrorRetryCount: 4,
@@ -92,7 +98,30 @@ describe("joinJsSdkSession", () => {
it("lifts the call intent out of the application data and sends sticky events in Matrix 2.0 mode", () => {
const s = session();
joinJsSdkSession(s, ownMemberMock, transport, {
joinJsSdkSession("!room:example.org", s, ownMemberMock, transport, {
encryptMedia: true,
matrixRTCMode: MatrixRTCMode.Matrix_2_0,
applicationData: { "m.call.intent": "audio", custom: 1 },
timings: defaultSessionTimings,
delayedLeave: defaultSessionTimings.delayedLeave,
});
expect(s.join).toHaveBeenCalledWith(
ownMemberMock,
[transportWithAlias],
expect.objectContaining({
callIntent: "audio",
applicationData: { custom: 1 },
unstableSendStickyEvents: true,
}),
);
});
it("does not add livekit_alias to newer transport format", () => {
const s = session();
const transport = { type: "livekit", url: "wss://lk.example.org" };
joinJsSdkSession("!room:example.org", s, ownMemberMock, transport, {
encryptMedia: true,
matrixRTCMode: MatrixRTCMode.Matrix_2_0,
applicationData: { "m.call.intent": "audio", custom: 1 },
+34 -16
View File
@@ -35,6 +35,7 @@ interface Options {
* @throws If the join could not be started.
*/
export function joinJsSdkSession(
roomId: string,
session: JsSdkRTCSession,
ownMembershipIdentity: CallMembershipIdentityParts,
transport: Transport,
@@ -58,20 +59,37 @@ export function joinJsSdkSession(
timings.syncDisconnectGracePeriodMs,
delayedLeave.delay_ms,
);
session.join(ownMembershipIdentity, [transport], {
notificationType: sendNotificationType,
callIntent,
applicationData: rest,
manageMediaKeys: encryptMedia,
delayedLeaveEventRestartMs: delayedLeave.restart_ms,
delayedLeaveEventDelayMs: delayedLeave.delay_ms,
delayedLeaveEventRestartLocalTimeoutMs: delayedLeave.restart_timeout_ms,
networkErrorRetryMs: timings.networkErrorRetryMs,
makeKeyDelay: timings.waitForKeyRotationMs,
membershipEventExpiryMs: timings.membershipEventExpiryMs,
keyRotationParticipantLimit: timings.keyRotationParticipantLimit,
unstableSendStickyEvents: matrixRTCMode === MatrixRTCMode.Matrix_2_0,
maximumNetworkErrorRetryCount:
Math.ceil(maxWaitMs / timings.networkErrorRetryMs) + 1,
});
session.join(
ownMembershipIdentity,
// TODO: `livekit_alias` should be unused these days, but some Ruma-based
// clients still expect it to be present. Remove from `publishedTransports`
// as soon as https://github.com/ruma/ruma/pull/2642 and
// https://github.com/matrix-org/matrix-rust-sdk/pull/7335 are widely
// deployed.
[
{
...("livekit_service_url" in transport
? { livekit_alias: roomId }
: {}),
...transport,
},
],
{
notificationType: sendNotificationType,
callIntent,
applicationData: rest,
manageMediaKeys: encryptMedia,
delayedLeaveEventRestartMs: delayedLeave.restart_ms,
delayedLeaveEventDelayMs: delayedLeave.delay_ms,
delayedLeaveEventRestartLocalTimeoutMs: delayedLeave.restart_timeout_ms,
networkErrorRetryMs: timings.networkErrorRetryMs,
makeKeyDelay: timings.waitForKeyRotationMs,
membershipEventExpiryMs: timings.membershipEventExpiryMs,
keyRotationParticipantLimit: timings.keyRotationParticipantLimit,
unstableSendStickyEvents: matrixRTCMode === MatrixRTCMode.Matrix_2_0,
maximumNetworkErrorRetryCount:
Math.ceil(maxWaitMs / timings.networkErrorRetryMs) + 1,
},
);
}
@@ -230,9 +230,10 @@ export function createLocalLivekitMediaTrack(
const track = videoTrack();
// A processor cannot be built on a track that has already ended
if (!track || track.mediaStreamTrack.readyState === "ended") return;
if (processor)
await track.setProcessor(convertToLivekitProcessor(processor));
else if (track.getProcessor()) await track.stopProcessor();
if (processor) {
const adapter = convertToLivekitProcessor(processor);
if (track.getProcessor() !== adapter) await track.setProcessor(adapter);
} else if (track.getProcessor()) await track.stopProcessor();
},
};
}
@@ -18,9 +18,10 @@ import {
import { logger } from "matrix-js-sdk/lib/logger";
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
import { type PublishRequest } from "../../media-api";
import { type PublishRequest, type VideoProcessor } from "../../media-api";
import { flushPromises } from "../../utils/test";
import { type DesiredMedia, Publisher } from "./Publisher";
import { convertToLivekitProcessor } from "./videoProcessor";
describe("Publisher", () => {
describe("setAudioOutputDevice", () => {
@@ -119,6 +120,24 @@ describe("Publisher", () => {
});
});
it("creates both with the options of requests made since the join", async () => {
const processor: VideoProcessor = {
name: "effect",
init: vi.fn(),
restart: vi.fn(),
destroy: vi.fn(),
};
want(microphone, { source: "camera", deviceId: "front", processor });
publisher.start();
await flushPromises();
expect(room.options.videoCaptureDefaults).toMatchObject({
deviceId: "front",
processor: convertToLivekitProcessor(processor),
});
});
it.each([
{ enabled: microphone, muted: camera, options: { audio: true } },
{ enabled: camera, muted: microphone, options: { video: true } },
+24 -1
View File
@@ -87,8 +87,10 @@ export class Publisher {
// permission once; the room defaults carry their options. LiveKit
// resolves these once the track is published, which may block on the
// connection; LocalTrackPublished is what tells us a track exists.
if (sources.has("microphone") && sources.has("camera"))
if (sources.has("microphone") && sources.has("camera")) {
this.takeCaptureDefaults(requests);
void this.room.localParticipant.enableCameraAndMicrophone();
}
for (const request of requests)
if (
request.source === "screenShare" ||
@@ -99,6 +101,27 @@ export class Publisher {
});
}
/**
* Makes the requests the room's capture defaults. The room took its defaults
* from the requests at the join, but one made since, such as a camera turned
* on before the start, has options of its own: a video processor dropped
* here would let the room see the camera before it is applied.
*/
private takeCaptureDefaults(requests: PublishRequest[]): void {
const options = this.room.options;
for (const request of requests)
if (request.source === "microphone")
options.audioCaptureDefaults = {
...options.audioCaptureDefaults,
...audioCaptureOptions(request),
};
else if (request.source === "camera")
options.videoCaptureDefaults = {
...options.videoCaptureDefaults,
...videoCaptureOptions(request),
};
}
/** Publishes one request and resolves with its publication once LiveKit has it. */
public async publish(
request: PublishRequest,
@@ -47,4 +47,11 @@ describe("convertToLivekitProcessor", () => {
await adapter.destroy();
expect(processor.destroy).toHaveBeenCalled();
});
it("gives one processor one adapter", () => {
const processor = fakeProcessor();
expect(convertToLivekitProcessor(processor)).toBe(
convertToLivekitProcessor(processor),
);
});
});
@@ -9,14 +9,24 @@ import { type Track, type TrackProcessor } from "livekit-client";
import { type VideoProcessor } from "../../media-api";
const adapters = new WeakMap<
VideoProcessor,
TrackProcessor<Track.Kind.Video>
>();
/**
* A `VideoProcessor` as livekit-client wants it. LiveKit reads the processed
* track off the processor after `init` and `restart` rather than taking it as
* their result, so the adapter keeps it there.
*
* One adapter per processor, so a track can tell that it already runs one: a
* processor set again would otherwise be torn down and built anew.
*/
export function convertToLivekitProcessor(
processor: VideoProcessor,
): TrackProcessor<Track.Kind.Video> {
const existing = adapters.get(processor);
if (existing) return existing;
const adapter: TrackProcessor<Track.Kind.Video> = {
name: processor.name,
init: async (opts) => {
@@ -30,5 +40,6 @@ export function convertToLivekitProcessor(
},
destroy: async () => processor.destroy(),
};
adapters.set(processor, adapter);
return adapter;
}
+3 -3
View File
@@ -35,7 +35,7 @@ import { ErrorPage, LoadingPage } from "./FullScreenView";
import { Initializer } from "./initializer";
import { type WidgetHelpers } from "./widget";
import { useTheme } from "./useTheme";
import { ProcessorProvider } from "./tracks/TrackProcessorContext";
import { BackgroundEffectsProvider } from "./tracks/BackgroundEffectsContext";
import { type AppViewModel } from "./state/AppViewModel";
import { MediaDevicesContext } from "./MediaDevicesContext";
import {
@@ -148,7 +148,7 @@ export const App: FC<Props> = ({ vm, widget }) => {
loaded && clientReady ? (
<ClientProvider client={widgetClient}>
<MediaDevicesContext value={vm.mediaDevices}>
<ProcessorProvider>
<BackgroundEffectsProvider effects={vm.backgroundEffects}>
<Sentry.ErrorBoundary
fallback={(error) => <ErrorPage error={error} />}
>
@@ -159,7 +159,7 @@ export const App: FC<Props> = ({ vm, widget }) => {
<SentryRoute path="*" element={<RoomPage />} />
</Routes>
</Sentry.ErrorBoundary>
</ProcessorProvider>
</BackgroundEffectsProvider>
</MediaDevicesContext>
</ClientProvider>
) : (
@@ -0,0 +1,92 @@
/*
Copyright 2026 Element Creations Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
.grid {
display: grid;
/* A zero floor, so a long label wraps inside its tile rather than widening
the grid past the menu. */
grid-template-columns: repeat(3, minmax(0, 1fr));
grid-auto-rows: 1fr;
gap: var(--cpd-space-3x);
padding: var(--cpd-space-2x) var(--cpd-space-4x) 0;
}
/* Within the grid, to outweigh the menu item styles the tiles are built on. */
.grid .tile {
position: relative;
display: flex;
flex-direction: column;
align-items: center;
justify-content: center;
gap: var(--cpd-space-1x);
min-inline-size: 0;
padding: var(--cpd-space-3x) var(--cpd-space-2x);
/* Not hidden: that makes the tile a scroll container, which focus scrolls
into view without its border, and so without the focus ring. */
overflow: clip;
border: var(--cpd-border-width-2) solid
var(--cpd-color-border-interactive-secondary);
border-radius: var(--cpd-space-2x);
background: none;
color: var(--cpd-color-icon-secondary);
font: inherit;
cursor: pointer;
}
/* As the menu items highlight on hover. */
.grid .tile[data-highlighted] {
background: var(--cpd-color-bg-action-secondary-hovered);
}
.grid .tile[aria-checked="true"] {
border-color: var(--cpd-color-border-accent);
}
.grid .tile[data-disabled] {
cursor: default;
opacity: 0.55;
}
.tile[data-disabled] .picture {
filter: grayscale(1);
}
.swatch {
display: flex;
align-items: center;
justify-content: center;
}
.picture {
position: absolute;
inset: 0;
inline-size: 100%;
block-size: 100%;
object-fit: cover;
}
/* Positioned, or the picture, positioned itself, would cover it. */
.mark {
position: relative;
display: inline-flex;
color: var(--cpd-color-icon-accent-primary);
}
/* On a picture the mark needs a ground of its own to be seen. */
.picture ~ .mark {
border-radius: 50%;
background: var(--cpd-color-bg-canvas-default);
box-shadow: 0 0 0 var(--cpd-border-width-2) var(--cpd-color-bg-canvas-default);
}
.label {
max-inline-size: 100%;
font: var(--cpd-font-body-sm-medium);
color: var(--cpd-color-text-primary);
text-align: center;
overflow-wrap: anywhere;
}
+142
View File
@@ -0,0 +1,142 @@
/*
Copyright 2026 Element Creations Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { type FC, type ReactNode, type ReactElement } from "react";
import { VisuallyHidden } from "@radix-ui/react-visually-hidden";
import { InlineSpinner, MenuItem } from "@vector-im/compound-web";
import {
BlockIcon,
BlurIcon,
CheckCircleSolidIcon,
} from "@vector-im/compound-design-tokens/assets/web/icons";
import styles from "./BackgroundEffectGrid.module.css";
/** One choice in the camera menu's Background effects section. */
export interface BackgroundEffectOption {
id: string;
/** The tile's accessible name; drawn only where the tile has no picture. */
label: string;
kind: "none" | "blur" | "image";
/** For kind "image". */
imageUrl?: string;
}
interface Props {
/** Names the group, whose heading is hidden from assistive technology. */
label: string;
heading: ReactNode;
effects: BackgroundEffectOption[];
selected: string | undefined;
/** Undefined where effects can't run: all but no effect are disabled. */
onSelect: ((id: string) => void) | undefined;
/** Shown on the selected tile until its effect is on screen. */
settling?: boolean;
/** Id of the text saying what the user should know before choosing. */
describedBy?: string;
}
/**
* The background effects as a grid of tiles: one radio choice, walked by the
* menu's arrow keys like the device rows.
*/
export const BackgroundEffectGrid: FC<Props> = ({
label,
heading,
effects,
selected,
onSelect,
settling = false,
describedBy,
}) => {
const choose = (id: string): void => {
if (id !== selected) onSelect?.(id);
};
const tiles = effects.map((effect) => {
const checked = effect.id === selected;
const disabled = onSelect === undefined && effect.kind !== "none";
return (
<MenuItem
key={effect.id}
role="menuitemradio"
aria-checked={checked}
aria-busy={checked && settling}
disabled={disabled}
data-disabled={disabled ? "" : undefined}
label={null}
hideChevron
className={styles.tile}
// Without a handler it isn't one of the menu's items, so the keyboard
// passes over it.
onSelect={
disabled
? null
: (e): void => {
// Kept open, so the user sees what they chose take effect.
e.preventDefault();
choose(effect.id);
}
}
>
<TileContent
effect={effect}
checked={checked}
settling={checked && settling}
/>
</MenuItem>
);
});
return (
<div role="group" aria-label={label} aria-describedby={describedBy}>
{heading}
<div role="none" className={styles.grid}>
{tiles}
</div>
</div>
);
};
function TileContent({
effect,
checked,
settling,
}: {
effect: BackgroundEffectOption;
checked: boolean;
settling: boolean;
}): ReactElement {
return (
<>
<span aria-hidden className={styles.swatch}>
{effect.kind === "image" && (
<img className={styles.picture} src={effect.imageUrl} alt="" />
)}
{settling ? (
<span className={styles.mark}>
<InlineSpinner size={20} />
</span>
) : checked ? (
<CheckCircleSolidIcon
className={styles.mark}
width={20}
height={20}
/>
) : effect.kind === "none" ? (
<BlockIcon width={20} height={20} />
) : effect.kind === "blur" ? (
<BlurIcon width={20} height={20} />
) : null}
</span>
{effect.kind === "image" ? (
<VisuallyHidden>{effect.label}</VisuallyHidden>
) : (
<span className={styles.label}>{effect.label}</span>
)}
</>
);
}
+14 -2
View File
@@ -22,6 +22,7 @@ import { MediaDevices } from "../state/MediaDevices";
import { globalScope } from "../state/ObservableScope";
import { constant } from "../state/Behavior";
import { type LayoutMode } from "../state/LayoutSwitchViewModel";
import { shippedBackgrounds } from "../tracks/backgroundEffects";
// consts for tests
const reactionIdentifier = "@user:example.com:DEVICE";
@@ -123,8 +124,19 @@ export const Default: Story = {
toggleAudio: fn(),
toggleVideo: fn(),
toggleScreenSharing: fn(),
toggleBlur: fn(),
videoBlurEnabled: true,
selectBackgroundEffect: fn(),
backgroundEffect: "blur",
backgroundEffects: [
{ id: "none", kind: "none" },
{ id: "blur", kind: "blur" },
...shippedBackgrounds.map((background) => ({
id: `image:${background.id}`,
kind: "image" as const,
imageUrl: background.imagePath,
})),
],
backgroundEffectNotice: undefined,
backgroundEffectSettling: false,
hangup: fn(),
buttonSize: "lg",
showFooter: true,
+59 -6
View File
@@ -7,6 +7,7 @@ Please see LICENSE in the repository root for full details.
import { type FC, type JSX, type Ref, useMemo } from "react";
import classNames from "classnames";
import { useTranslation } from "react-i18next";
import LogoMark from "../icons/LogoMark.svg?react";
import LogoType from "../icons/LogoType.svg?react";
@@ -31,6 +32,15 @@ import { type ViewModel } from "../state/ViewModel";
import { useBehavior } from "../useBehavior";
import { type LayoutSwitchViewModel } from "../state/LayoutSwitchViewModel";
import { LayoutSwitch } from "../room/LayoutSwitch";
import { type BackgroundEffectOption } from "./BackgroundEffectGrid";
/** A background effect on offer, which the view names. */
export interface BackgroundEffectChoice {
/** Its option id, as `backgroundEffect` gives the one in force. */
id: string;
kind: "none" | "blur" | "image";
imageUrl?: string;
}
export interface AudioOutputSwitcher {
targetOutput: string;
@@ -56,20 +66,29 @@ export interface FooterActions {
toggleAudio: (() => void) | undefined;
/** Also controls if the videoMute button is disabled */
toggleVideo: (() => void) | undefined;
toggleBlur: (() => void) | undefined;
/** Undefined where background effects can't be chosen. */
selectBackgroundEffect: ((id: string) => void) | undefined;
toggleScreenSharing: (() => void) | undefined;
/** Also controls if the settings button is visible */
openSettings: (() => void) | undefined;
/** Also controls if the hangup button is visible */
hangup: (() => void) | undefined;
}
// we do not use any ? optional properties so that the vm type is including all fields.
export interface FooterState {
audioEnabled: boolean;
audioBusy: boolean;
videoEnabled: boolean;
videoBusy: boolean;
videoBlurEnabled: boolean;
/** The background effect in force, as its option id. */
backgroundEffect: string;
/** Every effect on offer, in the order they are shown. */
backgroundEffects: BackgroundEffectChoice[];
/** What the user is told about effects here, if anything. */
backgroundEffectNotice: "unavailable" | "slow" | undefined;
/** Whether the first effect chosen is still being prepared. */
backgroundEffectSettling: boolean;
showFooter: boolean;
/* This is needed for WindowMode = "flat" */
@@ -117,12 +136,14 @@ export interface FooterProps {
children?: JSX.Element | JSX.Element[] | false;
vm: ViewModel<FooterSnapshot>;
}
export const CallFooter: FC<FooterProps> = ({
className,
ref,
children,
vm,
}) => {
const { t } = useTranslation();
const asOverlay = useBehavior(vm.asOverlay$);
const showFooter = useBehavior(vm.showFooter$);
const hideControls = useBehavior(vm.hideControls$);
@@ -151,8 +172,13 @@ export const CallFooter: FC<FooterProps> = ({
const selectedAudioOutput = useBehavior(vm.selectedAudioOutput$);
const selectAudioOutputOption = useBehavior(vm.selectAudioOutputOption$);
const selectVideoButtonOption = useBehavior(vm.selectVideoButtonOption$);
const toggleBlur = useBehavior(vm.toggleBlur$);
const videoBlurEnabled = useBehavior(vm.videoBlurEnabled$);
const backgroundEffect = useBehavior(vm.backgroundEffect$);
const selectBackgroundEffect = useBehavior(vm.selectBackgroundEffect$);
const backgroundEffectNotice = useBehavior(vm.backgroundEffectNotice$);
const backgroundEffectSettling = useBehavior(vm.backgroundEffectSettling$);
const backgroundEffects = useBackgroundEffectLabels(
useBehavior(vm.backgroundEffects$),
);
const buttonSize = useBehavior(vm.buttonSize$);
const showLogo = useBehavior(vm.showLogo$);
@@ -214,8 +240,17 @@ export const CallFooter: FC<FooterProps> = ({
options={videoOptions}
selectedOption={selectedVideo}
onSelect={selectVideoButtonOption}
videoBlurToggleClick={toggleBlur}
videoBlurEnabled={videoBlurEnabled}
backgroundEffects={backgroundEffects}
selectedBackgroundEffect={backgroundEffect}
onSelectBackgroundEffect={selectBackgroundEffect}
backgroundEffectSettling={backgroundEffectSettling}
backgroundEffectNotice={
backgroundEffectNotice === "unavailable"
? t("background_effects.unavailable")
: backgroundEffectNotice === "slow"
? t("background_effects.slow")
: undefined
}
/>,
);
} else {
@@ -340,3 +375,21 @@ const TilesDebugInfo: FC<TilesDebugInfoProps> = ({ generation$ }) => {
const generation = useBehavior(generation$);
return `Tiles generation: ${generation}`;
};
function useBackgroundEffectLabels(
choices: BackgroundEffectChoice[],
): BackgroundEffectOption[] {
const { t } = useTranslation();
return useMemo(() => {
let pictures = 0;
return choices.map((choice) => ({
...choice,
label:
choice.kind === "none"
? t("background_effects.none")
: choice.kind === "blur"
? t("background_effects.blur")
: t("background_effects.numbered", { n: ++pictures }),
}));
}, [choices, t]);
}
+139 -5
View File
@@ -5,7 +5,7 @@ SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { describe, expect, it, vi } from "vitest";
import { afterEach, describe, expect, it, vi } from "vitest";
import { BehaviorSubject } from "rxjs";
import { testScope, mockMuteStates, mockMediaDevices } from "../utils/test";
@@ -14,8 +14,13 @@ import type { CallViewModel } from "../state/CallViewModel/CallViewModel";
import type { Alignment, Layout } from "../state/layout-types";
import type { SpotlightTileViewModel } from "../state/TileViewModel";
import type { DeviceLabel } from "../state/MediaDevices";
import { createCallFooterViewModel } from "./CallFooterViewModel";
import {
createCallFooterViewModel,
createLobbyFooterViewModel,
} from "./CallFooterViewModel";
import { HeaderStyle } from "../UrlParams";
import { backgroundEffect as backgroundEffectSetting } from "../settings/settings";
import { shippedBackgrounds } from "../tracks/backgroundEffects";
const platformMock = vi.hoisted(() => vi.fn(() => "desktop"));
vi.mock("../Platform", () => ({
@@ -24,10 +29,12 @@ vi.mock("../Platform", () => ({
},
}));
// Prevent supportsBackgroundProcessors from throwing in jsdom – it is not
// exercised by these tests (only used in `videoToggles`, not `videoOptions`).
// The SDK's own check needs WebGL and canvas APIs jsdom does not have.
const sdkSupportMock = vi.hoisted(() => vi.fn(() => false));
const modernRouteMock = vi.hoisted(() => vi.fn(() => true));
vi.mock("@livekit/track-processors", () => ({
supportsBackgroundProcessors: (): boolean => false,
supportsBackgroundProcessors: (): boolean => sdkSupportMock(),
supportsModernBackgroundProcessors: (): boolean => modernRouteMock(),
}));
const outputSelectionMock = vi.hoisted(() => vi.fn(() => true));
@@ -109,6 +116,7 @@ describe("createCallFooterViewModel", () => {
buildMinimalCallViewModel(gridLayout),
mockMuteStates(),
twoMicsAndOneCamMediaDevices,
constant(false),
/* reactionIdentifier */ undefined,
{ showControls: true, header: HeaderStyle.Standard },
);
@@ -151,6 +159,7 @@ describe("createCallFooterViewModel", () => {
select: vi.fn(),
},
}),
constant(false),
/* reactionIdentifier */ undefined,
{ showControls: true, header: HeaderStyle.Standard },
);
@@ -169,6 +178,7 @@ describe("createCallFooterViewModel", () => {
buildMinimalCallViewModel(layout),
mockMuteStates(),
twoMicsAndOneCamMediaDevices,
constant(false),
/* reactionIdentifier */ undefined,
{ showControls: true, header: HeaderStyle.Standard },
);
@@ -191,6 +201,7 @@ describe("createCallFooterViewModel", () => {
buildMinimalCallViewModel(gridLayout),
mockMuteStates(),
twoMicsAndOneCamMediaDevices,
constant(false),
/* reactionIdentifier */ undefined,
{ showControls: true, header: HeaderStyle.Standard },
);
@@ -222,4 +233,127 @@ describe("createCallFooterViewModel", () => {
]);
});
});
describe("background effects", () => {
afterEach(() => backgroundEffectSetting.setValue("none"));
// The lobby, because in a call the whole device switcher is already
// withheld on a phone, whatever the verdict says.
function lobbyOn(
platform: string,
): ReturnType<typeof createLobbyFooterViewModel> {
platformMock.mockReturnValue(platform);
return createLobbyFooterViewModel(
testScope(),
mockMuteStates(),
twoMicsAndOneCamMediaDevices,
constant(false),
/* openSettings */ undefined,
/* hangup */ undefined,
/* showLogo */ false,
);
}
it("offers them on a phone whose browser can run them", () => {
sdkSupportMock.mockReturnValue(true);
expect(lobbyOn("ios").selectBackgroundEffect$.value).toBeDefined();
});
it("offers them on a desktop whose browser can run them", () => {
sdkSupportMock.mockReturnValue(true);
expect(lobbyOn("desktop").selectBackgroundEffect$.value).toBeDefined();
});
it("offers nothing where the browser itself cannot run them", () => {
sdkSupportMock.mockReturnValue(false);
expect(lobbyOn("desktop").selectBackgroundEffect$.value).toBeUndefined();
});
it("puts no effect in force where the browser cannot run them", () => {
backgroundEffectSetting.setValue("blur");
sdkSupportMock.mockReturnValue(false);
expect(lobbyOn("desktop").backgroundEffect$.value).toBe("none");
sdkSupportMock.mockReturnValue(true);
expect(lobbyOn("desktop").backgroundEffect$.value).toBe("blur");
});
it("stores the effect chosen", () => {
sdkSupportMock.mockReturnValue(true);
const vm = lobbyOn("desktop");
vm.selectBackgroundEffect$.value?.("image:arc");
expect(vm.backgroundEffect$.value).toBe("image:arc");
// What isn't an effect on offer is stored as none.
vm.selectBackgroundEffect$.value?.("image:gone");
expect(backgroundEffectSetting.getValue()).toBe("none");
});
it("says they run slowly where only the slower route exists", () => {
sdkSupportMock.mockReturnValue(true);
modernRouteMock.mockReturnValue(false);
const slow = lobbyOn("desktop");
expect(slow.backgroundEffectNotice$.value).toBe("slow");
expect(slow.selectBackgroundEffect$.value).toBeDefined();
modernRouteMock.mockReturnValue(true);
expect(lobbyOn("desktop").backgroundEffectNotice$.value).toBeUndefined();
});
it("offers every effect in order", () => {
sdkSupportMock.mockReturnValue(true);
expect(lobbyOn("desktop").backgroundEffects$.value).toEqual([
{ id: "none", kind: "none" },
{ id: "blur", kind: "blur" },
...shippedBackgrounds.map((background) => ({
id: `image:${background.id}`,
kind: "image",
imageUrl: background.imagePath,
})),
]);
});
it("says they are unavailable where they cannot be chosen", () => {
sdkSupportMock.mockReturnValue(false);
expect(lobbyOn("desktop").backgroundEffectNotice$.value).toBe(
"unavailable",
);
});
it("passes on the wait for the first effect", () => {
platformMock.mockReturnValue("desktop");
const settling$ = new BehaviorSubject(true);
const vm = createLobbyFooterViewModel(
testScope(),
mockMuteStates(),
twoMicsAndOneCamMediaDevices,
settling$,
/* openSettings */ undefined,
/* hangup */ undefined,
/* showLogo */ false,
);
expect(vm.backgroundEffectSettling$.value).toBe(true);
settling$.next(false);
expect(vm.backgroundEffectSettling$.value).toBe(false);
});
it("availability is the same before and during a call", () => {
for (const supported of [true, false]) {
sdkSupportMock.mockReturnValue(supported);
const lobby = lobbyOn("desktop");
const inCall = createCallFooterViewModel(
testScope(),
buildMinimalCallViewModel(gridLayout),
mockMuteStates(),
twoMicsAndOneCamMediaDevices,
constant(false),
/* reactionIdentifier */ undefined,
{ showControls: true, header: HeaderStyle.Standard },
);
const offeredInLobby =
lobby.selectBackgroundEffect$.value !== undefined;
expect(inCall.selectBackgroundEffect$.value !== undefined).toBe(
offeredInLobby,
);
expect(offeredInLobby).toBe(supported);
}
});
});
});
+88 -19
View File
@@ -5,15 +5,30 @@ SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { combineLatest, map, type Observable, switchMap } from "rxjs";
import { supportsBackgroundProcessors } from "@livekit/track-processors";
import {
combineLatest,
distinctUntilChanged,
map,
type Observable,
switchMap,
} from "rxjs";
import { supportsAudioOutputSelection } from "livekit-client";
import {
supportsBackgroundProcessors,
supportsModernBackgroundProcessors,
} from "@livekit/track-processors";
import {
parseEffect,
serializeEffect,
shippedBackgrounds,
} from "../tracks/backgroundEffects";
import { type CallViewModel } from "../state/CallViewModel/CallViewModel";
import { type MenuOptions } from "./MediaMuteAndSwitchButton";
import { type MediaDevices } from "../state/MediaDevices";
import {
backgroundBlur as backgroundBlurSettings,
backgroundEffect as backgroundEffectSetting,
debugTileLayout as debugTileLayoutSetting,
} from "../settings/settings";
import { type Behavior, constant } from "../state/Behavior";
@@ -22,7 +37,7 @@ import { type MuteStates } from "../state/MuteStates";
import { createStaticViewModel, type ViewModel } from "../state/ViewModel";
import { HeaderStyle } from "../UrlParams";
import { platform } from "../Platform";
import { type FooterSnapshot } from "./CallFooter";
import { type BackgroundEffectChoice, type FooterSnapshot } from "./CallFooter";
/**
* Shared helper: maps MuteStates into the audio/video enabled + toggle behaviors
@@ -56,13 +71,14 @@ function buildMuteBehaviors(
/**
* Shared helper: maps MediaDevices into the audio/video device-list behaviors
* needed by FooterSnapshot (options, selection, callbacks, blur toggle).
* needed by FooterSnapshot (options, selection, callbacks, background effect).
*/
function buildDeviceBehaviors(
scope: ObservableScope,
mediaDevices: MediaDevices,
/** return empty arrays for audioOptions and videoOptions*/
disableSwitcher$: Behavior<boolean>,
backgroundEffectSettling$: Observable<boolean>,
): Pick<
ViewModel<FooterSnapshot>,
| "audioOptions$"
@@ -74,8 +90,11 @@ function buildDeviceBehaviors(
| "videoOptions$"
| "selectedVideo$"
| "selectVideoButtonOption$"
| "toggleBlur$"
| "videoBlurEnabled$"
| "backgroundEffect$"
| "selectBackgroundEffect$"
| "backgroundEffects$"
| "backgroundEffectNotice$"
| "backgroundEffectSettling$"
> {
const options$ = (
available$: Behavior<Map<string, MenuOptions["label"]>>,
@@ -92,6 +111,11 @@ function buildDeviceBehaviors(
),
);
const supported = supportsBackgroundProcessors();
const slow = !supportsModernBackgroundProcessors();
const offered$ = disableSwitcher$.pipe(
map((switcherDisabled) => !switcherDisabled && supported),
);
return {
audioOptions$: scope.behavior(options$(mediaDevices.audioInput.available$)),
selectedAudio$: scope.behavior(
@@ -116,18 +140,35 @@ function buildDeviceBehaviors(
mediaDevices.videoInput.selected$.pipe(map((s) => s?.id)),
),
selectVideoButtonOption$: constant(mediaDevices.videoInput.select),
toggleBlur$: scope.behavior(
combineLatest([backgroundBlurSettings.value$, disableSwitcher$]).pipe(
map(([current, switcherDisabled]) => {
return !switcherDisabled && supportsBackgroundProcessors()
? (): void => {
backgroundBlurSettings.setValue(!current);
}
: undefined;
}),
backgroundEffect$: scope.behavior(
backgroundEffectSetting.value$.pipe(
map((raw) => (supported ? serializeEffect(parseEffect(raw)) : "none")),
),
),
videoBlurEnabled$: backgroundBlurSettings.value$,
selectBackgroundEffect$: scope.behavior(
offered$.pipe(
map((offered) =>
offered
? (id: string): void =>
backgroundEffectSetting.setValue(
serializeEffect(parseEffect(id)),
)
: undefined,
),
),
),
backgroundEffects$: constant(backgroundEffectChoices()),
backgroundEffectNotice$: scope.behavior(
offered$.pipe(
map((offered) =>
!offered ? "unavailable" : slow ? "slow" : undefined,
),
),
),
backgroundEffectSettling$: scope.behavior(
backgroundEffectSettling$.pipe(distinctUntilChanged()),
false,
),
};
}
@@ -138,6 +179,8 @@ function buildDeviceBehaviors(
* @param callModel - The root CallViewModel; provides layout, grid mode, reactions, etc.
* @param muteStates - Audio and video mute state + toggles.
* @param mediaDevices - Available and selected input devices.
* @param backgroundEffectSettling$ - Whether the first effect chosen is still
* being prepared.
* @param reactionIdentifier - The local user's reaction identifier string, or
* undefined when reactions are not supported (hides the reaction button).
* @param options - `showControls`: whether the call controls should be shown.
@@ -148,6 +191,7 @@ export function createCallFooterViewModel(
callModel: CallViewModel,
muteStates: MuteStates,
mediaDevices: MediaDevices,
backgroundEffectSettling$: Observable<boolean>,
reactionIdentifier: string | undefined,
options: { showControls: boolean; header: HeaderStyle },
): ViewModel<FooterSnapshot> {
@@ -162,7 +206,12 @@ export function createCallFooterViewModel(
);
return {
...buildMuteBehaviors(scope, muteStates),
...buildDeviceBehaviors(scope, mediaDevices, disableDeviceSwitcher$),
...buildDeviceBehaviors(
scope,
mediaDevices,
disableDeviceSwitcher$,
backgroundEffectSettling$,
),
// candidat to move into the FooterViewModel
showFooter$: callModel.showFooter$,
hideControls$: constant(!showControls),
@@ -224,6 +273,8 @@ export function createCallFooterViewModel(
* @param scope - ObservableScope that bounds the lifetime of derived behaviors.
* @param muteStates - Audio and video mute state + toggles.
* @param mediaDevices - Available and selected input devices.
* @param backgroundEffectSettling$ - Whether the first effect chosen is still
* being prepared.
* @param openSettings - Callback to open the settings modal, or undefined.
* @param hangup - Callback to leave/cancel, or undefined (hides the button).
* @param showLogo - Whether to show the Element Call logo.
@@ -232,6 +283,7 @@ export function createLobbyFooterViewModel(
scope: ObservableScope,
muteStates: MuteStates,
mediaDevices: MediaDevices,
backgroundEffectSettling$: Observable<boolean>,
openSettings: (() => void) | undefined,
hangup: (() => void) | undefined,
showLogo: boolean,
@@ -275,6 +327,23 @@ export function createLobbyFooterViewModel(
selectVideoButtonOption: undefined,
}),
...buildMuteBehaviors(scope, muteStates),
...buildDeviceBehaviors(scope, mediaDevices, constant(false)),
...buildDeviceBehaviors(
scope,
mediaDevices,
constant(false),
backgroundEffectSettling$,
),
};
}
function backgroundEffectChoices(): BackgroundEffectChoice[] {
return [
{ id: serializeEffect({ kind: "none" }), kind: "none" },
{ id: serializeEffect({ kind: "blur" }), kind: "blur" },
...shippedBackgrounds.map((background) => ({
id: serializeEffect({ kind: "shipped", id: background.id }),
kind: "image" as const,
imageUrl: background.imagePath,
})),
];
}
@@ -36,20 +36,19 @@ Please see LICENSE in the repository root for full details.
}
}
/* The width design sets: a long device name wraps. */
.menu {
display: flex;
flex-direction: column;
inline-size: 296px;
}
/* Only the device lists scroll, bounded by the measured call area: the menu is
portalled out of the root, so CSS can't size it against the call. */
/* Its share of the measured call area: the menu is portalled out of the root,
so CSS can't size it against the call. */
.deviceList {
overflow-y: auto;
min-block-size: 0;
max-block-size: var(--device-list-max-height);
/* Keeps a row reached by keyboard clear of the sticky heading and meter. */
scroll-padding-block: var(--device-list-scroll-padding-start, 0)
var(--device-list-scroll-padding-end, 0);
/* The raised headings stay raised within the list, under the menu's frame. */
isolation: isolate;
/* Keeps a row reached by keyboard clear of the sticky heading. */
scroll-padding-block-start: var(--device-list-scroll-padding-start, 0);
}
/* Compound's heading rule heads each section; the design adds space below it. */
@@ -62,23 +61,40 @@ Please see LICENSE in the repository root for full details.
margin-block-start: var(--cpd-space-7x);
}
/* Sticky within its own section. Opaque and inset a border width so it doesn't
paint over the menu's outline. */
/* Sticky within its own section: opaque, so rows scroll under it, and raised,
or the positioned tiles scroll over it. */
.sectionHeading {
position: sticky;
z-index: 1;
inset-block-start: 0;
background: var(--cpd-color-bg-canvas-default);
margin-inline: var(--cpd-border-width-1);
margin-block-start: var(--cpd-border-width-1);
}
/* Sticky at the foot of the microphone section; opaque and inset as above. */
/* Sticky at the foot of the microphone section; opaque as above. */
.stickyMeter {
position: sticky;
inset-block-end: 0;
background: var(--cpd-color-bg-canvas-default);
margin-inline: var(--cpd-border-width-1);
margin-block-end: var(--cpd-border-width-1);
}
/* Fills the band between the meter and the frame, or rows show through it. */
.stickyMeter::after {
content: "";
position: absolute;
inset-inline: 0;
inset-block-start: 100%;
block-size: var(--cpd-menu-padding-block-end, 0px);
background: var(--cpd-color-bg-canvas-default);
}
/* Not Compound's fade: that stays at the edge while a bounce carries the meter.
This one starts at the chevron, below the meter at rest. */
.deviceList.deviceListWithMeter[data-more-below] {
mask-image: linear-gradient(
to bottom,
#000 calc(100% - var(--cpd-space-6x)),
transparent calc(100% - var(--cpd-space-3x))
);
}
/* Radix focuses whatever the pointer is over, so the browser's ring would
@@ -92,3 +108,20 @@ Please see LICENSE in the repository root for full details.
outline: var(--cpd-border-width-2) solid var(--cpd-color-border-focused);
outline-offset: calc(-1 * var(--cpd-border-width-2));
}
.notice {
display: flex;
align-items: flex-start;
gap: var(--cpd-space-3x);
margin: var(--cpd-space-2x) var(--cpd-space-4x) 0;
padding: var(--cpd-space-3x);
border-radius: var(--cpd-space-2x);
background: var(--cpd-color-bg-subtle-secondary);
color: var(--cpd-color-text-primary);
font: var(--cpd-font-body-md-regular);
}
.notice > svg {
flex-shrink: 0;
color: var(--cpd-color-icon-secondary);
}
@@ -11,6 +11,8 @@ import { useEffect, useState, type FC, type JSX, type ReactNode } from "react";
import type { Meta, StoryObj } from "@storybook/react-vite";
import { MediaMuteAndSwitchButton } from "./MediaMuteAndSwitchButton";
import styles from "./MediaMuteAndSwitchButton.module.css";
import { type BackgroundEffectOption } from "./BackgroundEffectGrid";
import { shippedBackgrounds } from "../tracks/backgroundEffects";
import meterStyles from "./MicrophoneLevelMeter.module.css";
import { MediaDevicesContext } from "../MediaDevicesContext";
import { RootElementProvider } from "../RootElementContext";
@@ -59,14 +61,19 @@ const WithAMicrophone: FC<{ children: ReactNode }> = ({ children }) => {
};
/** Supplies a call-sized root. Without one the list is bounded by the whole Storybook frame. */
const WithACallArea: FC<{ children: ReactNode }> = ({ children }) => {
const WithACallArea: FC<{ children: ReactNode; blockSize?: number }> = ({
children,
// A call's size: anything smaller makes a short device list scroll, and a
// narrower one narrows the menu.
blockSize = 720,
}) => {
const [callArea, setCallArea] = useState<HTMLElement | null>(null);
return (
<div
ref={setCallArea}
style={{
// A call's size: anything smaller makes a short device list scroll.
blockSize: 720,
blockSize,
inlineSize: 1024,
display: "flex",
alignItems: "flex-end",
justifyContent: "center",
@@ -127,8 +134,6 @@ export const AudioMute: Story = {
{ label: { type: "name", name: "Microphone 1" }, id: "1" },
{ label: { type: "name", name: "Microphone 2" }, id: "2" },
],
videoBlurEnabled: true,
videoBlurToggleClick: fn(),
selectedOption: "2",
},
play: async ({ args, canvasElement }) => {
@@ -167,6 +172,17 @@ export const VideoMute: Story = {
},
};
const effects: BackgroundEffectOption[] = [
{ id: "none", kind: "none", label: "None" },
{ id: "blur", kind: "blur", label: "Blur" },
...shippedBackgrounds.map((background, i) => ({
id: `image:${background.id}`,
kind: "image" as const,
label: `Background ${i + 1}`,
imageUrl: background.imagePath,
})),
];
export const VideoUnmute: Story = {
args: {
iconsAndLabels: "video",
@@ -175,9 +191,210 @@ export const VideoUnmute: Story = {
{ label: { type: "name", name: "Camera 1" }, id: "1" },
{ label: { type: "name", name: "Camera 2" }, id: "2" },
],
videoBlurEnabled: true,
videoBlurToggleClick: fn(),
selectedOption: "2",
backgroundEffects: effects,
selectedBackgroundEffect: "none",
onSelectBackgroundEffect: fn(),
},
};
/** One choice among no effect, blur and the shipped images, under the cameras. */
export const BackgroundEffects: Story = {
args: VideoUnmute.args,
play: async ({ args, canvasElement }) => {
const canvas = within(canvasElement);
await userEvent.click(canvas.getByRole("button", { name: "Camera" }));
const menu = within(document.body);
const section = await menu.findByRole("group", {
name: "Background effects",
});
const tiles = within(section).getAllByRole("menuitemradio");
await expect(tiles.map((tile) => tile.textContent)).toEqual([
"None",
"Blur",
"Background 1",
"Background 2",
]);
await expect(
within(section).getByRole("menuitemradio", { checked: true }),
).toHaveTextContent("None");
const cameras = menu.getByRole("group", { name: "Camera" });
await expect(
cameras.compareDocumentPosition(section) &
Node.DOCUMENT_POSITION_FOLLOWING,
).toBeTruthy();
const [first, second, third, fourth] = tiles.map((tile) =>
tile.getBoundingClientRect(),
);
await expect(second.top).toBe(first.top);
await expect(third.top).toBe(first.top);
await expect(fourth.top).toBeGreaterThan(first.bottom);
const frame = document.body
.querySelector("[role='menu']")!
.getBoundingClientRect();
for (const tile of [first, second, third])
await expect(tile.left >= frame.left && tile.right <= frame.right).toBe(
true,
);
await userEvent.click(menu.getByRole("menuitemradio", { name: "Blur" }));
await expect(args.onSelectBackgroundEffect).toHaveBeenCalledWith("blur");
await expect(menu.getByRole("menu")).toBeVisible();
},
};
/** Where effects can't run: shown, disabled and explained, except no effect. */
export const BackgroundEffectsUnavailable: Story = {
args: {
...VideoUnmute.args,
onSelectBackgroundEffect: undefined,
backgroundEffectNotice:
"Background effects are not supported on this platform.",
},
play: async ({ canvasElement }) => {
const canvas = within(canvasElement);
await userEvent.click(canvas.getByRole("button", { name: "Camera" }));
const menu = within(document.body);
const section = await menu.findByRole("group", {
name: "Background effects",
});
await expect(section).toHaveAccessibleDescription(
"Background effects are not supported on this platform.",
);
for (const tile of within(section).getAllByRole("menuitemradio"))
if (tile.textContent === "None") await expect(tile).toBeEnabled();
else await expect(tile).toBeDisabled();
},
};
/** Where only the slower route exists: offered and choosable, with the cost said. */
export const BackgroundEffectsSlowInThisBrowser: Story = {
args: {
...VideoUnmute.args,
backgroundEffectNotice:
"Background effects run slowly on this platform, which may cause your video to stutter.",
},
play: async ({ args, canvasElement }) => {
const canvas = within(canvasElement);
await userEvent.click(canvas.getByRole("button", { name: "Camera" }));
const menu = within(document.body);
const section = await menu.findByRole("group", {
name: "Background effects",
});
await expect(section).toHaveAccessibleDescription(
"Background effects run slowly on this platform, which may cause your video to stutter.",
);
for (const tile of within(section).getAllByRole("menuitemradio"))
await expect(tile).not.toHaveAttribute("aria-disabled");
await userEvent.click(
within(section).getByRole("menuitemradio", { name: "Background 1" }),
);
await expect(args.onSelectBackgroundEffect).toHaveBeenCalledWith(
"image:arc",
);
const notice = document.body.querySelector(`.${styles.notice}`)!;
const frame = document.body
.querySelector("[role='menu']")!
.getBoundingClientRect();
await expect(frame.width).toBeLessThanOrEqual(296);
await expect(notice.getBoundingClientRect().right).toBeLessThanOrEqual(
frame.right,
);
},
};
/** The first effect of a session shows its wait on its own tile. */
export const BackgroundEffectsSettling: Story = {
args: {
...VideoUnmute.args,
selectedBackgroundEffect: "blur",
backgroundEffectSettling: true,
},
play: async ({ canvasElement }) => {
const canvas = within(canvasElement);
await userEvent.click(canvas.getByRole("button", { name: "Camera" }));
const menu = within(document.body);
const chosen = await menu.findByRole("menuitemradio", {
name: "Blur",
checked: true,
});
await expect(chosen).toHaveAttribute("aria-busy", "true");
const marks = chosen.querySelectorAll("svg");
await expect(marks).toHaveLength(1);
await expect(getComputedStyle(marks[0]).animationName).not.toBe("none");
const section = menu.getByRole("group", { name: "Background effects" });
for (const other of within(section).getAllByRole("menuitemradio", {
checked: false,
}))
await expect(other).toHaveAttribute("aria-busy", "false");
},
};
/** In a short call the effects scroll into view with the list. */
export const BackgroundEffectsScrollWhenTheyDoNotFit: Story = {
args: {
...VideoUnmute.args,
// It sits with the effects, so it must fit the same share of the call.
backgroundEffectNotice:
"Background effects run slowly on this platform, which may cause your video to stutter.",
},
decorators: [
(Story): JSX.Element => (
<WithACallArea blockSize={300}>
<Story />
</WithACallArea>
),
],
play: async ({ canvasElement }) => {
const canvas = within(canvasElement);
await userEvent.click(canvas.getByRole("button", { name: "Camera" }));
const menu = within(document.body);
const last = await menu.findByRole("menuitemradio", {
name: "Background 2",
});
const list = document.body.querySelector<HTMLElement>(
`.${styles.deviceList}`,
)!;
await expect(list.scrollHeight).toBeGreaterThan(list.clientHeight);
await expect(last.getBoundingClientRect().top).toBeGreaterThan(
list.getBoundingClientRect().bottom,
);
// Scrolled under the section's heading, the tiles pass beneath it.
const heading = [
...document.body.querySelectorAll<HTMLElement>(
`.${styles.sectionHeading}`,
),
].find((h) => h.textContent === "Background effects")!;
list.scrollTop = heading.offsetTop + 30;
await waitFor(async () => {
const box = heading.getBoundingClientRect();
const drawn = document.elementFromPoint(
box.left + box.width / 2,
box.bottom - 2,
);
await expect(heading.contains(drawn)).toBe(true);
});
list.scrollTop = 0;
for (let i = 0; i < 10 && document.activeElement !== last; i++)
await userEvent.keyboard("{ArrowDown}");
await expect(document.activeElement).toBe(last);
const scrollport = list.getBoundingClientRect();
const reached = last.getBoundingClientRect();
await expect(reached.bottom).toBeLessThanOrEqual(scrollport.bottom + 1);
await expect(reached.top).toBeGreaterThanOrEqual(scrollport.top - 1);
const callArea = canvasElement
.querySelector<HTMLElement>("[style*='block-size: 300px']")!
.getBoundingClientRect();
const frame = document.body
.querySelector("[role='menu']")!
.getBoundingClientRect();
await expect(frame.top).toBeGreaterThanOrEqual(callArea.top);
await expect(frame.bottom).toBeLessThanOrEqual(callArea.bottom);
},
};
@@ -256,6 +473,37 @@ export const SpeakerAndMicrophoneSections: Story = {
},
};
/** A long name wraps rather than widening the menu, which design sets. */
export const LongDeviceNameWraps: Story = {
args: {
...SpeakerAndMicrophoneSections.args,
outputOptions: [
{
label: {
type: "name",
name: "Logitech BRIO 4K Ultra HD Pro Business Webcam Speakers (046d:085e)",
},
id: "long",
},
{ label: { type: "name", name: "Headset" }, id: "spk2" },
],
selectedOutputOption: "long",
},
play: async ({ canvasElement }) => {
const canvas = within(canvasElement);
await userEvent.click(canvas.getByRole("button", { name: "Microphone" }));
const body = within(document.body);
const long = await body.findByRole("menuitemradio", { name: /Logitech/ });
const short = await body.findByRole("menuitemradio", { name: "Headset" });
const menu = document.body.querySelector("[role='menu']")!;
await expect(Math.round(menu.getBoundingClientRect().width)).toBe(296);
await expect(long.getBoundingClientRect().height).toBeGreaterThan(
short.getBoundingClientRect().height,
);
},
};
export const OutputCannotBeChosen: Story = {
args: {
...Default.args,
@@ -436,10 +684,6 @@ export const ManyDevices: Story = {
.getBoundingClientRect();
await expect(frame.top).toBeGreaterThanOrEqual(0);
await expect(frame.bottom).toBeLessThanOrEqual(window.innerHeight + 1);
// The meter is the menu's one opaque part, so it must stay inside the frame.
await expect(pinned.left).toBeGreaterThan(frame.left);
await expect(pinned.right).toBeLessThan(frame.right);
},
};
@@ -607,13 +851,6 @@ export const HeadingsStayWhileScrolling: Story = {
await expect(heading.getBoundingClientRect().bottom).toBeGreaterThan(
scrollport.top,
);
// Clear of the menu's frame.
const frame = document.body
.querySelector("[role='menu']")!
.getBoundingClientRect();
await expect(heading.getBoundingClientRect().left).toBeGreaterThan(
frame.left,
);
},
};
@@ -628,31 +865,23 @@ function overlapping(element: Element, overlays: Element[]): number {
}, 0);
}
/** The blur toggle gets the keyboard ring too: it is the menu's child, not the list's. */
export const FocusRingCoversTheBlurToggle: Story = {
args: {
...VideoUnmute.args,
iconsAndLabels: "video",
videoBlurEnabled: false,
videoBlurToggleClick: fn(),
},
/** The effect tiles get the keyboard ring the device rows get. */
export const FocusRingCoversTheEffectTiles: Story = {
args: VideoUnmute.args,
play: async ({ canvasElement }) => {
const canvas = within(canvasElement);
await userEvent.click(canvas.getByRole("button", { name: "Camera" }));
const toggle = await within(document.body).findByRole("menuitemcheckbox", {
name: /Blur background/,
const tile = await within(document.body).findByRole("menuitemradio", {
name: "Blur",
});
// Arrow down past the cameras to the toggle.
for (let i = 0; i < 6 && document.activeElement !== toggle; i++)
for (let i = 0; i < 6 && document.activeElement !== tile; i++)
await userEvent.keyboard("{ArrowDown}");
await expect(document.activeElement).toBe(toggle);
// The same ring the device rows get.
await expect(outlineWidth(toggle)).toBeGreaterThan(0);
await expect(document.activeElement).toBe(tile);
await expect(outlineWidth(tile)).toBeGreaterThan(0);
// And the pointer takes it away, with the toggle still focused.
await userEvent.hover(toggle);
await expect(document.activeElement).toBe(toggle);
await expect(outlineWidth(toggle)).toBe(0);
await userEvent.hover(tile);
await expect(document.activeElement).toBe(tile);
await expect(outlineWidth(tile)).toBe(0);
},
};
@@ -9,6 +9,7 @@ import { afterEach, describe, expect, test, vi } from "vitest";
import { axe } from "vitest-axe";
import {
act,
fireEvent,
render,
screen,
within,
@@ -22,10 +23,18 @@ import {
MediaMuteAndSwitchButton,
type MenuOptions,
} from "./MediaMuteAndSwitchButton";
import { type BackgroundEffectOption } from "./BackgroundEffectGrid";
import { MediaDevicesContext } from "../MediaDevicesContext";
import { type MediaDevices } from "../state/MediaDevices";
import { restoreAudioCapture, stubAudioCapture } from "../utils/test";
const effects: BackgroundEffectOption[] = [
{ id: "none", kind: "none", label: "None" },
{ id: "blur", kind: "blur", label: "Blur" },
{ id: "image:arc", kind: "image", label: "Background 1", imageUrl: "" },
{ id: "image:glow", kind: "image", label: "Background 2", imageUrl: "" },
];
interface RenderOptions {
requestDeviceNames: () => void;
}
@@ -305,30 +314,88 @@ describe("MediaMuteAndSwitchButton", () => {
expect(mic2ItemAfter.querySelector(".rotate")).toBeNull();
});
test("renders menu with toggle control and calls toggle callback", async () => {
test("background effects are one keyboard-operable labelled choice", async () => {
const user = userEvent.setup();
const onSelect = vi.fn();
const onVideoBlurToggle = vi.fn();
const onSelectBackgroundEffect = vi.fn();
const { getByRole } = renderComponent(
<MediaMuteAndSwitchButton
iconsAndLabels="video"
enabled={true}
videoBlurToggleClick={onVideoBlurToggle}
onSelect={onSelect}
options={[{ label: { type: "name", name: "Camera 1" }, id: "cam1" }]}
selectedOption="cam1"
onSelect={vi.fn()}
backgroundEffects={effects}
selectedBackgroundEffect="none"
onSelectBackgroundEffect={onSelectBackgroundEffect}
/>,
);
await user.click(getByRole("button", { name: "Camera" }));
const toggle = screen.getByRole("menuitemcheckbox", {
name: "Blur background",
const section = screen.getByRole("group", { name: "Background effects" });
within(section).getByRole("menuitemradio", { name: "None", checked: true });
within(section).getByRole("menuitemradio", {
name: "Background 1",
checked: false,
});
expect(toggle).toBeInTheDocument();
expect(toggle).toHaveAttribute("aria-checked", "false");
await user.click(toggle);
const blur = within(section).getByRole("menuitemradio", { name: "Blur" });
for (let i = 0; i < 6 && document.activeElement !== blur; i++)
await user.keyboard("{ArrowDown}");
expect(document.activeElement).toBe(blur);
await user.keyboard("{Enter}");
expect(onSelectBackgroundEffect).toHaveBeenCalledWith("blur");
expect(screen.getByRole("menu")).toBeInTheDocument();
});
expect(onVideoBlurToggle).toHaveBeenCalled();
test("does not choose the effect already in force again", async () => {
const user = userEvent.setup();
const onSelectBackgroundEffect = vi.fn();
const { getByRole } = renderComponent(
<MediaMuteAndSwitchButton
iconsAndLabels="video"
enabled={true}
options={[{ label: { type: "name", name: "Camera 1" }, id: "cam1" }]}
selectedOption="cam1"
backgroundEffects={effects}
selectedBackgroundEffect="blur"
onSelectBackgroundEffect={onSelectBackgroundEffect}
/>,
);
await user.click(getByRole("button", { name: "Camera" }));
await user.click(screen.getByRole("menuitemradio", { name: "Blur" }));
expect(onSelectBackgroundEffect).not.toHaveBeenCalled();
});
test("offers the background effects in a phone's drawer", async () => {
const userAgent = vi
.spyOn(navigator, "userAgent", "get")
.mockReturnValue("Mozilla/5.0 (Linux; Android 14)");
try {
const user = userEvent.setup();
const onSelectBackgroundEffect = vi.fn();
const { getByRole } = renderComponent(
<MediaMuteAndSwitchButton
iconsAndLabels="video"
enabled={true}
options={[{ label: { type: "name", name: "Camera 1" }, id: "cam1" }]}
selectedOption="cam1"
backgroundEffects={effects}
selectedBackgroundEffect="none"
onSelectBackgroundEffect={onSelectBackgroundEffect}
/>,
);
await user.click(getByRole("button", { name: "Camera" }));
const section = screen.getByRole("group", { name: "Background effects" });
// A click alone: the drawer's drag handling reads layout jsdom lacks.
fireEvent.click(
within(section).getByRole("menuitemradio", { name: "Background 2" }),
);
expect(onSelectBackgroundEffect).toHaveBeenCalledWith("image:glow");
} finally {
userAgent.mockRestore();
}
});
test("marks the selected menu item as checked", async () => {
@@ -552,7 +619,9 @@ describe("MediaMuteAndSwitchButton", () => {
]}
selectedOption="cam1"
onSelect={vi.fn()}
videoBlurToggleClick={vi.fn()}
backgroundEffects={effects}
selectedBackgroundEffect="none"
onSelectBackgroundEffect={vi.fn()}
/>,
);
@@ -560,9 +629,10 @@ describe("MediaMuteAndSwitchButton", () => {
screen.getByRole("menuitemradio", { name: "Camera 1", checked: true });
screen.getByRole("menuitemradio", { name: "Camera 2", checked: false });
expect(
screen.getByRole("menuitemcheckbox", { name: "Blur background" }),
).toBeInTheDocument();
within(screen.getByRole("group", { name: "Background effects" })).getByRole(
"menuitemradio",
{ name: "Blur" },
);
});
test("marks focus as keyboard-driven only when the keyboard moved it", async () => {
+64 -36
View File
@@ -7,6 +7,7 @@ Please see LICENSE in the repository root for full details.
import {
useCallback,
useId,
useState,
type CSSProperties,
type FC,
@@ -17,13 +18,14 @@ import {
Button,
Menu,
MenuItem,
MenuScrollArea,
MenuTitle,
RadioInput,
ToggleMenuItem,
} from "@vector-im/compound-web";
import {
ChevronUpIcon,
ChevronDownIcon,
InfoIcon,
SpinnerIcon,
} from "@vector-im/compound-design-tokens/assets/web/icons";
import classNames from "classnames";
@@ -40,6 +42,10 @@ import { useMediaDevices } from "../MediaDevicesContext";
import { useRootElement } from "../RootElementContext";
import { observeElementSize$ } from "../utils/elementSize";
import { LiveMicrophoneLevelMeter } from "./MicrophoneLevelMeter";
import {
BackgroundEffectGrid,
type BackgroundEffectOption,
} from "./BackgroundEffectGrid";
export interface MenuOptions {
label: DeviceLabel | AudioOutputDeviceLabel;
@@ -64,8 +70,15 @@ export interface MediaMuteAndSwitchButtonProps {
selectedOutputOption?: string;
/** Picks an output device. Undefined disables the speaker section. */
onSelectOutput?: (id: string) => void;
videoBlurToggleClick?: () => void;
videoBlurEnabled?: boolean;
/** Camera menu only. None or an empty list leaves the section out. */
backgroundEffects?: BackgroundEffectOption[];
selectedBackgroundEffect?: string;
/** Undefined where effects can't run: all but no effect are disabled. */
onSelectBackgroundEffect?: (id: string) => void;
/** What the user should know before choosing an effect, if anything. */
backgroundEffectNotice?: string;
/** Whether the first effect chosen is still being prepared. */
backgroundEffectSettling?: boolean;
/**
* For any toggle and option this method will be called.
* So toggles need to be implemented by listening here and setting the right toggle item to `enabled`
@@ -73,8 +86,6 @@ export interface MediaMuteAndSwitchButtonProps {
onSelect?: (id: string) => void;
}
const BLUR_ID = "blur";
/** Id of the placeholder "Default" row, shown when the platform lists no outputs. */
const DEFAULT_OUTPUT_ID = "default";
@@ -84,6 +95,9 @@ const LIST_SHARE_OF_CALL = 0.6;
/** Smallest device list height in px, so a short call still shows more than one device. */
const MIN_LIST_HEIGHT = 160;
/** Space kept between the menu and the call area's sides. */
const MENU_MARGIN = 16;
export const MediaMuteAndSwitchButton: FC<MediaMuteAndSwitchButtonProps> = ({
enabled,
busy,
@@ -94,8 +108,11 @@ export const MediaMuteAndSwitchButton: FC<MediaMuteAndSwitchButtonProps> = ({
outputOptions,
selectedOutputOption,
onSelectOutput,
videoBlurEnabled,
videoBlurToggleClick,
backgroundEffects,
selectedBackgroundEffect,
onSelectBackgroundEffect,
backgroundEffectNotice,
backgroundEffectSettling,
onSelect,
}) => {
// Requested but not yet selected. Keyed by kind too, since Chrome uses
@@ -113,6 +130,7 @@ export const MediaMuteAndSwitchButton: FC<MediaMuteAndSwitchButtonProps> = ({
const isBusy = busy ?? false;
const { t } = useTranslation();
const devices = useMediaDevices();
const noticeId = useId();
/**
* Records on the menu whether the keyboard or the pointer moved focus, for
@@ -185,16 +203,8 @@ export const MediaMuteAndSwitchButton: FC<MediaMuteAndSwitchButtonProps> = ({
/>
);
const toggles =
iconsAndLabels === "video" && videoBlurToggleClick !== undefined
? [
{
label: t("action.blur_background"),
enabled: videoBlurEnabled ?? false,
id: BLUR_ID,
},
]
: [];
const showEffects =
iconsAndLabels === "video" && (backgroundEffects?.length ?? 0) > 0;
let optionsButtonLabel: string;
let menuTitle: string;
@@ -326,6 +336,8 @@ export const MediaMuteAndSwitchButton: FC<MediaMuteAndSwitchButtonProps> = ({
open={menuOpen}
onOpenChange={onOpenChange}
side="top"
collisionBoundary={rootElement}
collisionPadding={MENU_MARGIN}
trigger={
<Button
iconOnly
@@ -340,18 +352,22 @@ export const MediaMuteAndSwitchButton: FC<MediaMuteAndSwitchButtonProps> = ({
/>
}
>
<div
<MenuScrollArea
ref={trackFocusSource}
// Keeps the items the menu's own children for assistive tech.
role="none"
className={styles.deviceList}
className={classNames(styles.deviceList, {
[styles.deviceListWithMeter]: iconsAndLabels === "audio",
})}
data-testid="device-list"
style={
{
"--device-list-max-height":
listMaxHeight === undefined ? undefined : `${listMaxHeight}px`,
"--device-list-scroll-padding-end":
meterHeight === undefined ? undefined : `${meterHeight}px`,
// A row reached by keyboard stays above the meter and the band its
// ground fills.
scrollPaddingBlockEnd:
meterHeight === undefined
? undefined
: `calc(${meterHeight}px + var(--cpd-menu-padding-block-end, 0px))`,
"--device-list-scroll-padding-start":
headingHeight === undefined ? undefined : `${headingHeight}px`,
} as CSSProperties
@@ -399,19 +415,31 @@ export const MediaMuteAndSwitchButton: FC<MediaMuteAndSwitchButtonProps> = ({
)}
</div>
</div>
</div>
{toggles.length > 0 && <hr />}
{toggles.map((toggle) => (
<ToggleMenuItem
label={toggle.label}
onSelect={(e) => {
videoBlurToggleClick?.();
e.preventDefault();
}}
checked={toggle.enabled}
key={toggle.id}
/>
))}
{showEffects && (
<BackgroundEffectGrid
label={t("settings.background_effects_header")}
heading={
<div aria-hidden className={styles.sectionHeading}>
<MenuTitle title={t("settings.background_effects_header")} />
</div>
}
effects={backgroundEffects ?? []}
selected={selectedBackgroundEffect}
onSelect={onSelectBackgroundEffect}
settling={backgroundEffectSettling}
describedBy={
backgroundEffectNotice === undefined ? undefined : noticeId
}
/>
)}
{/* In the list, so the menu grows no taller for it. */}
{showEffects && backgroundEffectNotice !== undefined && (
<div id={noticeId} role="none" className={styles.notice}>
<InfoIcon width={20} height={20} />
<span>{backgroundEffectNotice}</span>
</div>
)}
</MenuScrollArea>
</Menu>
</div>
);
@@ -16,10 +16,10 @@ Please see LICENSE in the repository root for full details.
align-items: center;
gap: var(--cpd-space-3x);
padding-block: var(--cpd-space-2x);
/* Centres the icon on the radio column: less the row's inset border and half
the icon's extra width. Checked in the MeterAlignsWithTheDeviceRows story. */
/* Centres the icon on the radio column: less half the icon's extra width.
Checked in the MeterAlignsWithTheDeviceRows story. */
padding-inline-start: calc(
var(--cpd-space-4x) - var(--cpd-border-width-1) -
var(--cpd-space-4x) -
(var(--meter-icon-size) - var(--device-control-size)) / 2
);
padding-inline-end: calc(var(--cpd-space-4x) * 2 + var(--cpd-space-2x));
@@ -9,7 +9,7 @@ exports[`MediaMuteAndSwitchButton > renders 1`] = `
aria-busy="false"
aria-checked="false"
aria-disabled="true"
aria-labelledby="_r_0_"
aria-labelledby="_r_1_"
class="_button_1nw83_8 _has-icon_1nw83_60 _icon-only_1nw83_53"
data-kind="primary"
data-size="lg"
@@ -39,7 +39,7 @@ exports[`MediaMuteAndSwitchButton > renders 1`] = `
data-kind="tertiary"
data-size="lg"
data-state="closed"
id="radix-_r_5_"
id="radix-_r_6_"
role="button"
tabindex="0"
type="button"
Binary file not shown.

After

Width:  |  Height:  |  Size: 27 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 22 KiB

+9 -3
View File
@@ -50,6 +50,7 @@ import {
mockMediaDevices,
mockRtcMembership,
MockRTCSession,
testScope,
} from "../utils/test";
import { CallView } from "./CallView";
import { GroupCallErrorBoundary } from "./GroupCallErrorBoundary";
@@ -61,7 +62,10 @@ import {
} from "../HostBridge";
import { type JoinCallData } from "../widget";
import { MatrixRTCTransportMissingError } from "../utils/errors";
import { ProcessorProvider } from "../tracks/TrackProcessorContext";
import {
BackgroundEffectsProvider,
createAppBackgroundEffects,
} from "../tracks/BackgroundEffectsContext";
import { MediaDevicesContext } from "../MediaDevicesContext";
import { constant } from "../state/Behavior";
@@ -193,7 +197,9 @@ function createCallView(
<HostBridgeProvider value={hostBridge}>
<TooltipProvider>
<MediaDevicesContext value={mockMediaDevices({})}>
<ProcessorProvider>
<BackgroundEffectsProvider
effects={createAppBackgroundEffects(testScope())}
>
{options.withErrorBoundary ? (
<GroupCallErrorBoundary recoveryActionHandler={vi.fn()}>
{callView}
@@ -201,7 +207,7 @@ function createCallView(
) : (
callView
)}
</ProcessorProvider>
</BackgroundEffectsProvider>
</MediaDevicesContext>
</TooltipProvider>
</HostBridgeProvider>
+13 -5
View File
@@ -28,6 +28,7 @@ import {
type MockRTCSession,
mockRTCSlot,
mockRTCMember,
testScope,
} from "../utils/test";
import { E2eeType } from "../e2ee/e2eeType";
import {
@@ -48,7 +49,10 @@ import { MediaDevicesContext } from "../MediaDevicesContext";
import { type MediaDevices as ECMediaDevices } from "../state/MediaDevices";
import { AppBar } from "../AppBar";
import { type MatrixInfo } from "./VideoPreview";
import { ProcessorProvider } from "../tracks/TrackProcessorContext";
import {
BackgroundEffectsProvider,
createAppBackgroundEffects,
} from "../tracks/BackgroundEffectsContext";
import { initializeWidget } from "../widget";
import { RootElementProvider } from "../RootElementContext";
@@ -234,7 +238,9 @@ describe("ActiveCall", () => {
const { findByTestId } = render(
<BrowserRouter>
<MediaDevicesContext value={mediaDevices}>
<ProcessorProvider>
<BackgroundEffectsProvider
effects={createAppBackgroundEffects(testScope())}
>
<TooltipProvider>
<ActiveCall
client={matrixRoom.client}
@@ -247,7 +253,7 @@ describe("ActiveCall", () => {
onLeft={(): void => {}}
/>
</TooltipProvider>
</ProcessorProvider>
</BackgroundEffectsProvider>
</MediaDevicesContext>
</BrowserRouter>,
);
@@ -282,7 +288,9 @@ describe("ActiveCall", () => {
<BrowserRouter>
<RootElementProvider value={root}>
<MediaDevicesContext value={mediaDevices}>
<ProcessorProvider>
<BackgroundEffectsProvider
effects={createAppBackgroundEffects(testScope())}
>
<TooltipProvider>
<ActiveCall
client={matrixRoom.client}
@@ -295,7 +303,7 @@ describe("ActiveCall", () => {
onLeft={(): void => {}}
/>
</TooltipProvider>
</ProcessorProvider>
</BackgroundEffectsProvider>
</MediaDevicesContext>
</RootElementProvider>
</BrowserRouter>,
+12 -6
View File
@@ -109,7 +109,7 @@ import { useBehavior } from "../useBehavior.ts";
import { useValueBehavior } from "../useValueBehavior.ts";
import { Toast } from "../Toast.tsx";
import overlayStyles from "../Overlay.module.css";
import { useTrackProcessorState$ } from "../tracks/TrackProcessorContext.tsx";
import { useBackgroundEffectsState$ } from "../tracks/BackgroundEffectsContext.tsx";
import { type Layout } from "../state/layout-types.ts";
import { ObservableScope } from "../state/ObservableScope.ts";
import { CallFooter, type FooterSnapshot } from "../components/CallFooter.tsx";
@@ -150,7 +150,7 @@ export const ActiveCall: FC<ActiveCallProps> = (props) => {
const urlParams = useUrlParams();
const hostBridge = useHostBridge();
const mediaDevices = useMediaDevices();
const trackProcessorState$ = useTrackProcessorState$();
const backgroundEffectsState$ = useBackgroundEffectsState$();
// The element we have to draw the call in: the page, or the container a host
// gave us. Its size, not the window's, decides how the call is laid out.
const rootElement = useRootElement();
@@ -170,7 +170,12 @@ export const ActiveCall: FC<ActiveCallProps> = (props) => {
);
const rtcParticipation = slot.join({
...rtcParticipationOptions(urlParams),
publish: initialPublishRequests(props.muteStates, mediaDevices, capture),
publish: initialPublishRequests(
props.muteStates,
mediaDevices,
capture,
backgroundEffectsState$.value.processor,
),
});
const vm = createCallViewModel$(
@@ -184,7 +189,7 @@ export const ActiveCall: FC<ActiveCallProps> = (props) => {
encryptionSystem: props.e2eeSystem,
capture,
videoProcessor$: scope.behavior(
trackProcessorState$.pipe(map((state) => state.processor)),
backgroundEffectsState$.pipe(map((state) => state.processor)),
),
hostBridge,
autoLeaveWhenOthersLeft,
@@ -214,7 +219,7 @@ export const ActiveCall: FC<ActiveCallProps> = (props) => {
urlParams,
hostBridge,
mediaDevices,
trackProcessorState$,
backgroundEffectsState$,
props.client,
rootElement,
]);
@@ -228,6 +233,7 @@ export const ActiveCall: FC<ActiveCallProps> = (props) => {
vm,
props.muteStates,
mediaDevices,
backgroundEffectsState$.pipe(map(({ settling }) => settling ?? false)),
`${props.client.getUserId()}:${props.client.getDeviceId()}`,
{ showControls: urlParams.showControls, header: urlParams.header },
);
@@ -245,7 +251,7 @@ export const ActiveCall: FC<ActiveCallProps> = (props) => {
props.onLeft,
urlParams,
mediaDevices,
trackProcessorState$,
backgroundEffectsState$,
props.client,
vm,
]);
+15 -6
View File
@@ -10,24 +10,33 @@ import { render, screen } from "@testing-library/react";
import userEvent from "@testing-library/user-event";
import { TooltipProvider } from "@vector-im/compound-web";
import { type MatrixClient, type RoomSummary } from "matrix-js-sdk";
import { type Observable } from "rxjs";
import { KnockLobbyView } from "./KnockLobbyView";
import { LeaveToHomeProvider } from "../LeaveToHomeContext";
import { MediaDevicesContext } from "../MediaDevicesContext";
import { type ProcessorState } from "../tracks/TrackProcessorContext";
import { type BackgroundEffectsState } from "../tracks/BackgroundEffectsContext";
import { mockMediaDevices } from "../utils/test";
vi.mock("@livekit/components-react", () => ({
usePreviewTracks: (): unknown[] => [],
}));
vi.mock("../tracks/TrackProcessorContext", () => ({
useTrackProcessor: (): ProcessorState => ({
vi.mock("../tracks/BackgroundEffectsContext", async () => {
const { of } = await import("rxjs");
const none: BackgroundEffectsState = {
supported: false,
processor: undefined,
}),
useTrackProcessorSync: (): void => {},
}));
};
// One observable, as the real hook keeps: a fresh one each render would
// rebuild the footer on every render.
const none$ = of(none);
return {
useBackgroundEffects: (): BackgroundEffectsState => none,
useBackgroundEffectsState$: (): Observable<BackgroundEffectsState> => none$,
useSyncBackgroundEffects: (): void => {},
};
});
vi.mock("react-use-measure", () => ({
default: (): [() => void, object] => [(): void => {}, {}],
+15 -7
View File
@@ -8,7 +8,7 @@ Please see LICENSE in the repository root for full details.
import { afterEach, describe, expect, it, vi } from "vitest";
import { render, screen } from "@testing-library/react";
import userEvent from "@testing-library/user-event";
import { of } from "rxjs";
import { type Observable, of } from "rxjs";
import { LeaveToHomeProvider } from "../LeaveToHomeContext";
import { TooltipProvider } from "@vector-im/compound-web";
import { type MatrixClient } from "matrix-js-sdk";
@@ -28,7 +28,7 @@ import {
} from "../utils/test";
import { type MediaDevices } from "../state/MediaDevices";
import { MediaDevicesContext } from "../MediaDevicesContext";
import { type ProcessorState } from "../tracks/TrackProcessorContext";
import { type BackgroundEffectsState } from "../tracks/BackgroundEffectsContext";
import { type EncryptionSystem } from "../e2ee/sharedKeyManagement";
import lobbyStyles from "./LobbyView.module.css";
import headerStyles from "../Header.module.css";
@@ -41,13 +41,21 @@ vi.mock("@livekit/components-react", () => ({
usePreviewTracks: (): unknown[] => [],
}));
vi.mock("../tracks/TrackProcessorContext", () => ({
useTrackProcessor: (): ProcessorState => ({
vi.mock("../tracks/BackgroundEffectsContext", async () => {
const { of } = await import("rxjs");
const none: BackgroundEffectsState = {
supported: false,
processor: undefined,
}),
useTrackProcessorSync: (): void => {},
}));
};
// One observable, as the real hook keeps: a fresh one each render would
// rebuild the footer on every render.
const none$ = of(none);
return {
useBackgroundEffects: (): BackgroundEffectsState => none,
useBackgroundEffectsState$: (): Observable<BackgroundEffectsState> => none$,
useSyncBackgroundEffects: (): void => {},
};
});
vi.mock("react-use-measure", () => ({
default: (): [() => void, object] => [(): void => {}, {}],
+17 -6
View File
@@ -25,6 +25,7 @@ import {
Track,
} from "livekit-client";
import { convertToLivekitProcessor } from "@element-hq/matrixrtc-sdk";
import { map } from "rxjs";
import inCallStyles from "./InCallView.module.css";
import styles from "./LobbyView.module.css";
@@ -41,9 +42,10 @@ import { useMediaDevices } from "../MediaDevicesContext";
import { ObservableScope } from "../state/ObservableScope";
import { useInitial } from "../useInitial";
import {
useTrackProcessor,
useTrackProcessorSync,
} from "../tracks/TrackProcessorContext";
useBackgroundEffects,
useBackgroundEffectsState$,
useSyncBackgroundEffects,
} from "../tracks/BackgroundEffectsContext";
import { getValue } from "../utils/observable";
import { useBehavior } from "../useBehavior";
import { CallFooter, type FooterSnapshot } from "../components/CallFooter";
@@ -137,7 +139,8 @@ export const LobbyView: FC<Props> = ({
},
);
const { processor } = useTrackProcessor();
const { processor } = useBackgroundEffects();
const backgroundEffectsState$ = useBackgroundEffectsState$();
const initialProcessor = useInitial(() => processor);
const localTrackOptions = useMemo<CreateLocalTracksOptions>(
@@ -185,7 +188,7 @@ export const LobbyView: FC<Props> = ({
}
}, [devices, videoInputId, videoTrack]);
useTrackProcessorSync(videoTrack);
useSyncBackgroundEffects(videoTrack);
const [footerVm, setFooterVm] = useState<ViewModel<FooterSnapshot> | null>(
null,
@@ -197,6 +200,7 @@ export const LobbyView: FC<Props> = ({
footerScope,
muteStates,
devices,
backgroundEffectsState$.pipe(map(({ settling }) => settling ?? false)),
openSettings,
hangup,
// Logo and header are connected: only show the logo in SPA with header.
@@ -206,7 +210,14 @@ export const LobbyView: FC<Props> = ({
return (): void => {
footerScope.end();
};
}, [devices, hangup, hideHeader, muteStates, openSettings]);
}, [
devices,
backgroundEffectsState$,
hangup,
hideHeader,
muteStates,
openSettings,
]);
// TODO: Unify this component with InCallView, so we can get slick joining
// animations and don't have to feel bad about reusing its CSS
+93
View File
@@ -0,0 +1,93 @@
/*
Copyright 2026 Element Creations Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { afterEach, describe, expect, it, vi } from "vitest";
import { render, screen } from "@testing-library/react";
import userEvent from "@testing-library/user-event";
import { type MatrixClient } from "matrix-js-sdk";
import { TooltipProvider } from "@vector-im/compound-web";
import { SettingsModal } from "./SettingsModal";
import { backgroundEffect } from "./settings";
import { MediaDevicesContext } from "../MediaDevicesContext";
import { type BackgroundEffectsState } from "../tracks/BackgroundEffectsContext";
import { mockMediaDevices } from "../utils/test";
import { constant } from "../state/Behavior";
import { type MediaDevices } from "../state/MediaDevices";
vi.mock("../tracks/BackgroundEffectsContext", () => ({
useBackgroundEffects: (): BackgroundEffectsState => ({
supported: true,
processor: undefined,
}),
}));
// Behaviors, as the device menus read them.
const noDevices = {
available$: constant(new Map()),
selected$: constant(undefined),
select: vi.fn(),
};
function renderVideoTab(): void {
render(
<TooltipProvider>
<MediaDevicesContext
value={mockMediaDevices({
requestDeviceNames: vi.fn(),
audioInput: noDevices,
audioOutput: noDevices,
videoInput: noDevices,
} as unknown as Partial<MediaDevices>)}
>
<SettingsModal
open
onDismiss={vi.fn()}
tab="video"
onTabChange={vi.fn()}
client={
{
getUserId: () => "@user:example.org",
getUser: () => null,
} as unknown as MatrixClient
}
/>
</MediaDevicesContext>
</TooltipProvider>,
);
}
const blurControl = (): HTMLElement =>
screen.getByRole("checkbox", { name: "Blur the background of the video" });
describe("SettingsModal", () => {
afterEach(() => backgroundEffect.setValue("none"));
it("blur control turns blur on and off", async () => {
const user = userEvent.setup();
renderVideoTab();
expect(blurControl()).not.toBeChecked();
await user.click(blurControl());
expect(backgroundEffect.getValue()).toBe("blur");
expect(blurControl()).toBeChecked();
await user.click(blurControl());
expect(backgroundEffect.getValue()).toBe("none");
expect(blurControl()).not.toBeChecked();
});
it("blur control reflects and replaces an image background", async () => {
const user = userEvent.setup();
backgroundEffect.setValue("image:arc");
renderVideoTab();
expect(blurControl()).not.toBeChecked();
await user.click(blurControl());
expect(backgroundEffect.getValue()).toBe("blur");
});
});
+13 -6
View File
@@ -21,13 +21,14 @@ import { useHostBridge } from "../HostBridge";
import {
useSetting,
soundEffectVolume as soundEffectVolumeSetting,
backgroundBlur as backgroundBlurSetting,
backgroundEffect as backgroundEffectSetting,
developerMode,
} from "./settings";
import { PreferencesSettingsTab } from "./PreferencesSettingsTab";
import { Slider } from "../Slider";
import { DeviceSelection } from "./DeviceSelection";
import { useTrackProcessor } from "../tracks/TrackProcessorContext";
import { useBackgroundEffects } from "../tracks/BackgroundEffectsContext";
import { parseEffect, serializeEffect } from "../tracks/backgroundEffects";
import {
DeveloperSettingsTab,
type DeveloperSettingsSnapshot,
@@ -78,9 +79,11 @@ export const SettingsModal: FC<Props> = ({
// Generate a `Checkbox` input to turn blur on or off.
const BlurCheckbox: React.FC = (): ReactNode => {
const { supported } = useTrackProcessor();
const { supported } = useBackgroundEffects();
const [blurActive, setBlurActive] = useSetting(backgroundBlurSetting);
// The camera menu's setting, of which this control sees only blur.
const [effect, setEffect] = useSetting(backgroundEffectSetting);
const blurActive = parseEffect(effect).kind === "blur";
return (
<>
@@ -94,8 +97,12 @@ export const SettingsModal: FC<Props> = ({
supported ? "" : t("settings.blur_not_supported_by_browser")
}
type="checkbox"
checked={!!blurActive}
onChange={(b): void => setBlurActive(b.target.checked)}
checked={blurActive}
onChange={(b): void =>
setEffect(
serializeEffect({ kind: b.target.checked ? "blur" : "none" }),
)
}
disabled={!supported}
/>
</FieldRow>
+9 -1
View File
@@ -13,6 +13,7 @@ import { type Behavior } from "../state/Behavior";
import { useBehavior } from "../useBehavior";
import { MatrixRTCMode } from "../config/ConfigOptions";
import { type UnstableLivekitTransport } from "matrix-js-sdk/lib/matrixrtc";
import { type EffectId } from "../tracks/backgroundEffects";
export class Setting<T> {
public constructor(
@@ -116,7 +117,14 @@ export const videoInput = new Setting<string | undefined>(
undefined,
);
export const backgroundBlur = new Setting<boolean>("background-blur", false);
// Only read, to carry blur over for anyone who had it on.
const backgroundBlur = new Setting<boolean>("background-blur", false);
/** The chosen background effect, in the form `parseEffect` reads. */
export const backgroundEffect = new Setting<EffectId>(
"background-effect",
backgroundBlur.getValue() ? "blur" : "none",
);
export const showHandRaisedTimer = new Setting<boolean>(
"hand-raised-show-timer",
+2
View File
@@ -6,6 +6,7 @@ Please see LICENSE in the repository root for full details.
*/
import { type AudioOutputOptions, MediaDevices } from "./MediaDevices";
import { createAppBackgroundEffects } from "../tracks/BackgroundEffectsContext";
import { type ObservableScope } from "./ObservableScope";
/**
@@ -16,6 +17,7 @@ export class AppViewModel {
this.scope,
this.audioOutputOptions,
);
public readonly backgroundEffects = createAppBackgroundEffects(this.scope);
// TODO: Move more application logic here. The CallViewModel, at the very
// least, ought to be accessible from this object.
+172
View File
@@ -0,0 +1,172 @@
/*
Copyright 2026 Element Creations Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { beforeEach, describe, expect, it, vi } from "vitest";
import { BehaviorSubject, distinctUntilChanged, map } from "rxjs";
import { type BackgroundProcessorWrapper } from "@livekit/track-processors";
import {
type BackgroundEffects,
createBackgroundEffects,
type BackgroundEffectsOptions,
} from "./BackgroundEffects";
import { type BackgroundEffectsState } from "../tracks/BackgroundEffectsContext";
import { shippedBackgrounds } from "../tracks/backgroundEffects";
import { flushPromises, testScope, withTestScheduler } from "../utils/test";
/** A pipeline that records what it is switched to. */
function fakePipeline(): {
pipeline: BackgroundProcessorWrapper;
switches: unknown[];
/** Holds the next switch until it resolves, as a picture loading does. */
holdNext: () => () => void;
} {
const switches: unknown[] = [];
let slow: Promise<void> | undefined;
const pipeline = {
switchTo: vi.fn(async (options: unknown) => {
switches.push(options);
const held = slow;
slow = undefined;
await held;
}),
} as unknown as BackgroundProcessorWrapper;
return {
pipeline,
switches,
holdNext: () => {
let release!: () => void;
slow = new Promise((resolve) => (release = resolve));
return release;
},
};
}
/** One letter per state: idle, waiting for a frame, attached. */
function letter(state: BackgroundEffectsState): string {
if (state.processor === undefined) return "i";
return state.settling ? "w" : "a";
}
describe("the pipeline's state", () => {
function testState({
effect,
firstFrame = "",
expected,
}: {
effect: string;
firstFrame?: string;
expected: string;
}): void {
withTestScheduler(({ behavior, schedule, expectObservable }) => {
const transformer = {
onFirstFrame: undefined as (() => void) | undefined,
};
const effects = createBackgroundEffects(testScope(), {
supported: true,
effect$: behavior(effect, { n: "none", b: "blur" }),
pipeline: fakePipeline().pipeline,
transformer,
});
schedule(firstFrame, { f: () => transformer.onFirstFrame?.() });
expectObservable(
effects.state$.pipe(map(letter), distinctUntilChanged()),
).toBe(expected);
});
}
it("defaults to no effect", () => testState({ effect: "n", expected: "i" }));
it("waits from the first effect until a frame carries it", () =>
testState({
effect: " nb-n-b",
firstFrame: " --f",
expected: " iwa",
}));
it("attaches on first use and stays attached", () =>
testState({ effect: "nb-n", firstFrame: "--f", expected: "iwa" }));
});
describe("background effects", () => {
let effect$: BehaviorSubject<string>;
let fake: ReturnType<typeof fakePipeline>;
function build(
options: Partial<BackgroundEffectsOptions> = {},
): BackgroundEffects {
return createBackgroundEffects(testScope(), {
supported: true,
effect$,
pipeline: fake.pipeline,
transformer: { onFirstFrame: undefined },
...options,
});
}
const choose = async (raw: string): Promise<void> => {
effect$.next(raw);
await flushPromises();
};
const blur = async (on: boolean): Promise<void> =>
choose(on ? "blur" : "none");
beforeEach(() => {
effect$ = new BehaviorSubject("none");
fake = fakePipeline();
});
it("puts a shipped background on as that picture", async () => {
build();
await choose(`image:${shippedBackgrounds[0].id}`);
expect(fake.switches).toEqual([
{
mode: "virtual-background",
imagePath: shippedBackgrounds[0].imagePath,
},
]);
});
it("switches in place rather than reattaching", async () => {
const effects = build();
await blur(true);
const pipeline = effects.state$.value.processor;
await blur(false);
await blur(true);
expect(effects.state$.value.processor).toBe(pipeline);
expect(fake.switches).toEqual([
{ mode: "background-blur", blurRadius: 15 },
{ mode: "disabled" },
{ mode: "background-blur", blurRadius: 15 },
]);
});
it("keeps switching after a switch fails", async () => {
vi.mocked(fake.pipeline.switchTo).mockRejectedValueOnce(new Error("lost"));
build();
await blur(true);
await blur(false);
expect(fake.switches).toEqual([{ mode: "disabled" }]);
});
it("switches one at a time, skipping those overtaken", async () => {
const finished = fake.holdNext();
build();
await blur(true);
await blur(false);
await blur(true);
await blur(false);
expect(fake.switches).toHaveLength(1);
finished();
await flushPromises();
expect(fake.switches).toEqual([
{ mode: "background-blur", blurRadius: 15 },
{ mode: "disabled" },
]);
});
});
+131
View File
@@ -0,0 +1,131 @@
/*
Copyright 2026 Element Creations Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import {
combineLatest,
distinctUntilChanged,
map,
Observable,
scan,
} from "rxjs";
import {
type BackgroundProcessorWrapper,
type SwitchBackgroundProcessorOptions,
} from "@livekit/track-processors";
import { logger } from "matrix-js-sdk/lib/logger";
import { deepCompare } from "matrix-js-sdk/lib/utils";
import { type Behavior } from "./Behavior";
import { type ObservableScope } from "./ObservableScope";
import {
fromLivekitProcessor,
type BackgroundEffectsState,
} from "../tracks/BackgroundEffectsContext";
import {
type BackgroundEffect,
blurRadius,
imagePathFor,
parseEffect,
} from "../tracks/backgroundEffects";
export interface BackgroundEffectsOptions {
/** Whether this browser can run a pipeline at all. */
supported: boolean;
/** The effect chosen, as the setting stores it. */
effect$: Behavior<string>;
/** The background processor pipeline to be switched between effects. */
pipeline: BackgroundProcessorWrapper;
/** Tells, once, that a frame carrying an effect has been drawn. */
transformer: { onFirstFrame: (() => void) | undefined };
}
/** The background effect pipeline, as the camera tracks and the menus see it. */
export interface BackgroundEffects {
readonly state$: Behavior<BackgroundEffectsState>;
}
/** Switches the pipeline as the choice changes, for as long as the scope lasts. */
export function createBackgroundEffects(
scope: ObservableScope,
{ supported, effect$, pipeline, transformer }: BackgroundEffectsOptions,
): BackgroundEffects {
// Wrapped once, so the call and the preview see one processor throughout.
const processor = fromLivekitProcessor(pipeline);
const choice$ = effect$.pipe(map(parseEffect));
const wanted$ = choice$.pipe(
map((effect) => effect.kind !== "none"),
distinctUntilChanged(),
);
const drewAFrame$ = scope.behavior(
new Observable<boolean>((subscriber) => {
subscriber.next(false);
transformer.onFirstFrame = (): void => subscriber.next(true);
return (): void => {
transformer.onFirstFrame = undefined;
};
}),
);
const state$ = scope.behavior(
combineLatest([wanted$, drewAFrame$]).pipe(
scan<[boolean, boolean], BackgroundEffectsState>(
(previous, [wanted, drewAFrame]) => {
// Attached the first time an effect is wanted and never detached
// after, so someone who never turns one on pays for none of it.
const enable =
previous.processor !== undefined || (supported && wanted);
return {
supported,
processor: enable ? processor : undefined,
settling: enable && !drewAFrame,
};
},
{ supported, processor: undefined },
),
),
);
const switchOptions$ = scope.behavior<
SwitchBackgroundProcessorOptions | undefined
>(
combineLatest([state$, choice$]).pipe(
map(([{ processor }, effect]) =>
processor === undefined ? undefined : switchOptionsFor(effect),
),
distinctUntilChanged(deepCompare),
),
);
// In turn: a picture's switch ends once it loads, so an earlier one could land last.
scope.reconcile(switchOptions$, async (options) => {
if (options === undefined) return;
try {
await pipeline.switchTo(options);
} catch (e) {
logger.warn("Failed to switch background effect", e);
}
});
return { state$ };
}
function switchOptionsFor(
effect: BackgroundEffect,
): SwitchBackgroundProcessorOptions {
switch (effect.kind) {
case "blur":
return { mode: "background-blur", blurRadius };
case "shipped": {
const imagePath = imagePathFor(effect.id);
return imagePath
? { mode: "virtual-background", imagePath }
: { mode: "disabled" };
}
default:
return { mode: "disabled" };
}
}
+81 -1
View File
@@ -23,6 +23,10 @@ import {
MatrixRTCSessionEvent,
} from "matrix-js-sdk/lib/matrixrtc";
import { deepCompare } from "matrix-js-sdk/lib/utils";
import {
type RTCParticipation,
type VideoProcessor,
} from "@element-hq/matrixrtc-sdk";
import { type Layout } from "../layout-types.ts";
import {
@@ -31,7 +35,9 @@ import {
mockRemoteParticipant,
withTestScheduler,
mockRtcMembership,
mockMuteStates,
testScope,
flushPromises,
} from "../../utils/test.ts";
import { E2eeType } from "../../e2ee/e2eeType.ts";
import {
@@ -62,7 +68,11 @@ import {
import { MatrixRTCMode } from "../../config/ConfigOptions.ts";
import { initializeWidget } from "../../widget.ts";
import { computeUrlParams } from "../../UrlParams.ts";
import { callViewModelOptionsFromParams } from "./CallViewModel.ts";
import {
callViewModelOptionsFromParams,
initialPublishRequests,
} from "./CallViewModel.ts";
import { type Handler, MuteState } from "../MuteStates.ts";
initializeWidget();
@@ -1664,3 +1674,73 @@ describe("callViewModelOptionsFromParams", () => {
expect(callViewModelOptionsFromParams(params).hideScreensharing).toBe(true);
});
});
describe("a camera with a video processor", () => {
const withCallViewModel = withCallViewModelInMode(MatrixRTCMode.Matrix_2_0);
const processor = { name: "effect" } as VideoProcessor;
/** The handler the view model gives the camera's mute switch. */
const cameraHandler = (setHandler: {
mock: { calls: [Handler][] };
}): Handler => setHandler.mock.calls[1][0];
it("starts each camera track with the processor", () => {
const muteStates = mockMuteStates();
vi.spyOn(muteStates.video.enabled$, "value", "get").mockReturnValue(true);
expect(
initialPublishRequests(
muteStates,
mockMediaDevices({}),
{},
processor,
).find((request) => request.source === "camera"),
).toMatchObject({ processor });
});
it("replaces a camera turned off before the processor was chosen", async () => {
const setHandler = vi.spyOn(MuteState.prototype, "setHandler");
onTestFinished(() => setHandler.mockRestore());
const videoProcessor$ = new BehaviorSubject<VideoProcessor | undefined>(
undefined,
);
let participation: RTCParticipation | undefined;
withCallViewModel(
{},
(_vm, _session, { rtcParticipation }) => {
participation = rtcParticipation;
},
{ videoProcessor$ },
);
// The camera is off, so it is stopped and can't take the processor
videoProcessor$.next(processor);
await cameraHandler(setHandler)(true);
await flushPromises();
expect(participation!.unpublish).toHaveBeenCalledWith("camera");
expect(participation!.publish).toHaveBeenCalledWith(
expect.objectContaining({ source: "camera", processor }),
);
});
it("turns on a camera that took the processor while running", async () => {
const setHandler = vi.spyOn(MuteState.prototype, "setHandler");
onTestFinished(() => setHandler.mockRestore());
const videoProcessor$ = new BehaviorSubject<VideoProcessor | undefined>(
undefined,
);
let participation: RTCParticipation | undefined;
withCallViewModel(
{ videoEnabled: new Map([[localParticipant, constant(true)]]) },
(_vm, _session, { rtcParticipation }) => {
participation = rtcParticipation;
},
{ videoProcessor$ },
);
videoProcessor$.next(processor);
await cameraHandler(setHandler)(true);
await flushPromises();
expect(participation!.unpublish).not.toHaveBeenCalled();
expect(participation!.publish).not.toHaveBeenCalled();
});
});
+36 -9
View File
@@ -135,14 +135,16 @@ export interface CaptureSettings {
}
/**
* What to publish for a microphone or camera: the device the user picked and
* the host's capture settings. The video processor is not in here; the call
* attaches it to the camera track once that exists.
* What to publish for a microphone or camera: the device the user picked, the
* host's capture settings and, for the camera, the video processor, so that
* the track starts with it rather than letting the room through while it is
* built.
*/
export function publishRequest(
source: "microphone" | "camera",
mediaDevices: MediaDevices,
capture: CaptureSettings,
processor?: VideoProcessor,
): PublishRequest {
return source === "microphone"
? {
@@ -154,6 +156,7 @@ export function publishRequest(
source,
deviceId: mediaDevices.videoInput.selected$.value?.id,
capture: capture.camera,
...(processor && { processor }),
};
}
@@ -162,12 +165,13 @@ export function initialPublishRequests(
muteStates: MuteStates,
mediaDevices: MediaDevices,
capture: CaptureSettings,
processor?: VideoProcessor,
): PublishRequest[] {
const requests: PublishRequest[] = [];
if (muteStates.audio.enabled$.value)
requests.push(publishRequest("microphone", mediaDevices, capture));
if (muteStates.video.enabled$.value)
requests.push(publishRequest("camera", mediaDevices, capture));
requests.push(publishRequest("camera", mediaDevices, capture, processor));
return requests;
}
@@ -503,6 +507,17 @@ export function createCallViewModel$(
"screenShare",
) as Behavior<LocalVideoMediaTrack | undefined>;
// Camera tracks the processor reached while they ran. A camera turned off is
// stopped, and a stopped track can't take one.
const tookProcessor = new WeakMap<LocalVideoMediaTrack, VideoProcessor>();
const missesProcessor = (track: LocalMediaTrack): boolean => {
const processor = videoProcessor$.value;
return (
processor !== undefined &&
tookProcessor.get(track as LocalVideoMediaTrack) !== processor
);
};
// The mute switches drive our tracks, which report back what the device
// allowed, so a denied permission flips the switch back. A source is muted
// and unmuted while its track exists and published when it does not.
@@ -520,14 +535,23 @@ export function createCallViewModel$(
muteState.setHandler(async (enabled) => {
desired = enabled;
const track = track$.value;
if (track !== undefined) {
// A camera turned off before the processor was chosen is replaced, so
// that the new one starts with it.
if (enabled && source === "camera" && track && missesProcessor(track))
await rtcParticipation.unpublish("camera");
else if (track !== undefined) {
desired = await track.setEnabled(enabled);
return desired;
}
if (!enabled) return false;
try {
await rtcParticipation.publish(
publishRequest(source, mediaDevices, capture),
publishRequest(
source,
mediaDevices,
capture,
source === "camera" ? videoProcessor$.value : undefined,
),
);
return true;
} catch (e) {
@@ -583,12 +607,15 @@ export function createCallViewModel$(
});
});
// Attached once the camera track exists rather than carried in the publish
// request, so that a new track and a change of processor take the same path
// Also set on each camera track as it appears and as the choice changes. A
// track made with the processor already runs it and is left alone.
combineLatest([camera$, videoProcessor$])
.pipe(scope.bind())
.subscribe(([camera, processor]) => {
camera?.setProcessor(processor).catch((e) => {
if (camera === undefined) return;
if (processor && !camera.muted$.value)
tookProcessor.set(camera, processor);
camera.setProcessor(processor).catch((e) => {
logger.error("Failed to set the video processor", e);
});
});
@@ -0,0 +1,81 @@
/*
Copyright 2026 Element Creations Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { afterEach, describe, expect, it, vi } from "vitest";
import {
BackgroundTransformer,
VideoTransformer,
type VideoTransformerInitOptions,
} from "@livekit/track-processors";
import { ImageSegmenter } from "@mediapipe/tasks-vision";
import { BackgroundEffectTransformer } from "./BackgroundEffectTransformer";
const frame = {} as VideoFrame;
const controller = {} as TransformStreamDefaultController<VideoFrame>;
describe("BackgroundEffectTransformer", () => {
afterEach(() => vi.restoreAllMocks());
it("loads a background chosen before it was built", async () => {
vi.spyOn(VideoTransformer.prototype, "init").mockResolvedValue();
vi.spyOn(ImageSegmenter, "createFromOptions").mockResolvedValue(
{} as ImageSegmenter,
);
const load = vi
.spyOn(BackgroundTransformer.prototype, "loadAndSetBackground")
.mockResolvedValue();
const transformer = new BackgroundEffectTransformer({
backgroundDisabled: true,
});
// Chosen before the camera is processed, as the first effect always is:
// loaded then, there is nothing yet to draw it with.
await transformer.update({ imagePath: "/background.jpg" });
load.mockClear();
await transformer.init({} as VideoTransformerInitOptions);
expect(load).toHaveBeenCalledWith("/background.jpg");
});
it("reports the first frame carrying an effect, once", async () => {
vi.spyOn(BackgroundTransformer.prototype, "transform").mockImplementation(
async function (this: BackgroundTransformer) {
this.isFirstFrame = false;
return Promise.resolve();
},
);
const transformer = new BackgroundEffectTransformer({});
transformer.onFirstFrame = vi.fn();
await transformer.transform(frame, controller);
await transformer.transform(frame, controller);
expect(transformer.onFirstFrame).toHaveBeenCalledTimes(1);
});
it("does not report a frame passed through untouched", async () => {
vi.spyOn(BackgroundTransformer.prototype, "transform").mockResolvedValue();
const transformer = new BackgroundEffectTransformer({
backgroundDisabled: true,
});
transformer.onFirstFrame = vi.fn();
await transformer.transform(frame, controller);
expect(transformer.onFirstFrame).not.toHaveBeenCalled();
});
it("passes frames on untouched once disabled, even after blurring", async () => {
const transformer = new BackgroundEffectTransformer({});
await transformer.update({ blurRadius: 15, backgroundDisabled: false });
await transformer.update({
imagePath: undefined,
backgroundDisabled: true,
});
// With neither, the library passes a frame on without segmenting it.
expect(transformer.options.blurRadius).toBeUndefined();
expect(transformer.options.imagePath).toBeUndefined();
});
});
@@ -1,5 +1,6 @@
/*
Copyright 2024-2025 New Vector Ltd.
Copyright 2026 Element Creations Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
@@ -9,7 +10,9 @@ import {
BackgroundTransformer,
VideoTransformer,
type VideoTransformerInitOptions,
type BackgroundOptions,
} from "@livekit/track-processors";
import { logger } from "matrix-js-sdk/lib/logger";
import { ImageSegmenter } from "@mediapipe/tasks-vision";
import modelAssetPath from "../mediapipe/imageSegmenter/selfie_segmenter.tflite?url";
@@ -49,7 +52,32 @@ const wasmFileset: WasmFileset = {
* loads the segmentation models from our own bundle rather than as an external
* resource fetched from the public internet.
*/
export class BlurBackgroundTransformer extends BackgroundTransformer {
export class BackgroundEffectTransformer extends BackgroundTransformer {
/** Called once, when the first frame carrying an effect has been drawn. */
public onFirstFrame: (() => void) | undefined;
public override async transform(
frame: VideoFrame,
controller: TransformStreamDefaultController<VideoFrame>,
): Promise<void> {
const priming = this.isFirstFrame;
await super.transform(frame, controller);
// Cleared only once a frame has been through the segmenter; a frame
// passed through untouched leaves it set.
if (priming && !this.isFirstFrame) this.onFirstFrame?.();
}
/**
* As the library's, except that disabling also drops the blur radius: kept,
* it has every frame segmented and the result thrown away, where with
* neither a radius nor a picture frames are passed on untouched.
*/
public override async update(opts: BackgroundOptions): Promise<void> {
await super.update(
opts.backgroundDisabled ? { ...opts, blurRadius: undefined } : opts,
);
}
public async init({
outputCanvas,
inputElement: inputVideo,
@@ -73,8 +101,15 @@ export class BlurBackgroundTransformer extends BackgroundTransformer {
outputConfidenceMasks: false,
});
// BackgroundTransformer's own init applies these, and this one replaces it.
if (this.options.imagePath) {
await this.loadAndSetBackground(this.options.imagePath).catch((e) =>
logger.warn("Failed to load the background image", e),
);
}
if (this.options.blurRadius) {
this.gl?.setBlurRadius(this.options.blurRadius);
}
this.gl?.setBackgroundDisabled(this.options.backgroundDisabled ?? false);
}
}
+296
View File
@@ -0,0 +1,296 @@
/*
Copyright 2026 Element Creations Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { act, createElement, type FC } from "react";
import { render } from "@testing-library/react";
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
import { type LocalVideoTrack } from "livekit-client";
import {
type BackgroundOptions,
type ProcessorWrapper,
} from "@livekit/track-processors";
import {
convertToLivekitProcessor,
type VideoProcessor,
} from "@element-hq/matrixrtc-sdk";
import {
applyProcessor,
fromLivekitProcessor,
BackgroundEffectsProvider,
type BackgroundEffectsState,
syncBackgroundEffects,
useBackgroundEffects,
useSyncBackgroundEffects,
createAppBackgroundEffects,
} from "./BackgroundEffectsContext";
import { type BackgroundEffects } from "../state/BackgroundEffects";
import { backgroundEffect } from "../settings/settings";
import { constant } from "../state/Behavior";
import { flushPromises, testScope } from "../utils/test";
const pipelines = vi.hoisted(() => ({
built: 0,
destroyed: 0,
switches: [] as unknown[],
}));
// The pipeline itself needs WebGL and MediaPipe; what these tests are about is
// what the provider asks of it.
vi.mock("@livekit/track-processors", () => ({
BackgroundProcessorWrapper: vi.fn(function (this: Record<string, unknown>) {
pipelines.built += 1;
this.switchTo = vi.fn(async (options: unknown) => {
pipelines.switches.push(options);
return Promise.resolve();
});
this.destroy = vi.fn(async () => {
pipelines.destroyed += 1;
return Promise.resolve();
});
}),
supportsBackgroundProcessors: (): boolean => true,
}));
const platformMock = vi.hoisted(() => ({ platform: "desktop" }));
vi.mock("../Platform", () => platformMock);
vi.mock("./BackgroundEffectTransformer", () => ({
BackgroundEffectTransformer: vi.fn(),
}));
const processor = { name: "blur" } as VideoProcessor;
function mockTrack(
readyState: MediaStreamTrackState,
current?: VideoProcessor,
): LocalVideoTrack {
return {
mediaStreamTrack: { readyState },
getProcessor: vi.fn().mockReturnValue(current),
setProcessor: vi.fn().mockResolvedValue(undefined),
stopProcessor: vi.fn().mockResolvedValue(undefined),
} as unknown as LocalVideoTrack;
}
describe("applyProcessor", () => {
it("attaches the processor to a live track", () => {
const track = mockTrack("live");
applyProcessor(track, processor);
expect(track.setProcessor).toHaveBeenCalledWith(
expect.objectContaining({ name: "blur" }),
);
});
it("does not attach the processor to an ended track", () => {
const track = mockTrack("ended");
applyProcessor(track, processor);
expect(track.setProcessor).not.toHaveBeenCalled();
});
it("does not surface a rejected setProcessor", async () => {
const track = mockTrack("live");
vi.mocked(track.setProcessor).mockRejectedValue(
new TypeError("Input track cannot be ended"),
);
const unhandled = vi.fn();
process.on("unhandledRejection", unhandled);
applyProcessor(track, processor);
await flushPromises();
process.off("unhandledRejection", unhandled);
expect(unhandled).not.toHaveBeenCalled();
});
it("stops the processor when none is wanted", () => {
const track = mockTrack("live", processor);
applyProcessor(track, undefined);
expect(track.stopProcessor).toHaveBeenCalled();
});
it("does not surface a rejected stopProcessor", async () => {
const track = mockTrack("live", processor);
vi.mocked(track.stopProcessor).mockRejectedValue(new Error("nope"));
const unhandled = vi.fn();
process.on("unhandledRejection", unhandled);
applyProcessor(track, undefined);
await flushPromises();
process.off("unhandledRejection", unhandled);
expect(unhandled).not.toHaveBeenCalled();
});
});
describe("syncBackgroundEffects", () => {
it("applies the processor to the current track", () => {
const track = mockTrack("live");
syncBackgroundEffects(
testScope(),
constant(track),
constant({ supported: true, processor }),
);
expect(track.setProcessor).toHaveBeenCalledWith(
expect.objectContaining({ name: "blur" }),
);
});
});
describe("fromLivekitProcessor", () => {
const captured = { id: "captured" } as MediaStreamTrack;
const processed = { id: "processed" } as MediaStreamTrack;
function mockWrapper(
processedTrack: MediaStreamTrack | undefined,
): ProcessorWrapper<BackgroundOptions> {
return {
name: "background-blur",
processedTrack,
init: vi.fn().mockResolvedValue(undefined),
restart: vi.fn().mockResolvedValue(undefined),
destroy: vi.fn().mockResolvedValue(undefined),
} as unknown as ProcessorWrapper<BackgroundOptions>;
}
it("resolves with the track the wrapper produced", async () => {
const wrapper = mockWrapper(processed);
const sdkProcessor = fromLivekitProcessor(wrapper);
expect(sdkProcessor.name).toBe("background-blur");
await expect(sdkProcessor.init(captured)).resolves.toBe(processed);
expect(wrapper.init).toHaveBeenCalledWith({
kind: "video",
track: captured,
});
await expect(sdkProcessor.restart(captured)).resolves.toBe(processed);
await sdkProcessor.destroy();
expect(wrapper.destroy).toHaveBeenCalled();
});
it("rejects where the wrapper produced no track", async () => {
await expect(
fromLivekitProcessor(mockWrapper(undefined)).init(captured),
).rejects.toThrow("produced no track");
});
});
describe("BackgroundEffectsProvider", () => {
/** A camera track that holds whatever processor it is given. */
function cameraTrack(): LocalVideoTrack {
let current: unknown;
return {
mediaStreamTrack: { readyState: "live" },
getProcessor: vi.fn(() => current),
setProcessor: vi.fn(async (next: unknown) => {
current = next;
return Promise.resolve();
}),
stopProcessor: vi.fn(async () => {
current = undefined;
return Promise.resolve();
}),
} as unknown as LocalVideoTrack;
}
let seen: BackgroundEffectsState[];
const Surface: FC<{ track: LocalVideoTrack | null }> = ({ track }) => {
seen.push(useBackgroundEffects());
useSyncBackgroundEffects(track);
return null;
};
// One component for every render, so a rerender updates the tree rather than
// mounting a second provider with a pipeline of its own.
let effects: BackgroundEffects | undefined;
const Surfaces: FC<{ tracks: (LocalVideoTrack | null)[] }> = ({ tracks }) =>
createElement(BackgroundEffectsProvider, {
// Made on first render, once a test has set up the platform.
effects: (effects ??= createAppBackgroundEffects(testScope())),
children: createElement(
"div",
null,
...tracks.map((track, key) => createElement(Surface, { key, track })),
),
});
const surfaces = (
...tracks: (LocalVideoTrack | null)[]
): ReturnType<typeof createElement> => createElement(Surfaces, { tracks });
const blur = async (on: boolean): Promise<void> => {
await act(async () => {
backgroundEffect.setValue(on ? "blur" : "none");
await flushPromises();
});
};
const latest = (): BackgroundEffectsState => seen[seen.length - 1];
beforeEach(() => {
seen = [];
effects = undefined;
pipelines.built = 0;
pipelines.destroyed = 0;
pipelines.switches = [];
platformMock.platform = "desktop";
backgroundEffect.setValue("none");
});
afterEach(() => backgroundEffect.setValue("none"));
// A pipeline primes itself when it is built and when it is destroyed, and a
// primed pipeline lets one frame through untouched: so the frame is spent
// once, as long as it is built once and never taken off the camera.
it("spends one priming frame on first use and none afterwards", async () => {
const track = cameraTrack();
render(surfaces(track));
for (const on of [true, false, true, false, true]) await blur(on);
expect(pipelines.built).toBe(1);
expect(pipelines.destroyed).toBe(0);
expect(track.setProcessor).toHaveBeenCalledTimes(1);
expect(track.stopProcessor).not.toHaveBeenCalled();
});
it("runs on a phone whose browser can run it", async () => {
platformMock.platform = "ios";
const track = cameraTrack();
render(surfaces(track));
await blur(true);
expect(latest().supported).toBe(true);
expect(track.setProcessor).toHaveBeenCalledWith(
convertToLivekitProcessor(latest().processor!),
);
});
it("preview and call share one pipeline", async () => {
const preview = cameraTrack();
const call = cameraTrack();
render(surfaces(preview, call));
await blur(true);
const pipeline = latest().processor;
expect(preview.setProcessor).toHaveBeenCalledWith(
convertToLivekitProcessor(pipeline!),
);
expect(call.setProcessor).toHaveBeenCalledWith(
convertToLivekitProcessor(pipeline!),
);
expect(pipelines.built).toBe(1);
});
it("applies an effect chosen while the camera is off", async () => {
const view = render(surfaces(null));
await blur(true);
// Already switched to blur, with nothing yet to attach it to.
expect(pipelines.switches).toEqual([
{ mode: "background-blur", blurRadius: 15 },
]);
const track = cameraTrack();
await act(async () => {
view.rerender(surfaces(track));
await flushPromises();
});
expect(track.setProcessor).toHaveBeenCalledTimes(1);
expect(track.setProcessor).toHaveBeenCalledWith(
convertToLivekitProcessor(latest().processor!),
);
expect(pipelines.built).toBe(1);
expect(pipelines.switches).toHaveLength(1);
});
});
@@ -1,23 +1,17 @@
/*
Copyright 2024-2025 New Vector Ltd.
Copyright 2026 Element Creations Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import {
ProcessorWrapper,
supportsBackgroundProcessors as supportsBackgroundProcessorsLivekitSdk,
type ProcessorWrapper,
type BackgroundOptions,
supportsBackgroundProcessors,
} from "@livekit/track-processors";
import {
createContext,
type FC,
type JSX,
use,
useEffect,
useMemo,
} from "react";
import { createContext, type FC, type JSX, use, useEffect } from "react";
import { type LocalVideoTrack, Track } from "livekit-client";
import { logger } from "matrix-js-sdk/lib/logger";
import { combineLatest } from "rxjs";
@@ -26,23 +20,26 @@ import {
type VideoProcessor,
} from "@element-hq/matrixrtc-sdk";
import {
backgroundBlur as backgroundBlurSettings,
useSetting,
} from "../settings/settings";
import { BlurBackgroundTransformer } from "./BlurBackgroundTransformer";
import { backgroundEffect as backgroundEffectSetting } from "../settings/settings";
import { BackgroundEffectTransformer } from "./BackgroundEffectTransformer";
import { OneStepPipeline } from "./OneStepPipeline";
import { type Behavior } from "../state/Behavior";
import { type ObservableScope } from "../state/ObservableScope";
import { platform } from "../Platform";
import { useValueBehavior } from "../useValueBehavior";
import {
type BackgroundEffects,
createBackgroundEffects,
} from "../state/BackgroundEffects";
import { useBehavior } from "../useBehavior";
//TODO-MULTI-SFU: This is not yet fully there.
// it is a combination of exposing observable and react hooks.
// preferably we should not make this a context anymore and instead just a vm?
export type ProcessorState = {
export type BackgroundEffectsState = {
supported: boolean | undefined;
processor: undefined | VideoProcessor;
/** From the first effect chosen until a frame carrying it is drawn. */
settling?: boolean;
};
/**
@@ -72,19 +69,21 @@ export function fromLivekitProcessor(
};
}
const ProcessorContext = createContext<ProcessorState | undefined>(undefined);
const BackgroundEffectsContext = createContext<BackgroundEffects | undefined>(
undefined,
);
export function useTrackProcessor(): ProcessorState {
const state = use(ProcessorContext);
if (state === undefined)
throw new Error(
"useTrackProcessor must be used within a ProcessorProvider",
);
return state;
export function useBackgroundEffects(): BackgroundEffectsState {
return useBehavior(useBackgroundEffectsState$());
}
export function useTrackProcessorState$(): Behavior<ProcessorState> {
return useValueBehavior(useTrackProcessor());
export function useBackgroundEffectsState$(): Behavior<BackgroundEffectsState> {
const effects = use(BackgroundEffectsContext);
if (effects === undefined)
throw new Error(
"useBackgroundEffects must be used within a BackgroundEffectsProvider",
);
return effects.state$;
}
/**
@@ -104,8 +103,11 @@ export function applyProcessor(
logger.debug("Not attaching video processor to an ended track");
return;
}
const track = videoTrack.mediaStreamTrack;
videoTrack.setProcessor(convertToLivekitProcessor(processor)).catch((e) => {
logger.warn("Failed to attach video processor", e);
if (track.readyState === "ended")
logger.debug("Video processor not attached: the track ended first");
else logger.warn("Failed to attach video processor", e);
});
}
if (!processor && videoTrack.getProcessor()) {
@@ -118,61 +120,52 @@ export function applyProcessor(
/**
* Updates your video tracks to always use the given processor.
*/
export const trackProcessorSync = (
export const syncBackgroundEffects = (
scope: ObservableScope,
videoTrack$: Behavior<LocalVideoTrack | null>,
processor$: Behavior<ProcessorState>,
backgroundEffectsState$: Behavior<BackgroundEffectsState>,
): void => {
combineLatest([videoTrack$, processor$])
combineLatest([videoTrack$, backgroundEffectsState$])
.pipe(scope.bind())
.subscribe(([videoTrack, processorState]) => {
if (!processorState) return;
.subscribe(([videoTrack, backgroundEffectsState]) => {
if (!backgroundEffectsState) return;
if (!videoTrack) return;
applyProcessor(videoTrack, processorState.processor);
applyProcessor(videoTrack, backgroundEffectsState.processor);
});
};
export const useTrackProcessorSync = (
export const useSyncBackgroundEffects = (
videoTrack: LocalVideoTrack | null,
): void => {
const { processor } = useTrackProcessor();
const { processor } = useBackgroundEffects();
useEffect(() => {
if (!videoTrack) return;
applyProcessor(videoTrack, processor);
}, [processor, videoTrack]);
};
/** The app's one pipeline, shared by every camera track it opens. */
export function createAppBackgroundEffects(
scope: ObservableScope,
): BackgroundEffects {
const transformer = new BackgroundEffectTransformer({
backgroundDisabled: true,
});
return createBackgroundEffects(scope, {
supported: supportsBackgroundProcessors(),
effect$: backgroundEffectSetting.value$,
pipeline: new OneStepPipeline(transformer, "background-effect"),
transformer,
});
}
interface Props {
effects: BackgroundEffects;
children: JSX.Element;
}
function supportsBackgroundProcessors(): boolean {
return supportsBackgroundProcessorsLivekitSdk() && platform === "desktop";
}
export const ProcessorProvider: FC<Props> = ({ children }) => {
// The setting the user wants to have
const [blurActivated] = useSetting(backgroundBlurSettings);
const supported = useMemo(() => supportsBackgroundProcessors(), []);
const blur = useMemo(
() =>
fromLivekitProcessor(
new ProcessorWrapper(
new BlurBackgroundTransformer({ blurRadius: 15 }),
"background-blur",
),
),
[],
);
// This is the actual state exposed through the context
const processorState = useMemo(
() => ({
supported,
processor: supported && blurActivated ? blur : undefined,
}),
[supported, blurActivated, blur],
);
return <ProcessorContext value={processorState}>{children}</ProcessorContext>;
};
export const BackgroundEffectsProvider: FC<Props> = ({ effects, children }) => (
<BackgroundEffectsContext value={effects}>
{children}
</BackgroundEffectsContext>
);
+80
View File
@@ -0,0 +1,80 @@
/*
Copyright 2026 Element Creations Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { afterEach, describe, expect, it, vi } from "vitest";
import { ProcessorWrapper } from "@livekit/track-processors";
import { OneStepPipeline } from "./OneStepPipeline";
import { BackgroundEffectTransformer } from "./BackgroundEffectTransformer";
import { flushPromises } from "../utils/test";
const opts = {} as Parameters<OneStepPipeline["init"]>[0];
describe("OneStepPipeline", () => {
afterEach(() => vi.restoreAllMocks());
it("stops and builds again only once a build has finished", async () => {
let finishBuild!: () => void;
const init = vi
.spyOn(ProcessorWrapper.prototype, "init")
.mockImplementationOnce(
async () => new Promise((resolve) => (finishBuild = resolve)),
)
.mockResolvedValue();
const destroy = vi
.spyOn(ProcessorWrapper.prototype, "destroy")
.mockResolvedValue();
const pipeline = new OneStepPipeline(
new BackgroundEffectTransformer({}),
"test",
);
const first = pipeline.init(opts);
const stopped = pipeline.destroy();
const second = pipeline.init(opts);
await flushPromises();
expect(init).toHaveBeenCalledTimes(1);
expect(destroy).not.toHaveBeenCalled();
finishBuild();
await Promise.all([first, stopped, second]);
expect(init).toHaveBeenCalledTimes(2);
expect(destroy.mock.invocationCallOrder[0]).toBeLessThan(
init.mock.invocationCallOrder[1],
);
});
it("restarts in one step, without waiting on itself", async () => {
const init = vi
.spyOn(ProcessorWrapper.prototype, "init")
.mockResolvedValue();
const destroy = vi
.spyOn(ProcessorWrapper.prototype, "destroy")
.mockResolvedValue();
const pipeline = new OneStepPipeline(
new BackgroundEffectTransformer({}),
"test",
);
await pipeline.restart(opts);
expect(destroy).toHaveBeenCalledWith({ willProcessorRestart: true });
expect(init).toHaveBeenCalledOnce();
});
it("carries on after a step that failed", async () => {
vi.spyOn(ProcessorWrapper.prototype, "init")
.mockRejectedValueOnce(new Error("no GPU"))
.mockResolvedValue();
const pipeline = new OneStepPipeline(
new BackgroundEffectTransformer({}),
"test",
);
await expect(pipeline.init(opts)).rejects.toThrow("no GPU");
await expect(pipeline.init(opts)).resolves.toBeUndefined();
});
});
+43
View File
@@ -0,0 +1,43 @@
/*
Copyright 2026 Element Creations Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { BackgroundProcessorWrapper } from "@livekit/track-processors";
type InitOptions = Parameters<BackgroundProcessorWrapper["init"]>[0];
type DestroyOptions = Parameters<BackgroundProcessorWrapper["destroy"]>[0];
/**
* A background pipeline that is built, rebuilt and destroyed one step at a
* time. The camera tracks sharing it attach and stop it independently, and
* the SDK's wrapper holds a single track's streams: two builds at once
* overwrite each other's, and a destroy during a build does nothing.
*/
export class OneStepPipeline extends BackgroundProcessorWrapper {
private steps: Promise<void> = Promise.resolve();
public override async init(opts: InitOptions): Promise<void> {
return this.inTurn(async () => super.init(opts));
}
// The SDK's restart calls init and destroy, which would wait on itself.
public override async restart(opts: InitOptions): Promise<void> {
return this.inTurn(async () => {
await super.destroy({ willProcessorRestart: true });
await super.init(opts);
});
}
public override async destroy(options?: DestroyOptions): Promise<void> {
return this.inTurn(async () => super.destroy(options));
}
private async inTurn(step: () => Promise<void>): Promise<void> {
const turn = this.steps.then(step);
this.steps = turn.catch(() => {});
return turn;
}
}
-133
View File
@@ -1,133 +0,0 @@
/*
Copyright 2026 Element Creations Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { describe, expect, it, vi } from "vitest";
import { type LocalVideoTrack } from "livekit-client";
import {
type BackgroundOptions,
type ProcessorWrapper,
} from "@livekit/track-processors";
import { type VideoProcessor } from "@element-hq/matrixrtc-sdk";
import {
applyProcessor,
fromLivekitProcessor,
trackProcessorSync,
} from "./TrackProcessorContext";
import { constant } from "../state/Behavior";
import { flushPromises, testScope } from "../utils/test";
const processor = { name: "blur" } as VideoProcessor;
function mockTrack(
readyState: MediaStreamTrackState,
current?: VideoProcessor,
): LocalVideoTrack {
return {
mediaStreamTrack: { readyState },
getProcessor: vi.fn().mockReturnValue(current),
setProcessor: vi.fn().mockResolvedValue(undefined),
stopProcessor: vi.fn().mockResolvedValue(undefined),
} as unknown as LocalVideoTrack;
}
describe("applyProcessor", () => {
it("attaches the processor to a live track", () => {
const track = mockTrack("live");
applyProcessor(track, processor);
expect(track.setProcessor).toHaveBeenCalledWith(
expect.objectContaining({ name: "blur" }),
);
});
it("does not attach the processor to an ended track", () => {
const track = mockTrack("ended");
applyProcessor(track, processor);
expect(track.setProcessor).not.toHaveBeenCalled();
});
it("does not surface a rejected setProcessor", async () => {
const track = mockTrack("live");
vi.mocked(track.setProcessor).mockRejectedValue(
new TypeError("Input track cannot be ended"),
);
const unhandled = vi.fn();
process.on("unhandledRejection", unhandled);
applyProcessor(track, processor);
await flushPromises();
process.off("unhandledRejection", unhandled);
expect(unhandled).not.toHaveBeenCalled();
});
it("stops the processor when none is wanted", () => {
const track = mockTrack("live", processor);
applyProcessor(track, undefined);
expect(track.stopProcessor).toHaveBeenCalled();
});
it("does not surface a rejected stopProcessor", async () => {
const track = mockTrack("live", processor);
vi.mocked(track.stopProcessor).mockRejectedValue(new Error("nope"));
const unhandled = vi.fn();
process.on("unhandledRejection", unhandled);
applyProcessor(track, undefined);
await flushPromises();
process.off("unhandledRejection", unhandled);
expect(unhandled).not.toHaveBeenCalled();
});
});
describe("trackProcessorSync", () => {
it("applies the processor to the current track", () => {
const track = mockTrack("live");
trackProcessorSync(
testScope(),
constant(track),
constant({ supported: true, processor }),
);
expect(track.setProcessor).toHaveBeenCalledWith(
expect.objectContaining({ name: "blur" }),
);
});
});
describe("fromLivekitProcessor", () => {
const captured = { id: "captured" } as MediaStreamTrack;
const processed = { id: "processed" } as MediaStreamTrack;
function mockWrapper(
processedTrack: MediaStreamTrack | undefined,
): ProcessorWrapper<BackgroundOptions> {
return {
name: "background-blur",
processedTrack,
init: vi.fn().mockResolvedValue(undefined),
restart: vi.fn().mockResolvedValue(undefined),
destroy: vi.fn().mockResolvedValue(undefined),
} as unknown as ProcessorWrapper<BackgroundOptions>;
}
it("resolves with the track the wrapper produced", async () => {
const wrapper = mockWrapper(processed);
const sdkProcessor = fromLivekitProcessor(wrapper);
expect(sdkProcessor.name).toBe("background-blur");
await expect(sdkProcessor.init(captured)).resolves.toBe(processed);
expect(wrapper.init).toHaveBeenCalledWith({
kind: "video",
track: captured,
});
await expect(sdkProcessor.restart(captured)).resolves.toBe(processed);
await sdkProcessor.destroy();
expect(wrapper.destroy).toHaveBeenCalled();
});
it("rejects where the wrapper produced no track", async () => {
await expect(
fromLivekitProcessor(mockWrapper(undefined)).init(captured),
).rejects.toThrow("produced no track");
});
});
+53
View File
@@ -0,0 +1,53 @@
/*
Copyright 2026 Element Creations Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { readFile } from "node:fs/promises";
import { describe, expect, test } from "vitest";
import {
type BackgroundEffect,
imagePathFor,
parseEffect,
serializeEffect,
shippedBackgrounds,
} from "./backgroundEffects";
describe("the chosen background effect", () => {
test("round-trips through its stored form", () => {
const effects: BackgroundEffect[] = [
{ kind: "none" },
{ kind: "blur" },
...shippedBackgrounds.map(({ id }) => ({ kind: "shipped" as const, id })),
];
for (const effect of effects)
expect(parseEffect(serializeEffect(effect))).toEqual(effect);
});
test("falls back to no effect when the stored form names nothing we ship", () => {
expect(parseEffect("image:gone")).toEqual({ kind: "none" });
expect(parseEffect("")).toEqual({ kind: "none" });
});
test("reads a stored value that is not a string as no effect", () => {
for (const raw of [null, 1, {}, ["blur"]])
expect(parseEffect(raw)).toEqual({ kind: "none" });
});
test("gives every shipped background an image to draw", () => {
for (const { id } of shippedBackgrounds)
expect(imagePathFor(id)).toBeTruthy();
});
// A background has to cover what is behind it: JPEG has no alpha channel.
test("ships only opaque backgrounds", async () => {
for (const { imagePath } of shippedBackgrounds) {
const name = imagePath.split("/").pop()!.split("?")[0];
const bytes = await readFile(`src/graphics/${name}`);
expect([bytes[0], bytes[1], bytes[2]]).toEqual([0xff, 0xd8, 0xff]);
}
});
});
+62
View File
@@ -0,0 +1,62 @@
/*
Copyright 2026 Element Creations Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import arc from "../graphics/background-arc.jpg?url";
import glow from "../graphics/background-glow.jpg?url";
export const blurRadius = 15;
export interface ShippedBackground {
/** Stored in the user's settings when chosen, so it must not change. */
id: string;
imagePath: string;
}
export const shippedBackgrounds: ShippedBackground[] = [
{ id: "arc", imagePath: arc },
{ id: "glow", imagePath: glow },
];
export type BackgroundEffect =
| { kind: "none" }
| { kind: "blur" }
| { kind: "shipped"; id: string };
/** An effect as the setting stores it, and as the menu names its option. */
export type EffectId = "none" | "blur" | `image:${string}`;
export const noEffect: BackgroundEffect = { kind: "none" };
/**
* Reads a stored effect; one we no longer ship reads as no effect. Stored by
* hand or by another build, the value need not even be a string.
*/
export function parseEffect(raw: unknown): BackgroundEffect {
if (typeof raw !== "string") return noEffect;
if (raw === "blur") return { kind: "blur" };
if (raw.startsWith("image:")) {
const id = raw.slice("image:".length);
if (shippedBackgrounds.some((b) => b.id === id))
return { kind: "shipped", id };
}
return noEffect;
}
export function serializeEffect(effect: BackgroundEffect): EffectId {
switch (effect.kind) {
case "blur":
return "blur";
case "shipped":
return `image:${effect.id}`;
default:
return "none";
}
}
export function imagePathFor(id: string): string | undefined {
return shippedBackgrounds.find((b) => b.id === id)?.imagePath;
}
+1
View File
@@ -217,6 +217,7 @@ export function getBasicCallViewModelEnvironment(
vm,
muteStates,
mediaDevices,
constant(false),
"reactionId",
{ showControls: true, header: HeaderStyle.Standard },
);
+15 -2
View File
@@ -38,8 +38,21 @@ export default defineConfig(({ mode }) => {
...base,
root: "component/dev",
// The same scoping the library build applies, so the harness shows what a
// host will get — including whether its own page is left alone
css: { postcss: { plugins: [scopeStylesToRoot()] } },
// host will get — including whether its own page is left alone. The host's
// own stylesheets, Compound's among them, stay unscoped, as on a real host.
css: {
postcss: {
plugins: [
scopeStylesToRoot({
exclude: [
/component[\\/]dev[\\/]/,
/@vector-im[\\/]compound-/,
/@fontsource[\\/]/,
],
}),
],
},
},
// So that the harness can read the same config.json the standalone app
// does, if the developer has written one
publicDir: "../../public",
+35 -4
View File
@@ -7,6 +7,7 @@ Please see LICENSE in the repository root for full details.
import {
loadEnv,
type Plugin,
type PluginOption,
searchForWorkspaceRoot,
type ConfigEnv,
@@ -43,10 +44,12 @@ export const vitePluginsConfig = ({
}),
react(),
wasm(),
nodePolyfills({
// Enables the 'events' module, which is required by the matrix-js-sdk
include: ["events"],
}),
exceptMediaPipeWasm(
nodePolyfills({
// Enables the 'events' module, which is required by the matrix-js-sdk
include: ["events"],
}),
),
svgrPlugin({
svgrOptions: {
// This enables ref forwarding on SVGR components, which is needed, for
@@ -191,3 +194,31 @@ export default ({
},
};
};
/**
* Keeps a plugin's transform off MediaPipe's WASM loader. The loader is a
* classic script that sets `self.ModuleFactory`; transformed into a module it
* sets nothing, and the segmenter fails with "ModuleFactory not set".
*/
function exceptMediaPipeWasm(plugins: PluginOption): PluginOption {
const isLoader = (id: string): boolean =>
id.includes("tasks-vision") && id.includes("wasm");
return (Array.isArray(plugins) ? plugins : [plugins]).map((plugin) => {
if (
!plugin ||
typeof plugin !== "object" ||
!("transform" in plugin) ||
typeof plugin.transform !== "function"
)
return plugin;
const transform = plugin.transform;
const wrapped: Plugin = {
...plugin,
transform(code, id, options) {
if (isLoader(id)) return null;
return transform.call(this, code, id, options);
},
};
return wrapped;
});
}