Compare commits

...
Author SHA1 Message Date
fkwpandClaude Opus 5.5 5ca9cedf30 Create the background effects with the app's view model
- AppViewModel creates them, next to its media devices, as review asked;
  BackgroundEffectsProvider takes them as a prop rather than building
  them in an effect.
- The component build, which has no AppViewModel of its own, now makes one
  in the effect that made its media devices, so both builds get their
  state the same way.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-08 14:34:45 +02:00
fkwpandClaude Opus 5.5 de2b98251c Switch the background effect with ObservableScope.reconcile
- The pipeline's switches run through reconcile, which already runs them
  one at a time and skips those overtaken, in place of a queue of our own.
- A failed switch is logged and the next still runs, now tested.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-08 12:54:54 +02:00
fkwpandClaude Opus 5.5 32e7c407c3 Make the background effects state an interface built by a function
- BackgroundEffects is an interface, built by createBackgroundEffects,
  following the convention for state holders, as review asked.
- Same behaviour: the body moves from the constructor unchanged.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-08 12:22:34 +02:00
fkwpandClaude Opus 5.5 e9365b85b5 Name the background effect pieces after background effects
- One name, as review asked: TrackProcessorContext becomes
  BackgroundEffectsContext, with BackgroundEffectsProvider,
  BackgroundEffectsState, useBackgroundEffects,
  useBackgroundEffectsState$ and useSyncBackgroundEffects.
- The state's Behavior is backgroundEffectsState$ wherever it is passed,
  in place of trackProcessorState$, trackerProcessorState$ and
  processorState$.
- Renames only: nothing changes in behaviour.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-08 12:22:34 +02:00
fkwpandClaude Opus 5.5 cbb22a0094 Keep background effects to desktops until the menu says what they cost
- The one support check asks for a desktop again, as #4059 had the
  pipeline do: menu and pipeline still agree, so a phone is neither
  offered blur nor refused it.
- Lifting it moves to the PR that tells the user where effects run
  slowly, so it is their choice to make.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-08 12:22:34 +02:00
fkwpandClaude Opus 5.5 ce41fee6ff Tidy the background effect pipeline after review
- The Vite transform wrapper takes Vite's own types, with no cast.
- The pipeline's options are compared with deepCompare, not as JSON.
- Renames `attached` to `enable` and shortens the pipeline's doc comment.
- Drops a comment that blamed an ended track on the pipeline's queue.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-08 12:22:34 +02:00
fkwpandClaude Opus 5.5 e2b89a64b9 Ask one question about whether background effects can run
- The menu asked the browser; the pipeline asked the browser and for a
  desktop. On a phone whose browser can run them, blur was offered and
  then refused: the toggle moved and the video did not change.
- Both now ask one function, and it asks the browser alone. The platform
  was the wrong measure: a phone on the fast route held 99% of its frame
  rate and was refused, a desktop on the slow one held 73% and was
  allowed.
- The in-call device switcher is still withheld on a phone; that is the
  device menu's own rule and is unchanged.
- The provider tests run as a phone, and fail with the desktop test back.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-08 12:22:34 +02:00
fkwpandClaude Opus 5.5 9823103ef7 Attach background effects through one switchable pipeline
- One pipeline, shared by the pre-join preview and the call, attached the
  first time an effect is wanted and never detached after; turning blur
  off switches it to a disabled mode in place. Main dropped the processor
  whenever blur went off, and a pipeline taken off the camera is destroyed
  and re-primed, so each time blur came back one unprocessed frame went
  out. Built once, the priming frame is spent once.
- Still driven by the existing blur setting: nothing a user sees changes.
- Its logic is BackgroundEffects in src/state/, which the provider only
  builds, so the tests drive it without rendering anything.
- Disabled drops the blur radius too. The SDK's disabled mode keeps it,
  and with a radius every frame is still segmented and thrown away; with
  neither a radius nor a picture, frames pass untouched.
- Switches run one at a time and skip any overtaken, so an earlier,
  slower switch can't land after a later choice.
- Builds, rebuilds and teardowns of the pipeline run one at a time
  (OneStepPipeline). The preview's and the call's tracks attach and stop
  it independently, and the SDK's wrapper holds one track's streams: two
  builds at once overwrite each other's, and a stop during a build does
  nothing.
- Every camera track starts with the pipeline on: the room's capture
  defaults follow it, and a camera turned off before the first effect is
  replaced on turning it on, since a stopped track can't take one. One
  attached after publishing lets the room through while it is built.
- BlurBackgroundTransformer becomes BackgroundEffectTransformer: it also
  sits disabled, and its init applies that state as the SDK's would.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-08 12:22:34 +02:00
fkwpandClaude Opus 5.5 98152f162d Let background effects run under the dev server
- nodePolyfills rewrites any module that mentions process, and MediaPipe's
  WASM loader does. The loader is a classic script that sets
  self.ModuleFactory; as a module it sets nothing, so under pnpm dev the
  segmenter failed with "ModuleFactory not set". Blur on main fails the
  same way there. Production is unaffected: the loader ships as a static
  asset.
- The plugin's transform now skips that loader. Checked with main's own
  BlurBackgroundTransformer: it initialises with this and fails without.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-08 12:22:34 +02:00
fkwpandClaude Opus 5.5 b1255f0424 Size and scroll the device menus with Compound's menu
- Compound 10.3.0 bounds a menu by the space it has and scrolls what
  doesn't fit, within a boundary the app passes. The menus pass the call
  area, 16px in from its sides, rather than measuring its width.
- The 296px width is the menu's own style. Compound gives that style only
  to the dropdown, so menuIsDrawer, which copied its phone rule, goes.
- Compound draws the menu's frame over its content, so nothing that
  scrolls can hide it in Safari: the list, headings and meter no longer
  keep a border width clear of it, and the checks that they did go.
- The device list is Compound's scroll area, so it fades out at the
  foot with Compound's chevron while there is more below.
- The microphone menu's meter sticks at that foot, with rows scrolling
  under it as before, so the list has no fade above it: Compound's is a
  mask fixed to the list's edge, while a bounce at the top carries the
  meter with the rows. The band under the meter is filled, or rows would
  show there. The edge fades from where the chevron starts, so a bounce
  fades the meter out before it reaches the chevron. A row reached by
  keyboard stays above the meter.
- The list's height is still its share of the call.
- The component harness leaves the host's stylesheets unscoped, Compound's
  among them, as a host does: scoped, they never reached the menu, which is
  portalled out of Element Call's root.
- Its specs check only Element Call's own document styles on the host
  page, and find the device list within Compound's scroll area.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-08 12:22:34 +02:00
fkwpandClaude Opus 5.5 65686ee9fc Hold both device menus at 296px, framed in Safari too
- The camera and microphone menus were as wide as their longest device
  name, so a long one widened the menu over more of the call: 340px for a
  long speaker name. Both are now 296px, as design sets them, and a long
  name wraps.
- Narrower only where the call area is, so the menu still fits inside it:
  268px in a 300px call. Taken from the observation of the call area that
  already bounds the list's height.
- Held only for the dropdown. On Android and iOS Compound renders the menu
  as a drawer as wide as the screen; menuIsDrawer follows its rule.
- Once the list is long enough to scroll, Safari paints it above the
  outline the menu is framed with, and the frame vanished down both sides
  of it: 384px of it on the many-devices microphone menu, in WebKit. The
  list now keeps a border width clear of the frame, so the headings and
  the meter inside it no longer need their own.
- Guarded with a long speaker name, which fails at 340px on main's sizing;
  a unit test run as a phone, which fails with the width held there; and
  the many-devices story's inset, which fails on the old stylesheet, since
  the stories run in Chromium, which never showed the Safari bug.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-07 09:57:28 +02:00
renovate[bot] 2faa6c3dac Update ghcr.io/element-hq/element-web:develop Docker digest to 589c490 (#4328)
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
2026-10-07 01:42:16 +00:00
48 changed files with 1205 additions and 394 deletions
@@ -109,4 +109,12 @@ describe("scopeStylesToRoot", () => {
".a .b_abc12:hover { border: 0 }",
);
});
it("leaves an excluded stylesheet as it is", async () => {
const css = "body { margin: 0 } .menu { color: red }";
const result = await postcss([
scopeStylesToRoot({ exclude: [/host\.css$/] }),
]).process(css, { from: "/app/host.css" });
expect(result.css).toBe(css);
});
});
+9 -3
View File
@@ -60,12 +60,18 @@ const IS_ROOT = `:where(${ROOT_SELECTOR})`;
* tokens, the component styles, the fonts — is not in this build at all: the
* host supplies it, unscoped, along with its copy of Compound.
*/
export function scopeStylesToRoot(): Plugin {
interface Options {
/** Stylesheets left as they are: a host's own, which it doesn't scope. */
exclude?: RegExp[];
}
export function scopeStylesToRoot({ exclude = [] }: Options = {}): Plugin {
return {
postcssPlugin: "element-call-scope-styles-to-root",
Once(root) {
const isModule =
root.source?.input.file?.endsWith(".module.css") ?? false;
const file = root.source?.input.file ?? "";
if (exclude.some((pattern) => pattern.test(file))) return;
const isModule = file.endsWith(".module.css");
root.walkRules((rule) => {
if (isRelative(rule)) return;
rule.selector = (isModule ? scopeBare : scopeAll).processSync(
+11 -12
View File
@@ -72,9 +72,9 @@ import {
useUrlParams,
} from "../src/UrlParams";
import { MediaDevicesContext } from "../src/MediaDevicesContext";
import { MediaDevices } from "../src/state/MediaDevices";
import { AppViewModel } from "../src/state/AppViewModel";
import { ObservableScope } from "../src/state/ObservableScope";
import { ProcessorProvider } from "../src/livekit/TrackProcessorContext";
import { BackgroundEffectsProvider } from "../src/livekit/BackgroundEffectsContext";
import { Config } from "../src/config/Config";
import { type ConfigOptions } from "../src/config/ConfigOptions";
import { i18n } from "../src/utils/i18n";
@@ -192,14 +192,12 @@ export const ElementCall: FC<ElementCallProps> = ({
// observers running for the rest of the page's life. Null until then, which
// is one render.
const { controlledAudioDevices, callIntent } = params;
const [mediaDevices, setMediaDevices] = useState<MediaDevices | null>(null);
const [vm, setVm] = useState<AppViewModel | null>(null);
useEffect(() => {
const scope = new ObservableScope();
setMediaDevices(
new MediaDevices(scope, { controlledAudioDevices, callIntent }),
);
setVm(new AppViewModel(scope, { controlledAudioDevices, callIntent }));
return (): void => {
setMediaDevices(null);
setVm(null);
scope.end();
};
}, [controlledAudioDevices, callIntent]);
@@ -220,8 +218,7 @@ export const ElementCall: FC<ElementCallProps> = ({
// is told that Element Call has loaded, as the widget tells its client once
// its own initialisation is over. Once per mount, however often the pieces
// are later swapped out.
const ready =
container !== null && rtcSession !== null && mediaDevices !== null;
const ready = container !== null && rtcSession !== null && vm !== null;
const announcedLoaded = useRef(false);
useEffect(() => {
if (!ready || announcedLoaded.current) return;
@@ -251,8 +248,10 @@ export const ElementCall: FC<ElementCallProps> = ({
<Decoration>
<TooltipProvider>
<ClientProvider client={client}>
<MediaDevicesContext value={mediaDevices}>
<ProcessorProvider>
<MediaDevicesContext value={vm.mediaDevices}>
<BackgroundEffectsProvider
effects={vm.backgroundEffects}
>
<CallView
client={client}
rtcSession={rtcSession}
@@ -261,7 +260,7 @@ export const ElementCall: FC<ElementCallProps> = ({
preload={params.preload}
skipLobby={params.skipLobby}
/>
</ProcessorProvider>
</BackgroundEffectsProvider>
</MediaDevicesContext>
</ClientProvider>
</TooltipProvider>
+2 -2
View File
@@ -19,7 +19,7 @@ services:
- ./backend/playwright_homeserver-othersite.yaml:/data/cfg/homeserver.yaml:Z
element-web:
# Pin to a SHA so that upstream cannot break our tests. Renovate handles regular updates.
image: ghcr.io/element-hq/element-web:develop@sha256:39c94f0c29e7c5614ac7264a872bb3643320158b3bf1730fc4742ddf672c27d6
image: ghcr.io/element-hq/element-web:develop@sha256:589c4909fd5844a9db7dd730f9796186eabbb981580cdf9bfa597186a2fb81f3
element-web-1:
# Pin to a SHA so that upstream cannot break our tests. Renovate handles regular updates.
image: ghcr.io/element-hq/element-web:develop@sha256:39c94f0c29e7c5614ac7264a872bb3643320158b3bf1730fc4742ddf672c27d6
image: ghcr.io/element-hq/element-web:develop@sha256:589c4909fd5844a9db7dd730f9796186eabbb981580cdf9bfa597186a2fb81f3
+1 -1
View File
@@ -84,7 +84,7 @@
"@typescript-eslint/utils": "^8.61.0",
"@use-gesture/react": "^10.2.11",
"@vector-im/compound-design-tokens": "^11.0.0",
"@vector-im/compound-web": "^10.0.0",
"@vector-im/compound-web": "^10.3.0",
"@vitejs/plugin-react": "^6.0.2",
"@vitest/browser-playwright": "^4.1.5",
"@vitest/coverage-v8": "^4.0.18",
+4 -1
View File
@@ -129,7 +129,10 @@ async function openDeviceList(page: Page, pane: Locator): Promise<Locator> {
.getByRole("button", { name: "Microphone" })
.click({ timeout: 60_000 });
await expect(page.getByRole("menu")).toBeVisible();
const list = page.locator("[role='menu'] div[role='none']").first();
// The area of the menu holding the device groups, not the menu's own.
const list = page.locator(
"[role='menu'] div[role='none']:has(> [role='group'])",
);
await expect(list).toBeVisible();
return list;
}
+9 -15
View File
@@ -125,25 +125,19 @@ test("leaves the host's own page unstyled", async ({ page }) => {
timeout: 60_000,
});
// Element Call's stylesheet is written for a page of its own: normalize.css
// gives `html` a line height, Compound gives `body` its font and feature
// settings, and the design tokens live on `:root`. None of that may reach the
// host's document — the harness sets none of these itself, so anything other
// than the browser's defaults here came from us.
// Compound's document styles reach the host, but it loads them itself.
const host = await page.evaluate(() => {
const html = getComputedStyle(document.documentElement);
const body = getComputedStyle(document.body);
const sub = document.body.appendChild(document.createElement("sub"));
const verticalAlign = getComputedStyle(sub).verticalAlign;
sub.remove();
return {
lineHeight: html.lineHeight,
fontFeatureSettings: body.fontFeatureSettings,
token: html.getPropertyValue("--cpd-color-text-primary"),
verticalAlign,
token: getComputedStyle(document.documentElement).getPropertyValue(
"--font-scale",
),
};
});
expect(host).toEqual({
lineHeight: "normal",
fontFeatureSettings: "normal",
token: "",
});
expect(host).toEqual({ verticalAlign: "sub", token: "" });
// While inside the container, the same rules do apply
const root = panes.first().locator("[data-element-call-root]");
+6 -6
View File
@@ -134,8 +134,8 @@ importers:
specifier: ^11.0.0
version: 11.1.0(@types/react@19.2.18)(react@19.2.8)
'@vector-im/compound-web':
specifier: ^10.0.0
version: 10.1.0(@fontsource/inconsolata@5.3.0)(@fontsource/inter@5.3.0)(@types/react-dom@19.2.5(@types/react@19.2.18))(@types/react@19.2.18)(@vector-im/compound-design-tokens@11.1.0(@types/react@19.2.18)(react@19.2.8))(react-dom@19.2.8(react@19.2.8))(react@19.2.8)
specifier: ^10.3.0
version: 10.3.0(@fontsource/inconsolata@5.3.0)(@fontsource/inter@5.3.0)(@types/react-dom@19.2.5(@types/react@19.2.18))(@types/react@19.2.18)(@vector-im/compound-design-tokens@11.1.0(@types/react@19.2.18)(react@19.2.8))(react-dom@19.2.8(react@19.2.8))(react@19.2.8)
'@vitejs/plugin-react':
specifier: ^6.0.2
version: 6.1.1(@rolldown/plugin-babel@0.2.3(@babel/core@7.29.7(supports-color@7.2.0))(@babel/runtime@8.0.0)(rolldown@1.2.7)(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(sass@1.103.1)(terser@5.46.1)(yaml@2.9.0)))(babel-plugin-react-compiler@1.0.0)(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(sass@1.103.1)(terser@5.46.1)(yaml@2.9.0))
@@ -3240,13 +3240,13 @@ packages:
react:
optional: true
'@vector-im/compound-web@10.1.0':
resolution: {integrity: sha512-Lhw9Q+RZo1GtDYjF5PXWTD0zdfbrz/vnSVBYFHC8Hq+YeDule2QfGbTRo8IVHeV7jK6ZzsLc8zb6553eeIRLYg==}
'@vector-im/compound-web@10.3.0':
resolution: {integrity: sha512-tce5hsNpdvYXfB5nYQgTK7WjTDqjdxpV1BeGsw67gYAWSFKJr9p6X5gzE79o+Tp/q5uykLuO4ozTBdjLsmUWMw==}
peerDependencies:
'@fontsource/inconsolata': ^5
'@fontsource/inter': ^5
'@types/react': '*'
'@vector-im/compound-design-tokens': '>=1.6.1 <11.0.0'
'@vector-im/compound-design-tokens': '>=1.6.1 <12.0.0'
react: ^18 || ^19.0.0
peerDependenciesMeta:
'@types/react':
@@ -8672,7 +8672,7 @@ snapshots:
'@types/react': 19.2.18
react: 19.2.8
'@vector-im/compound-web@10.1.0(@fontsource/inconsolata@5.3.0)(@fontsource/inter@5.3.0)(@types/react-dom@19.2.5(@types/react@19.2.18))(@types/react@19.2.18)(@vector-im/compound-design-tokens@11.1.0(@types/react@19.2.18)(react@19.2.8))(react-dom@19.2.8(react@19.2.8))(react@19.2.8)':
'@vector-im/compound-web@10.3.0(@fontsource/inconsolata@5.3.0)(@fontsource/inter@5.3.0)(@types/react-dom@19.2.5(@types/react@19.2.18))(@types/react@19.2.18)(@vector-im/compound-design-tokens@11.1.0(@types/react@19.2.18)(react@19.2.8))(react-dom@19.2.8(react@19.2.8))(react@19.2.8)':
dependencies:
'@floating-ui/react': 0.27.20(react-dom@19.2.8(react@19.2.8))(react@19.2.8)
'@floating-ui/react-dom': 2.1.9(react-dom@19.2.8(react@19.2.8))(react@19.2.8)
+3 -3
View File
@@ -35,7 +35,7 @@ import { ErrorPage, LoadingPage } from "./FullScreenView";
import { Initializer } from "./initializer";
import { type WidgetHelpers } from "./widget";
import { useTheme } from "./useTheme";
import { ProcessorProvider } from "./livekit/TrackProcessorContext";
import { BackgroundEffectsProvider } from "./livekit/BackgroundEffectsContext";
import { type AppViewModel } from "./state/AppViewModel";
import { MediaDevicesContext } from "./MediaDevicesContext";
import {
@@ -148,7 +148,7 @@ export const App: FC<Props> = ({ vm, widget }) => {
loaded && clientReady ? (
<ClientProvider client={widgetClient}>
<MediaDevicesContext value={vm.mediaDevices}>
<ProcessorProvider>
<BackgroundEffectsProvider effects={vm.backgroundEffects}>
<Sentry.ErrorBoundary
fallback={(error) => <ErrorPage error={error} />}
>
@@ -159,7 +159,7 @@ export const App: FC<Props> = ({ vm, widget }) => {
<SentryRoute path="*" element={<RoomPage />} />
</Routes>
</Sentry.ErrorBoundary>
</ProcessorProvider>
</BackgroundEffectsProvider>
</MediaDevicesContext>
</ClientProvider>
) : (
+2 -2
View File
@@ -15,7 +15,7 @@ exports[`AppBar > renders 1`] = `
tabindex="0"
>
<div
class="_indicator-icon_147l5_17"
class="_indicator-icon_j097f_17"
style="--cpd-icon-button-size: 100%;"
>
<svg
@@ -58,7 +58,7 @@ exports[`AppBar > renders with title and subtitle 1`] = `
tabindex="0"
>
<div
class="_indicator-icon_147l5_17"
class="_indicator-icon_j097f_17"
style="--cpd-icon-button-size: 100%;"
>
<svg
+58 -4
View File
@@ -14,7 +14,10 @@ import type { CallViewModel } from "../state/CallViewModel/CallViewModel";
import type { Alignment, Layout } from "../state/layout-types";
import type { SpotlightTileViewModel } from "../state/TileViewModel";
import type { DeviceLabel } from "../state/MediaDevices";
import { createCallFooterViewModel } from "./CallFooterViewModel";
import {
createCallFooterViewModel,
createLobbyFooterViewModel,
} from "./CallFooterViewModel";
import { HeaderStyle } from "../UrlParams";
const platformMock = vi.hoisted(() => vi.fn(() => "desktop"));
@@ -24,10 +27,10 @@ vi.mock("../Platform", () => ({
},
}));
// Prevent supportsBackgroundProcessors from throwing in jsdom – it is not
// exercised by these tests (only used in `videoToggles`, not `videoOptions`).
// The SDK's own check needs WebGL and canvas APIs jsdom does not have.
const sdkSupportMock = vi.hoisted(() => vi.fn(() => false));
vi.mock("@livekit/track-processors", () => ({
supportsBackgroundProcessors: (): boolean => false,
supportsBackgroundProcessors: (): boolean => sdkSupportMock(),
}));
const outputSelectionMock = vi.hoisted(() => vi.fn(() => true));
@@ -222,4 +225,55 @@ describe("createCallFooterViewModel", () => {
]);
});
});
describe("background effects", () => {
// The lobby, because in a call the whole device switcher is already
// withheld on a phone, whatever the verdict says.
function lobbyOn(
platform: string,
): ReturnType<typeof createLobbyFooterViewModel> {
platformMock.mockReturnValue(platform);
return createLobbyFooterViewModel(
testScope(),
mockMuteStates(),
twoMicsAndOneCamMediaDevices,
/* openSettings */ undefined,
/* hangup */ undefined,
/* showLogo */ false,
);
}
it("offers nothing on a phone, where the pipeline would refuse them", () => {
sdkSupportMock.mockReturnValue(true);
expect(lobbyOn("ios").toggleBlur$.value).toBeUndefined();
});
it("offers them where the pipeline will honour them", () => {
sdkSupportMock.mockReturnValue(true);
expect(lobbyOn("desktop").toggleBlur$.value).toBeDefined();
});
it("offers nothing where the browser itself cannot run them", () => {
sdkSupportMock.mockReturnValue(false);
expect(lobbyOn("desktop").toggleBlur$.value).toBeUndefined();
});
it("availability is the same before and during a call", () => {
for (const supported of [true, false]) {
sdkSupportMock.mockReturnValue(supported);
const lobby = lobbyOn("desktop");
const inCall = createCallFooterViewModel(
testScope(),
buildMinimalCallViewModel(gridLayout),
mockMuteStates(),
twoMicsAndOneCamMediaDevices,
/* reactionIdentifier */ undefined,
{ showControls: true, header: HeaderStyle.Standard },
);
const offeredInLobby = lobby.toggleBlur$.value !== undefined;
expect(inCall.toggleBlur$.value !== undefined).toBe(offeredInLobby);
expect(offeredInLobby).toBe(supported);
}
});
});
});
+2 -1
View File
@@ -6,9 +6,10 @@ Please see LICENSE in the repository root for full details.
*/
import { combineLatest, map, type Observable, switchMap } from "rxjs";
import { supportsBackgroundProcessors } from "@livekit/track-processors";
import { supportsAudioOutputSelection } from "livekit-client";
import { supportsBackgroundProcessors } from "../livekit/backgroundProcessing";
import { type CallViewModel } from "../state/CallViewModel/CallViewModel";
import { type MenuOptions } from "./MediaMuteAndSwitchButton";
import { type MediaDevices } from "../state/MediaDevices";
@@ -36,20 +36,17 @@ Please see LICENSE in the repository root for full details.
}
}
/* The width design sets: a long device name wraps. */
.menu {
display: flex;
flex-direction: column;
inline-size: 296px;
}
/* Only the device lists scroll, bounded by the measured call area: the menu is
portalled out of the root, so CSS can't size it against the call. */
/* Its share of the measured call area: the menu is portalled out of the root,
so CSS can't size it against the call. */
.deviceList {
overflow-y: auto;
min-block-size: 0;
max-block-size: var(--device-list-max-height);
/* Keeps a row reached by keyboard clear of the sticky heading and meter. */
scroll-padding-block: var(--device-list-scroll-padding-start, 0)
var(--device-list-scroll-padding-end, 0);
/* Keeps a row reached by keyboard clear of the sticky heading. */
scroll-padding-block-start: var(--device-list-scroll-padding-start, 0);
}
/* Compound's heading rule heads each section; the design adds space below it. */
@@ -62,23 +59,38 @@ Please see LICENSE in the repository root for full details.
margin-block-start: var(--cpd-space-7x);
}
/* Sticky within its own section. Opaque and inset a border width so it doesn't
paint over the menu's outline. */
/* Sticky within its own section, and opaque so rows scroll under it. */
.sectionHeading {
position: sticky;
inset-block-start: 0;
background: var(--cpd-color-bg-canvas-default);
margin-inline: var(--cpd-border-width-1);
margin-block-start: var(--cpd-border-width-1);
}
/* Sticky at the foot of the microphone section; opaque and inset as above. */
/* Sticky at the foot of the microphone section; opaque as above. */
.stickyMeter {
position: sticky;
inset-block-end: 0;
background: var(--cpd-color-bg-canvas-default);
margin-inline: var(--cpd-border-width-1);
margin-block-end: var(--cpd-border-width-1);
}
/* Fills the band between the meter and the frame, or rows show through it. */
.stickyMeter::after {
content: "";
position: absolute;
inset-inline: 0;
inset-block-start: 100%;
block-size: var(--cpd-menu-padding-block-end, 0px);
background: var(--cpd-color-bg-canvas-default);
}
/* Not Compound's fade: that stays at the edge while a bounce carries the meter.
This one starts at the chevron, below the meter at rest. */
.deviceList.deviceListWithMeter[data-more-below] {
mask-image: linear-gradient(
to bottom,
#000 calc(100% - var(--cpd-space-6x)),
transparent calc(100% - var(--cpd-space-3x))
);
}
/* Radix focuses whatever the pointer is over, so the browser's ring would
@@ -65,8 +65,10 @@ const WithACallArea: FC<{ children: ReactNode }> = ({ children }) => {
<div
ref={setCallArea}
style={{
// A call's size: anything smaller makes a short device list scroll.
// A call's size: anything smaller makes a short device list scroll, and
// a narrower one narrows the menu.
blockSize: 720,
inlineSize: 1024,
display: "flex",
alignItems: "flex-end",
justifyContent: "center",
@@ -256,6 +258,37 @@ export const SpeakerAndMicrophoneSections: Story = {
},
};
/** A long name wraps rather than widening the menu, which design sets. */
export const LongDeviceNameWraps: Story = {
args: {
...SpeakerAndMicrophoneSections.args,
outputOptions: [
{
label: {
type: "name",
name: "Logitech BRIO 4K Ultra HD Pro Business Webcam Speakers (046d:085e)",
},
id: "long",
},
{ label: { type: "name", name: "Headset" }, id: "spk2" },
],
selectedOutputOption: "long",
},
play: async ({ canvasElement }) => {
const canvas = within(canvasElement);
await userEvent.click(canvas.getByRole("button", { name: "Microphone" }));
const body = within(document.body);
const long = await body.findByRole("menuitemradio", { name: /Logitech/ });
const short = await body.findByRole("menuitemradio", { name: "Headset" });
const menu = document.body.querySelector("[role='menu']")!;
await expect(Math.round(menu.getBoundingClientRect().width)).toBe(296);
await expect(long.getBoundingClientRect().height).toBeGreaterThan(
short.getBoundingClientRect().height,
);
},
};
export const OutputCannotBeChosen: Story = {
args: {
...Default.args,
@@ -436,10 +469,6 @@ export const ManyDevices: Story = {
.getBoundingClientRect();
await expect(frame.top).toBeGreaterThanOrEqual(0);
await expect(frame.bottom).toBeLessThanOrEqual(window.innerHeight + 1);
// The meter is the menu's one opaque part, so it must stay inside the frame.
await expect(pinned.left).toBeGreaterThan(frame.left);
await expect(pinned.right).toBeLessThan(frame.right);
},
};
@@ -607,13 +636,6 @@ export const HeadingsStayWhileScrolling: Story = {
await expect(heading.getBoundingClientRect().bottom).toBeGreaterThan(
scrollport.top,
);
// Clear of the menu's frame.
const frame = document.body
.querySelector("[role='menu']")!
.getBoundingClientRect();
await expect(heading.getBoundingClientRect().left).toBeGreaterThan(
frame.left,
);
},
};
+17 -7
View File
@@ -17,6 +17,7 @@ import {
Button,
Menu,
MenuItem,
MenuScrollArea,
MenuTitle,
RadioInput,
ToggleMenuItem,
@@ -84,6 +85,9 @@ const LIST_SHARE_OF_CALL = 0.6;
/** Smallest device list height in px, so a short call still shows more than one device. */
const MIN_LIST_HEIGHT = 160;
/** Space kept between the menu and the call area's sides. */
const MENU_MARGIN = 16;
export const MediaMuteAndSwitchButton: FC<MediaMuteAndSwitchButtonProps> = ({
enabled,
busy,
@@ -326,6 +330,8 @@ export const MediaMuteAndSwitchButton: FC<MediaMuteAndSwitchButtonProps> = ({
open={menuOpen}
onOpenChange={onOpenChange}
side="top"
collisionBoundary={rootElement}
collisionPadding={MENU_MARGIN}
trigger={
<Button
iconOnly
@@ -340,17 +346,21 @@ export const MediaMuteAndSwitchButton: FC<MediaMuteAndSwitchButtonProps> = ({
/>
}
>
<div
<MenuScrollArea
ref={trackFocusSource}
// Keeps the items the menu's own children for assistive tech.
role="none"
className={styles.deviceList}
className={classNames(styles.deviceList, {
[styles.deviceListWithMeter]: iconsAndLabels === "audio",
})}
style={
{
"--device-list-max-height":
listMaxHeight === undefined ? undefined : `${listMaxHeight}px`,
"--device-list-scroll-padding-end":
meterHeight === undefined ? undefined : `${meterHeight}px`,
// A row reached by keyboard stays above the meter and the band its
// ground fills.
scrollPaddingBlockEnd:
meterHeight === undefined
? undefined
: `calc(${meterHeight}px + var(--cpd-menu-padding-block-end, 0px))`,
"--device-list-scroll-padding-start":
headingHeight === undefined ? undefined : `${headingHeight}px`,
} as CSSProperties
@@ -398,7 +408,7 @@ export const MediaMuteAndSwitchButton: FC<MediaMuteAndSwitchButtonProps> = ({
)}
</div>
</div>
</div>
</MenuScrollArea>
{toggles.length > 0 && <hr />}
{toggles.map((toggle) => (
<ToggleMenuItem
@@ -16,10 +16,10 @@ Please see LICENSE in the repository root for full details.
align-items: center;
gap: var(--cpd-space-3x);
padding-block: var(--cpd-space-2x);
/* Centres the icon on the radio column: less the row's inset border and half
the icon's extra width. Checked in the MeterAlignsWithTheDeviceRows story. */
/* Centres the icon on the radio column: less half the icon's extra width.
Checked in the MeterAlignsWithTheDeviceRows story. */
padding-inline-start: calc(
var(--cpd-space-4x) - var(--cpd-border-width-1) -
var(--cpd-space-4x) -
(var(--meter-icon-size) - var(--device-control-size)) / 2
);
padding-inline-end: calc(var(--cpd-space-4x) * 2 + var(--cpd-space-2x));
@@ -0,0 +1,24 @@
/*
Copyright 2026 Element Creations Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { describe, expect, it } from "vitest";
import { BackgroundEffectTransformer } from "./BackgroundEffectTransformer";
describe("BackgroundEffectTransformer", () => {
it("passes frames on untouched once disabled, even after blurring", async () => {
const transformer = new BackgroundEffectTransformer({});
await transformer.update({ blurRadius: 15, backgroundDisabled: false });
await transformer.update({
imagePath: undefined,
backgroundDisabled: true,
});
// With neither, the library passes a frame on without segmenting it.
expect(transformer.options.blurRadius).toBeUndefined();
expect(transformer.options.imagePath).toBeUndefined();
});
});
@@ -1,5 +1,6 @@
/*
Copyright 2024-2025 New Vector Ltd.
Copyright 2026 Element Creations Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
@@ -9,6 +10,7 @@ import {
BackgroundTransformer,
VideoTransformer,
type VideoTransformerInitOptions,
type BackgroundOptions,
} from "@livekit/track-processors";
import { ImageSegmenter } from "@mediapipe/tasks-vision";
@@ -49,7 +51,18 @@ const wasmFileset: WasmFileset = {
* loads the segmentation models from our own bundle rather than as an external
* resource fetched from the public internet.
*/
export class BlurBackgroundTransformer extends BackgroundTransformer {
export class BackgroundEffectTransformer extends BackgroundTransformer {
/**
* As the library's, except that disabling also drops the blur radius: kept,
* it has every frame segmented and the result thrown away, where with
* neither a radius nor a picture frames are passed on untouched.
*/
public override async update(opts: BackgroundOptions): Promise<void> {
await super.update(
opts.backgroundDisabled ? { ...opts, blurRadius: undefined } : opts,
);
}
public async init({
outputCanvas,
inputElement: inputVideo,
@@ -73,8 +86,10 @@ export class BlurBackgroundTransformer extends BackgroundTransformer {
outputConfidenceMasks: false,
});
// BackgroundTransformer's own init applies these, and this one replaces it.
if (this.options.blurRadius) {
this.gl?.setBlurRadius(this.options.blurRadius);
}
this.gl?.setBackgroundDisabled(this.options.backgroundDisabled ?? false);
}
}
@@ -0,0 +1,242 @@
/*
Copyright 2026 Element Creations Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { act, createElement, type FC } from "react";
import { render } from "@testing-library/react";
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
import { type LocalVideoTrack } from "livekit-client";
import {
type BackgroundOptions,
type ProcessorWrapper,
} from "@livekit/track-processors";
import {
applyProcessor,
BackgroundEffectsProvider,
type BackgroundEffectsState,
syncBackgroundEffects,
useBackgroundEffects,
useSyncBackgroundEffects,
createAppBackgroundEffects,
} from "./BackgroundEffectsContext";
import { type BackgroundEffects } from "../state/BackgroundEffects";
import { backgroundBlur } from "../settings/settings";
import { constant } from "../state/Behavior";
import { flushPromises, testScope } from "../utils/test";
const pipelines = vi.hoisted(() => ({
built: 0,
destroyed: 0,
switches: [] as unknown[],
}));
// The pipeline itself needs WebGL and MediaPipe; what these tests are about is
// what the provider asks of it.
vi.mock("@livekit/track-processors", () => ({
BackgroundProcessorWrapper: vi.fn(function (this: Record<string, unknown>) {
pipelines.built += 1;
this.switchTo = vi.fn(async (options: unknown) => {
pipelines.switches.push(options);
return Promise.resolve();
});
this.destroy = vi.fn(async () => {
pipelines.destroyed += 1;
return Promise.resolve();
});
}),
supportsBackgroundProcessors: (): boolean => true,
}));
const platformMock = vi.hoisted(() => ({ platform: "desktop" }));
vi.mock("../Platform", () => platformMock);
vi.mock("./BackgroundEffectTransformer", () => ({
BackgroundEffectTransformer: vi.fn(),
}));
const processor = {} as ProcessorWrapper<BackgroundOptions>;
function mockTrack(
readyState: MediaStreamTrackState,
current?: ProcessorWrapper<BackgroundOptions>,
): LocalVideoTrack {
return {
mediaStreamTrack: { readyState },
getProcessor: vi.fn().mockReturnValue(current),
setProcessor: vi.fn().mockResolvedValue(undefined),
stopProcessor: vi.fn().mockResolvedValue(undefined),
} as unknown as LocalVideoTrack;
}
describe("applyProcessor", () => {
it("attaches the processor to a live track", () => {
const track = mockTrack("live");
applyProcessor(track, processor);
expect(track.setProcessor).toHaveBeenCalledWith(processor);
});
it("does not attach the processor to an ended track", () => {
const track = mockTrack("ended");
applyProcessor(track, processor);
expect(track.setProcessor).not.toHaveBeenCalled();
});
it("does not surface a rejected setProcessor", async () => {
const track = mockTrack("live");
vi.mocked(track.setProcessor).mockRejectedValue(
new TypeError("Input track cannot be ended"),
);
const unhandled = vi.fn();
process.on("unhandledRejection", unhandled);
applyProcessor(track, processor);
await flushPromises();
process.off("unhandledRejection", unhandled);
expect(unhandled).not.toHaveBeenCalled();
});
it("stops the processor when none is wanted", () => {
const track = mockTrack("live", processor);
applyProcessor(track, undefined);
expect(track.stopProcessor).toHaveBeenCalled();
});
it("does not surface a rejected stopProcessor", async () => {
const track = mockTrack("live", processor);
vi.mocked(track.stopProcessor).mockRejectedValue(new Error("nope"));
const unhandled = vi.fn();
process.on("unhandledRejection", unhandled);
applyProcessor(track, undefined);
await flushPromises();
process.off("unhandledRejection", unhandled);
expect(unhandled).not.toHaveBeenCalled();
});
});
describe("syncBackgroundEffects", () => {
it("applies the processor to the current track", () => {
const track = mockTrack("live");
syncBackgroundEffects(
testScope(),
constant(track),
constant({ supported: true, processor }),
);
expect(track.setProcessor).toHaveBeenCalledWith(processor);
});
});
describe("BackgroundEffectsProvider", () => {
/** A camera track that holds whatever processor it is given. */
function cameraTrack(): LocalVideoTrack {
let current: unknown;
return {
mediaStreamTrack: { readyState: "live" },
getProcessor: vi.fn(() => current),
setProcessor: vi.fn(async (next: unknown) => {
current = next;
return Promise.resolve();
}),
stopProcessor: vi.fn(async () => {
current = undefined;
return Promise.resolve();
}),
} as unknown as LocalVideoTrack;
}
let seen: BackgroundEffectsState[];
const Surface: FC<{ track: LocalVideoTrack | null }> = ({ track }) => {
seen.push(useBackgroundEffects());
useSyncBackgroundEffects(track);
return null;
};
// One component for every render, so a rerender updates the tree rather than
// mounting a second provider with a pipeline of its own.
let effects: BackgroundEffects | undefined;
const Surfaces: FC<{ tracks: (LocalVideoTrack | null)[] }> = ({ tracks }) =>
createElement(BackgroundEffectsProvider, {
// Made on first render, once a test has set up the platform.
effects: (effects ??= createAppBackgroundEffects(testScope())),
children: createElement(
"div",
null,
...tracks.map((track, key) => createElement(Surface, { key, track })),
),
});
const surfaces = (
...tracks: (LocalVideoTrack | null)[]
): ReturnType<typeof createElement> => createElement(Surfaces, { tracks });
const blur = async (on: boolean): Promise<void> => {
await act(async () => {
backgroundBlur.setValue(on);
await flushPromises();
});
};
const latest = (): BackgroundEffectsState => seen[seen.length - 1];
beforeEach(() => {
seen = [];
effects = undefined;
pipelines.built = 0;
pipelines.destroyed = 0;
pipelines.switches = [];
platformMock.platform = "desktop";
backgroundBlur.setValue(false);
});
afterEach(() => backgroundBlur.setValue(false));
// A pipeline primes itself when it is built and when it is destroyed, and a
// primed pipeline lets one frame through untouched: so the frame is spent
// once, as long as it is built once and never taken off the camera.
it("spends one priming frame on first use and none afterwards", async () => {
const track = cameraTrack();
render(surfaces(track));
for (const on of [true, false, true, false, true]) await blur(on);
expect(pipelines.built).toBe(1);
expect(pipelines.destroyed).toBe(0);
expect(track.setProcessor).toHaveBeenCalledTimes(1);
expect(track.stopProcessor).not.toHaveBeenCalled();
});
it("does not run on a phone, even where its browser could", async () => {
platformMock.platform = "ios";
const track = cameraTrack();
render(surfaces(track));
await blur(true);
expect(latest().supported).toBe(false);
expect(track.setProcessor).not.toHaveBeenCalled();
});
it("preview and call share one pipeline", async () => {
const preview = cameraTrack();
const call = cameraTrack();
render(surfaces(preview, call));
await blur(true);
const pipeline = latest().processor;
expect(preview.setProcessor).toHaveBeenCalledWith(pipeline);
expect(call.setProcessor).toHaveBeenCalledWith(pipeline);
expect(pipelines.built).toBe(1);
});
it("applies an effect chosen while the camera is off", async () => {
const view = render(surfaces(null));
await blur(true);
// Already switched to blur, with nothing yet to attach it to.
expect(pipelines.switches).toEqual([
{ mode: "background-blur", blurRadius: 15 },
]);
const track = cameraTrack();
await act(async () => {
view.rerender(surfaces(track));
await flushPromises();
});
expect(track.setProcessor).toHaveBeenCalledTimes(1);
expect(track.setProcessor).toHaveBeenCalledWith(latest().processor);
expect(pipelines.built).toBe(1);
expect(pipelines.switches).toHaveLength(1);
});
});
+137
View File
@@ -0,0 +1,137 @@
/*
Copyright 2024-2025 New Vector Ltd.
Copyright 2026 Element Creations Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import {
type ProcessorWrapper,
type BackgroundOptions,
} from "@livekit/track-processors";
import { createContext, type FC, type JSX, use, useEffect } from "react";
import { type LocalVideoTrack } from "livekit-client";
import { logger } from "matrix-js-sdk/lib/logger";
import { combineLatest } from "rxjs";
import { backgroundBlur as backgroundBlurSettings } from "../settings/settings";
import { BackgroundEffectTransformer } from "./BackgroundEffectTransformer";
import { OneStepPipeline } from "./OneStepPipeline";
import { supportsBackgroundProcessors } from "./backgroundProcessing";
import { type Behavior } from "../state/Behavior";
import { type ObservableScope } from "../state/ObservableScope";
import {
type BackgroundEffects,
createBackgroundEffects,
} from "../state/BackgroundEffects";
import { useBehavior } from "../useBehavior";
//TODO-MULTI-SFU: This is not yet fully there.
// it is a combination of exposing observable and react hooks.
// preferably we should not make this a context anymore and instead just a vm?
export type BackgroundEffectsState = {
supported: boolean | undefined;
processor: undefined | ProcessorWrapper<BackgroundOptions>;
};
const BackgroundEffectsContext = createContext<BackgroundEffects | undefined>(
undefined,
);
export function useBackgroundEffects(): BackgroundEffectsState {
return useBehavior(useBackgroundEffectsState$());
}
export function useBackgroundEffectsState$(): Behavior<BackgroundEffectsState> {
const effects = use(BackgroundEffectsContext);
if (effects === undefined)
throw new Error(
"useBackgroundEffects must be used within a BackgroundEffectsProvider",
);
return effects.state$;
}
/**
* Attaches or detaches the processor so that the track matches the desired
* state, without throwing.
*/
export function applyProcessor(
videoTrack: LocalVideoTrack,
processor: ProcessorWrapper<BackgroundOptions> | undefined,
): void {
if (processor && !videoTrack.getProcessor()) {
// A MediaStreamTrackProcessor cannot be constructed on an ended track
// (e.g. the camera was stopped while the processor was being applied),
// and setProcessor rejects with a TypeError. The track is going away
// anyway, so there is nothing to attach to.
if (videoTrack.mediaStreamTrack.readyState === "ended") {
logger.debug("Not attaching video processor to an ended track");
return;
}
const track = videoTrack.mediaStreamTrack;
videoTrack.setProcessor(processor).catch((e) => {
if (track.readyState === "ended")
logger.debug("Video processor not attached: the track ended first");
else logger.warn("Failed to attach video processor", e);
});
}
if (!processor && videoTrack.getProcessor()) {
videoTrack.stopProcessor().catch((e) => {
logger.warn("Failed to stop video processor", e);
});
}
}
/**
* Updates your video tracks to always use the given processor.
*/
export const syncBackgroundEffects = (
scope: ObservableScope,
videoTrack$: Behavior<LocalVideoTrack | null>,
backgroundEffectsState$: Behavior<BackgroundEffectsState>,
): void => {
combineLatest([videoTrack$, backgroundEffectsState$])
.pipe(scope.bind())
.subscribe(([videoTrack, backgroundEffectsState]) => {
if (!backgroundEffectsState) return;
if (!videoTrack) return;
applyProcessor(videoTrack, backgroundEffectsState.processor);
});
};
export const useSyncBackgroundEffects = (
videoTrack: LocalVideoTrack | null,
): void => {
const { processor } = useBackgroundEffects();
useEffect(() => {
if (!videoTrack) return;
applyProcessor(videoTrack, processor);
}, [processor, videoTrack]);
};
/** The app's one pipeline, shared by every camera track it opens. */
export function createAppBackgroundEffects(
scope: ObservableScope,
): BackgroundEffects {
return createBackgroundEffects(scope, {
supported: supportsBackgroundProcessors(),
blur$: backgroundBlurSettings.value$,
pipeline: new OneStepPipeline(
new BackgroundEffectTransformer({ backgroundDisabled: true }),
"background-effect",
),
});
}
interface Props {
effects: BackgroundEffects;
children: JSX.Element;
}
export const BackgroundEffectsProvider: FC<Props> = ({ effects, children }) => (
<BackgroundEffectsContext value={effects}>
{children}
</BackgroundEffectsContext>
);
+80
View File
@@ -0,0 +1,80 @@
/*
Copyright 2026 Element Creations Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { afterEach, describe, expect, it, vi } from "vitest";
import { ProcessorWrapper } from "@livekit/track-processors";
import { OneStepPipeline } from "./OneStepPipeline";
import { BackgroundEffectTransformer } from "./BackgroundEffectTransformer";
import { flushPromises } from "../utils/test";
const opts = {} as Parameters<OneStepPipeline["init"]>[0];
describe("OneStepPipeline", () => {
afterEach(() => vi.restoreAllMocks());
it("stops and builds again only once a build has finished", async () => {
let finishBuild!: () => void;
const init = vi
.spyOn(ProcessorWrapper.prototype, "init")
.mockImplementationOnce(
async () => new Promise((resolve) => (finishBuild = resolve)),
)
.mockResolvedValue();
const destroy = vi
.spyOn(ProcessorWrapper.prototype, "destroy")
.mockResolvedValue();
const pipeline = new OneStepPipeline(
new BackgroundEffectTransformer({}),
"test",
);
const first = pipeline.init(opts);
const stopped = pipeline.destroy();
const second = pipeline.init(opts);
await flushPromises();
expect(init).toHaveBeenCalledTimes(1);
expect(destroy).not.toHaveBeenCalled();
finishBuild();
await Promise.all([first, stopped, second]);
expect(init).toHaveBeenCalledTimes(2);
expect(destroy.mock.invocationCallOrder[0]).toBeLessThan(
init.mock.invocationCallOrder[1],
);
});
it("restarts in one step, without waiting on itself", async () => {
const init = vi
.spyOn(ProcessorWrapper.prototype, "init")
.mockResolvedValue();
const destroy = vi
.spyOn(ProcessorWrapper.prototype, "destroy")
.mockResolvedValue();
const pipeline = new OneStepPipeline(
new BackgroundEffectTransformer({}),
"test",
);
await pipeline.restart(opts);
expect(destroy).toHaveBeenCalledWith({ willProcessorRestart: true });
expect(init).toHaveBeenCalledOnce();
});
it("carries on after a step that failed", async () => {
vi.spyOn(ProcessorWrapper.prototype, "init")
.mockRejectedValueOnce(new Error("no GPU"))
.mockResolvedValue();
const pipeline = new OneStepPipeline(
new BackgroundEffectTransformer({}),
"test",
);
await expect(pipeline.init(opts)).rejects.toThrow("no GPU");
await expect(pipeline.init(opts)).resolves.toBeUndefined();
});
});
+43
View File
@@ -0,0 +1,43 @@
/*
Copyright 2026 Element Creations Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { BackgroundProcessorWrapper } from "@livekit/track-processors";
type InitOptions = Parameters<BackgroundProcessorWrapper["init"]>[0];
type DestroyOptions = Parameters<BackgroundProcessorWrapper["destroy"]>[0];
/**
* A background pipeline that is built, rebuilt and destroyed one step at a
* time. The camera tracks sharing it attach and stop it independently, and
* the SDK's wrapper holds a single track's streams: two builds at once
* overwrite each other's, and a destroy during a build does nothing.
*/
export class OneStepPipeline extends BackgroundProcessorWrapper {
private steps: Promise<void> = Promise.resolve();
public override async init(opts: InitOptions): Promise<void> {
return this.inTurn(async () => super.init(opts));
}
// The SDK's restart calls init and destroy, which would wait on itself.
public override async restart(opts: InitOptions): Promise<void> {
return this.inTurn(async () => {
await super.destroy({ willProcessorRestart: true });
await super.init(opts);
});
}
public override async destroy(options?: DestroyOptions): Promise<void> {
return this.inTurn(async () => super.destroy(options));
}
private async inTurn(step: () => Promise<void>): Promise<void> {
const turn = this.steps.then(step);
this.steps = turn.catch(() => {});
return turn;
}
}
-87
View File
@@ -1,87 +0,0 @@
/*
Copyright 2026 Element Creations Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { describe, expect, it, vi } from "vitest";
import { type LocalVideoTrack } from "livekit-client";
import {
type BackgroundOptions,
type ProcessorWrapper,
} from "@livekit/track-processors";
import { applyProcessor, trackProcessorSync } from "./TrackProcessorContext";
import { constant } from "../state/Behavior";
import { flushPromises, testScope } from "../utils/test";
const processor = {} as ProcessorWrapper<BackgroundOptions>;
function mockTrack(
readyState: MediaStreamTrackState,
current?: ProcessorWrapper<BackgroundOptions>,
): LocalVideoTrack {
return {
mediaStreamTrack: { readyState },
getProcessor: vi.fn().mockReturnValue(current),
setProcessor: vi.fn().mockResolvedValue(undefined),
stopProcessor: vi.fn().mockResolvedValue(undefined),
} as unknown as LocalVideoTrack;
}
describe("applyProcessor", () => {
it("attaches the processor to a live track", () => {
const track = mockTrack("live");
applyProcessor(track, processor);
expect(track.setProcessor).toHaveBeenCalledWith(processor);
});
it("does not attach the processor to an ended track", () => {
const track = mockTrack("ended");
applyProcessor(track, processor);
expect(track.setProcessor).not.toHaveBeenCalled();
});
it("does not surface a rejected setProcessor", async () => {
const track = mockTrack("live");
vi.mocked(track.setProcessor).mockRejectedValue(
new TypeError("Input track cannot be ended"),
);
const unhandled = vi.fn();
process.on("unhandledRejection", unhandled);
applyProcessor(track, processor);
await flushPromises();
process.off("unhandledRejection", unhandled);
expect(unhandled).not.toHaveBeenCalled();
});
it("stops the processor when none is wanted", () => {
const track = mockTrack("live", processor);
applyProcessor(track, undefined);
expect(track.stopProcessor).toHaveBeenCalled();
});
it("does not surface a rejected stopProcessor", async () => {
const track = mockTrack("live", processor);
vi.mocked(track.stopProcessor).mockRejectedValue(new Error("nope"));
const unhandled = vi.fn();
process.on("unhandledRejection", unhandled);
applyProcessor(track, undefined);
await flushPromises();
process.off("unhandledRejection", unhandled);
expect(unhandled).not.toHaveBeenCalled();
});
});
describe("trackProcessorSync", () => {
it("applies the processor to the current track", () => {
const track = mockTrack("live");
trackProcessorSync(
testScope(),
constant(track),
constant({ supported: true, processor }),
);
expect(track.setProcessor).toHaveBeenCalledWith(processor);
});
});
-145
View File
@@ -1,145 +0,0 @@
/*
Copyright 2024-2025 New Vector Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import {
ProcessorWrapper,
supportsBackgroundProcessors as supportsBackgroundProcessorsLivekitSdk,
type BackgroundOptions,
} from "@livekit/track-processors";
import {
createContext,
type FC,
type JSX,
use,
useEffect,
useMemo,
} from "react";
import { type LocalVideoTrack } from "livekit-client";
import { logger } from "matrix-js-sdk/lib/logger";
import { combineLatest } from "rxjs";
import {
backgroundBlur as backgroundBlurSettings,
useSetting,
} from "../settings/settings";
import { BlurBackgroundTransformer } from "./BlurBackgroundTransformer";
import { type Behavior } from "../state/Behavior";
import { type ObservableScope } from "../state/ObservableScope";
import { platform } from "../Platform";
import { useValueBehavior } from "../useValueBehavior";
//TODO-MULTI-SFU: This is not yet fully there.
// it is a combination of exposing observable and react hooks.
// preferably we should not make this a context anymore and instead just a vm?
export type ProcessorState = {
supported: boolean | undefined;
processor: undefined | ProcessorWrapper<BackgroundOptions>;
};
const ProcessorContext = createContext<ProcessorState | undefined>(undefined);
export function useTrackProcessor(): ProcessorState {
const state = use(ProcessorContext);
if (state === undefined)
throw new Error(
"useTrackProcessor must be used within a ProcessorProvider",
);
return state;
}
export function useTrackProcessorState$(): Behavior<ProcessorState> {
return useValueBehavior(useTrackProcessor());
}
/**
* Attaches or detaches the processor so that the track matches the desired
* state, without throwing.
*/
export function applyProcessor(
videoTrack: LocalVideoTrack,
processor: ProcessorWrapper<BackgroundOptions> | undefined,
): void {
if (processor && !videoTrack.getProcessor()) {
// A MediaStreamTrackProcessor cannot be constructed on an ended track
// (e.g. the camera was stopped while the processor was being applied),
// and setProcessor rejects with a TypeError. The track is going away
// anyway, so there is nothing to attach to.
if (videoTrack.mediaStreamTrack.readyState === "ended") {
logger.debug("Not attaching video processor to an ended track");
return;
}
videoTrack.setProcessor(processor).catch((e) => {
logger.warn("Failed to attach video processor", e);
});
}
if (!processor && videoTrack.getProcessor()) {
videoTrack.stopProcessor().catch((e) => {
logger.warn("Failed to stop video processor", e);
});
}
}
/**
* Updates your video tracks to always use the given processor.
*/
export const trackProcessorSync = (
scope: ObservableScope,
videoTrack$: Behavior<LocalVideoTrack | null>,
processor$: Behavior<ProcessorState>,
): void => {
combineLatest([videoTrack$, processor$])
.pipe(scope.bind())
.subscribe(([videoTrack, processorState]) => {
if (!processorState) return;
if (!videoTrack) return;
applyProcessor(videoTrack, processorState.processor);
});
};
export const useTrackProcessorSync = (
videoTrack: LocalVideoTrack | null,
): void => {
const { processor } = useTrackProcessor();
useEffect(() => {
if (!videoTrack) return;
applyProcessor(videoTrack, processor);
}, [processor, videoTrack]);
};
interface Props {
children: JSX.Element;
}
function supportsBackgroundProcessors(): boolean {
return supportsBackgroundProcessorsLivekitSdk() && platform === "desktop";
}
export const ProcessorProvider: FC<Props> = ({ children }) => {
// The setting the user wants to have
const [blurActivated] = useSetting(backgroundBlurSettings);
const supported = useMemo(() => supportsBackgroundProcessors(), []);
const blur = useMemo(
() =>
new ProcessorWrapper(
new BlurBackgroundTransformer({ blurRadius: 15 }),
"background-blur",
),
[],
);
// This is the actual state exposed through the context
const processorState = useMemo(
() => ({
supported,
processor: supported && blurActivated ? blur : undefined,
}),
[supported, blurActivated, blur],
);
return <ProcessorContext value={processorState}>{children}</ProcessorContext>;
};
+15
View File
@@ -0,0 +1,15 @@
/*
Copyright 2026 Element Creations Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { supportsBackgroundProcessors as supportsBackgroundProcessorsLivekitSdk } from "@livekit/track-processors";
import { platform } from "../Platform";
/** One answer for the controls and the pipeline, so neither offers what the other refuses. */
export function supportsBackgroundProcessors(): boolean {
return supportsBackgroundProcessorsLivekitSdk() && platform === "desktop";
}
+9 -3
View File
@@ -50,6 +50,7 @@ import {
mockMediaDevices,
mockRtcMembership,
MockRTCSession,
testScope,
} from "../utils/test";
import { CallView } from "./CallView";
import { GroupCallErrorBoundary } from "./GroupCallErrorBoundary";
@@ -61,7 +62,10 @@ import {
} from "../HostBridge";
import { type JoinCallData } from "../widget";
import { MatrixRTCTransportMissingError } from "../utils/errors";
import { ProcessorProvider } from "../livekit/TrackProcessorContext";
import {
BackgroundEffectsProvider,
createAppBackgroundEffects,
} from "../livekit/BackgroundEffectsContext";
import { MediaDevicesContext } from "../MediaDevicesContext";
import { constant } from "../state/Behavior";
@@ -193,7 +197,9 @@ function createCallView(
<HostBridgeProvider value={hostBridge}>
<TooltipProvider>
<MediaDevicesContext value={mockMediaDevices({})}>
<ProcessorProvider>
<BackgroundEffectsProvider
effects={createAppBackgroundEffects(testScope())}
>
{options.withErrorBoundary ? (
<GroupCallErrorBoundary recoveryActionHandler={vi.fn()}>
{callView}
@@ -201,7 +207,7 @@ function createCallView(
) : (
callView
)}
</ProcessorProvider>
</BackgroundEffectsProvider>
</MediaDevicesContext>
</TooltipProvider>
</HostBridgeProvider>
+13 -5
View File
@@ -31,6 +31,7 @@ import {
mockRemoteParticipant,
mockRtcMembership,
type MockRTCSession,
testScope,
} from "../utils/test";
import { E2eeType } from "../e2ee/e2eeType";
import {
@@ -49,7 +50,10 @@ import { MediaDevicesContext } from "../MediaDevicesContext";
import { type MediaDevices as ECMediaDevices } from "../state/MediaDevices";
import { AppBar } from "../AppBar";
import { type MatrixInfo } from "./VideoPreview";
import { ProcessorProvider } from "../livekit/TrackProcessorContext";
import {
BackgroundEffectsProvider,
createAppBackgroundEffects,
} from "../livekit/BackgroundEffectsContext";
import { initializeWidget } from "../widget";
import { RootElementProvider } from "../RootElementContext";
@@ -242,7 +246,9 @@ describe("ActiveCall", () => {
const { findByTestId } = render(
<BrowserRouter>
<MediaDevicesContext value={mediaDevices}>
<ProcessorProvider>
<BackgroundEffectsProvider
effects={createAppBackgroundEffects(testScope())}
>
<TooltipProvider>
<RoomContext value={mockLivekitRoom({ localParticipant })}>
<ActiveCall
@@ -257,7 +263,7 @@ describe("ActiveCall", () => {
/>
</RoomContext>
</TooltipProvider>
</ProcessorProvider>
</BackgroundEffectsProvider>
</MediaDevicesContext>
</BrowserRouter>,
);
@@ -292,7 +298,9 @@ describe("ActiveCall", () => {
<BrowserRouter>
<RootElementProvider value={root}>
<MediaDevicesContext value={mediaDevices}>
<ProcessorProvider>
<BackgroundEffectsProvider
effects={createAppBackgroundEffects(testScope())}
>
<TooltipProvider>
<RoomContext value={mockLivekitRoom({ localParticipant })}>
<ActiveCall
@@ -307,7 +315,7 @@ describe("ActiveCall", () => {
/>
</RoomContext>
</TooltipProvider>
</ProcessorProvider>
</BackgroundEffectsProvider>
</MediaDevicesContext>
</RootElementProvider>
</BrowserRouter>,
+5 -5
View File
@@ -79,7 +79,7 @@ import { useValueBehavior } from "../useValueBehavior.ts";
import { constant } from "../state/Behavior.ts";
import { Toast } from "../Toast.tsx";
import overlayStyles from "../Overlay.module.css";
import { useTrackProcessorState$ } from "../livekit/TrackProcessorContext.tsx";
import { useBackgroundEffectsState$ } from "../livekit/BackgroundEffectsContext.tsx";
import { type Layout } from "../state/layout-types.ts";
import { ObservableScope } from "../state/ObservableScope.ts";
import { CallFooter, type FooterSnapshot } from "../components/CallFooter.tsx";
@@ -120,7 +120,7 @@ export const ActiveCall: FC<ActiveCallProps> = (props) => {
const urlParams = useUrlParams();
const hostBridge = useHostBridge();
const mediaDevices = useMediaDevices();
const trackProcessorState$ = useTrackProcessorState$();
const backgroundEffectsState$ = useBackgroundEffectsState$();
// The element we have to draw the call in: the page, or the container a host
// gave us. Its size, not the window's, decides how the call is laid out.
const rootElement = useRootElement();
@@ -150,7 +150,7 @@ export const ActiveCall: FC<ActiveCallProps> = (props) => {
},
reactionsReader.raisedHands$,
reactionsReader.reactions$,
trackProcessorState$,
backgroundEffectsState$,
);
// TODO move this somewhere else once we use the callViewModel in the lobby as well!
vm.join();
@@ -170,7 +170,7 @@ export const ActiveCall: FC<ActiveCallProps> = (props) => {
urlParams,
hostBridge,
mediaDevices,
trackProcessorState$,
backgroundEffectsState$,
props.client,
rootElement,
]);
@@ -201,7 +201,7 @@ export const ActiveCall: FC<ActiveCallProps> = (props) => {
props.onLeft,
urlParams,
mediaDevices,
trackProcessorState$,
backgroundEffectsState$,
props.client,
vm,
]);
+4 -4
View File
@@ -14,19 +14,19 @@ import { type MatrixClient, type RoomSummary } from "matrix-js-sdk";
import { KnockLobbyView } from "./KnockLobbyView";
import { LeaveToHomeProvider } from "../LeaveToHomeContext";
import { MediaDevicesContext } from "../MediaDevicesContext";
import { type ProcessorState } from "../livekit/TrackProcessorContext";
import { type BackgroundEffectsState } from "../livekit/BackgroundEffectsContext";
import { mockMediaDevices } from "../utils/test";
vi.mock("@livekit/components-react", () => ({
usePreviewTracks: (): unknown[] => [],
}));
vi.mock("../livekit/TrackProcessorContext", () => ({
useTrackProcessor: (): ProcessorState => ({
vi.mock("../livekit/BackgroundEffectsContext", () => ({
useBackgroundEffects: (): BackgroundEffectsState => ({
supported: false,
processor: undefined,
}),
useTrackProcessorSync: (): void => {},
useSyncBackgroundEffects: (): void => {},
}));
vi.mock("react-use-measure", () => ({
+4 -4
View File
@@ -28,7 +28,7 @@ import {
} from "../utils/test";
import { type MediaDevices } from "../state/MediaDevices";
import { MediaDevicesContext } from "../MediaDevicesContext";
import { type ProcessorState } from "../livekit/TrackProcessorContext";
import { type BackgroundEffectsState } from "../livekit/BackgroundEffectsContext";
import { type EncryptionSystem } from "../e2ee/sharedKeyManagement";
import lobbyStyles from "./LobbyView.module.css";
import headerStyles from "../Header.module.css";
@@ -41,12 +41,12 @@ vi.mock("@livekit/components-react", () => ({
usePreviewTracks: (): unknown[] => [],
}));
vi.mock("../livekit/TrackProcessorContext", () => ({
useTrackProcessor: (): ProcessorState => ({
vi.mock("../livekit/BackgroundEffectsContext", () => ({
useBackgroundEffects: (): BackgroundEffectsState => ({
supported: false,
processor: undefined,
}),
useTrackProcessorSync: (): void => {},
useSyncBackgroundEffects: (): void => {},
}));
vi.mock("react-use-measure", () => ({
+5 -5
View File
@@ -40,9 +40,9 @@ import { useMediaDevices } from "../MediaDevicesContext";
import { ObservableScope } from "../state/ObservableScope";
import { useInitial } from "../useInitial";
import {
useTrackProcessor,
useTrackProcessorSync,
} from "../livekit/TrackProcessorContext";
useBackgroundEffects,
useSyncBackgroundEffects,
} from "../livekit/BackgroundEffectsContext";
import { getValue } from "../utils/observable";
import { useBehavior } from "../useBehavior";
import { CallFooter, type FooterSnapshot } from "../components/CallFooter";
@@ -136,7 +136,7 @@ export const LobbyView: FC<Props> = ({
},
);
const { processor } = useTrackProcessor();
const { processor } = useBackgroundEffects();
const initialProcessor = useInitial(() => processor);
const localTrackOptions = useMemo<CreateLocalTracksOptions>(
@@ -183,7 +183,7 @@ export const LobbyView: FC<Props> = ({
}
}, [devices, videoInputId, videoTrack]);
useTrackProcessorSync(videoTrack);
useSyncBackgroundEffects(videoTrack);
const [footerVm, setFooterVm] = useState<ViewModel<FooterSnapshot> | null>(
null,
@@ -181,7 +181,7 @@ exports[`InCallView > rendering > renders 1`] = `
tabindex="0"
>
<div
class="_indicator-icon_147l5_17"
class="_indicator-icon_j097f_17"
style="--cpd-icon-button-size: 100%;"
>
<svg
@@ -15,7 +15,7 @@ exports[`LobbyView > renders with AppBar android 1`] = `
tabindex="0"
>
<div
class="_indicator-icon_147l5_17"
class="_indicator-icon_j097f_17"
style="--cpd-icon-button-size: 100%;"
>
<svg
@@ -111,7 +111,7 @@ exports[`LobbyView > renders with AppBar android 1`] = `
tabindex="0"
>
<div
class="_indicator-icon_147l5_17"
class="_indicator-icon_j097f_17"
style="--cpd-icon-button-size: 100%;"
>
<svg
@@ -247,7 +247,7 @@ exports[`LobbyView > renders with AppBar ios 1`] = `
tabindex="0"
>
<div
class="_indicator-icon_147l5_17"
class="_indicator-icon_j097f_17"
style="--cpd-icon-button-size: 100%;"
>
<svg
@@ -343,7 +343,7 @@ exports[`LobbyView > renders with AppBar ios 1`] = `
tabindex="0"
>
<div
class="_indicator-icon_147l5_17"
class="_indicator-icon_j097f_17"
style="--cpd-icon-button-size: 100%;"
>
<svg
@@ -621,7 +621,7 @@ exports[`LobbyView > renders with header and participant count 1`] = `
tabindex="0"
>
<div
class="_indicator-icon_147l5_17"
class="_indicator-icon_j097f_17"
style="--cpd-icon-button-size: 100%;"
>
<svg
+2 -2
View File
@@ -28,7 +28,7 @@ import {
import { PreferencesSettingsTab } from "./PreferencesSettingsTab";
import { Slider } from "../Slider";
import { DeviceSelection } from "./DeviceSelection";
import { useTrackProcessor } from "../livekit/TrackProcessorContext";
import { useBackgroundEffects } from "../livekit/BackgroundEffectsContext";
import {
DeveloperSettingsTab,
type DeveloperSettingsSnapshot,
@@ -81,7 +81,7 @@ export const SettingsModal: FC<Props> = ({
// Generate a `Checkbox` input to turn blur on or off.
const BlurCheckbox: React.FC = (): ReactNode => {
const { supported } = useTrackProcessor();
const { supported } = useBackgroundEffects();
const [blurActive, setBlurActive] = useSetting(backgroundBlurSetting);
+2
View File
@@ -6,6 +6,7 @@ Please see LICENSE in the repository root for full details.
*/
import { type AudioOutputOptions, MediaDevices } from "./MediaDevices";
import { createAppBackgroundEffects } from "../livekit/BackgroundEffectsContext";
import { type ObservableScope } from "./ObservableScope";
/**
@@ -16,6 +17,7 @@ export class AppViewModel {
this.scope,
this.audioOutputOptions,
);
public readonly backgroundEffects = createAppBackgroundEffects(this.scope);
// TODO: Move more application logic here. The CallViewModel, at the very
// least, ought to be accessible from this object.
+140
View File
@@ -0,0 +1,140 @@
/*
Copyright 2026 Element Creations Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { beforeEach, describe, expect, it, vi } from "vitest";
import { BehaviorSubject, distinctUntilChanged, map } from "rxjs";
import { type BackgroundProcessorWrapper } from "@livekit/track-processors";
import {
type BackgroundEffects,
createBackgroundEffects,
type BackgroundEffectsOptions,
} from "./BackgroundEffects";
import { type BackgroundEffectsState } from "../livekit/BackgroundEffectsContext";
import { flushPromises, testScope, withTestScheduler } from "../utils/test";
/** A pipeline that records what it is switched to. */
function fakePipeline(): {
pipeline: BackgroundProcessorWrapper;
switches: unknown[];
/** Holds the next switch until it resolves, as a picture loading does. */
holdNext: () => () => void;
} {
const switches: unknown[] = [];
let slow: Promise<void> | undefined;
const pipeline = {
switchTo: vi.fn(async (options: unknown) => {
switches.push(options);
const held = slow;
slow = undefined;
await held;
}),
} as unknown as BackgroundProcessorWrapper;
return {
pipeline,
switches,
holdNext: () => {
let release!: () => void;
slow = new Promise((resolve) => (release = resolve));
return release;
},
};
}
/** One letter per state: idle, attached. */
function letter(state: BackgroundEffectsState): string {
return state.processor === undefined ? "i" : "a";
}
describe("the pipeline's state", () => {
function testState({
effect,
expected,
}: {
effect: string;
expected: string;
}): void {
withTestScheduler(({ behavior, expectObservable }) => {
const effects = createBackgroundEffects(testScope(), {
supported: true,
blur$: behavior(effect, { n: false, b: true }),
pipeline: fakePipeline().pipeline,
});
expectObservable(
effects.state$.pipe(map(letter), distinctUntilChanged()),
).toBe(expected);
});
}
it("attaches on first use and stays attached", () =>
testState({ effect: "nbn", expected: "ia" }));
});
describe("background effects", () => {
let blur$: BehaviorSubject<boolean>;
let fake: ReturnType<typeof fakePipeline>;
function build(
options: Partial<BackgroundEffectsOptions> = {},
): BackgroundEffects {
return createBackgroundEffects(testScope(), {
supported: true,
blur$,
pipeline: fake.pipeline,
...options,
});
}
const blur = async (on: boolean): Promise<void> => {
blur$.next(on);
await flushPromises();
};
beforeEach(() => {
blur$ = new BehaviorSubject(false);
fake = fakePipeline();
});
it("switches in place rather than reattaching", async () => {
const effects = build();
await blur(true);
const pipeline = effects.state$.value.processor;
await blur(false);
await blur(true);
expect(effects.state$.value.processor).toBe(pipeline);
expect(fake.switches).toEqual([
{ mode: "background-blur", blurRadius: 15 },
{ mode: "disabled" },
{ mode: "background-blur", blurRadius: 15 },
]);
});
it("keeps switching after a switch fails", async () => {
vi.mocked(fake.pipeline.switchTo).mockRejectedValueOnce(new Error("lost"));
build();
await blur(true);
await blur(false);
expect(fake.switches).toEqual([{ mode: "disabled" }]);
});
it("switches one at a time, skipping those overtaken", async () => {
const finished = fake.holdNext();
build();
await blur(true);
await blur(false);
await blur(true);
await blur(false);
expect(fake.switches).toHaveLength(1);
finished();
await flushPromises();
expect(fake.switches).toEqual([
{ mode: "background-blur", blurRadius: 15 },
{ mode: "disabled" },
]);
});
});
+81
View File
@@ -0,0 +1,81 @@
/*
Copyright 2026 Element Creations Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { combineLatest, distinctUntilChanged, map, scan } from "rxjs";
import {
type BackgroundProcessorWrapper,
type SwitchBackgroundProcessorOptions,
} from "@livekit/track-processors";
import { logger } from "matrix-js-sdk/lib/logger";
import { deepCompare } from "matrix-js-sdk/lib/utils";
import { type Behavior } from "./Behavior";
import { type ObservableScope } from "./ObservableScope";
import { type BackgroundEffectsState } from "../livekit/BackgroundEffectsContext";
const blurRadius = 15;
export interface BackgroundEffectsOptions {
/** Whether this browser can run a pipeline at all. */
supported: boolean;
/** Whether blur is chosen. */
blur$: Behavior<boolean>;
/** The background processor pipeline to be switched between effects. */
pipeline: BackgroundProcessorWrapper;
}
/** The background effect pipeline, as the camera tracks and the menus see it. */
export interface BackgroundEffects {
readonly state$: Behavior<BackgroundEffectsState>;
}
/** Switches the pipeline as the choice changes, for as long as the scope lasts. */
export function createBackgroundEffects(
scope: ObservableScope,
{ supported, blur$, pipeline }: BackgroundEffectsOptions,
): BackgroundEffects {
const state$ = scope.behavior(
blur$.pipe(
scan<boolean, BackgroundEffectsState>(
(previous, wanted) => {
// Attached the first time an effect is wanted and never detached
// after, so someone who never turns one on pays for none of it.
const enable =
previous.processor !== undefined || (supported && wanted);
return { supported, processor: enable ? pipeline : undefined };
},
{ supported, processor: undefined },
),
),
);
const switchOptions$ = scope.behavior<
SwitchBackgroundProcessorOptions | undefined
>(
combineLatest([state$, blur$]).pipe(
map(([{ processor }, blur]) =>
processor === undefined
? undefined
: blur
? { mode: "background-blur", blurRadius }
: { mode: "disabled" },
),
distinctUntilChanged(deepCompare),
),
);
// In turn: a picture's switch ends once it loads, so an earlier one could land last.
scope.reconcile(switchOptions$, async (options) => {
if (options === undefined) return;
try {
await pipeline.switchTo(options);
} catch (e) {
logger.warn("Failed to switch background effect", e);
}
});
return { state$ };
}
+4 -4
View File
@@ -89,7 +89,7 @@ import { E2eeType } from "../../e2ee/e2eeType";
import { MatrixKeyProvider } from "../../e2ee/matrixKeyProvider";
import { type MuteStates } from "../MuteStates";
import { HeaderStyle, type UrlParams } from "../../UrlParams";
import { type ProcessorState } from "../../livekit/TrackProcessorContext";
import { type BackgroundEffectsState } from "../../livekit/BackgroundEffectsContext";
import { type HostBridge, nullHostBridge } from "../../HostBridge";
import {
layoutShallowEquals,
@@ -501,7 +501,7 @@ export function createCallViewModel$(
options: CallViewModelOptions,
handsRaisedSubject$: Observable<Record<string, RaisedHandInfo>>,
reactionsSubject$: Observable<Record<string, ReactionInfo>>,
trackProcessorState$: Behavior<ProcessorState>,
backgroundEffectsState$: Behavior<BackgroundEffectsState>,
): CallViewModel {
const logger = rootLogger.getChild("[CallViewModel]");
const client = matrixRoom.client;
@@ -590,7 +590,7 @@ export function createCallViewModel$(
client,
matrixRoom.roomId,
mediaDevices,
trackProcessorState$,
backgroundEffectsState$,
livekitKeyProvider,
controlledAudioDevices,
options.livekitRoomFactory,
@@ -645,7 +645,7 @@ export function createCallViewModel$(
connection,
mediaDevices,
muteStates,
trackProcessorState$,
backgroundEffectsState$,
logger.getChild(
"[Publisher " + connection.transport.livekit_service_url + "]",
),
@@ -54,7 +54,7 @@ import {
localRtcMember,
} from "../../utils/test-fixtures";
import { type Behavior, constant } from "../Behavior";
import { type ProcessorState } from "../../livekit/TrackProcessorContext";
import { type BackgroundEffectsState } from "../../livekit/BackgroundEffectsContext";
import { type MediaDevices } from "../MediaDevices";
import { type MatrixRTCMode } from "../../config/ConfigOptions";
@@ -240,7 +240,7 @@ export function withCallViewModel(mode: MatrixRTCMode) {
},
raisedHands$,
reactions$,
new BehaviorSubject<ProcessorState>({
new BehaviorSubject<BackgroundEffectsState>({
processor: undefined,
supported: undefined,
}),
@@ -209,7 +209,7 @@ describe("LocalMembership", () => {
leave: vi.fn(),
} as unknown as MatrixRTCSession,
muteStates: mockMuteStates(),
trackProcessorState$: constant({
backgroundEffectsState$: constant({
supported: false,
processor: undefined,
}),
@@ -11,11 +11,16 @@ import {
LocalParticipant,
type LocalTrack,
type LocalTrackPublication,
LocalVideoTrack,
ParticipantEvent,
Track,
} from "livekit-client";
import { BehaviorSubject } from "rxjs";
import { logger } from "matrix-js-sdk/lib/logger";
import {
type BackgroundOptions,
type ProcessorWrapper,
} from "@livekit/track-processors";
import { ObservableScope } from "../../ObservableScope";
import { constant } from "../../Behavior";
@@ -25,6 +30,7 @@ import {
mockMediaDevices,
} from "../../../utils/test";
import { Publisher } from "./Publisher";
import { type BackgroundEffectsState } from "../../../livekit/BackgroundEffectsContext";
import { type Connection } from "../remoteMembers/Connection";
import { type MuteStates } from "../../MuteStates";
@@ -400,3 +406,67 @@ describe("Bug fix", () => {
await publisher.destroy();
});
});
describe("turning the camera on with an effect chosen", () => {
const processor = {} as ProcessorWrapper<BackgroundOptions>;
let state$: BehaviorSubject<BackgroundEffectsState>;
let publisher: Publisher;
/** The processor the SDK would give each camera track it made. */
let madeWith: unknown[];
beforeEach(() => {
state$ = new BehaviorSubject<BackgroundEffectsState>({
supported: true,
processor: undefined,
});
publisher = new Publisher(
connection,
mockMediaDevices({}),
muteStates,
state$,
logger,
false,
);
madeWith = [];
vi.spyOn(localParticipant, "setCameraEnabled").mockImplementation(
async () => {
madeWith.push(
connection.livekitRoom.options.videoCaptureDefaults?.processor,
);
return Promise.resolve(undefined);
},
);
vi.spyOn(localParticipant, "unpublishTrack").mockResolvedValue(undefined);
});
afterEach(async () => {
await publisher.destroy();
});
it("makes each camera track with the pipeline attached since", async () => {
await publisher.createAndSetupTracks();
state$.next({ supported: true, processor });
videoEnabled$.next(true);
await flushPromises();
expect(madeWith).toEqual([processor]);
});
it("replaces a camera track turned off before the effect was chosen", async () => {
state$.next({ supported: true, processor });
const stopped = Object.assign(Object.create(LocalVideoTrack.prototype), {
source: Track.Source.Camera,
getProcessor: (): undefined => undefined,
}) as LocalVideoTrack;
trackPublications.push({
track: stopped,
source: Track.Source.Camera,
} as Partial<LocalTrackPublication> as LocalTrackPublication);
await publisher.createAndSetupTracks();
videoEnabled$.next(true);
await flushPromises();
expect(localParticipant.unpublishTrack).toHaveBeenCalledWith(stopped);
expect(madeWith).toEqual([processor]);
});
});
@@ -26,9 +26,9 @@ import type { Behavior } from "../../Behavior.ts";
import type { MediaDevices, SelectedDevice } from "../../MediaDevices.ts";
import type { MuteStates } from "../../MuteStates.ts";
import {
type ProcessorState,
trackProcessorSync,
} from "../../../livekit/TrackProcessorContext.tsx";
type BackgroundEffectsState,
syncBackgroundEffects,
} from "../../../livekit/BackgroundEffectsContext.tsx";
import { observeTrackReference$ } from "../../observeTrackReference";
import { type Connection } from "../remoteMembers/Connection.ts";
@@ -54,7 +54,7 @@ export class Publisher {
* @param connection - The connection to use for publishing.
* @param devices - The media devices to use for audio and video input.
* @param muteStates - The mute states for audio and video.
* @param trackerProcessorState$ - The processor state for the video track processor (e.g. background blur).
* @param backgroundEffectsState$ - The processor state for the video track processor (e.g. background blur).
* @param logger - The logger to use for logging :D.
* @param controlledAudioDevices - Whether the app hosting Element Call
* controls the audio output devices, rather than the browser.
@@ -63,7 +63,7 @@ export class Publisher {
private connection: Pick<Connection, "livekitRoom" | "state$">, //setE2EEEnabled,
devices: MediaDevices,
private readonly muteStates: MuteStates,
trackerProcessorState$: Behavior<ProcessorState>,
private readonly backgroundEffectsState$: Behavior<BackgroundEffectsState>,
private logger: Logger,
controlledAudioDevices: boolean,
) {
@@ -73,8 +73,11 @@ export class Publisher {
this.logger.error("Failed to set E2EE enabled on room", e);
});
// Setup track processor syncing (blur)
this.observeTrackProcessors(this.scope, room, trackerProcessorState$);
this.observeBackgroundEffects(
this.scope,
room,
this.backgroundEffectsState$,
);
// Observe media device changes and update LiveKit active devices accordingly
this.observeMediaDevices(this.scope, devices, controlledAudioDevices);
@@ -412,6 +415,8 @@ export class Publisher {
this.muteStates.video.setHandler(async (enable) => {
try {
this.logger.debug(`handler: Setting LiveKit camera enabled: ${enable}`);
const { processor } = this.backgroundEffectsState$.value;
if (enable && processor) await this.dropCameraWithoutEffect(lkRoom);
await lkRoom.localParticipant.setCameraEnabled(enable);
// Unmute will restart the track if it was paused upstream,
// but until explicitly requested, we want to keep it paused.
@@ -426,10 +431,23 @@ export class Publisher {
});
}
private observeTrackProcessors(
/**
* Unpublishes a camera track that has no effect on, so that turning the
* camera on creates one with it. A camera turned off is stopped, and a
* stopped track can't take an effect chosen while it was off.
*/
private async dropCameraWithoutEffect(lkRoom: LivekitRoom): Promise<void> {
const track = lkRoom.localParticipant.getTrackPublication(
Track.Source.Camera,
)?.track;
if (track instanceof LocalVideoTrack && !track.getProcessor())
await lkRoom.localParticipant.unpublishTrack(track);
}
private observeBackgroundEffects(
scope: ObservableScope,
room: LivekitRoom,
trackerProcessorState$: Behavior<ProcessorState>,
backgroundEffectsState$: Behavior<BackgroundEffectsState>,
): void {
const track$ = scope.behavior(
observeTrackReference$(room.localParticipant, Track.Source.Camera).pipe(
@@ -440,6 +458,15 @@ export class Publisher {
),
null,
);
trackProcessorSync(scope, track$, trackerProcessorState$);
syncBackgroundEffects(scope, track$, backgroundEffectsState$);
// Every camera track the SDK makes, on joining or on turning the camera
// on, then starts with the pipeline on. One attached after publishing lets
// the room through while it is built.
backgroundEffectsState$.pipe(scope.bind()).subscribe(({ processor }) => {
room.options.videoCaptureDefaults = {
...room.options.videoCaptureDefaults,
processor,
};
});
}
}
@@ -655,7 +655,7 @@ describe("remote track logging", () => {
// function createSetup(): void {
// setUpPublishConnection();
// const fakeTrackProcessorSubject$ = new BehaviorSubject<ProcessorState>({
// const fakeTrackProcessorSubject$ = new BehaviorSubject<BackgroundEffectsState>({
// supported: true,
// processor: undefined,
// });
@@ -26,7 +26,7 @@ import type {
} from "../../../livekit/openIDSFU.ts";
import type { MediaDevices } from "../../MediaDevices.ts";
import type { Behavior } from "../../Behavior.ts";
import type { ProcessorState } from "../../../livekit/TrackProcessorContext.tsx";
import type { BackgroundEffectsState } from "../../../livekit/BackgroundEffectsContext.tsx";
import { getLiveKitOptions } from "../../../livekit/options.ts";
import {
advancedCamera,
@@ -60,7 +60,7 @@ export class ECConnectionFactory implements ConnectionFactory {
* @param client - The OpenID client parts for authentication, needed to get openID and JWT tokens.
* @param roomId - The current room ID.
* @param devices - Used for video/audio out/in capture options.
* @param processorState$ - Effects like background blur (only for publishing connection?)
* @param backgroundEffectsState$ - Effects like background blur (only for publishing connection?)
* @param livekitKeyProvider - Optional key provider for end-to-end encryption.
* @param controlledAudioDevices - Option to indicate whether audio output device is controlled externally (native mobile app).
* @param livekitRoomFactory - Optional factory function (for testing) to create LivekitRoom instances. If not provided, a default factory is used.
@@ -69,7 +69,7 @@ export class ECConnectionFactory implements ConnectionFactory {
private client: OpenIDClientParts,
private readonly roomId: string,
private devices: MediaDevices,
private processorState$: Behavior<ProcessorState>,
private backgroundEffectsState$: Behavior<BackgroundEffectsState>,
livekitKeyProvider: BaseKeyProvider | undefined,
private controlledAudioDevices: boolean,
livekitRoomFactory?: () => LivekitRoom,
@@ -77,7 +77,7 @@ export class ECConnectionFactory implements ConnectionFactory {
const defaultFactory = (): LivekitRoom => {
const roomOptions = generateRoomOption({
devices: this.devices,
processorState: this.processorState$.value,
backgroundEffectsState: this.backgroundEffectsState$.value,
e2eeLivekitOptions: livekitKeyProvider && {
keyProvider: livekitKeyProvider,
// It's important that every room use a separate E2EE worker.
@@ -129,12 +129,12 @@ export class ECConnectionFactory implements ConnectionFactory {
*/
function generateRoomOption({
devices,
processorState,
backgroundEffectsState,
e2eeLivekitOptions,
controlledAudioDevices,
}: {
devices: MediaDevices;
processorState: ProcessorState;
backgroundEffectsState: BackgroundEffectsState;
e2eeLivekitOptions:
| E2EEManagerOptions
| { e2eeManager: BaseE2EEManager }
@@ -147,7 +147,7 @@ function generateRoomOption({
let videoCaptureDefaults = {
...liveKitOptions.videoCaptureDefaults,
deviceId: devices.videoInput.selected$.value?.id,
processor: processorState.processor,
processor: backgroundEffectsState.processor,
};
let publishDefaults = liveKitOptions.publishDefaults;
@@ -20,7 +20,7 @@ import {
mockMediaDevices,
ownMemberMock,
} from "../../../utils/test.ts";
import type { ProcessorState } from "../../../livekit/TrackProcessorContext.tsx";
import type { BackgroundEffectsState } from "../../../livekit/BackgroundEffectsContext.tsx";
import { constant } from "../../Behavior";
import {
echoCancellationSetting,
@@ -80,7 +80,7 @@ describe("ECConnectionFactory - Audio inputs options", () => {
mockClient,
"!roomid:example.org",
mockMediaDevices({}),
new BehaviorSubject<ProcessorState>({
new BehaviorSubject<BackgroundEffectsState>({
supported: true,
processor: undefined,
}),
@@ -128,7 +128,7 @@ describe("ECConnectionFactory - ControlledAudioDevice", () => {
select: () => {},
},
}),
new BehaviorSubject<ProcessorState>({
new BehaviorSubject<BackgroundEffectsState>({
supported: true,
processor: undefined,
}),
@@ -163,7 +163,7 @@ describe("ECConnectionFactory - Camera quality settings", () => {
mockClient,
"!roomid:example.org",
mockMediaDevices({}),
new BehaviorSubject<ProcessorState>({
new BehaviorSubject<BackgroundEffectsState>({
supported: true,
processor: undefined,
}),
@@ -200,7 +200,7 @@ describe("ECConnectionFactory - Camera quality settings", () => {
mockClient,
"!roomid:example.org",
mockMediaDevices({}),
new BehaviorSubject<ProcessorState>({
new BehaviorSubject<BackgroundEffectsState>({
supported: true,
processor: undefined,
}),
@@ -238,7 +238,7 @@ describe("ECConnectionFactory - Camera quality settings", () => {
mockClient,
"!roomid:example.org",
mockMediaDevices({}),
new BehaviorSubject<ProcessorState>({
new BehaviorSubject<BackgroundEffectsState>({
supported: true,
processor: undefined,
}),
@@ -26,7 +26,7 @@ import {
ownMemberMock,
withTestScheduler,
} from "../../../utils/test.ts";
import { type ProcessorState } from "../../../livekit/TrackProcessorContext.tsx";
import { type BackgroundEffectsState } from "../../../livekit/BackgroundEffectsContext.tsx";
import {
areLivekitTransportsEqual,
createRemoteMatrixLivekitMembers$,
@@ -72,7 +72,7 @@ beforeEach(() => {
mockClient,
"!roomid:example.org",
mockMediaDevices({}),
new BehaviorSubject<ProcessorState>({
new BehaviorSubject<BackgroundEffectsState>({
supported: true,
processor: undefined,
}),
+15 -2
View File
@@ -37,8 +37,21 @@ export default defineConfig(({ mode }) => {
...vitePluginsConfig({ mode, html: false }),
root: "component/dev",
// The same scoping the library build applies, so the harness shows what a
// host will get — including whether its own page is left alone
css: { postcss: { plugins: [scopeStylesToRoot()] } },
// host will get — including whether its own page is left alone. The host's
// own stylesheets, Compound's among them, stay unscoped, as on a real host.
css: {
postcss: {
plugins: [
scopeStylesToRoot({
exclude: [
/component[\\/]dev[\\/]/,
/@vector-im[\\/]compound-/,
/@fontsource[\\/]/,
],
}),
],
},
},
// So that the harness can read the same config.json the standalone app
// does, if the developer has written one
publicDir: "../../public",
+35 -4
View File
@@ -7,6 +7,7 @@ Please see LICENSE in the repository root for full details.
import {
loadEnv,
type Plugin,
type PluginOption,
searchForWorkspaceRoot,
type ConfigEnv,
@@ -42,10 +43,12 @@ export const vitePluginsConfig = ({
}),
react(),
wasm(),
nodePolyfills({
// Enables the 'events' module, which is required by the matrix-js-sdk
include: ["events"],
}),
exceptMediaPipeWasm(
nodePolyfills({
// Enables the 'events' module, which is required by the matrix-js-sdk
include: ["events"],
}),
),
svgrPlugin({
svgrOptions: {
// This enables ref forwarding on SVGR components, which is needed, for
@@ -176,3 +179,31 @@ export default ({
},
};
};
/**
* Keeps a plugin's transform off MediaPipe's WASM loader. The loader is a
* classic script that sets `self.ModuleFactory`; transformed into a module it
* sets nothing, and the segmenter fails with "ModuleFactory not set".
*/
function exceptMediaPipeWasm(plugins: PluginOption): PluginOption {
const isLoader = (id: string): boolean =>
id.includes("tasks-vision") && id.includes("wasm");
return (Array.isArray(plugins) ? plugins : [plugins]).map((plugin) => {
if (
!plugin ||
typeof plugin !== "object" ||
!("transform" in plugin) ||
typeof plugin.transform !== "function"
)
return plugin;
const transform = plugin.transform;
const wrapped: Plugin = {
...plugin,
transform(code, id, options) {
if (isLoader(id)) return null;
return transform.call(this, code, id, options);
},
};
return wrapped;
});
}