Commit Graph
3686 Commits
Author SHA1 Message Date
Timo K. 0475f2b2b0 cleanup widget sdk
- dont use the callViewModel anymore (bundle size 6 MB to 12 MB, and
   the entry from 2.1 MB to 1.1 MB)
 - better docs and readme information
2026-11-13 15:50:06 +01:00
Timo K.andClaude Fable 5.1 3100c1c410 Cover the standalone client's initialisation and session restore
Neither had a test before the move, so the moved lines showed up as
uncovered. The client initialisation is tested through a fake client
and mocked stores: which store it syncs into with and without
IndexedDB, with and without the worker, the fallback to memory when
the IndexedDB store cannot start, clearing the stores only when not
restoring, carrying on without crypto, and counting as ready only
after a sync from the network rather than the one restored from the
store. The session restore is tested for no stored session, a
restored one, one the homeserver no longer knows, and any other
failure.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-11 19:24:21 +01:00
Timo K.andClaude Fable 5.1 fc1e91f54c Keep the store worker out of the component package
The package carried the standalone app's IndexedDB store worker: 11 MB
with a copy of the Matrix SDK and its crypto wasm, referenced by an
absolute path no host could serve, and never used by the component,
which takes the host's client. The bundler emits the worker file as
soon as it loads the `?worker` import, used or not, and the import sat
in the Matrix utilities the component reaches for the room URLs.

The client initialisation moves into a module of its own, with the
worker import, which only the standalone login, registration and
session restore paths import. ClientProvider takes the session loader
through a `loadClient` prop instead of importing it itself, since even
a dynamic import keeps a module in the package: the standalone app
passes a lazy `initSPA` from App.tsx, the component passes nothing,
and given neither a client nor a loader no one is signed in.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-11 19:24:21 +01:00
Johannes Marbach 73826ec4f9 Reformat 2026-11-11 15:29:03 +01:00
Johannes Marbach 1135c02e6f Merge branch 'main' into johannes/flakes 2026-11-11 13:53:30 +01:00
Johannes MarbachandTimo 9dc425b5f8 Apply batched suggestions from code review
Improve comments

Co-authored-by: Timo <16718859+toger5@users.noreply.github.com>
2026-11-11 13:52:05 +01:00
Johannes Marbach fa6efd9dad Merge branch 'main' into johannes/intent 2026-11-11 10:45:51 +01:00
Johannes Marbach 98e8343d0b Stabilise m.call.intent 2026-11-11 10:10:33 +01:00
Timo 5f59283b4a Merge pull request #4428 from element-hq/toger5/component-assets-as-files
Ship the component's assets as files beside the module
2026-11-10 14:18:30 +01:00
Johannes Marbach cfd8312e46 Try to fix flaky background tests 2026-11-09 12:20:00 +01:00
Johannes Marbach ab4f0f0f77 Make one site use the embedded build in playwright tests 2026-11-09 10:05:04 +01:00
Timo K.andClaude Fable 5.1 a5374791e4 Resolve asset references from the chunk that holds them
The runtime path was `./` plus the asset's path from the package root,
which is right for every chunk the build emits today, all of which sit
at that root, and wrong for one emitted under a directory. Vite names
the referencing chunk, so the path is now computed relative to it.

The sound loader kept a branch for sounds inlined as data URLs, with a
comment saying the component build produces them. No build does any
more, and no host would inline a sound of this size, so the branch and
its tests go.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-07 16:06:05 +01:00
Timo K. cd5c367ce9 fix race in processor setup vs camera start 2026-11-06 16:54:42 +01:00
Timo K.andClaude Fable 5.1 190902eec0 Test the session where the pipeline cannot be built
- The effects state's timeline through preparing, settling and attached,
  and that a failed trial build withholds the effects for the session while
  keeping the choice.
- The footer offers nothing once the pipeline fails to build, and passes
  on the wait as the state's single wait field.
- The call leaves the camera out of the join while the pipeline is
  prepared, publishes it once that is known, and turns on a camera turned
  off before the processor was chosen; the lobby opens no camera meanwhile.
  Both view models are driven by one state subject.
- The transformer's trial build answers whether a segmenter can be made.
- End to end, a browser whose pipeline cannot be built sees the effects
  withheld.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-06 16:54:42 +01:00
Timo K.andClaude Opus 5.5 88f748ec21 Offer no effects for the session where the pipeline cannot be built
- A segmenter that can't be built, with the model unreachable or a GPU
  its delegate refuses, took the camera down with the pipeline: the SDK
  fails the track it was attaching to, and the lobby muted camera and
  microphone, on every load while the choice was kept.
- One is now built on trial when an effect is first wanted, before the
  pipeline is attached to anything, on the CPU where the GPU refuses.
  Where it can't be built, the options are disabled with the existing
  notice for the rest of the session. The choice is kept: a browser
  update may fix what failed.
- The view model takes that verdict from the pipeline's state rather
  than
  asking the SDK, so both say the same, and it no longer builds a WebGL
  context of its own to ask.
- Until the trial is over, the lobby opens no camera and the call makes
  no
  camera track, so neither starts without the effect.

Rebased onto main, where the Publisher is gone and the call view model
publishes the camera through the SDK:
- CallViewModel takes videoProcessorPreparing$. While it is true, the
  camera's mute handler waits before making a track, and the join leaves
  the camera out of its requests (initialPublishRequests); the view
  model
  publishes it once it is known whether the pipeline builds. The
  Publisher's test became CallViewModel tests.

Cherry-picked from spec/background-effects/12-unbuildable-pipeline onto
toger5/background-effect/camera-preview, where the camera menu's preview
takes the camera from the footer's caller rather than from a track
synced
on the background-effects state. The footer factories take the
pipeline's
state again, as this commit needs its verdict and not only the settling
flag; the synced camera and the preview's wrapper type are not brought
along.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-06 16:54:42 +01:00
Timo K. f2f67019b4 Show the self preview in the video mute button context menu 2026-11-06 15:35:04 +01:00
Timo K. 51fac690f4 Hand the camera track to the footer view model 2026-11-06 15:35:04 +01:00
Timo K.andClaude Fable 5.1 797a86815b Ship the component's assets as files beside the module
The component module was 18 MB, of which 15 MB was MediaPipe's runtime
as a base64 data URL, with the segmentation model, every sound and the
background pictures inlined the same way: a library build inlines every
asset unless its import says `no-inline`. Every host downloaded all of
it before the first call, whether or not a background effect was ever
switched on, and a host with a Content Security Policy without
`script-src data:` could not run effects at all, since MediaPipe
injects its loader as a script tag.

The imports now say `no-inline`, and the build renders each reference
as `new URL("./assets/…", import.meta.url)`, which webpack 5, Vite and
Rollup follow as an asset of the host's own; the `./` is spelled out
because a bare `assets/…` is read as a package name. The module is
2.3 MB. The runtime is fetched when an effect is first switched on and
cached on its own. MediaPipe's files come through the package's
`exports` rather than a path into node_modules.

MediaPipe's JavaScript (160 kB) stays in the module: LiveKit's track
processors import it statically, so a dynamic import moves nothing.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-06 14:39:25 +01:00
fkwpandClaude Opus 5.5 fb4cbc72e2 Remove a background of one's own
- A cross on the tile's top-right corner removes an added background,
  critical-coloured at rest and named by its tooltip, as design draws it.
  The one in force offers none, since taking it away would leave nothing
  chosen; the view model refuses it too.
- The cross sits beside the item rather than in it, which would take the
  press first and choose the tile, and isn't a control of its own, so the
  menu keeps one keyboard model: Delete or Backspace on the tile does the
  same, and the tile says so with aria-keyshortcuts.
- Removed from IndexedDB and its picture released. End to end on
  Chromium and Firefox, a background removed by Delete is still gone
  after a reload.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-06 10:33:22 +01:00
fkwp bac9811829 Merge pull request #4288 from element-hq/spec/background-effects/8-own-backgrounds
Add a background of one's own, kept on this device
2026-11-06 10:33:17 +01:00
Valere Fedronic dcb2b9e8f6 Merge pull request #4418 from element-hq/valere/bug_fix_no_config_uncaught_error
Fix: Uncaught exception in downloadConfig
2026-11-05 16:30:40 +01:00
fkwpandClaude Opus 5.5 373b1e0fc0 Guard what can be added as a background
- Five of one's own at most, so the grid is whole: at the limit the add
  tile goes, and it comes back once one is removed. The store holds the
  limit too.
- An added image is refused if it isn't an image, is animated, or can't
  be decoded; reduced to 1920px on its longest edge; and laid on an opaque
  ground, since a transparent pixel would be a hole in the background.
- Animation is read from the file, as a still and an animated WebP share
  a type. Chromium has no selected track until the decoder's tracks are
  ready; without waiting for them an animated GIF got through.
- A refusal says why, in place of the notice, and is scrolled into view.
  It can be dismissed, is gone once the menu closes, and the next shows
  even if it reads the same. What was in force stays.
- End to end on Chromium and Firefox, from what the device then keeps: an
  animated GIF is refused with its message, a 4000x3000 image is kept at
  1920x1440, and a half transparent one is kept opaque. All three fail
  with the preparation bypassed.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-05 16:20:33 +01:00
fkwpandClaude Opus 5.5 1bc372b587 Add a background of one's own, kept on this device
- An add tile after the pictures opens the file picker, and the image
  chosen joins the backgrounds on offer without being put on; it is in
  force once chosen, like a shipped one. The tile is a command rather
  than a choice, so a menu item beside the radios, and it isn't offered
  where effects can't run.
- Kept in IndexedDB on this device and never sent anywhere, behind a
  small interface, so the unit tests use an in-memory store; the
  IndexedDB store itself is tested against fake-indexeddb, a new dev
  dependency.
- The picker takes the focus, which the menu reads as a click elsewhere,
  so the menu holds itself open until a file is chosen or the picker is
  cancelled.
- A chosen image the device no longer keeps, as after its storage is
  cleared, falls back to no effect once the device has said what it
  keeps.
- Switching is checked between any two kinds of effect. End to end on
  Chromium and Firefox: an added image shows in the preview once chosen;
  once it is in force, no request carries the file or the copy kept,
  which a deliberate upload fails; chosen, it is in force again after a
  reload, gone once storage is cleared, and can be added again.
- Kept as given here; what can be added is checked in the commit after.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-05 16:20:33 +01:00
Valere e9cffb60dd Fix: Uncaught exception in downloadConfig 2026-11-05 16:08:41 +01:00
fkwpandClaude Opus 5.5 741dab6dbb Hold the chosen effect across calls, and forget one that can't run
- A remembered effect this browser can't run is cleared rather than kept,
  as the menu already showed no effect in force there.
- A camera turned off and on again with an effect chosen gets the same
  pipeline as its track arrives, neither rebuilt nor switched again.
- End to end, on Chromium and Firefox: the effect chosen before a call is
  still chosen, and worn, in the next call after leaving; and a second
  tab still shuts the first down with the existing "Opened in another
  tab" error while an effect runs.
- Switching camera and sharing a screen are checked by hand: they need
  two cameras and a screen, which the fake devices don't give yet.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-05 15:30:25 +01:00
fkwpandClaude Opus 5.5 4e779ac4f5 Offer background effects on any platform whose browser can run them
- The desktop rule from #4059 goes: now that the menu says where effects
  run slowly, using them there is the user's choice. A phone on the fast
  route held 99% of its frame rate and was refused; a desktop on the slow
  one held 73% and was allowed.
- Asks @livekit/track-processors directly; the wrapper module goes.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-05 13:48:29 +01:00
fkwpandClaude Opus 5.5 7fb222ce60 Show the wait for the first background effect on its tile
- The first effect chosen in a session shows a spinner on its own tile,
  in place of the tick, until a frame carrying it is drawn. Later changes
  show nothing, having nothing to prepare.
- Ended by the frame, not the promises, which resolve while the segmenter
  is still being built. The transformer reports the first frame that went
  through the segmenter; one passed through untouched doesn't count.
- The pipeline publishes the wait with its state, and both footer view
  models take it as an input.
- The lobby tests' provider mock gains the observable hook, built once: a
  fresh observable per render rebuilt the footer on every render.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-05 13:48:29 +01:00
fkwpandClaude Opus 5.5 5fdb920ea8 Say where background effects run slowly
- Where the browser has no MediaStreamTrackProcessor, effects draw each
  frame through a canvas: slower, and the first build stalls the page.
  They stay offered and choosable there, with a notice under the grid
  that says so and describes the group.
- Decided in the view model, beside whether effects can run at all, so
  the lobby and the call give the same answer.
- The notice stays within the menu's width however long the sentence,
  which the story checks.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-05 13:48:22 +01:00
Timo K. 813dd4a49e remove unnecassary type 2026-11-05 13:45:59 +01:00
Timo K. 21b54501de rename BackgroundEffectOption to BackgroundEffectGridItem
This makes it more clear that this is a UI struct and not related to the
items in the catalogue
2026-11-05 13:45:59 +01:00
fkwpandClaude Opus 5.5 3d57920556 Offer background effects as a grid in the camera menu
- The camera menu's blur toggle becomes a Background effects section
  under the cameras: no effect, blur and the two shipped backgrounds as
  one radio choice of tiles, three across. It scrolls with the device
  list, so it stays reachable in a short call, and choosing keeps the
  menu open. The lobby gets it too, as both render the same footer.
- The footer's view model gives the options, the notice and the actions;
  the footer only names them.
- Each tile is Compound's MenuItem as a menuitemradio, styled as a tile.
  Compound makes it an item of the dropdown, so the arrow keys walk the
  tiles with the device rows, and takes its tap in the phone drawer. A
  tile that can't be chosen has no select handler, so the keyboard passes
  over it.
- Where the browser can't run effects the tiles are shown, disabled and
  explained by a notice the group is described by. No effect stays
  choosable and shows as in force. The notice is in the scrolling list,
  so a short call doesn't push the menu out of it.
- Section headings are raised, or the positioned tiles scroll over the
  one they sit under; raised within the list only, so the menu's frame
  stays over them. Tiles clip with overflow: clip; with hidden they are
  scroll containers, which focus scrolls into view without their border.
- End to end on Chromium and Firefox: a shipped background chosen in the
  lobby shows in the preview, and None after Blur draws nothing, counted
  in WebGL draws (Blur: about 1400 a second on Chromium, 2300 on
  Firefox).
- The Quick Audio Menu's test of the camera menu keeps its name and now
  finds blur among the effects rather than as a toggle.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-05 13:45:59 +01:00
Timo K. cc994abeb3 review duplicated BackgoundEffects files
One is now called backgroundEffectCatalog
2026-11-05 10:39:57 +01:00
fkwpandClaude Opus 5.5 33f4d5043f Choose a background effect, not only blur on or off
- A catalogue of what can be chosen: no effect, blur, and two shipped
  backgrounds, the app's own gradients flattened onto its canvas colour.
  Their ids are what a saved choice stores, so they are named for the
  pictures.
- One setting for the choice, typed as the ids an effect can have and
  defaulting to blur for anyone who had it on. A stored value we no
  longer ship, or one that isn't even a string, reads as no effect: it
  comes from storage another build or a hand may have written.
- The pipeline follows it, switching in place to a picture as readily as
  to blur, and the camera menu's blur toggle reads and writes it.
- So does Settings' blur checkbox: a picture reads as blur off, and
  checking it replaces the picture with blur. The old on/off setting is
  only read, to carry blur over. First tests for the Settings modal,
  which fail with the checkbox on the old setting.
- The transformer's own init loads a picture chosen before it was built,
  as the base init does and it had not: the first effect always is, and
  the picture was drawn black.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-05 10:39:57 +01:00
Timo K.andClaude Opus 5.5 0d60eabef2 Name the upstream cause behind the transformer's update override
@livekit/track-processors skips a frame on
`isDisabled ?? backgroundDisabled`, and `isDisabled` is never
undefined, so a disabled effect that keeps its blur radius is still
segmented and drawn on every frame. Still the case in 0.8.2. The
override stays until that is fixed upstream.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-04 20:17:22 +01:00
Timo K.andClaude Opus 5.5 22083299fa Turn a camera back on rather than replace it for its processor
The SDK now builds a processor chosen while the camera was off when the
camera turns on, so the call view model no longer unpublishes the
camera and publishes a new one. TrackSupply.replace, the record of
which camera took which processor, and followProcessor's applied
callback go with it.

Answers robintown's review on #4282 ("what's wrong with attaching
processors to already published camera tracks?").

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-04 20:17:22 +01:00
fkwpandClaude Fable 5.1 a0aaa8e09a Ask one question about whether background effects can run
The controls and the pipeline each asked the SDK on their own whether
background processing is supported, so one could offer what the other
refused. One function answers both, and it asks the browser, kept to
desktops until the menu says what the effects cost elsewhere.

Ported from spec/background-effects/2-pipeline (#4282): f78e9c05 and
87638325, the tests on the lobby and call footers as they are.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-04 20:17:22 +01:00
fkwpandClaude Fable 5.1 146dd3aea1 Start every camera track with the pipeline on
A camera track made without the processor lets the room see the camera
until the processor is built. So the processor travels in the publish
request: the lobby opens the camera with it, the join publishes with it,
the room's capture defaults follow the requests made since the join, and
a camera turned off before the processor was chosen is replaced on turning
it on rather than unmuted, so that the new track starts with it.

Ported from spec/background-effects/2-pipeline (#4282): the publish half
of a91dab8f. The replacement lives in the mute switch's track supply,
which the call view model shares with the lobby.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-04 20:17:22 +01:00
fkwpandClaude Fable 5.1 3f5f707ca7 Attach background effects through one switchable pipeline
Blur runs through one pipeline, shared by the pre-join preview and the
call: attached the first time an effect is wanted and switched in place
after, never detached. The logic is BackgroundEffects in src/state;
BackgroundEffectsContext only builds it, from the app's view model, so
the standalone app and the component hand the same instance to every
camera. Disabled drops the blur radius, so no effect costs no
segmentation. The SDK gains fromLivekitProcessor beside the adapter,
now toLivekitProcessor.

Ported from spec/background-effects/2-pipeline (#4282): a91dab8f with
its review tidy-ups 445e010c, bbc73d07, 51729bcf, f52d2b79, 813f1e6f
and 584bbaa0. Left out as already solved here: OneStepPipeline (the SDK
runs a processor's steps in turn) and the LiveKit-track sync helpers
(the lobby's camera is an SDK track, synced by followProcessor).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-04 20:17:22 +01:00
Timo K.andClaude Fable 5.1 69c57bd534 Run the lobby on the SDK's local media and keep its tracks through the
join

The lobby captured its own LiveKit tracks and the join captured again:
two permission prompts, a grey tile while the camera restarted, and the
background pipeline built twice. The lobby now opens the devices on the
SDK's `LocalMedia`, follows the mute switches as they change (they start
off until the devices are enumerated), the device choice and the
processor through controls shared with the call view model, and the call
joins with that media, so the tile carries the very track the lobby
previewed.

`useBehavior` subscribes without `useSyncExternalStore`, which left a
view without a re-render for an emission React had been told about,
about one time in three, when its store had switched. The LiveKit
preview hook and the processor sync helpers go, with
`@livekit/components-react`.

Background effects run under the dev server: the node polyfill no longer
rewrites MediaPipe's WASM loader. Taken from fkwp's 2fad4e08 on
spec/background-effects/2-pipeline (#4282).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-04 18:59:11 +01:00
Timo K. 1aa91bf27c Add the old useBehavior implementation as reference 2026-11-04 16:27:52 +01:00
Timo 818906eef8 Merge pull request #4404 from element-hq/toger5/fix-storybook-docgen-parse
Use a generic instead of a type assertion in useRoomEncryptionSystem
2026-11-03 16:00:58 +01:00
Timo K. e339098939 Merge branch 'main' into spec/background-effects/1-device-menus 2026-11-03 14:40:13 +01:00
Timo K.andClaude Opus 5.5 1dad38edd2 Use a generic instead of a type assertion in useRoomEncryptionSystem
Storybook's react-docgen plugin parses .ts files with Babel's jsx plugin
enabled, so the angle-bracket assertion `<EncryptionSystem>useMemo(...)`
reads as an opening JSX tag and the Storybook build fails with
"Unexpected token". Passing the type as useMemo's generic is equivalent
and parses either way.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 14:38:42 +01:00
Timo K. 38ef1add48 use compound web sticky patch 2026-10-28 19:29:08 +01:00
Timo K. cffdb58521 rop the device list's custom fade under the mete 2026-10-28 16:33:31 +01:00
Johannes Marbach b7bc218b19 Merge pull request #4367 from RaphaelFakhri/RaphaelFakhri/shared-key-storage-key
Read and write the room shared key under its single-prefixed storage key
2026-10-28 16:21:11 +01:00
Timo K. 17caded88b clean up css calc in ts code 2026-10-28 16:11:49 +01:00
Raphael Fakhri 85424f76b1 Ignore the URL password when the parameters are for another room 2026-10-28 14:33:07 +00:00
Timo K. c713b13994 review: audioMeter instead of iconsAndLabels === "audio" 2026-10-28 15:17:44 +01:00
Robin 5780597b83 Fix crash when joining call with connection stats enabled
As demonstrated by the test. This crashed with "Behavior failed to synchronously emit an initial value."
2026-10-27 16:17:58 +01:00