Commit Graph
100 Commits
Author SHA1 Message Date
Timo K.andClaude Opus 5.5 186e1bdc9f Split the media menu into a button and per-device context menus
MediaButtonWithContextMenu holds the mute button, the chevron and the
menu's open state; AudioContextMenu and CameraContextMenu are its content,
and AudioMenuButton and CameraMenuButton wire each to its own view model.
The view model is split the same way: the audio menu carries the output
devices, the camera menu the background effects and the preview.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 16:33:27 +01:00
Timo K. 586154772e Update track mute comment 2026-11-18 15:23:29 +01:00
Timo K. 3acf5d9a28 review: toggle comments confusing 2026-11-18 14:58:03 +01:00
Timo K. ac461840a3 comment changes to MediaMenuViewModel 2026-11-18 14:55:28 +01:00
Timo K. 438d14d1a1 fix facing mode
Use default fallback instead of inversed check.
2026-11-17 21:11:34 +01:00
Timo K. 889828763c Review from the background effects 2026-11-07 16:18:26 +01:00
Timo K.andClaude Fable 5.1 190902eec0 Test the session where the pipeline cannot be built
- The effects state's timeline through preparing, settling and attached,
  and that a failed trial build withholds the effects for the session while
  keeping the choice.
- The footer offers nothing once the pipeline fails to build, and passes
  on the wait as the state's single wait field.
- The call leaves the camera out of the join while the pipeline is
  prepared, publishes it once that is known, and turns on a camera turned
  off before the processor was chosen; the lobby opens no camera meanwhile.
  Both view models are driven by one state subject.
- The transformer's trial build answers whether a segmenter can be made.
- End to end, a browser whose pipeline cannot be built sees the effects
  withheld.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-06 16:54:42 +01:00
Timo K.andClaude Opus 5.5 88f748ec21 Offer no effects for the session where the pipeline cannot be built
- A segmenter that can't be built, with the model unreachable or a GPU
  its delegate refuses, took the camera down with the pipeline: the SDK
  fails the track it was attaching to, and the lobby muted camera and
  microphone, on every load while the choice was kept.
- One is now built on trial when an effect is first wanted, before the
  pipeline is attached to anything, on the CPU where the GPU refuses.
  Where it can't be built, the options are disabled with the existing
  notice for the rest of the session. The choice is kept: a browser
  update may fix what failed.
- The view model takes that verdict from the pipeline's state rather
  than
  asking the SDK, so both say the same, and it no longer builds a WebGL
  context of its own to ask.
- Until the trial is over, the lobby opens no camera and the call makes
  no
  camera track, so neither starts without the effect.

Rebased onto main, where the Publisher is gone and the call view model
publishes the camera through the SDK:
- CallViewModel takes videoProcessorPreparing$. While it is true, the
  camera's mute handler waits before making a track, and the join leaves
  the camera out of its requests (initialPublishRequests); the view
  model
  publishes it once it is known whether the pipeline builds. The
  Publisher's test became CallViewModel tests.

Cherry-picked from spec/background-effects/12-unbuildable-pipeline onto
toger5/background-effect/camera-preview, where the camera menu's preview
takes the camera from the footer's caller rather than from a track
synced
on the background-effects state. The footer factories take the
pipeline's
state again, as this commit needs its verdict and not only the settling
flag; the synced camera and the preview's wrapper type are not brought
along.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-06 16:54:42 +01:00
Timo K. f2f67019b4 Show the self preview in the video mute button context menu 2026-11-06 15:35:04 +01:00
Timo K. 51fac690f4 Hand the camera track to the footer view model 2026-11-06 15:35:04 +01:00
fkwpandClaude Opus 5.5 fb4cbc72e2 Remove a background of one's own
- A cross on the tile's top-right corner removes an added background,
  critical-coloured at rest and named by its tooltip, as design draws it.
  The one in force offers none, since taking it away would leave nothing
  chosen; the view model refuses it too.
- The cross sits beside the item rather than in it, which would take the
  press first and choose the tile, and isn't a control of its own, so the
  menu keeps one keyboard model: Delete or Backspace on the tile does the
  same, and the tile says so with aria-keyshortcuts.
- Removed from IndexedDB and its picture released. End to end on
  Chromium and Firefox, a background removed by Delete is still gone
  after a reload.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-06 10:33:22 +01:00
fkwpandClaude Opus 5.5 373b1e0fc0 Guard what can be added as a background
- Five of one's own at most, so the grid is whole: at the limit the add
  tile goes, and it comes back once one is removed. The store holds the
  limit too.
- An added image is refused if it isn't an image, is animated, or can't
  be decoded; reduced to 1920px on its longest edge; and laid on an opaque
  ground, since a transparent pixel would be a hole in the background.
- Animation is read from the file, as a still and an animated WebP share
  a type. Chromium has no selected track until the decoder's tracks are
  ready; without waiting for them an animated GIF got through.
- A refusal says why, in place of the notice, and is scrolled into view.
  It can be dismissed, is gone once the menu closes, and the next shows
  even if it reads the same. What was in force stays.
- End to end on Chromium and Firefox, from what the device then keeps: an
  animated GIF is refused with its message, a 4000x3000 image is kept at
  1920x1440, and a half transparent one is kept opaque. All three fail
  with the preparation bypassed.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-05 16:20:33 +01:00
fkwpandClaude Opus 5.5 1bc372b587 Add a background of one's own, kept on this device
- An add tile after the pictures opens the file picker, and the image
  chosen joins the backgrounds on offer without being put on; it is in
  force once chosen, like a shipped one. The tile is a command rather
  than a choice, so a menu item beside the radios, and it isn't offered
  where effects can't run.
- Kept in IndexedDB on this device and never sent anywhere, behind a
  small interface, so the unit tests use an in-memory store; the
  IndexedDB store itself is tested against fake-indexeddb, a new dev
  dependency.
- The picker takes the focus, which the menu reads as a click elsewhere,
  so the menu holds itself open until a file is chosen or the picker is
  cancelled.
- A chosen image the device no longer keeps, as after its storage is
  cleared, falls back to no effect once the device has said what it
  keeps.
- Switching is checked between any two kinds of effect. End to end on
  Chromium and Firefox: an added image shows in the preview once chosen;
  once it is in force, no request carries the file or the copy kept,
  which a deliberate upload fails; chosen, it is in force again after a
  reload, gone once storage is cleared, and can be added again.
- Kept as given here; what can be added is checked in the commit after.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-05 16:20:33 +01:00
fkwpandClaude Opus 5.5 4e779ac4f5 Offer background effects on any platform whose browser can run them
- The desktop rule from #4059 goes: now that the menu says where effects
  run slowly, using them there is the user's choice. A phone on the fast
  route held 99% of its frame rate and was refused; a desktop on the slow
  one held 73% and was allowed.
- Asks @livekit/track-processors directly; the wrapper module goes.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-05 13:48:29 +01:00
fkwpandClaude Opus 5.5 7fb222ce60 Show the wait for the first background effect on its tile
- The first effect chosen in a session shows a spinner on its own tile,
  in place of the tick, until a frame carrying it is drawn. Later changes
  show nothing, having nothing to prepare.
- Ended by the frame, not the promises, which resolve while the segmenter
  is still being built. The transformer reports the first frame that went
  through the segmenter; one passed through untouched doesn't count.
- The pipeline publishes the wait with its state, and both footer view
  models take it as an input.
- The lobby tests' provider mock gains the observable hook, built once: a
  fresh observable per render rebuilt the footer on every render.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-05 13:48:29 +01:00
fkwpandClaude Opus 5.5 5fdb920ea8 Say where background effects run slowly
- Where the browser has no MediaStreamTrackProcessor, effects draw each
  frame through a canvas: slower, and the first build stalls the page.
  They stay offered and choosable there, with a notice under the grid
  that says so and describes the group.
- Decided in the view model, beside whether effects can run at all, so
  the lobby and the call give the same answer.
- The notice stays within the menu's width however long the sentence,
  which the story checks.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-05 13:48:22 +01:00
Timo K. 813dd4a49e remove unnecassary type 2026-11-05 13:45:59 +01:00
Timo K. 21b54501de rename BackgroundEffectOption to BackgroundEffectGridItem
This makes it more clear that this is a UI struct and not related to the
items in the catalogue
2026-11-05 13:45:59 +01:00
fkwpandClaude Opus 5.5 3d57920556 Offer background effects as a grid in the camera menu
- The camera menu's blur toggle becomes a Background effects section
  under the cameras: no effect, blur and the two shipped backgrounds as
  one radio choice of tiles, three across. It scrolls with the device
  list, so it stays reachable in a short call, and choosing keeps the
  menu open. The lobby gets it too, as both render the same footer.
- The footer's view model gives the options, the notice and the actions;
  the footer only names them.
- Each tile is Compound's MenuItem as a menuitemradio, styled as a tile.
  Compound makes it an item of the dropdown, so the arrow keys walk the
  tiles with the device rows, and takes its tap in the phone drawer. A
  tile that can't be chosen has no select handler, so the keyboard passes
  over it.
- Where the browser can't run effects the tiles are shown, disabled and
  explained by a notice the group is described by. No effect stays
  choosable and shows as in force. The notice is in the scrolling list,
  so a short call doesn't push the menu out of it.
- Section headings are raised, or the positioned tiles scroll over the
  one they sit under; raised within the list only, so the menu's frame
  stays over them. Tiles clip with overflow: clip; with hidden they are
  scroll containers, which focus scrolls into view without their border.
- End to end on Chromium and Firefox: a shipped background chosen in the
  lobby shows in the preview, and None after Blur draws nothing, counted
  in WebGL draws (Blur: about 1400 a second on Chromium, 2300 on
  Firefox).
- The Quick Audio Menu's test of the camera menu keeps its name and now
  finds blur among the effects rather than as a toggle.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-05 13:45:59 +01:00
Timo K. cc994abeb3 review duplicated BackgoundEffects files
One is now called backgroundEffectCatalog
2026-11-05 10:39:57 +01:00
fkwpandClaude Opus 5.5 33f4d5043f Choose a background effect, not only blur on or off
- A catalogue of what can be chosen: no effect, blur, and two shipped
  backgrounds, the app's own gradients flattened onto its canvas colour.
  Their ids are what a saved choice stores, so they are named for the
  pictures.
- One setting for the choice, typed as the ids an effect can have and
  defaulting to blur for anyone who had it on. A stored value we no
  longer ship, or one that isn't even a string, reads as no effect: it
  comes from storage another build or a hand may have written.
- The pipeline follows it, switching in place to a picture as readily as
  to blur, and the camera menu's blur toggle reads and writes it.
- So does Settings' blur checkbox: a picture reads as blur off, and
  checking it replaces the picture with blur. The old on/off setting is
  only read, to carry blur over. First tests for the Settings modal,
  which fail with the checkbox on the old setting.
- The transformer's own init loads a picture chosen before it was built,
  as the base init does and it had not: the first effect always is, and
  the picture was drawn black.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-05 10:39:57 +01:00
fkwpandClaude Fable 5.1 a0aaa8e09a Ask one question about whether background effects can run
The controls and the pipeline each asked the SDK on their own whether
background processing is supported, so one could offer what the other
refused. One function answers both, and it asks the browser, kept to
desktops until the menu says what the effects cost elsewhere.

Ported from spec/background-effects/2-pipeline (#4282): f78e9c05 and
87638325, the tests on the lobby and call footers as they are.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-04 20:17:22 +01:00
Timo K. e339098939 Merge branch 'main' into spec/background-effects/1-device-menus 2026-11-03 14:40:13 +01:00
Timo K. 38ef1add48 use compound web sticky patch 2026-10-28 19:29:08 +01:00
Timo K. cffdb58521 rop the device list's custom fade under the mete 2026-10-28 16:33:31 +01:00
Timo K. 17caded88b clean up css calc in ts code 2026-10-28 16:11:49 +01:00
Timo K. c713b13994 review: audioMeter instead of iconsAndLabels === "audio" 2026-10-28 15:17:44 +01:00
fkwpandClaude Opus 5.5 9e6f2e89a8 Size and scroll the device menus with Compound's menu
- Compound 10.3.0 bounds a menu by the space it has and scrolls what
  doesn't fit, within a boundary the app passes. The menus pass the call
  area, 16px in from its sides, rather than measuring its width.
- The 296px width is the menu's own style. Compound gives that style only
  to the dropdown, so menuIsDrawer, which copied its phone rule, goes.
- Compound draws the menu's frame over its content, so nothing that
  scrolls can hide it in Safari: the list, headings and meter no longer
  keep a border width clear of it, and the checks that they did go.
- The device list is Compound's scroll area, so it fades out at the
  foot with Compound's chevron while there is more below.
- The microphone menu's meter sticks at that foot, with rows scrolling
  under it as before, so the list has no fade above it: Compound's is a
  mask fixed to the list's edge, while a bounce at the top carries the
  meter with the rows. The band under the meter is filled, or rows would
  show there. The edge fades from where the chevron starts, so a bounce
  fades the meter out before it reaches the chevron. A row reached by
  keyboard stays above the meter.
- The list's height is still its share of the call.
- The component harness leaves the host's stylesheets unscoped, Compound's
  among them, as a host does: scoped, they never reached the menu, which is
  portalled out of Element Call's root.
- Its specs check only Element Call's own document styles on the host
  page, and find the device list within Compound's scroll area.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-26 15:48:52 +01:00
fkwpandClaude Opus 5.5 7a48b09c4c Hold both device menus at 296px, framed in Safari too
- The camera and microphone menus were as wide as their longest device
  name, so a long one widened the menu over more of the call: 340px for a
  long speaker name. Both are now 296px, as design sets them, and a long
  name wraps.
- Narrower only where the call area is, so the menu still fits inside it:
  268px in a 300px call. Taken from the observation of the call area that
  already bounds the list's height.
- Held only for the dropdown. On Android and iOS Compound renders the menu
  as a drawer as wide as the screen; menuIsDrawer follows its rule.
- Once the list is long enough to scroll, Safari paints it above the
  outline the menu is framed with, and the frame vanished down both sides
  of it: 384px of it on the many-devices microphone menu, in WebKit. The
  list now keeps a border width clear of the frame, so the headings and
  the meter inside it no longer need their own.
- Guarded with a long speaker name, which fails at 340px on main's sizing;
  a unit test run as a phone, which fails with the width held there; and
  the many-devices story's inset, which fails on the old stylesheet, since
  the stories run in Chromium, which never showed the Safari bug.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-26 15:48:07 +01:00
Johannes Marbach 96333dbc3a Merge branch 'main' into floating-tile 2026-10-26 13:46:54 +01:00
Timo K. 674692c926 Fix breaking Ci tests due to compound-web 10.1.0 -> 10.3.1 bump 2026-10-09 11:09:41 +02:00
Robin f2f49315eb Make drag & drop performant in all layouts
While also giving the logic for drag & drop of a floating tile a common home in the view model layer.
2026-09-28 19:53:30 +02:00
fkwpandClaude Opus 5 b7bd7d18b2 [Feature] Quick audio menu (#4275)
* Give the microphone level its own observable
* Add a microphone level meter
* Show speakers and microphones in the quick audio menu
* Wire speaker selection through the call footer
* Add stories for the meter and the device menu
* Add end-to-end specs for the quick audio menu
* Keep the footer while a menu opened from it is open
* Use the shared audio capture stub in the lobby test
---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-09-24 09:52:02 +00:00
Timo K. 365204d04c fix media queries in component based element call. 2026-09-07 14:50:36 +02:00
Valere 6e44332fd1 Stop reading URL parameters from inside the call path
The view models reached for getUrlParams() — and so window.location —
from deep inside the call path: CallViewModel, MediaDevices, Publisher,
LocalMember and the footer view model. An embedded Element Call has no URL
of its own, so these values have to arrive as arguments instead.

Add the relevant options to CallViewModelOptions, to the MediaDevices and
Publisher constructors, to createLocalMembership$ and enterRTCSession, and
to createCallFooterViewModel. The remaining React consumers read the
context added in the previous commit. AppViewModel now takes its audio
output options too, moving that URL read out to main.tsx, where the app
shell can act as the adapter.

The new CallViewModelOptions fields are optional, defaulting to what the
URL parameters resolve to outside widget mode; the MediaDevices and
Publisher arguments are required, so that every construction site has to
be explicit.

useTheme.test.ts mocked the UrlParams module with a factory, so it needed
updating to mock the hook rather than getUrlParams.

No functional change.
2026-09-02 10:31:48 +02:00
Robin 1c03ad335c Hide reactions interface in PiP mode 2026-08-19 17:44:45 +02:00
Robin b554fa63d1 Performance: Isolate tile store debug info in its own component
It's very uncommon to have this debug option enabled, and yet it currently causes the footer to re-render on every layout update, which is a small but avoidable cost.
2026-08-17 16:45:26 +02:00
Robin a443a55bec Factor out a layout switch view + view model
So it can easily be shown and hidden wholesale.
2026-07-28 15:06:15 +02:00
Robin bff41ad66a Update spotlight icon 2026-07-13 12:22:44 +02:00
Robin 32e30dd4f5 Merge branch 'livekit' into mobile-gradient 2026-07-09 12:56:28 +02:00
Robin 11fadebb87 Fix test failures due to scoped class names 2026-07-08 17:06:25 +02:00
Robin 943ce5b853 Update test snapshots to use scoped class names 2026-07-08 17:06:09 +02:00
Robin 900d0abd27 Let page background shine through the footer
By visually separating it from other page content with an additional gradient background only when necessary.
2026-06-30 16:29:20 +02:00
Robin e11c04ac87 Move ringing status indicator to header on mobile
On mobile, the ringing status indicator is supposed to display in the header rather than on a tile. The exact layout differs between Android and iOS. To get it right I had to refactor AppBar to use CSS grid templates.

(Also, I changed my mind about the exact ringing data I needed out of CallViewModel - sorry. A little move of the ringtone audio renderer into its own component was necessary to accommodate that.)
2026-06-18 17:41:35 +02:00
Valere 62a3ec8319 review: cleanup removed redundant default properties 2026-06-05 12:50:25 +02:00
Valere acc5a440c7 review: remove unneeded early return 2026-06-05 12:35:21 +02:00
Valere b5b238ffd3 storybook: Add storybook for audio/video busy 2026-06-04 19:13:27 +02:00
Valere 4606373e5b feat(mute): add syncing state and disable toggle during async mute 2026-06-04 18:55:31 +02:00
Robin cec78a383d Fix footer stories 2026-06-02 11:14:53 +02:00
Robin 5fed562db2 Request full device names when device switchers are open 2026-06-02 10:36:57 +02:00