Compare commits

...
Author SHA1 Message Date
fkwpandClaude Opus 5.5 993991105f Keep five backgrounds of their own, which fills the grid
- At four the grid stopped at eight tiles, since the add tile goes at the
  limit, and left its last corner empty. At five it is nine at the limit,
  and four with the add tile is nine one short of it.
- Guarded as the rule rather than the number: the grid is whole at the
  limit and one short of it. Fails at four.
- The stylesheet had claimed a whole three-by-three at the limit all along;
  now it is one.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-23 13:09:17 +02:00
fkwpandClaude Opus 5.5 f2782c565e Add a background without putting it on, before joining too
- Added to the backgrounds on offer and in force once chosen, in the lobby
  exactly as in a call. Choosing a file and choosing to wear it are two
  decisions, and one rule for both places is one less thing to know.
- The footer's before-joining flag existed only for the other rule, so it
  goes, from both view models and the story.
- The edge-fade story waits longer for a scroll to settle: under the full
  parallel run a wait could outlast the default second.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-23 13:07:17 +02:00
fkwpandClaude Opus 5.5 fd972a717e Show that the camera menu scrolls, whatever the scrollbar does
- The platform fades its scrollbar out after a moment, and with a trackpad
  on a Mac there was then nothing to say the list scrolled: the backgrounds
  seemed simply to stop. Keeping the scrollbar would mean drawing our own,
  overriding a setting the user chose, and still not working in Firefox.
- Instead a fade at each edge of the list while there is more that way, as
  iOS, Android and Material do: the rows and tiles fade into it at the foot,
  and at the top it stands just below whichever heading is stuck there,
  where the content comes out from under it.
- Worked out from the list on scrolling, on resizing and after every render,
  since adding or removing a background changes its length without either.
- The camera menu only: in the microphone menu the level meter holds the
  foot of the list.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-23 12:51:41 +02:00
fkwpandClaude Opus 5.5 b7ca586f78 Keep the headings over the grid, and the preview where there is no room
- The effect tiles are positioned, to carry their tick and the remove
  cross, and positioned content painted after a sticky heading slid over
  it: the grid ran across "Background effects" instead of under it. The
  headings now have a layer of their own.
- Where the call is too short to pin the preview, it is no longer dropped:
  it goes first into the list and scrolls away with the devices and the
  effects, seen on opening and out of the way once choosing.
- The list moves up over the menu's padding to bring it flush with the
  top, since a scrolling container cuts off what is pulled above it, and
  rounds its top corners with the frame's so a stuck heading cannot show
  past the curve.
- Guarded: the heading is what is on top where it stands, failing without
  the fix; and the preview sits inside the list and scrolls away.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-23 12:07:46 +02:00
fkwpandClaude Opus 5.5 efe6274636 Keep the menu's frame visible beside a scrolling list in Safari
- Once the device list had enough in it to scroll, Safari painted it above
  the outline the menu is framed with, and the frame vanished beside every
  device. Not new: the microphone menu with many devices lost 384px of it.
  The camera menu only met it because the self-preview shortens its list.
- The list now holds itself a border width clear of the frame, so nothing
  inside it has to: the headings and the meter lose their own clearance.
- Frame intact in both engines in every case measured. Belongs to the
  device menu, so it can be cherry-picked on its own.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-23 10:30:40 +02:00
fkwpandClaude Opus 5.5 8d95c76b8b Hold the camera menu at 296px, as design sets it
- A long device name now wraps instead of widening the menu over more of
  the picture the user is judging. Narrower only where the call area is,
  so it still fits inside it: 268px in a 300px call.
- The self-preview is always the sixteen by nine it is drawn at, 294 by
  166 inside the frame, and Safari's narrowed preview has nothing left to
  happen to.
- Tiles are thirds of the set width, about the 80px design draws, where
  their floor was there only to push the menu wider.
- Nothing else in the menu gets to set its width either: the picture
  sits over its box, and the notice and the refusal fit the menu rather
  than asking for their sentence on one line, which took it to 340px.
- The microphone menu is unchanged.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-23 10:29:56 +02:00
fkwpandClaude Opus 5.5 31f057726f Fill the preview's width in Safari
- A ratio with a height cap let Safari keep the ratio by narrowing the box:
  once a long device name widened the menu past 313px, the preview stopped
  short and left a strip of menu at its right. Chromium stretched it.
- A fixed height instead, cropped by the picture: sixteen by nine at the
  menu's narrowest, wider as the menu widens, the same in both engines.
  Measured in WebKit before and after, 313px against a 398px menu, then
  398px.
- Guarded in a wide menu, though in Chromium only: no test tier here runs
  WebKit, so the story holds the requirement rather than the engine.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-23 09:58:42 +02:00
fkwpandClaude Opus 5.5 45469f3828 Show how the user looks while they choose a background
- A live self-preview at the top of the camera menu, as design drew it:
  full width, flush with the top, sixteen by nine. Both sections scroll
  beneath it, because it is what the choosing below is for.
- Only where the call leaves room. It is paid for out of the device list's
  share rather than on top of it, and only where the list keeps its floor
  after paying, so the menu is never taller for having one.
- Attached to the track the pipeline is synced to, which both the lobby and
  the call already report, so it shows the effect in force without a capture
  or a pipeline of its own. Mirrored like the pre-join preview, a camera-off
  mark while there is no camera, and the wait while the first effect builds.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-23 09:35:57 +02:00
fkwpandClaude Opus 5 8ffc8e2720 Sit the notice as far from the menu's foot as from its sides
- The menu carries bottom padding of its own, so setting one on the notice
  as well made the gap under it twice the gap beside it.
- Guarded as the relationship rather than the number, which is what anyone
  looking at it would notice. Fails at 32 against 16 without the fix.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-18 15:04:40 +02:00
fkwpandClaude Opus 5 508fde1029 Say what the browser costs without raising its voice
- The design's second pass: the subtle ground, an outline mark and text at
  its ordinary weight, sixteen pixels in from the menu's sides like the
  grid above it. Measured against the drawing, the ground and the inset
  land on it exactly, and the padding gives the same box for the same
  number of lines.
- No longer an alert. An alert is for something that just happened; this
  was true before the menu opened and will be true after it closes, which
  is also why it cannot be dismissed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-18 14:59:25 +02:00
fkwpandClaude Opus 5 a2fc94a882 Guard that a shipped background can cover something
- Reads each shipped file and requires a format with no alpha channel in it.
  Fails on the overlay scrims that started this, which is the point.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-18 14:10:26 +02:00
fkwpandClaude Opus 5 dfd42e2edd Give a background something to cover with
- The stand-in backgrounds were the app's own gradients, which are overlay
  scrims: measured, not one pixel in either is opaque, and where they look
  dark they are transparent. As a background their dark half was simply
  absent, which is what showed on the slow path.
- They are now laid on the canvas colour they are painted over and kept as
  JPEG, which cannot carry transparency at all.
- The same hole was waiting for anyone's own picture: a canvas starts
  transparent, WebP keeps an alpha channel, and a file small enough to keep
  as it was never passed through either. Every picture now goes through an
  opaque ground, whatever its size.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-18 13:59:07 +02:00
fkwpandClaude Opus 5 bfc42d8d7f Lay the background effect states out for review
- A docs page putting the four states side by side: offered, offered with
  what it costs, being set up, and not possible at all.
- The settling story plays the sequence rather than posing it, shortened so
  it is watchable.
- The notices carry the real strings, so a reviewer reads the words a user
  would rather than a paraphrase.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-18 13:49:40 +02:00
fkwpandClaude Opus 5 5b8e7d8cfd Offer effects wherever they run, and say what they cost
- Drops the desktop test. Measured, it sorted devices by the wrong thing:
  a phone on the fast path held 99% of its frame rate and was refused, a
  desktop on the slow one held 73% and was allowed.
- What the cost tracks is the processing path, and Safari and Firefox take
  the slow one on every platform. That is told, not enforced: the cost is
  smoothness, the reason to pay it is privacy, and only the user knows
  whether they would rather show the room they are sitting in.
- The first effect of a session takes a while to prepare, so its own tile
  spins where its tick will go, until a frame has actually come out of the
  pipeline. The promises say nothing useful about when that is: attaching
  resolves in about three seconds and switching in none at all, and the
  slow path then holds the page still for twelve to fifteen more. The
  spinner turns by rotating, which a browser composites, so it keeps moving
  through a pause that has stopped everything else.
- Copy is the exploration's and needs an owner (S3).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-18 13:49:40 +02:00
fkwpandClaude Opus 5 2f214c8ca2 Ask the bench where the startup went
- Splits attaching from switching: building the segmenter fetches, compiles
  and sets up the GPU, while switching only tells a built one what to draw.
  One of those can be paid ahead of time.
- Reports the segmenter's assets as the browser saw them. A transfer size
  of zero means cache, which is what separates the first run on a device
  from every run.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-18 11:52:31 +02:00
fkwpandClaude Opus 5 6a4bfa1b9a Put the bench's buckets on the clock
- They closed only when a frame arrived, so a freeze could not be reported
  as one: the iPhone's fourteen frozen seconds came back as a single
  bucket at a seventh of a frame per second.
- Adds the two numbers that say it plainly: time to the first frame, and
  the longest the picture simply stopped.
- A run now ends even if no further frame ever arrives.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-18 11:49:07 +02:00
fkwpandClaude Opus 5 1e445a4d14 Measure whether a device can run background effects
- The camera and the pipeline, no call in the way: if a device cannot hold
  a frame rate here it cannot hold one in a call either.
- Reports mean and worst five-second frame rate, startup, and the worst
  stall on the main thread. Run no effect first; every number is only
  meaningful beside that device's own baseline.
- Ignores the desktop gate on purpose. Measuring what it forbids is the
  point, and the gate was written from reports rather than numbers.
- Exploration only, on an unlisted route, and one path segment deep: the
  config is fetched relative to the page, so a nested path never starts.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-18 11:35:42 +02:00
fkwpandClaude Opus 5 aabd9a7022 Ask one question about whether effects can run
- The menu asked the browser for the APIs; the pipeline asked for those and
  a desktop. A phone browser sat between the two: every effect offered,
  none applied, nothing said.
- Both now ask the same function, so what is offered and what can be
  delivered cannot drift apart again.
- Tested against the lobby, where the switcher stays enabled on a phone —
  in a call it is already withheld there, so a check would pass whatever
  the verdict said. The test fails on the old wiring.
- This also fixes the blur toggle, which had the same split.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-18 11:14:44 +02:00
fkwpandClaude Opus 5 1299dad4b1 Draw the remove button as design drew it
- Critical from the moment it appears, not only under the pointer: the
  ground carries the hover instead, and gains a ring where the shadow was.
- 28px on the tile's corner, overhanging it by a fifth of itself, with the
  glyph a third of the button as the mock draws it.
- It names itself: a Remove tooltip, so the cross has words.
- Every colour is a Compound token and lands on the mock's own value in
  both themes, bar the dark hover ground, which is one step up the red
  ramp from what the mock measures.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-18 10:58:13 +02:00
fkwpandClaude Opus 5 4cea64bd07 Format the branch
- oxfmt over the six files the exploration added or changed.
- Wrapping only, no behaviour: the formatter is a separate CI step from
  pnpm lint, which is why it went unnoticed on the branch.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 23:24:12 +02:00
fkwpandClaude Opus 5 61cc12d937 Take the section rules from the device menu's new design
- The camera menu drew its own rule under each heading, because the menu
  suppressed Compound's. It no longer does, so ours drew a second one:
  the separator, its class and the heading override all go.
- The grid is its section's first control, so it starts under its rule
  where a device row's control starts under that one.
- Guarded in the story: two headings, each ruled edge to edge, nothing
  else dividing them, and both sections' first controls level.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 23:12:29 +02:00
fkwpandClaude Opus 5 c793cd5fc1 Open up the space between the effect tiles
- Measured off the design file: its gap is about a sixth of a tile's
  width, ours was under a tenth
- A step wider brings it to 14.3% against the design's 15.6%, the closest
  a spacing token reaches
- The tiles keep their width and the menu grows instead, which is the
  column floor doing its job; the grid still sits 16px from the frame
- Gives the remove cross on the corner room of its own, which is what
  prompted it

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 23:01:54 +02:00
fkwpandClaude Opus 5 8a2ad2226d Move the remove cross to the tile's corner
- On the top-right corner, where Zoom and Teams both put it
- Centred, it sat inside the target for choosing the tile: the one
  destructive action lived exactly where the ordinary one does, which is
  what was confusing people
- Red under the pointer. That alone would not have fixed it: colour says
  what a control does, it does not stop the press landing on the wrong one
- The one in force still offers no removal (FR-025). The corner would
  allow it, since cross and tick no longer compete for the middle, but
  that stays product's call

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 23:01:54 +02:00
fkwpandClaude Opus 5 c7c44eea4e Put an added background on before joining, not during a call
- Before joining nobody sees the change, so the picture goes on at once
- In a call it waits to be chosen: otherwise choosing a file would change
  what everyone sees with no further word from the user
- Teams draws the line in the same place, which is what settled it: a
  background added before joining applies by itself there, one added in a
  meeting has to be applied
- The footer learns which side of joining it is on from its view model,
  since the lobby and the call build their own

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 23:01:54 +02:00
fkwpandClaude Opus 5 ea25cde725 Add a background without putting it on
- Whether choosing a file should also wear it is still open, so this does
  the smaller thing: the image joins the others and the user says when
- Reverts the auto-selection added earlier, which was never proven: the
  stories carry selection as a fixed arg, so nothing there could show it

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 23:01:54 +02:00
fkwpandClaude Opus 5 31e67577b0 Keep the refusal out of the scrolling list
- It sat inside the device list, which scrolls and is bounded by the size
  of the call, so the menu's own height cut it off
- Now below that list, where nothing can clip it or leave it under the
  fold: no way to tell someone something failed
- 16px from the menu on every side, a step of which the menu's own column
  gap supplies

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 23:01:54 +02:00
fkwpandClaude Opus 5 72ad732771 Space the refusal evenly, and let it go when the menu does
- It sat flush against the tiles with twice that gap beneath; now 16px
  all round, the same as everything else in the menu keeps from the frame
- It can be dismissed, and it does not outlive the menu it appeared in.
  Before, it stayed until the next attempt: closing the chevron and
  opening it again showed a message about a file chosen long ago
- Checked: shown on refusal, gone when dismissed, shown again on the next
  refusal, gone on reopening

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 23:01:54 +02:00
fkwpandClaude Opus 5 fb223bcc4b Say why a file cannot be used as a background
- The reason appears with the grid, where the user chose the file, using
  the same Alert the reaction menu already reports errors with
- A reason for each: not an image, animated, could not be opened, and a
  fallback for anything the store itself refuses
- Named one by one rather than built from the reason, so the extractor
  finds every string it has to translate
- The background in force is untouched, which is the other half of
  FR-021: being refused a file changes nothing
- Covered by a story, and checked end to end by offering a text file:
  the message appears, the menu stays open, nothing is kept

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 23:01:54 +02:00
fkwpandClaude Opus 5 3ac42a735b Make the cross remove by mouse, not just by keyboard
- The cross now sits beside the tile rather than inside it. Inside, the
  menu reached the item before any handler on the cross, whichever phase
  it was in: the tile was chosen, became the one in force, and the cross
  was taken away before it had acted
- That is why the keyboard worked and the mouse did not, and why the
  static stories missed it: with selection fixed by an arg, nothing
  re-rendered and the click survived
- A story with live selection covers it now: the cross removes the tile
  and does not make it the one in force on the way out

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 23:01:54 +02:00
fkwpandClaude Opus 5 e3aaad8615 Let a user remove a background they added
- A cross where the tick would be, on hovering one of their own images
- Not offered on the one in force: that is what they are wearing, and
  taking it away would leave them with nothing chosen (FR-025)
- Delete or Backspace on a focused tile does the same, announced through
  aria-keyshortcuts. The cross is drawn rather than focusable: a control
  inside a menu item is invalid, and the item is what the arrows walk
- Covered by a story: the one in force offers no removal, the others say
  Delete and answer to it

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 23:01:54 +02:00
fkwpandClaude Opus 5 943cae0129 Sit the grid evenly in the menu, and the headings closer to their rules
- The grid now stands the same distance from the menu's frame on every
  side: 16px left, right, bottom, and above from its rule. It was 16 at
  the sides but 12 above and 28 below, since its own bottom padding and
  the menu's were both in play
- A ruled heading sits closer to its rule than the menu's own spacing
  allows, which is set for a heading with nothing under it: 14.5px from
  the heading's text rather than 18.5px, and the same in both sections
- Scoped to headings that are ruled off, so the microphone menu keeps the
  spacing it was drawn with

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 23:01:54 +02:00
fkwpandClaude Opus 5 79eb5c8606 Draw the active frame at one width on every tile
- Thinner: colour alone, at the width every tile already carries
- The same on an added background as on no effect. It was thickened with
  an inset shadow, which draws inside the padding box where an image
  tile's picture sits, so the picture covered it and image tiles wore a
  thinner frame than the rest

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 23:01:54 +02:00
fkwpandClaude Opus 5 35f9893b0a Even up the space below the two section rules
- The tile grid added its own room on top of the rule's, so the tiles sat
  20px below their rule while the device rows sat 12px below theirs
- The rule sets that distance now, and it is the same in both sections

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 23:01:54 +02:00
fkwpandClaude Opus 5 304373e735 Rule off under each heading in the camera menu
- A rule under "Camera" as well as under "Background effects"
- Both run edge to edge: a separator insets itself by default, which
  leaves it floating short of the menu's sides
- The divider that separated the two groups goes; the rules do that work
- Closer to what they name than a separator's own spacing allows, which
  is set for one dividing two sections
- Scoped to the camera menu: the microphone menu still divides its two
  sections instead, which is the design the audio menu was drawn to

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 23:01:54 +02:00
fkwpandClaude Opus 5 1af968748b Keep the menu open when adding a background, and choose it
- The picker moved into the menu: a native file dialog takes the focus,
  which the menu read as a click elsewhere and closed behind it
- Close requests are ignored while the picker is up, and a cancelled
  pick releases that through the input's own `cancel` event, which React
  does not type
- Adding answers with the background it kept, so the footer can choose
  it straight away rather than asking the user to pick it twice

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 23:01:54 +02:00
fkwpandClaude Opus 5 81fe5d355f Let a user add a background of their own
- Added backgrounds are kept on the device in IndexedDB; nothing uploads
  them and no identifier would let anything fetch one
- Refuses what cannot be used and reduces what is larger than we keep:
  a 4000x3000 photo became 1920x1440 and 238KB became 5KB
- Animation is read from the file, not guessed from its type: an
  animated WebP and a still one share a type, and a .png may be an APNG
- Does not crop. The pipeline already covers the camera's frame with
  whatever it is given, recomputed as the frame changes, so cropping on
  the way in would bake in one shape and lose the rest for good
- The add tile is withheld once four are kept, rather than failing
- Added backgrounds live in their own context, not in ProcessorState:
  the publisher has no use for them and every stub of it would carry them

Not done: removing an added background, which the design gives no
affordance for, and telling the user why a file was refused, which has no
surface in the menu yet. Both are logged for now.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 23:01:54 +02:00
fkwpandClaude Opus 5 8a7eb5c7fd Make the settings blur control agree with the camera menu
- Both now read and write one setting, so they cannot disagree
- An image background reads as blur off there; turning blur on replaces
  the image, which is what FR-022 and EC-011 ask for
- The new setting defaults to whatever blur the user already had, so
  nobody loses the blur they turned on before this existed
- Covers the derivation with tests: what the checkbox shows, what it
  writes, and the fallback when a stored background is one we no longer
  ship

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 23:01:54 +02:00
fkwpandClaude Opus 5 6a6605c233 Let background effects run under the dev server
- nodePolyfills rewrote MediaPipe's WASM loader, which mentions `process`
- That loader is a classic script whose job is to set self.ModuleFactory;
  as an ES module it sets nothing, so the segmenter threw
  "ModuleFactory not set" and no effect could run under `pnpm dev`
- Its transform now skips that one directory
- Affects blur on main too, not only this feature: neither could be
  exercised locally before
- Production was never affected, since the loader is emitted as an asset
  there; build still green and the assets still emitted

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 23:01:54 +02:00
fkwpandClaude Opus 5 9179b1cccb Wire the background effects grid to the pipeline
- Footer view model exposes the chosen effect and a setter for it
- The footer names the catalogue: a view model holds no translated text
- The camera menu shows the grid in place of the blur toggle
- Shipped images are numbered, not named, since the art is a stand-in
- Withholding the setter is what renders the section disabled, as with
  the speaker section where no output can be chosen

Exploration for FEATURES_SPEC/2026-09_Background_Effects.md.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 23:01:54 +02:00
fkwpandClaude Opus 5 e823fbc85b Show background effects as a grid in the camera menu
Adds the Background effects section from the design: no effect, blur, the
shipped images and a tile for adding your own, under the camera list.

The tiles are menuitemradio like the device rows above rather than a new
control, so the menu keeps one keyboard model throughout and the selection is
announced rather than only drawn. An image tile is the picture, filling the
tile, and its name goes to assistive technology alone; the add tile is a plus,
named the same way. The tick stands where the glyph would, carrying its own
ground on a picture so it reads against whatever is behind it.

Repurposing a menu item as a tile means overriding its row layout on three
axes, each found by measuring rather than by eye: it lays itself out as a grid
of chevron, icon and label, so flex-direction does nothing; it carries the gap
between them as a right margin on each, which pushes both off the tile's
centre when they are stacked; and it insets its own text sixteen pixels, so
the grid needs the same inset to line its tiles up with the heading above them.

Sized against the design file rather than judged: glyph and tick a quarter of
the tile wide, an ink gap of about a seventh of its height to the label, and
the tile's proportion carried by its padding. Rows are equal height, set by
the tallest tile — a fixed ratio clipped the label as soon as the menu was
narrow enough that the ratio could not hold it — and the columns carry a
minimum width so the menu grows wide enough for three tiles rather than
shrinking them to whatever the device names need.

The blur hatch is a stand-in: the design system has no blur icon, and the
design's own is an icon whose strokes shorten towards the corners. A uniform
hatch fills its box, so this one is drawn smaller and finer to carry the same
weight rather than the same size.

Where background processing is unavailable, no effect stays choosable: it
needs none and it is what the user has. The rest are disabled, and the images
lose their colour as well as their contrast, since a washed-out photograph
still reads as one you could pick.

The existing blur toggle stays for callers that pass no effects, so nothing
that renders this component has to change yet.

Exploration for FEATURES_SPEC/2026-09_Background_Effects.md. Not for merge: the
image tiles are stand-in gradients drawn in the story, and nothing wires the
grid to the pipeline yet.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 23:01:53 +02:00
fkwpandClaude Opus 5 b90f267009 Attach background effects through one switchable pipeline
Replace the blur-only processor with a single wrapper that switches between no
effect, blur and a background image in place. Switching in place rather than
rebuilding matters: destroy() resets the transformer's first-frame flag, and
that flag makes it emit one unprocessed frame when it next starts processing,
so a rebuild per change would leak a frame of the real background every time.

Attach the pipeline the first time an effect is chosen and keep it attached
afterwards, including at no effect. Measured against a production build, the
segmentation assets are about 11.4MB and take ~143ms to initialise, so always
attaching would charge that to every user on every camera-on call, including
the majority who never choose an effect. Keeping it attached once it is there
costs about 25us per frame, which is within noise of no pipeline at all.

Complete init() while moving it. The old subclass replaced the base class's
init wholesale and dropped its image loading and disabled-mode setup, which
went unnoticed while blur was the only effect and blurRadius was the only
option that mattered.

Keep loading the segmenter from our own bundle. The library's assetPaths option
resolves WASM through MediaPipe's FilesetResolver from a directory, which is
not the same thing, and its defaults reach jsDelivr and googleapis.

Exploration for FEATURES_SPEC/2026-09_Background_Effects.md. Not for merge:
the two shipped backgrounds are stand-in gradients and there is no picker yet,
recorded as S1 and S2 in the spec.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 23:01:53 +02:00
fkwpandClaude Opus 5 5fff49670b Expect the default output entry to carry no name
- main stopped labelling the virtual default output with the first device's
  name (426acd22): a pinned sink is not re-routed, and which device the browser
  default resolves to is not knowable from here.
- The exploration still carried the older file, so this check passed locally
  and failed in CI, which tests the branch merged with main.
- AC8 is unchanged — Default is still a distinct entry beside the device it
  resolves to. Only the label it carries has moved.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-17 22:52:08 +02:00
fkwp 0633b24880 Merge remote-tracking branch 'origin/main' into fkwp/explore/quick-audio-menu 2026-09-17 22:50:39 +02:00
fkwpandClaude Opus 5 099950a100 Update the device menu to the latest design
- Each section is headed by its own rule, edge to edge, instead of one inset
  divider between the sections. Compound draws that rule under a menu heading
  already, so this removes our Separator, its spacing rule and its import.
- Our heading padding override goes too: Compound's menu heading is the design
  already — 13px/600 on a 19.5px line, #656D77, 16px either side, 7px from the
  text to the rule. Ours had been moving that 7px above the text.
- Two spacings do differ from Compound, both measured off the mock rather than
  guessed: 20px below the rule before a section's first device (27px rule to
  control, against Compound's 16px), and 28px above a heading that follows
  another section (41px from the last control to the next heading's text,
  against 22px). The first heading keeps Compound's spacing — the menu's own
  padding sits above it.
- Guarded by relationships, not numbers: the first control sits further below
  the rule than from the menu's edge, and a section stands further from the one
  above it than a heading does from its own first device. Both fail if either
  spacing goes back to Compound's 8px.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-17 22:43:48 +02:00
fkwp 194bfea287 Cover the speaker section sitting above the microphone section
- Covers AC28, which `ux_design_shevron.png` settles.
- Not written in this session: it was uncommitted in the tree beforehand, and
  is committed here so it is not lost.
2026-09-17 11:56:31 +02:00
fkwp 4f8f71e861 Cover the device menu with stories and end-to-end specs
- The stories supply what the app supplies rather than what a harness has
  lying about: a microphone, and a call-sized root element.
- Invented device ids meant the menu asked for hardware that does not exist,
  so the meter reported no microphone. A tone played into a real MediaStream
  fixes it, and the meter runs its own analyser over it.
- Without a root element the list was bounded by Storybook's whole frame and
  the menu ran off the canvas; without a call-sized one, four devices scrolled.
- Standalone specs: the switch that does not drop the call, the meter alive
  while muted, the pinned meter, the focus ring.
- Component specs: the list sized to the call and not the window, resized
  while open, every device reachable, and the modality belonging to one call.
- A headless browser has one microphone and one speaker, so fake-devices.ts
  adds synthetic ones. It is explicit that it does not route audio.
2026-09-17 11:56:31 +02:00
fkwp d7e0444117 Write the mute button once instead of twice per kind
- Both arms of the switch were the same eleven lines of click, disabled, busy
  and enabled wiring; only the component and the test id differed.
- Two copies that had to be kept in sync by hand, for nothing.
- `toggles` becomes an expression rather than a `let` mutated in a switch,
  which makes the optional chaining downstream dead.
- The labels switch stays: i18n extraction needs literal `t("…")` keys, and it
  gives exhaustiveness checking on the union.
2026-09-17 11:56:31 +02:00
fkwp a6e00895a0 Track the focus modality on the menu itself
- Was watched at the document, so a key pressed in another Element Call on the
  same page — or in the host's own page — lit this menu's ring. Element Call
  can be mounted more than once, and the menu is portalled out of the root.
- Watched on the menu instead, which is the element that holds focus.
- A callback ref, not an effect on `menuOpen`: that state is ours and the open
  menu is Radix's, and they do not commit together, so the effect could run
  before there was anything to attach to.
- Recorded on the menu rather than in state, so every item it can focus answers
  to it — the camera menu's blur toggle kept the browser's own ring otherwise,
  which follows the pointer.
- Nothing re-renders when the modality changes now, so the reset folds in with
  the listeners and the second effect on that state goes.
- Comments moved to the code they describe; three unrelated ones had stacked up
  on one declaration.
2026-09-17 11:56:31 +02:00
fkwp 8f3506782a Keep each section's heading in view while the list scrolls
- The heading of the section you are reading stays at the top of the list and
  leaves with its own section, so a long list never leaves you guessing which
  kind of device a row is.
- Opaque and held a border width clear of the sides, like the meter: a
  positioned element paints over the outline the menu draws its frame with.
- Sticky alone hides rows: scrolling a row flush to either edge puts it under
  the heading or under the meter, which is how a row reached by keyboard ends
  up half-readable. scroll-padding-block keeps both heights clear.
- Measured from the real elements, not from tokens — the meter's failure states
  are two lines where a level is one.
2026-09-17 11:56:31 +02:00
fkwp 8508d88714 Size the device list to the call as it changes
- The bound was taken once when the menu opened, so a host resizing the space
  Element Call is drawn in while the menu is open left it describing a call
  area that no longer exists.
- Follow it with observeElementSize$, which was already here and already used
  by useRootSizeMatches, and only while the menu is open.
- Quantised before it reaches React, so a drag-resize re-renders only when the
  bound itself moves.
- 160 and 0.6 are named now, with the reason each exists.
2026-09-17 11:56:31 +02:00
fkwp ace7e947b4 Show a default speaker where the platform lists none
- Safari enumerates no output devices at all, so the list arrived empty — and
  an empty array is truthy, so the Speaker heading rendered with nothing under
  it, which reads as a broken feature.
- Show one entry instead, named as the default, disabled, and marked as the
  selection: audio is going somewhere, and an unchecked lone row reads as
  nothing being chosen.
- D2 already says a row that cannot be chosen is shown disabled, never hidden.
- Covered by its condition rather than by the browser, so Linux CI checks it:
  one test that the view model hands over an empty list rather than none, one
  that the menu then draws the default.
2026-09-17 11:56:31 +02:00
fkwp 6c3012a40b Line the level meter up with the device rows
- Measured: the icon's centre sat 3px right of the radio controls above it.
- Two causes, both off the padding: the row is inset a border width to keep
  the menu's frame visible, and the icon is wider than a radio control.
- Vertically the icon was right and the text was not — a paragraph's bottom
  margin means the margin box gets centred, lifting the words.
- Guarded in a story, not a unit test: jsdom lays nothing out.
2026-09-17 11:56:31 +02:00
fkwp 6c9550da0a Give the level meter one bar size at every width
- A bar and the gap beside it are fixed; how many bars there are follows the
  space. Spread gaps made the meter a different shape everywhere, and closed
  the bars into one block when space ran short.
- Bars that touch cannot be counted, and the count is what carries the level
  without colour.
- `contain: inline-size` and `min-inline-size: 0` so the width decides the
  count and not the reverse — without them the bars set the menu's width.
- Gap is half again the bar width, measured off the design mock: 10px bars
  14px apart, a ratio of 1.4, nearest token pair 4px and 6px.
- Sizes stay in the stylesheet; the component reads them back off a rendered
  bar rather than repeating them.
2026-09-17 11:56:31 +02:00
fkwp d8211c0c61 Own the microphone level in the state layer
- The capture, the analyser loop and the loudness maths were in a component
  hook. Call logic belongs in the state layer; components render.
- observeMicrophoneState$ opens the device when something subscribes and
  releases it when nothing does. The hook is left as a bridge, no domain logic.
- Fixes a leak. A permission prompt outlives the menu that opened it, so the
  browser could hand over a microphone after cleanup had already run — with the
  variable that cleanup would have released still unset. The device stayed
  held, indicator lit, no meter on screen.
- Expressed now as the subscriber being closed, which the observable can always
  answer.
- The hook had no tests at all, which is why this survived. Four now, and the
  capture stub they need is in the shared factories rather than hand-rolled.
- One counts emissions across twenty animation frames and expects one: the
  redraw cost AGENTS.md asks continuous drawing to account for.
2026-09-17 11:56:25 +02:00
fkwp 0802649375 Show the focus ring for the keyboard alone, and fix the radio styling
- Show the focus ring only when the keyboard moved the focus. Radix focuses
  whatever the pointer is over, so the browser cannot answer that question
  here: Chromium treats every focus after a key press as keyboard-driven,
  Firefox treats no programmatic focus as keyboard-driven. The menu records
  which modality arrived, watched at the document while it is open, since the
  first arrow key lands on the menu itself rather than on anything we render.
- Suppress the browser's own ring on menu items, so there is one answer to
  that question rather than two.
- Drop readOnly from the device radios. Compound paints a read-only control
  muted, and that rule comes after the checked rule, so it overrode the accent
  fill marking the selection and the menu stopped matching settings.
- Make the decorative radio inert rather than aria-hidden. A negative tabindex
  inside an interactive control stays reachable to assistive technology even
  when hidden, which axe rejects.
- Give each section a labelled group and mark its heading decorative. A menu
  may contain only items, separators and groups, and the headings were direct
  children of it. The list wrappers are role="none" for the same reason.
- Run axe over the open menu, which is what found both of the above.

Spec: FEATURES_SPEC/2026-09_Quick_Audio_Menu.md — AC23, AC24, AC25
2026-09-16 17:44:55 +02:00
fkwp a76ef6215b more spacing between player and mic section 2026-09-16 16:27:17 +02:00
fkwp 7f3b98ed6a Size the device menu against the call, and cover its criteria
- Bound the scrolling device list by the measured height of the call area.
  It was capped with a viewport unit and a Radix variable, both of which
  measure the window: as a component in a corner of a host's page, the menu
  would have been sized against the whole page. The menu is portalled outside
  the root, so no container query reaches it and it has to be measured.
- Leave a single divider between the two lists, as the design has. Compound
  underlines every menu heading, so the headings drop theirs, and the line
  closing the microphone section goes: the section wrapper already bounds the
  meter's stickiness without one.
- Cover device persistence, hot-plug, fallback when the device in use is
  removed, fallback to a default when a remembered device is gone, numbered
  labels before permission, "Default" listed as its own entry, one device kind
  not disturbing another, and camera parity. Their acceptance criteria named
  check commands for tests that did not exist, which makes the criteria
  unenforceable and blocks the next drift check.

Spec: FEATURES_SPEC/2026-09_Quick_Audio_Menu.md — AC3, AC4, AC5, AC8, AC9,
AC10, AC11, AC19, AC20
2026-09-16 16:23:02 +02:00
fkwp 3f97ec8619 Show a live microphone level in the audio menu
- Draw the level beneath the microphone list as 24 segments, announced
  through role="meter", so it reads by count and not by hue alone.
- Give denied permission and no input device their own message and next
  action, rather than a flat meter that reads as silence.
- Read the selected device, and only while the menu is open, so nothing holds
  a second capture for the length of a call.
- Follow a rise quickly and a fall slowly, in elapsed time rather than frames,
  so the meter does not chase the gaps between syllables.
- Ignore a noise floor, below which a quiet room's hiss would light the first
  segments permanently. Smoothing changes how the meter moves, not where it
  settles, so it does not replace this.
- Scroll only the device lists, and keep the meter pinned to the foot of the
  microphone section while its rows are in view.
- Withhold the output select callback where the platform cannot route audio
  to a chosen device, which is what renders the speaker section disabled.
- Track the in-flight device by kind as well as id: Chrome names both an
  input and an output "default", so the spinner lit the wrong row.
- Disable every device while a selection is settling, so a second request
  cannot overtake the first.

Two CSS choices are load-bearing and look arbitrary: the meter's wrapper is
unpositioned, because a positioned one paints above the menu's outline and
swallows the frame along that section, and the meter is held clear of that
outline, because it is the only opaque thing in the menu.

The meter opens its own short-lived capture rather than tapping the call's
audio track. That lets it follow the picker instantly and avoids the pre-join
track, which is frozen to the device selected when the screen mounted. The
cost is a second capture while the menu is open. See the notes sidecar.

Spec: FEATURES_SPEC/2026-09_Quick_Audio_Menu.md — AC1, AC6, AC7, AC11, AC13,
AC14, AC15, AC17, AC18, AC21, AC22
2026-09-16 15:37:05 +02:00
fkwp 9a68de193f Show a live microphone level in the audio menu
- Draw the input level beneath the microphone list as 32 discrete segments,
  so the level reads by count and not by hue alone.
- Expose it through role="meter" with aria-valuenow and aria-valuetext, so a
  screen reader announces it.
- Give denied permission and no input device their own message and next
  action, rather than a flat meter that reads as silence.
- Read the level only while the menu is open, so nothing holds a second
  capture of the microphone for the length of a call.
- Read the device the picker has selected, so the meter re-points as soon as
  the selection changes.
- Ignore anything below a noise floor: a quiet room is never digitally
  silent, and its hiss otherwise lights the first segments permanently, which
  reads as "it can hear me" when nobody is speaking.
- Quantise loudness with a square root above that floor: amplitude puts
  ordinary speech near the bottom of the range, where a linear meter barely
  moves.
- Withhold the output select callback where the platform cannot route audio
  to a chosen device, which is what renders the speaker section disabled.
  Safari and most Firefox builds land here.
- Track the in-flight device by kind as well as id: Chrome names both an
  input and an output "default", so the spinner appeared on the wrong row.
- Disable every device in the menu while a selection is settling, so a second
  request cannot overtake the first.

The meter opens its own short-lived capture rather than tapping the call's
audio track. That is what lets it follow the picker instantly, and it avoids
the pre-join track, which is deliberately frozen to the device selected when
the screen mounted. The cost is a second capture of the same device while the
menu is open. See the spec's notes sidecar.

Spec: FEATURES_SPEC/2026-09_Quick_Audio_Menu.md — AC1, AC6, AC7, AC13, AC14,
AC15, AC17, AC18, AC21, AC22
2026-09-16 14:52:59 +02:00
fkwp 664fac104f Use radio controls to mark the selected device in the quick menu
- Mark the selection with a radio control instead of a trailing check icon
  beside a device glyph, matching the design and the settings modal, so both
  device pickers read the same way.
- Take the visual from Compound's RadioInput rather than restyling a span of
  our own. RadioControl is wrapped in a Radix form control and needs a Form
  ancestor, which a dropdown menu has no business providing; RadioInput does
  not.
- Keep rows as menuitemradio with aria-checked, rendered through
  MenuItem as="div" so an input is never nested inside a button.
- Render the radio aria-hidden and not focusable: Radix owns focus inside the
  menu, and aria-checked carries the state.
- Leave the activating spinner unchanged.
- Keep every scope.behavior call inside the function that owns the scope,
  which no-observablescope-leak requires.
- Cover the speaker section, an output that cannot be chosen, and a lone
  device shown disabled, with unit tests and stories for each.

Spec: FEATURES_SPEC/2026-09_Quick_Audio_Menu.md — AC1, AC7
2026-09-16 14:02:01 +02:00
fkwp ef2cbbbaea Translate the device menu titles and drop the duplicated heading
Spec: FEATURES_SPEC/2026-09_Quick_Audio_Menu.md — AC1
2026-09-16 13:14:23 +02:00
fkwp 27789131ee Add speaker selection to the microphone quick menu
The microphone chevron listed input devices only, so changing the output
device meant leaving the call controls for the settings modal.

Render the audio menu as two sections, Speaker above Microphone, and wire
the output device list, selection and select callback through the footer
view model. Entries are shown but disabled where the platform cannot switch
that kind of device, or where only one exists, so the menu keeps the same
shape everywhere.

Spec: FEATURES_SPEC/2026-09_Quick_Audio_Menu.md — AC1, AC6, AC7, AC8
2026-09-16 12:58:32 +02:00
40 changed files with 6678 additions and 212 deletions
+21
View File
@@ -3,6 +3,10 @@
"user_menu": "User menu"
},
"action": {
"add_background_image": "Add image",
"background_effect_blur": "Blur",
"background_effect_none": "None",
"background_effect_numbered": "Background {{n}}",
"blur_background": "Blur background",
"close": "Close",
"copy_link": "Copy link",
@@ -23,6 +27,9 @@
"upload_file": "Upload file"
},
"analytics_notice": "By participating in this beta, you consent to the collection of anonymous data, which we use to improve the product. You can find more information about which data we track in our <2>Privacy Policy</2> and our <6>Cookie Policy</6>.",
"background_effects": {
"slow_in_this_browser": "This browser runs background effects slowly, so other people may see your video stutter."
},
"call_ended_view": {
"create_account_button": "Create account",
"create_account_prompt": "<0>Why not finish by setting up a password to keep your account?</0><1>You'll be able to keep your name and set an avatar for use on future calls</1>",
@@ -92,6 +99,10 @@
},
"disconnected_banner": "Connectivity to the server has been lost.",
"error": {
"background_animated": "Animated images cannot be used as a background",
"background_not_an_image": "That file is not an image",
"background_not_kept": "That background could not be saved",
"background_undecodable": "That image could not be opened",
"call_is_not_supported": "Call is not supported",
"call_not_found": "Call not found",
"call_not_found_description": "<0>That link doesn't appear to belong to any existing call. Check that you have the right link, or <2>create a new one</2>.</0>",
@@ -163,6 +174,12 @@
"login_auth_links_prompt": "Not registered yet?",
"login_subheading": "To continue to Element",
"login_title": "Login",
"microphone_level": {
"label": "Microphone level",
"no_device": "No microphone found. Connect one, then try again.",
"permission_denied": "Element Call cannot use your microphone. Allow access in your browser settings.",
"value": "{{level}} of {{max}}"
},
"microphone_off": "Microphone off",
"microphone_on": "Microphone on",
"mute_microphone_button_label": "Mute microphone",
@@ -206,6 +223,7 @@
"auto_gain_control_label": "Automatic gain control",
"background_blur_header": "Background",
"background_blur_label": "Blur the background of the video",
"background_effects_header": "Background effects",
"bitrate_label": "Bitrate",
"blur_not_supported_by_browser": "(Background blur is not supported by this device.)",
"camera_header": "Camera quality",
@@ -215,11 +233,14 @@
"activating": "Activating…",
"camera": "Camera",
"camera_numbered": "Camera {{n}}",
"camera_source": "Camera Source",
"change_device_button": "Change audio device",
"default": "Default",
"default_named": "Default <2>({{name}})</2>",
"default_named_plain": "Default ({{name}})",
"handset": "Handset",
"loudspeaker": "Loudspeaker",
"mic_source": "Mic Source",
"microphone": "Microphone",
"microphone_numbered": "Microphone {{n}}",
"speaker": "Speaker",
+253
View File
@@ -0,0 +1,253 @@
/*
Copyright 2026 Element Creations Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { expect, test, type Locator, type Page } from "@playwright/test";
import { SpaHelpers } from "./spa-helpers.ts";
import { installFakeDevices } from "./utils/fake-devices.ts";
test.describe("the quick audio menu", () => {
test("lists speakers and microphones with a live level meter", async ({
page,
}) => {
await installFakeDevices(page, { microphones: 3, speakers: 3 });
await joinACall(page, "Menu user", "Audio menu");
await openAudioMenu(page);
// The speaker list is what the settings modal used to be the only home of.
await expect(page.getByRole("group", { name: "Speaker" })).toBeVisible();
await expect(page.getByRole("group", { name: "Microphone" })).toBeVisible();
// Named rather than counted: the browser contributes its own fake output
// and a "Default" entry, so a total would be a fact about the browser.
for (const n of [1, 2, 3])
await expect(
page
.getByRole("group", { name: "Speaker" })
.getByRole("menuitemradio", { name: `Fake Speaker ${n}` }),
).toBeVisible();
// Only one entry of a kind is marked, and the meter reports a number
// rather than a colour.
await expect(
page.getByRole("menuitemradio", { checked: true }),
).toHaveCount(2);
const meter = page.getByRole("meter", { name: "Microphone level" });
await expect(meter).toBeVisible();
await expect(meter).toHaveAttribute("aria-valuenow", /\d+/);
await expect(meter).toHaveAttribute("aria-valuetext", /\d+ of \d+/);
// Both browsers in the matrix can route audio to a chosen output, so the
// section offers a real choice. The case where a platform cannot — Safari,
// and anything without setSinkId — is covered by a unit check, since no
// browser here can reach it.
await expect(
page
.getByRole("group", { name: "Speaker" })
.getByRole("menuitemradio")
.first(),
).toHaveAttribute("aria-disabled", "false");
});
test("moves the microphone and the speaker without disturbing the call", async ({
browser,
}) => {
// Two browsers, two joins and a real call between them.
test.slow();
const hostContext = await browser.newContext({ reducedMotion: "reduce" });
const host = await hostContext.newPage();
await installFakeDevices(host, { microphones: 3, speakers: 3 });
await joinACall(host, "Host", "Device switch");
const inviteLink = await SpaHelpers.getCallInviteLink(host);
const guestContext = await browser.newContext({ reducedMotion: "reduce" });
const guest = await guestContext.newPage();
await SpaHelpers.joinCallFromInviteLink(guest, inviteLink, "Guest");
await SpaHelpers.expectVideoTilesCount(guest, 2);
await openAudioMenu(host);
await selectDevice(host, "Microphone", "Fake Microphone 2");
await openAudioMenu(host);
await selectDevice(host, "Speaker", "Fake Speaker 2");
// The point of the criterion: the switch is not a rejoin. Neither side
// sees the call drop, and the guest still has both tiles — so the host
// never left and came back.
await expect(
host.getByRole("dialog", { name: "Reconnecting…" }),
).not.toBeVisible();
await expect(
guest.getByRole("dialog", { name: "Reconnecting…" }),
).not.toBeVisible();
await SpaHelpers.expectVideoTilesCount(guest, 2);
await expect(guest.getByText("Waiting for media...")).not.toBeVisible();
await hostContext.close();
await guestContext.close();
});
test("keeps the meter moving while muted, and sends nothing", async ({
browser,
}) => {
// Two browsers, two joins and a real call between them.
test.slow();
const hostContext = await browser.newContext({ reducedMotion: "reduce" });
const host = await hostContext.newPage();
await installFakeDevices(host);
await joinACall(host, "Muted host", "Muted meter");
const inviteLink = await SpaHelpers.getCallInviteLink(host);
const guestContext = await browser.newContext({ reducedMotion: "reduce" });
const guest = await guestContext.newPage();
await SpaHelpers.joinCallFromInviteLink(guest, inviteLink, "Listener");
await SpaHelpers.expectVideoTilesCount(guest, 2);
const mute = host.getByTestId("incall_mute");
await mute.click();
await expect(mute).toHaveAttribute("aria-checked", "false");
await openAudioMenu(host);
// The microphone is held open while muted, so the meter still reports the
// hardware. The mute control is what says nothing is being transmitted.
const meter = host.getByRole("meter", { name: "Microphone level" });
await expect(meter).toBeVisible();
// Queried by test id, not by role: the menu is modal, so Radix takes the
// rest of the call out of the accessibility tree while it is open.
await expect(mute).toHaveAttribute("aria-checked", "false");
await expect(mute).toBeVisible();
// And the listener is told so, rather than being left to guess from silence.
await expect(
guest.getByTestId("videoTile").filter({ hasText: "Muted host" }),
).toBeVisible();
await hostContext.close();
await guestContext.close();
});
test("keeps the meter at the foot of the list while it scrolls", async ({
page,
}) => {
await installFakeDevices(page, { microphones: 20, speakers: 4 });
await joinACall(page, "Scroller", "Long device list");
await openAudioMenu(page);
const meter = page.getByRole("meter", { name: "Microphone level" });
await expect(meter).toBeVisible();
// Scrolled so the microphones start at the top of the list and run past its
// bottom: the position that tells a pinned meter from one that merely
// happens to be last.
const list = page.locator("[role='menu'] div[role='none']").first();
await list.evaluate((element) => {
const group = element.querySelector("[role='group'][aria-label*='icro']");
element.scrollTop +=
group!.getBoundingClientRect().top -
element.getBoundingClientRect().top;
});
await expect(meter).toBeInViewport();
await expectPinnedInside(meter, list);
// Every entry stays reachable, which is what the scroll is for.
await expect(
page.getByRole("menuitemradio", { name: "Fake Microphone 20" }),
).toBeVisible();
});
test("shows the focus ring only when the keyboard moved the focus", async ({
page,
browserName,
}) => {
test.skip(
browserName === "firefox",
"Headless Firefox does not deliver synthetic key presses reliably; see reconnect.spec.ts",
);
await installFakeDevices(page);
await joinACall(page, "Keyboard user", "Focus ring");
await openAudioMenu(page);
const first = page.getByRole("menuitemradio").first();
// Opened by pointer, so no ring, even though Radix has moved focus into the
// menu already.
await expect.poll(async () => outlineWidth(first)).toBe(0);
await page.keyboard.press("ArrowDown");
const focused = page.locator("[role='menuitemradio']:focus");
await expect.poll(async () => outlineWidth(focused)).toBeGreaterThan(0);
// The pointer takes it away again: the menu focuses whatever it is over, so
// a ring that followed focus alone would trail the mouse.
await first.hover();
await expect.poll(async () => outlineWidth(focused)).toBe(0);
});
});
/** Creates a call and joins it, leaving the page in the call. */
async function joinACall(
page: Page,
userName: string,
callName: string,
): Promise<void> {
await page.goto("/");
await SpaHelpers.createCall(page, userName, callName, true);
await expect(page.getByTestId("name_tag")).toContainText(userName);
// The media controls stay disabled until the devices have enumerated, and
// every test here drives them.
await expect(page.getByTestId("incall_mute")).toBeEnabled({
timeout: 10_000,
});
}
async function openAudioMenu(page: Page): Promise<void> {
await page.getByRole("button", { name: "Microphone" }).click();
await expect(page.getByRole("menu")).toBeVisible();
}
async function selectDevice(
page: Page,
section: "Speaker" | "Microphone",
name: string,
): Promise<void> {
const item = page
.getByRole("group", { name: section })
.getByRole("menuitemradio", { name });
await item.click();
// Selecting does not close the menu — the component prevents the default so
// the list survives a mis-click — so it is dismissed explicitly.
await page.keyboard.press("Escape");
await expect(page.getByRole("menu")).not.toBeVisible();
}
/**
* Asserts the meter sits within the scrollport, and inside the menu's frame.
*
* The meter is the one opaque element in the menu, so it is the one thing that
* can paint over the border. Whether it actually does needs a screenshot; this
* pins the geometry that decides it.
*/
async function expectPinnedInside(
meter: Locator,
list: Locator,
): Promise<void> {
const meterBox = (await meter.boundingBox())!;
const listBox = (await list.boundingBox())!;
const frame = (await meter.page().getByRole("menu").boundingBox())!;
expect(meterBox.y + meterBox.height).toBeLessThanOrEqual(
listBox.y + listBox.height + 1,
);
expect(meterBox.y).toBeGreaterThanOrEqual(listBox.y - 1);
expect(meterBox.x).toBeGreaterThan(frame.x);
expect(meterBox.x + meterBox.width).toBeLessThan(frame.x + frame.width);
}
/** The painted outline width in pixels, however the stylesheet spells it. */
async function outlineWidth(item: Locator): Promise<number> {
if ((await item.count()) === 0) return 0;
return item.first().evaluate((element) => {
const { outlineStyle, outlineWidth } = getComputedStyle(element);
return outlineStyle === "none" ? 0 : Number.parseFloat(outlineWidth) || 0;
});
}
+165
View File
@@ -0,0 +1,165 @@
/*
Copyright 2026 Element Creations Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { expect, type Locator, type Page, test } from "@playwright/test";
import { createUserAndRoom, resizeContainer, startHarness } from "./harness.ts";
import { installFakeDevices } from "../utils/fake-devices.ts";
/**
* The device menu where Element Call is a component in a host's page rather
* than the whole of one.
*
* This is the case the stylesheets cannot describe: the menu is portalled to
* the document, so a container query and a viewport unit both measure the wrong
* thing — the first has no container to resolve against out there, the second
* measures a page Element Call does not own. The menu has to be sized against
* the space the call is actually drawn in.
*
* Driven from the lobby rather than a joined call. The footer builds the same
* menu from the same device behaviours in both, and the container is the same
* size either way, so joining would only add two connections' worth of flake.
*/
// Signing in, setting up crypto and syncing happen twice before anything is on
// screen, as in component-call.spec.ts
test.describe.configure({ timeout: 180_000 });
test("sizes the device list against the call, not the window", async ({
page,
}) => {
await installFakeDevices(page, { microphones: 20, speakers: 4 });
const { username, roomId } = await createUserAndRoom("menusize");
const panes = await startHarness(page, username, roomId);
const pane = panes.first();
// A short call in a much taller page: the difference between measuring the
// call and measuring the window.
const container = pane.getByTestId("call-container");
await resizeContainer(container, { width: 900, height: 400 });
await expect(pane.getByTestId("lobby_joinCall")).toBeVisible({
timeout: 60_000,
});
const list = await openDeviceList(page, pane);
const callHeight = (await container.boundingBox())!.height;
const windowHeight = page.viewportSize()!.height;
const listHeight = (await list.boundingBox())!.height;
// Sized against the call. Were it sized against the window the list would be
// half as tall again, and the assertion below would not be able to tell.
expect(callHeight).toBeLessThan(windowHeight * 0.75);
expect(listHeight).toBeLessThanOrEqual(callHeight);
expect(listHeight).toBeLessThan(windowHeight * 0.6);
});
test("follows the call area when the host resizes it", async ({ page }) => {
await installFakeDevices(page, { microphones: 20, speakers: 4 });
const { username, roomId } = await createUserAndRoom("menuresize");
const panes = await startHarness(page, username, roomId);
const pane = panes.first();
const container = pane.getByTestId("call-container");
await resizeContainer(container, { width: 900, height: 360 });
await expect(pane.getByTestId("lobby_joinCall")).toBeVisible({
timeout: 60_000,
});
const list = await openDeviceList(page, pane);
const whenShort = (await list.boundingBox())!.height;
// The host grows the space Element Call is drawn in while the menu is open —
// a panel opening, a window dragged, a phone turned. A bound taken once on
// opening would still describe the smaller call.
await resizeContainer(container, { width: 900, height: 700 });
await expect
.poll(async () => (await list.boundingBox())!.height)
.toBeGreaterThan(whenShort);
});
test("keeps every device reachable in a small container", async ({ page }) => {
await installFakeDevices(page, { microphones: 20, speakers: 4 });
const { username, roomId } = await createUserAndRoom("menureach");
const panes = await startHarness(page, username, roomId);
const pane = panes.first();
const container = pane.getByTestId("call-container");
// Narrow as well as short, which is where entries get pushed out of reach.
await resizeContainer(container, { width: 400, height: 360 });
await expect(pane.getByTestId("lobby_joinCall")).toBeVisible({
timeout: 60_000,
});
const list = await openDeviceList(page, pane);
// More devices than the space allows, so the list has to scroll rather than
// put entries somewhere they cannot be got at.
expect(await list.evaluate((el) => el.scrollHeight > el.clientHeight)).toBe(
true,
);
const last = page.getByRole("menuitemradio", { name: "Fake Microphone 20" });
await last.scrollIntoViewIfNeeded();
await expect(last).toBeInViewport();
// Reachable means usable, not merely painted: D11 accepts that the menu may
// be drawn outside the call area, so this asserts reach rather than
// containment.
await last.click();
await expect(last).toHaveAttribute("aria-checked", "true");
});
test("tracks the focus modality of its own call, not the page", async ({
page,
}) => {
await installFakeDevices(page);
const { username, roomId } = await createUserAndRoom("menufocus");
const panes = await startHarness(page, username, roomId);
const pane = panes.first();
const other = panes.nth(1);
await expect(pane.getByTestId("lobby_joinCall")).toBeVisible({
timeout: 60_000,
});
await openDeviceList(page, pane);
// The menu owns the modality, because every item it can focus has to answer
// to it — the device rows and the camera menu's blur toggle alike.
const menu = page.getByRole("menu");
// Asserted on the attribute rather than the painted ring, which cannot be
// read here: the menu is portalled outside the call root, and the component
// build scopes the stylesheet to it, so neither the ring nor the rule that
// suppresses the browser's own reaches this menu. The paint is asserted
// standalone instead — in the story and in audio-menu.spec.ts. What is on
// trial here is which call the tracking answers for.
await expect(menu).toHaveAttribute("data-focus-modality", "pointer");
// A key pressed in the other call on this page — or anywhere in the host's
// own page — says nothing about how this menu is being used.
await other.evaluate((element) =>
element.dispatchEvent(
new KeyboardEvent("keydown", { key: "ArrowDown", bubbles: true }),
),
);
await expect(menu).toHaveAttribute("data-focus-modality", "pointer");
// A key pressed in this menu does.
await page.keyboard.press("ArrowDown");
await expect(menu).toHaveAttribute("data-focus-modality", "keyboard");
});
/**
* Opens the microphone menu of one component and returns its scrolling device
* list, which lives outside the component: the menu is portalled to the page.
*/
async function openDeviceList(page: Page, pane: Locator): Promise<Locator> {
await pane
.getByRole("button", { name: "Microphone" })
.click({ timeout: 60_000 });
await expect(page.getByRole("menu")).toBeVisible();
const list = page.locator("[role='menu'] div[role='none']").first();
await expect(list).toBeVisible();
return list;
}
+98
View File
@@ -0,0 +1,98 @@
/*
Copyright 2026 Element Creations Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { type Page } from "@playwright/test";
/**
* Gives the browser more fake devices than it ships with.
*
* A headless browser's fake capture offers one microphone and one speaker,
* which is one short of what a device menu is for: with no choice to make, every
* entry renders disabled. These are synthetic entries on top of the real fake
* device, so the menu has a list to show and a selection to move, and the app
* runs its real device pipeline against them.
*
* What they do not do is route audio: every entry is backed by the same capture,
* and `setSinkId` is accepted rather than honoured. A test can prove that
* choosing a device changes the app's state and does not disturb the call. That
* a listener hears the change needs hardware, and stays a manual check.
*
* Must be called before the page navigates.
*/
export async function installFakeDevices(
page: Page,
{ microphones = 2, speakers = 2 } = {},
): Promise<void> {
await page.addInitScript(
({ microphones, speakers }) => {
const synthetic = (
kind: MediaDeviceKind,
count: number,
name: string,
): MediaDeviceInfo[] =>
Array.from({ length: count }, (_, i) => {
const info = {
deviceId: `${kind}-${i + 1}`,
groupId: `${kind}-group-${i + 1}`,
kind,
label: `${name} ${i + 1}`,
};
return { ...info, toJSON: () => info } as MediaDeviceInfo;
});
const ids = new Set(
[
...synthetic("audioinput", microphones, ""),
...synthetic("audiooutput", speakers, ""),
].map((d) => d.deviceId),
);
const devices = navigator.mediaDevices;
const enumerate = devices.enumerateDevices.bind(devices);
devices.enumerateDevices = async (): Promise<MediaDeviceInfo[]> => [
...(await enumerate()),
...synthetic("audioinput", microphones, "Fake Microphone"),
...synthetic("audiooutput", speakers, "Fake Speaker"),
];
// Our ids name no hardware, so an exact-device constraint on one would be
// rejected. Drop it and let the one real fake device answer.
const getUserMedia = devices.getUserMedia.bind(devices);
devices.getUserMedia = async (
constraints?: MediaStreamConstraints,
): Promise<MediaStream> => {
const audio = constraints?.audio;
if (typeof audio === "object") {
const requested = audio.deviceId;
const id =
typeof requested === "object" && requested !== null
? ((requested as ConstrainDOMStringParameters).exact as string)
: (requested as string | undefined);
if (id !== undefined && ids.has(id))
return getUserMedia({ ...constraints, audio: true });
}
return getUserMedia(constraints);
};
// Routing to a device that does not exist would reject, and the app
// treats that as a failed switch. Both sinks are patched: Element Call
// routes its own AudioContext as well as the media elements, and leaving
// that one alone logs a NotFoundError for every switch.
for (const proto of [
HTMLMediaElement.prototype,
AudioContext.prototype,
]) {
const sink = proto as { setSinkId?: (id: string) => Promise<void> };
const setSinkId = sink.setSinkId;
if (setSinkId === undefined) continue;
sink.setSinkId = async function (id: string): Promise<void> {
if (!ids.has(id)) await setSinkId.call(this, id);
};
}
},
{ microphones, speakers },
);
}
+11
View File
@@ -22,6 +22,8 @@ import {
} from "react-router-dom";
import * as Sentry from "@sentry/react";
import { TooltipProvider } from "@vector-im/compound-web";
import { BackgroundEffectsBench } from "./livekit/BackgroundEffectsBench";
import { logger } from "matrix-js-sdk/lib/logger";
import { type MatrixClient } from "matrix-js-sdk";
import { I18nextProvider } from "react-i18next";
@@ -156,6 +158,15 @@ export const App: FC<Props> = ({ vm, widget }) => {
<SentryRoute path="/" element={<HomePage />} />
<SentryRoute path="/login" element={<LoginPage />} />
<SentryRoute path="/register" element={<RegisterPage />} />
{/* Exploration only: measures whether this device can run
background effects, on a device the app refuses to run them
on. One path segment, not two: the config is fetched
relative to the page, so a nested path looks for it in a
directory that does not exist and the app never starts. */}
<SentryRoute
path="/background-effects-bench"
element={<BackgroundEffectsBench />}
/>
<SentryRoute path="*" element={<RoomPage />} />
</Routes>
</Sentry.ErrorBoundary>
+20
View File
@@ -12,6 +12,7 @@ import { Link } from "@vector-im/compound-web";
import type { Meta, StoryObj } from "@storybook/react-vite";
import { CallFooter, type FooterSnapshot } from "./CallFooter";
import { ProcessorProvider } from "../livekit/TrackProcessorContext";
import inCallViewStyles from "../room/InCallView.module.css";
import { useStaticViewModel } from "../state/ViewModel";
import { ReactionsSenderContext } from "../reactions/useReactionsSender";
@@ -82,6 +83,15 @@ const fnArgType = {
const meta = {
component: CallFooterStoryWrapper,
// The footer reads the backgrounds this device keeps from the processor
// provider, the same way the lobby and the call do.
decorators: [
(Story): JSX.Element => (
<ProcessorProvider>
<Story />
</ProcessorProvider>
),
],
argTypes: {
layout: {
control: "radio",
@@ -124,6 +134,8 @@ export const Default: Story = {
toggleScreenSharing: fn(),
toggleBlur: fn(),
videoBlurEnabled: true,
backgroundEffect: "none",
selectBackgroundEffect: fn(),
hangup: fn(),
buttonSize: "lg",
showFooter: true,
@@ -137,10 +149,13 @@ export const Default: Story = {
debugTileLayout: false,
tileStoreGeneration: undefined,
audioOptions: [],
audioOutputOptions: [],
videoOptions: [],
selectedAudio: undefined,
selectedAudioOutput: undefined,
selectedVideo: undefined,
selectAudioButtonOption: undefined,
selectAudioOutputOption: undefined,
selectVideoButtonOption: undefined,
},
parameters: {
@@ -158,11 +173,16 @@ export const WithAudioAndVideoOptions: Story = {
{ label: { type: "name", name: "Microphone 1" }, id: "1" },
{ label: { type: "name", name: "Microphone 2" }, id: "2" },
],
audioOutputOptions: [
{ label: { type: "default", name: "Built-in Output" }, id: "default" },
{ label: { type: "name", name: "Headset" }, id: "2" },
],
videoOptions: [
{ label: { type: "name", name: "Camera 1" }, id: "1" },
{ label: { type: "name", name: "Camera 2" }, id: "2" },
],
selectedAudio: "2",
selectedAudioOutput: "default",
selectedVideo: "1",
},
};
+152 -7
View File
@@ -5,8 +5,17 @@ SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { type FC, type JSX, type Ref, useMemo } from "react";
import {
type FC,
type JSX,
type Ref,
useCallback,
useMemo,
useState,
} from "react";
import classNames from "classnames";
import { useTranslation } from "react-i18next";
import { logger } from "matrix-js-sdk/lib/logger";
import LogoMark from "../icons/LogoMark.svg?react";
import LogoType from "../icons/LogoType.svg?react";
@@ -23,9 +32,25 @@ import {
} from "../button";
import styles from "./CallFooter.module.css";
import {
type BackgroundEffectOption,
MediaMuteAndSwitchButton,
type MenuOptions,
} from "./MediaMuteAndSwitchButton";
import {
parseEffect,
serializeEffect,
shippedBackgrounds,
} from "../livekit/backgroundEffects";
import {
maxAddedBackgrounds,
UnusableImage,
} from "../livekit/backgroundImages";
import {
useAddedBackgrounds,
useBackgroundProcessing,
} from "../livekit/TrackProcessorContext";
import { usesFallbackProcessing } from "../livekit/backgroundProcessing";
import { SelfPreview } from "./SelfPreview";
import { type Behavior } from "../state/Behavior";
import { type ViewModel } from "../state/ViewModel";
import { useBehavior } from "../useBehavior";
@@ -57,6 +82,8 @@ export interface FooterActions {
/** Also controls if the videoMute button is disabled */
toggleVideo: (() => void) | undefined;
toggleBlur: (() => void) | undefined;
/** Undefined where background processing is unavailable. */
selectBackgroundEffect: ((id: string) => void) | undefined;
toggleScreenSharing: (() => void) | undefined;
/** Also controls if the settings button is visible */
openSettings: (() => void) | undefined;
@@ -70,6 +97,8 @@ export interface FooterState {
videoEnabled: boolean;
videoBusy: boolean;
videoBlurEnabled: boolean;
/** The chosen background effect, in its stored form. */
backgroundEffect: string;
showFooter: boolean;
/* This is needed for WindowMode = "flat" */
@@ -99,11 +128,15 @@ export interface FooterState {
/** Providing no options `[]` or `undefined` will imply that we dont have a audio fast switcher */
audioOptions: MenuOptions[];
/** Output devices shown as their own section in the audio menu. */
audioOutputOptions: MenuOptions[];
/** Providing no options `[]` or `undefined` will imply that we dont have a audio fast switcher */
videoOptions: MenuOptions[];
selectedAudio: string | undefined;
selectedAudioOutput: string | undefined;
selectedVideo: string | undefined;
selectAudioButtonOption: ((deviceId: string) => void) | undefined;
selectAudioOutputOption: ((deviceId: string) => void) | undefined;
selectVideoButtonOption: ((option: string) => void) | undefined;
}
@@ -119,6 +152,7 @@ export const CallFooter: FC<FooterProps> = ({
children,
vm,
}) => {
const { t } = useTranslation();
const asOverlay = useBehavior(vm.asOverlay$);
const showFooter = useBehavior(vm.showFooter$);
const hideControls = useBehavior(vm.hideControls$);
@@ -143,9 +177,106 @@ export const CallFooter: FC<FooterProps> = ({
const audioOptions = useBehavior(vm.audioOptions$);
const selectedAudio = useBehavior(vm.selectedAudio$);
const selectAudioButtonOption = useBehavior(vm.selectAudioButtonOption$);
const audioOutputOptions = useBehavior(vm.audioOutputOptions$);
const selectedAudioOutput = useBehavior(vm.selectedAudioOutput$);
const selectAudioOutputOption = useBehavior(vm.selectAudioOutputOption$);
const selectVideoButtonOption = useBehavior(vm.selectVideoButtonOption$);
const toggleBlur = useBehavior(vm.toggleBlur$);
const videoBlurEnabled = useBehavior(vm.videoBlurEnabled$);
const backgroundEffect = useBehavior(vm.backgroundEffect$);
const selectBackgroundEffect = useBehavior(vm.selectBackgroundEffect$);
const { added, addBackground, removeBackground } = useAddedBackgrounds();
const { settling, cameraTrack } = useBackgroundProcessing();
// Said, not decided. Where only the slow path exists the effect still works,
// it costs frames — and the user is the only one who knows whether they would
// rather show the room they are sitting in. It stays put while the pipeline
// is being built: that wait is shown on the tile that was pressed, where a
// message that came and went would only flash once the assets are cached.
const backgroundEffectNotice = useMemo(
() =>
usesFallbackProcessing()
? t("background_effects.slow_in_this_browser")
: undefined,
[t],
);
const [backgroundEffectError, setBackgroundEffectError] = useState<
string | undefined
>(undefined);
// Spelled out rather than built from the reason, so the extractor can find
// every string it has to translate.
const whyRefused = useCallback(
(e: unknown): string => {
if (e instanceof UnusableImage)
switch (e.reason) {
case "not-an-image":
return t("error.background_not_an_image");
case "animated":
return t("error.background_animated");
case "undecodable":
return t("error.background_undecodable");
}
return t("error.background_not_kept");
},
[t],
);
const onRemoveBackgroundEffect = useCallback(
(id: string): void => {
const effect = parseEffect(id);
if (effect.kind !== "added") return;
removeBackground(effect.id).catch((e) =>
logger.warn("Could not remove that background", e),
);
},
[removeBackground],
);
const onAddBackgroundImage = useCallback(
(file: File): void => {
setBackgroundEffectError(undefined);
// Added, not put on — before joining and during a call alike. It joins
// the backgrounds on offer and is in force once the user chooses it:
// choosing a file and choosing to wear it are two decisions, and one
// rule for both places is one less thing to know.
addBackground(file).catch((e) => {
setBackgroundEffectError(whyRefused(e));
logger.warn(
e instanceof UnusableImage
? `Cannot use that file as a background: ${e.reason}`
: "Could not keep that background",
e,
);
});
},
[addBackground, whyRefused],
);
// The catalogue is named here rather than in the view model: the names are
// for reading, and a view model has no business holding translated text.
const backgroundEffects = useMemo(
(): BackgroundEffectOption[] => [
{ id: "none", kind: "none", label: t("action.background_effect_none") },
{ id: "blur", kind: "blur", label: t("action.background_effect_blur") },
...shippedBackgrounds.map((background, i) => ({
id: serializeEffect({ kind: "shipped", id: background.id }),
kind: "image" as const,
// Numbered rather than named: the images are stand-ins, and naming
// them here would invent names the design has not given them.
label: t("action.background_effect_numbered", { n: i + 1 }),
imageUrl: background.imagePath,
})),
...added.map((background, i) => ({
id: serializeEffect({ kind: "added", id: background.id }),
kind: "image" as const,
removable: true,
label: t("action.background_effect_numbered", {
n: shippedBackgrounds.length + i + 1,
}),
imageUrl: background.url,
})),
],
[t, added],
);
const buttonSize = useBehavior(vm.buttonSize$);
const showLogo = useBehavior(vm.showLogo$);
@@ -168,7 +299,6 @@ export const CallFooter: FC<FooterProps> = ({
if ((audioOptions?.length ?? 0) > 0) {
buttons.push(
<MediaMuteAndSwitchButton
title={"Mic Source"}
key="audio"
iconsAndLabels="audio"
enabled={audioEnabled ?? false}
@@ -178,6 +308,9 @@ export const CallFooter: FC<FooterProps> = ({
options={audioOptions}
selectedOption={selectedAudio}
onSelect={selectAudioButtonOption}
outputOptions={audioOutputOptions}
selectedOutputOption={selectedAudioOutput}
onSelectOutput={selectAudioOutputOption}
/>,
);
} else {
@@ -197,7 +330,6 @@ export const CallFooter: FC<FooterProps> = ({
if ((videoOptions?.length ?? 0) > 0) {
buttons.push(
<MediaMuteAndSwitchButton
title={"Camera Source"}
key="video"
iconsAndLabels="video"
enabled={videoEnabled ?? false}
@@ -206,8 +338,21 @@ export const CallFooter: FC<FooterProps> = ({
options={videoOptions}
selectedOption={selectedVideo}
onSelect={selectVideoButtonOption}
videoBlurToggleClick={toggleBlur}
videoBlurEnabled={videoBlurEnabled}
backgroundEffects={backgroundEffects}
selectedBackgroundEffect={backgroundEffect}
onSelectBackgroundEffect={selectBackgroundEffect}
// Withheld once the device keeps as many as it will, which is what
// renders the add tile unavailable rather than letting it fail.
onAddBackgroundImage={
selectBackgroundEffect && added.length < maxAddedBackgrounds
? onAddBackgroundImage
: undefined
}
onRemoveBackgroundEffect={onRemoveBackgroundEffect}
backgroundEffectError={backgroundEffectError}
backgroundEffectNotice={backgroundEffectNotice}
backgroundEffectSettling={settling}
selfPreview={<SelfPreview track={cameraTrack ?? null} />}
/>,
);
} else {
+149 -4
View File
@@ -14,7 +14,10 @@ import type { CallViewModel } from "../state/CallViewModel/CallViewModel";
import type { Alignment, Layout } from "../state/layout-types";
import type { SpotlightTileViewModel } from "../state/TileViewModel";
import type { DeviceLabel } from "../state/MediaDevices";
import { createCallFooterViewModel } from "./CallFooterViewModel";
import {
createCallFooterViewModel,
createLobbyFooterViewModel,
} from "./CallFooterViewModel";
import { HeaderStyle } from "../UrlParams";
const platformMock = vi.hoisted(() => vi.fn(() => "desktop"));
@@ -24,10 +27,17 @@ vi.mock("../Platform", () => ({
},
}));
// Prevent supportsBackgroundProcessors from throwing in jsdom – it is not
// exercised by these tests (only used in `videoToggles`, not `videoOptions`).
// The SDK's own check needs WebGL and canvas APIs jsdom does not have. The
// tests below drive it directly, because what they are about is the answer the
// app gives on top of it.
const sdkSupportMock = vi.hoisted(() => vi.fn(() => true));
vi.mock("@livekit/track-processors", () => ({
supportsBackgroundProcessors: (): boolean => false,
supportsBackgroundProcessors: (): boolean => sdkSupportMock(),
}));
const outputSelectionMock = vi.hoisted(() => vi.fn(() => true));
vi.mock("livekit-client", () => ({
supportsAudioOutputSelection: (): boolean => outputSelectionMock(),
}));
/**
@@ -96,6 +106,71 @@ const twoMicsAndOneCamMediaDevices = mockMediaDevices({
});
describe("createCallFooterViewModel", () => {
describe("selectAudioOutputOption", () => {
function buildFooterVm(): ReturnType<typeof createCallFooterViewModel> {
platformMock.mockReturnValue("desktop");
return createCallFooterViewModel(
testScope(),
buildMinimalCallViewModel(gridLayout),
mockMuteStates(),
twoMicsAndOneCamMediaDevices,
/* reactionIdentifier */ undefined,
{ showControls: true, header: HeaderStyle.Standard },
);
}
it("is withheld where the platform cannot route audio to a chosen device", () => {
outputSelectionMock.mockReturnValue(false);
// Undefined is what renders the speaker section disabled.
expect(buildFooterVm().selectAudioOutputOption$.value).toBeUndefined();
});
it("is offered where the platform can route audio to a chosen device", () => {
outputSelectionMock.mockReturnValue(true);
expect(buildFooterVm().selectAudioOutputOption$.value).toBeDefined();
});
});
describe("audioOutputOptions", () => {
it("is an empty list, not absent, where the platform enumerates no outputs", () => {
platformMock.mockReturnValue("desktop");
outputSelectionMock.mockReturnValue(true);
const vm = createCallFooterViewModel(
testScope(),
buildMinimalCallViewModel(gridLayout),
mockMuteStates(),
mockMediaDevices({
audioInput: {
available$: constant(
new Map<string, DeviceLabel>([
["mic1", { type: "name", name: "Microphone 1" }],
]),
),
selected$: constant(undefined),
select: vi.fn(),
},
// Safari enumerates no output devices whatsoever. Reproduced by the
// condition rather than by the browser, so it is checked on the
// Linux CI runners that have no Safari to check it with.
audioOutput: {
available$: constant(new Map<string, DeviceLabel>()),
selected$: constant(undefined),
select: vi.fn(),
},
}),
/* reactionIdentifier */ undefined,
{ showControls: true, header: HeaderStyle.Standard },
);
// Empty rather than undefined: undefined means this menu has no notion
// of outputs at all, as the camera menu has none, and hides the section.
// Empty means there are none to list, and the menu still shows the
// section with a default in it, disabled.
expect(vm.audioOutputOptions$.value).toEqual([]);
});
});
describe("audioOptions and videoOptions", () => {
function checkEmptyFor(platform: string, layout: Layout): void {
platformMock.mockReturnValue(platform);
@@ -158,4 +233,74 @@ describe("createCallFooterViewModel", () => {
]);
});
});
describe("background effects", () => {
// The lobby, deliberately: in a call the whole switcher is already
// withheld on a phone, so a check there passes whatever the verdict says.
// The lobby keeps its switcher, which is where a phone browser was offered
// every effect, given none of them, and told nothing.
function lobbyFor(
platform: string,
): ReturnType<typeof createLobbyFooterViewModel> {
platformMock.mockReturnValue(platform);
return createLobbyFooterViewModel(
testScope(),
mockMuteStates(),
twoMicsAndOneCamMediaDevices,
/* openSettings */ undefined,
/* hangup */ undefined,
/* showLogo */ false,
);
}
// A phone is offered them, and that is the point of asking the browser
// rather than the platform: measured, a phone on the fast path held 99% of
// its frame rate while a desktop on the slow one held 73%, so refusing by
// platform refused the better device and allowed the worse.
it("offers them on a phone whose browser can run them", () => {
sdkSupportMock.mockReturnValue(true);
const vm = lobbyFor("ios");
expect(vm.selectBackgroundEffect$.value).toBeDefined();
expect(vm.toggleBlur$.value).toBeDefined();
});
it("offers them where the pipeline will honour them", () => {
sdkSupportMock.mockReturnValue(true);
const vm = lobbyFor("desktop");
expect(vm.selectBackgroundEffect$.value).toBeDefined();
expect(vm.toggleBlur$.value).toBeDefined();
});
it("offers nothing where the browser itself cannot run them", () => {
sdkSupportMock.mockReturnValue(false);
const vm = lobbyFor("desktop");
expect(vm.selectBackgroundEffect$.value).toBeUndefined();
expect(vm.toggleBlur$.value).toBeUndefined();
});
it("availability is the same before and during a call", () => {
for (const supported of [true, false]) {
sdkSupportMock.mockReturnValue(supported);
platformMock.mockReturnValue("desktop");
const inCall = createCallFooterViewModel(
testScope(),
buildMinimalCallViewModel(gridLayout),
mockMuteStates(),
twoMicsAndOneCamMediaDevices,
/* reactionIdentifier */ undefined,
{ showControls: true, header: HeaderStyle.Standard },
);
const lobby = lobbyFor("desktop");
const offeredInCall =
inCall.selectBackgroundEffect$.value !== undefined;
const offeredInLobby =
lobby.selectBackgroundEffect$.value !== undefined;
expect(offeredInLobby).toBe(offeredInCall);
expect(offeredInLobby).toBe(supported);
}
});
});
});
+55 -32
View File
@@ -5,14 +5,17 @@ SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { combineLatest, map, switchMap } from "rxjs";
import { supportsBackgroundProcessors } from "@livekit/track-processors";
import { combineLatest, map, type Observable, switchMap } from "rxjs";
import { supportsAudioOutputSelection } from "livekit-client";
import { supportsBackgroundProcessors } from "../livekit/backgroundProcessing";
import { type CallViewModel } from "../state/CallViewModel/CallViewModel";
import { type MenuOptions } from "./MediaMuteAndSwitchButton";
import { type MediaDevices } from "../state/MediaDevices";
import {
backgroundBlur as backgroundBlurSettings,
backgroundEffect as backgroundEffectSetting,
debugTileLayout as debugTileLayoutSetting,
} from "../settings/settings";
import { type Behavior, constant } from "../state/Behavior";
@@ -67,49 +70,52 @@ function buildDeviceBehaviors(
| "audioOptions$"
| "selectedAudio$"
| "selectAudioButtonOption$"
| "audioOutputOptions$"
| "selectedAudioOutput$"
| "selectAudioOutputOption$"
| "videoOptions$"
| "selectedVideo$"
| "selectVideoButtonOption$"
| "toggleBlur$"
| "videoBlurEnabled$"
| "backgroundEffect$"
| "selectBackgroundEffect$"
> {
return {
audioOptions$: scope.behavior(
disableSwitcher$.pipe(
switchMap((disable) =>
disable
? constant([] as MenuOptions[])
: mediaDevices.audioInput.available$.pipe(
map((available) =>
[...available.entries()].map(([id, label]) => ({
id,
label,
})),
),
const options$ = (
available$: Behavior<Map<string, MenuOptions["label"]>>,
): Observable<MenuOptions[]> =>
disableSwitcher$.pipe(
switchMap((disable) =>
disable
? constant([] as MenuOptions[])
: available$.pipe(
map((available) =>
[...available.entries()].map(([id, label]) => ({ id, label })),
),
),
),
),
),
);
return {
audioOptions$: scope.behavior(options$(mediaDevices.audioInput.available$)),
selectedAudio$: scope.behavior(
mediaDevices.audioInput.selected$.pipe(map((s) => s?.id)),
),
selectAudioButtonOption$: constant(mediaDevices.audioInput.select),
videoOptions$: scope.behavior(
disableSwitcher$.pipe(
switchMap((disable) =>
disable
? constant([] as MenuOptions[])
: mediaDevices.videoInput.available$.pipe(
map((available) =>
[...available.entries()].map(([id, label]) => ({
id,
label,
})),
),
),
),
),
audioOutputOptions$: scope.behavior(
options$(mediaDevices.audioOutput.available$),
),
selectedAudioOutput$: scope.behavior(
mediaDevices.audioOutput.selected$.pipe(map((s) => s?.id)),
),
// Safari and most Firefox builds cannot route audio to a chosen device at
// all. Withholding the callback is what renders the section disabled.
selectAudioOutputOption$: constant(
supportsAudioOutputSelection()
? mediaDevices.audioOutput.select
: undefined,
),
videoOptions$: scope.behavior(options$(mediaDevices.videoInput.available$)),
selectedVideo$: scope.behavior(
mediaDevices.videoInput.selected$.pipe(map((s) => s?.id)),
),
@@ -126,6 +132,20 @@ function buildDeviceBehaviors(
),
),
videoBlurEnabled$: backgroundBlurSettings.value$,
backgroundEffect$: backgroundEffectSetting.value$,
// Withholding the callback is what renders the section disabled, the same
// way the speaker section is disabled where no output can be chosen.
selectBackgroundEffect$: scope.behavior(
disableSwitcher$.pipe(
map((switcherDisabled) =>
!switcherDisabled && supportsBackgroundProcessors()
? (id: string): void => {
backgroundEffectSetting.setValue(id);
}
: undefined,
),
),
),
};
}
@@ -263,10 +283,13 @@ export function createLobbyFooterViewModel(
reactionData: undefined,
tileStoreGeneration: undefined,
audioOptions: undefined,
audioOutputOptions: undefined,
videoOptions: undefined,
selectedAudio: undefined,
selectedAudioOutput: undefined,
selectedVideo: undefined,
selectAudioButtonOption: undefined,
selectAudioOutputOption: undefined,
selectVideoButtonOption: undefined,
}),
...buildMuteBehaviors(scope, muteStates),
@@ -0,0 +1,88 @@
{/**
Copyright 2026 Element Creations Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
**/}
{/**
A custom doc page, replacing the default autodocs for this component, so that
the background effects states can be reviewed together rather than hunted for
in the sidebar. Same filename as the component, which is how Storybook picks
it up.
**/}
import {
Meta,
Primary,
Controls,
Stories,
Title,
Subtitle,
Canvas,
} from "@storybook/addon-docs/blocks";
import * as MenuStories from "./MediaMuteAndSwitchButton.stories";
<Meta of={MenuStories} />
<Title> Device menu </Title>
The menu behind the chevron on the microphone and camera buttons: the devices
to choose from, and — on the camera — the background effects.
<Subtitle> Background effects: the four states worth reviewing </Subtitle>
**Offered.** The ordinary case. No effect, blur, the images we ship, the ones
the user added, and a tile for adding another.
<Canvas of={MenuStories.BackgroundEffects} />
**Offered, with what it costs.** Browsers without the fast path — Safari and
Firefox, on every platform — draw each frame through a canvas instead.
Measured across four of them that costs between a sixth and a quarter of the
frame rate, on a phone and on a desktop alike.
It is said rather than enforced. The cost is smoothness; the reason to pay it
is privacy; and only the person in the room knows which they would rather
have. Everything stays choosable.
Drawn quietly on purpose, to the design: the subtle ground, an outline mark and
the text at its ordinary weight, held the same sixteen pixels from the menu's
sides as the grid above it. Not an alert — an alert is for something that just
happened, and this was true before the menu opened and will be true after it
closes.
<Canvas of={MenuStories.BackgroundEffectsSlowInThisBrowser} />
**Being set up.** Only on the first effect of a session. The tile that was
pressed spins where its tick will go, until a frame has actually come out of
the pipeline — about three seconds of building, and on Safari another twelve to
fifteen while the page holds still. The spinner turns by rotating, which a
browser composites, so it keeps moving through a pause that has stopped
everything else: the difference between looking busy and looking crashed.
On the tile rather than in a message, because once the assets are cached the
wait is a fraction of a second, and a message that appears and vanishes that
fast is noise. A spinner that brief just reads as a control responding. The
notice about what the browser costs stays put throughout — it is true before,
during and after.
This story plays the sequence rather than posing it, shortened to a second and
a half so it is watchable. Later effects skip it entirely: switching a built
pipeline costs nothing.
<Canvas of={MenuStories.BackgroundEffectsSettling} />
**Not possible at all.** Where the browser cannot run background processing on
any path, the tiles are shown and disabled — except no effect, which needs no
processing and stays choosable, so there is always a way back to plain video.
<Canvas of={MenuStories.BackgroundEffectsUnavailable} />
<Subtitle> Mobile layouts </Subtitle>
This component is reactive. To check the mobile layout, open the story from the
sidebar: the summaries on this page do not render it.
<Primary />
<Controls of={MenuStories.Default} />
<Stories />
@@ -35,3 +35,555 @@ Please see LICENSE in the repository root for full details.
transform: rotate(360deg);
}
}
.menu {
display: flex;
flex-direction: column;
}
/* Only the device lists scroll; the level meter stays put beneath them.
The bound comes from the measured height of the call area, set by the
component: the menu is portalled outside the root, so neither a container
query nor a viewport unit describes the space it is allowed to fill. */
.deviceList {
/* Held one border width clear of the frame on each side. The menu draws its
frame as an outline just inside its own edge, and once this list has
enough in it to scroll, Safari paints it — rows and all — above that
outline, so the frame vanished beside every device; measured on the
microphone menu with many devices, 384 pixels of it. Everything the list
holds is inside this, so nothing in it needs to keep clear on its own. */
margin-inline: var(--cpd-border-width-1);
/* The camera menu's width, set by the component, less that clearance so the
menu still comes to it. Without the variable the calc is invalid and the
width falls back to auto, which is what every other menu wants. */
inline-size: calc(
var(--device-list-inline-size) - 2 * var(--cpd-border-width-1)
);
overflow-y: auto;
min-block-size: 0;
max-block-size: var(--device-list-max-height);
/* The heading stands over the head of this list and the level meter over its
foot. Scrolling a row flush to either edge would put it underneath one of
them, which is how a row reached by the keyboard ends up half-readable;
this keeps both their heights clear for anything the browser scrolls to.
Set by the component, which is the only thing that can measure them. */
scroll-padding-block: var(--device-list-scroll-padding-start, 0)
var(--device-list-scroll-padding-end, 0);
}
/* Each section is headed by its own rule, which Compound draws under a menu
heading already. The heading spans the whole menu — the menu has no padding
of its own — so that rule runs edge to edge, stopping only where the frame
is, rather than inset as a separator between the sections would be.
Compound's spacing around it is the design's too, bar one thing: the design
leaves a section's first device further below the rule than Compound does —
measured off the mock, 27px from the rule to the top of the control, against
the 16px that Compound's 8px margin and the row's own 8px padding give. The
difference goes below the rule, so the rule stays tight under its own text
where Compound put it. */
.menu h3 {
margin-block-end: var(--cpd-space-5x);
}
/* One section is set much further from the one above it than Compound's 8px
heading margin allows: measured off the mock, 41px from the last device's
control to the next heading's text, against the 22px we had. Only between
sections — the first heading keeps Compound's spacing, because the menu's
own padding is already above it. */
.deviceList [role="group"] + [role="group"] h3 {
margin-block-start: var(--cpd-space-7x);
}
/* Each section's heading stays at the top of the scrollport while any of that
section is still in view, so a long list never leaves you wondering which
kind of device you are looking at. It leaves with its own section, because
sticky only holds while the group it belongs to is in view.
Opaque, and held a border width clear of the sides, for the same reason as
the meter below: this is a positioned element, so it paints above the
outline the menu draws its frame with, and would swallow it. */
.sectionHeading {
position: sticky;
inset-block-start: 0;
/* Over what scrolls beneath it. Without a layer of its own it is painted in
order with everything else positioned, and the effect tiles — positioned,
to carry their tick and the remove cross — come after it, so the grid
slid over the heading instead of under it. */
z-index: 1;
background: var(--cpd-color-bg-canvas-default);
/* Clear of the frame's top edge; the list it sits in already keeps it clear
of the sides. */
margin-block-start: var(--cpd-border-width-1);
}
/* The meter belongs to the microphone section: it stays at the bottom of the
scrollport while that section is in view, and leaves with it when the list
is scrolled up to the speakers. The wrapper is deliberately unpositioned —
a positioned one paints above the menu's outline and swallows the frame
along this whole section. Sticky is resolved against the scroll container,
so it does not need one. */
.stickyMeter {
position: sticky;
inset-block-end: 0;
/* Opaque, so the list does not show through it as it scrolls past, and held
clear of the frame's bottom edge. The list it sits in already keeps it
clear of the sides. */
background: var(--cpd-color-bg-canvas-default);
margin-block-end: var(--cpd-border-width-1);
}
/* Radix focuses whatever the pointer is over, so the browser's own ring marks
the item under the mouse. It cannot distinguish the two modalities here, so
it is suppressed and replaced by one that can.
Every kind of item the menu can focus, not only the device rows: the camera
menu's blur toggle is a checkbox item and a child of the menu rather than of
the list, and left out it kept the browser's own ring and followed the
pointer with it. */
.menu [role^="menuitem"]:focus,
.menu [role^="menuitem"]:focus-visible {
outline: none;
}
/* Shown only when the keyboard is what moved the focus. */
.menu[data-focus-modality="keyboard"] [role^="menuitem"]:focus {
outline: var(--cpd-border-width-2) solid var(--cpd-color-border-focused);
outline-offset: calc(-1 * var(--cpd-border-width-2));
}
/*
* The Background effects grid.
*
* Three across matches the design and keeps a thumbnail readable at a glance;
* with no effect, blur and two shipped images, the five a user may add fill it
* to a whole three-by-three at the limit, where the add tile goes away, and
* four and the add tile fill it one short of it.
*
* minmax(0, 1fr) rather than 1fr: a grid track will not shrink below its
* content's minimum width by default, and a tile carrying a label is wider
* than a third of the menu, so plain 1fr lays out three full-width columns and
* clips two of them.
*/
.effectGrid {
display: grid;
/* Thirds of a menu whose width is set, so the tiles are the size design
draws them — about eighty pixels — whatever the device names are. Zero
rather than the content as the floor, so a long label wraps inside its
tile instead of pushing the grid wider than the menu. */
grid-template-columns: repeat(3, minmax(0, 1fr));
/* Every tile the same height, set by the tallest — the ones carrying a
label. Forcing a ratio instead clips the label as soon as the menu is
narrow enough that the ratio cannot hold it. */
grid-auto-rows: 1fr;
/* Measured against the design, where the gap is about a sixth of a tile's
width. A step less than this read as tight, and left the remove cross on
the corner with little room of its own. */
gap: var(--cpd-space-3x);
/* The same inset a menu heading and a device row carry, so the tiles line
their left edge up with the "B" of the heading above them and with the
radio buttons above that. Matching the surrounding boxes instead puts the
tiles to the left of both, since a heading insets its own text. */
/* The sides carry the full inset to the menu's frame. Above, the grid is
the first control of its section, so it starts where a device row's
control starts under its own rule: the heading's spacing supplies most of
that, and this adds the same step a device row holds above its control.
Below, the menu's own padding is already the inset, so nothing is set. */
/* Sixteen from the menu's edge, as the notice below it is and as design
draws both: the list this sits in is already held a border width in from
the frame, so that much less here. */
padding: var(--cpd-space-2x)
calc(var(--cpd-space-4x) - var(--cpd-border-width-1)) 0;
}
/*
* A menu item lays itself out as a grid of chevron, icon and label in a row.
* A tile centres its glyph with the label beneath instead, so this has to
* outweigh that rule rather than sit beside it.
*
* The tiles are a little wider than tall, as the design draws them, and hold
* that shape whatever the menu measures.
*/
.effectGrid .effectTile {
position: relative;
display: flex;
flex-direction: column;
align-items: center;
/* Glyph and label centre together as one group. Pinning the label to the
floor instead leaves the glyph riding high in the space above it. */
justify-content: center;
/* Measured against the design: the glyph sits a quarter of the tile wide,
with about a seventh of the tile's height between its ink and the label's.
The label's own leading is part of that distance, so the step here is
smaller than the ink gap it produces. */
gap: var(--cpd-space-1x);
block-size: auto;
min-block-size: 0;
min-inline-size: 0;
/* More room above and below than at the sides: the tile is wider than tall,
and the padding is what carries that proportion now the gap is set by the
design's spacing rather than by filling the box. */
padding: var(--cpd-space-3x) var(--cpd-space-2x);
overflow: hidden;
border: 2px solid var(--cpd-color-border-interactive-secondary);
border-radius: var(--cpd-space-2x);
}
/* The selection is drawn here and announced by the item's aria-checked, so it
never rests on colour alone. Matched to the specificity of the base tile
rule above, which also sets a border-color and would otherwise win.
Colour alone, at the width every tile already carries: thickening it with an
inset shadow drew it inside the padding box, where an image tile's picture
sits, so the picture covered it and an added background wore a thinner frame
than no effect did. */
.effectGrid .effectTileSelected {
border-color: var(--cpd-color-border-accent);
}
.effectGrid .effectTile > * {
/* A menu item spaces its icon and label for a row, with the gap carried as a
right margin on each. Stacked in a tile that margin is dead space on one
side, which shifts the glyph and the label off the tile's centre. */
margin: 0;
min-inline-size: 0;
max-inline-size: 100%;
text-align: center;
overflow-wrap: anywhere;
}
.effectSwatch {
flex: 0 0 auto;
inline-size: 100%;
display: flex;
align-items: center;
justify-content: center;
color: var(--cpd-color-icon-secondary);
}
/* An image tile is the picture: it fills the tile rather than sitting in a box
inside one, which is what the design draws and what makes a background
recognisable at this size. */
.effectThumb {
position: absolute;
inset: 0;
inline-size: 100%;
block-size: 100%;
object-fit: cover;
}
/* Blur has no icon in the design system, and a hatch reads as "softened" in a
way a glyph does not. */
.effectBlurGlyph {
/* Smaller than the glyph beside it, and than the design's own hatch measures.
The design's is an icon whose strokes shorten towards the corners, so it
leaves them empty and reads lighter; a uniform hatch fills its box and
reads heavier at the same size. */
inline-size: 18px;
block-size: 18px;
border-radius: var(--cpd-space-1x);
background-image: repeating-linear-gradient(
135deg,
var(--cpd-color-icon-secondary) 0 1.2px,
transparent 1.2px 3.15px
);
}
/* The tick marking the tile in force, standing where the glyph would.
Positioned, so it paints above the image, which is positioned itself and
would otherwise cover anything left in normal flow. */
.effectCheck {
position: relative;
color: var(--cpd-color-icon-accent-primary);
}
/* On a picture it needs its own ground: a tick alone disappears into whatever
the image happens to be behind it. */
.effectCheckOnImage {
border-radius: 50%;
background: var(--cpd-color-bg-canvas-default);
box-shadow: 0 0 0 var(--cpd-border-width-2) var(--cpd-color-bg-canvas-default);
}
/* A menu item's label is sized for a row; in a tile the design sets it a step
smaller, against the glyph above it. */
.effectLabel {
font: var(--cpd-font-body-sm-medium);
/* A menu item labels itself in the secondary colour, which reads as muted
under a glyph the design draws at full strength. */
color: var(--cpd-color-text-primary);
}
/* Named for assistive technology, not drawn: an image labels itself, and the
design gives the add tile a plus alone. */
.effectLabelUnseen {
position: absolute;
inline-size: 1px;
block-size: 1px;
overflow: hidden;
clip-path: inset(50%);
white-space: nowrap;
}
/*
* Unavailable. Opacity alone is not enough on an image tile: a washed-out
* photograph still reads as a photograph you could pick. Draining the colour
* as well says plainly that there is nothing to choose here.
*/
.effectGrid .effectTile[aria-disabled="true"] {
opacity: 0.55;
}
.effectGrid .effectTile[aria-disabled="true"] .effectThumb {
filter: grayscale(1);
}
/*
* The cross that removes one of the user's own backgrounds.
*
* Shown on hover, and whenever the tile has the keyboard's attention, so the
* two ways of reaching it look the same. It carries its own ground for the
* same reason the tick does: it sits on a picture nobody chose.
*/
.effectRemove {
position: absolute;
/* On the tile's top-right corner, where Zoom and Teams both put it, and
clear of the middle: centred, it sat inside the target for choosing the
tile, so the one destructive action lived where the ordinary one does. */
inset-block-start: 0;
inset-inline-end: 0;
/* It overhangs the corner by about a fifth of itself, as the design draws
it: enough to read as sitting on the tile rather than in it, little
enough that it never reaches the tile beside it. */
translate: 20% -20%;
display: none;
align-items: center;
justify-content: center;
inline-size: 28px;
block-size: 28px;
border-radius: 50%;
background: var(--cpd-color-bg-canvas-default);
/* Red at rest, not only under the pointer. The design says what the button
does before you are on it, and the ground carries the hover instead. */
color: var(--cpd-color-icon-critical-primary);
/* Raised off the tile rather than outlined against it — Compound's own value
for a small raised surface, there being no shadow token. In a dark theme
it falls on a near-black ground and reads as nothing, which is how the
design draws it there too. */
box-shadow: 0 1.2px 2.4px 0 rgb(0 0 0 / 15%);
cursor: pointer;
}
/* Under the pointer the ground turns critical and the shadow gives way to a
ring, so the button reads as pressable and as destructive at once. */
.effectRemove:hover {
background: var(--cpd-color-bg-critical-subtle-hovered);
box-shadow: 0 0 0 var(--cpd-border-width-1)
var(--cpd-color-border-critical-subtle);
}
.effectGrid .effectTileWrap:hover .effectRemove,
.effectGrid .effectTileWrap:focus-within .effectRemove {
display: flex;
}
/* Holds a tile that can be removed, so the cross can sit over it without
being inside the menu item. */
.effectTileWrap {
position: relative;
display: flex;
min-inline-size: 0;
}
.effectGrid .effectTileWrap > .effectTile {
flex: 1;
min-inline-size: 0;
}
/* Sits under the tiles, inside the same margin they keep from the frame, and
with the same distance above it as below: the menu's own bottom padding
supplies the lower half, so only the upper half is set here. Without it the
message sits flush against the tiles with a gap twice its size underneath. */
.effectError {
/* For the same reason as the notice below: a refusal's title is a sentence,
and would otherwise set the menu's width rather than fit inside it. */
contain: inline-size;
/* A step short of the margin it keeps at the sides: the menu is a column
with a gap of its own, which makes up the difference above. */
padding: var(--cpd-space-3x) var(--cpd-space-4x) 0;
}
/* The spinner stands exactly where the tick will, so the tile does not shift
when one becomes the other. */
.effectBusy {
display: inline-flex;
align-items: center;
justify-content: center;
inline-size: 20px;
block-size: 20px;
}
/*
* What the browser costs, said quietly.
*
* Measured off the design: the subtle secondary ground, held the same sixteen
* pixels from the menu's sides as the grid above it, with the mark and the
* text on one line's baseline and the text at its ordinary weight. Not an
* alert — an alert is for something that just happened, and this was true
* before the menu opened and will be true after it closes.
*/
.effectNotice {
/* Takes the width the menu has rather than lending it one. The menu is as
wide as its widest content, and a sentence asks for its whole length on
one line: without this the notice widened the camera menu from 296 to 340
pixels, past the width the grid and the preview are drawn for. */
contain: inline-size;
display: flex;
align-items: flex-start;
gap: var(--cpd-space-3x);
/* Nothing below: the menu's own bottom padding is already the inset, and
setting one here as well made the gap under the notice twice the gap
beside it. The same reason the grid leaves its own bottom alone. */
margin: var(--cpd-space-2x) var(--cpd-space-4x) 0;
padding: var(--cpd-space-3x);
border-radius: var(--cpd-space-2x);
background: var(--cpd-color-bg-subtle-secondary);
color: var(--cpd-color-text-primary);
font: var(--cpd-font-body-md-regular);
}
.effectNotice > svg {
flex-shrink: 0;
color: var(--cpd-color-icon-secondary);
}
/*
* How the user looks, at the top of the camera menu.
*
* Edge to edge and flush with the top, as the design draws it: it takes the
* menu's own top padding, which is why it pulls itself up over it. Held one
* border width clear of the frame and rounded to match what is left of the
* frame's corner, because a video paints on a layer of its own and would
* otherwise draw over the outline the menu is framed with.
*
* Sixteen by nine across the camera menu's fixed width, less its frame, and
* written as a height rather than a ratio: with a ratio and a cap Safari kept
* the ratio by narrowing the box, and the preview stopped short of the menu's
* right edge. Cropped rather than letterboxed where the call is narrow enough
* to narrow the menu — a face cut at the edge reads as a self-view, a face in
* bars reads as a broken one.
*/
.selfPreview {
position: relative;
display: flex;
align-items: center;
justify-content: center;
margin-block-end: var(--cpd-space-2x);
block-size: 166px;
overflow: hidden;
border-start-start-radius: calc(
var(--cpd-space-3x) - var(--cpd-border-width-1)
);
border-start-end-radius: calc(
var(--cpd-space-3x) - var(--cpd-border-width-1)
);
background: var(--cpd-color-bg-subtle-secondary);
color: var(--cpd-color-icon-secondary);
}
/* Above the list: it takes the menu's own top padding, which is why it pulls
itself up over it, and it keeps clear of the frame's sides itself, since it
is not inside the list that otherwise does. */
.selfPreviewPinned {
margin-block-start: calc(
-1 * var(--cpd-space-5x) + var(--cpd-border-width-1)
);
margin-inline: var(--cpd-border-width-1);
}
/* Inside the list, where there was no room to pin it. A scrolling container
cuts off whatever is pulled above its top, so it is the list that moves up
over the menu's padding instead, to bring the preview flush with the top.
That puts the list's corners where the frame's are rounded, so they are
rounded too: square, a stuck heading would show past the curve. */
.deviceListLeadsWithPreview {
margin-block-start: calc(
-1 * var(--cpd-space-5x) + var(--cpd-border-width-1)
);
border-start-start-radius: calc(
var(--cpd-space-3x) - var(--cpd-border-width-1)
);
border-start-end-radius: calc(
var(--cpd-space-3x) - var(--cpd-border-width-1)
);
}
/* Laid over the box rather than in it. In flow, a sixteen-by-nine picture
166 pixels tall asks for 295 of width, and the menu, sized to its widest
content, grew by the pixel that asked for. Out of flow it only fills what
the menu's width already gives it. */
.selfPreview > video,
.selfPreview > img {
position: absolute;
inset: 0;
inline-size: 100%;
block-size: 100%;
object-fit: cover;
}
/* The camera faces the user, so the picture is what a mirror would show —
the same as the pre-join preview, or the two would disagree. */
.selfPreview > .mirrored {
transform: scaleX(-1);
}
/*
* A fade at each edge of the list while there is more that way.
*
* In place of the scrollbar, which the platform fades out after a moment: with
* a trackpad on a Mac there was otherwise nothing to say that the list scrolls,
* and it looked as though the backgrounds simply stopped. Sticky, so each holds
* its edge of what is in view, and taking no room: each gives back its own
* height in a margin. Over the content, so the rows and tiles fade into it
* rather than past it, since they are opaque and a background would not show.
*
* The top one stands below whichever heading holds the top of the list, which
* is where the content is coming out from under.
*/
.scrollEdge {
position: sticky;
z-index: 1;
block-size: var(--cpd-space-6x);
pointer-events: none;
opacity: 0;
}
@media (prefers-reduced-motion: no-preference) {
.scrollEdge {
transition: opacity 150ms ease-out;
}
}
.scrollEdgeShown {
opacity: 1;
}
.scrollEdgeTop {
inset-block-start: var(--device-list-stuck-heading-height, 0);
margin-block-end: calc(-1 * var(--cpd-space-6x));
background: linear-gradient(
to bottom,
var(--cpd-color-bg-canvas-default),
transparent
);
}
.scrollEdgeBottom {
inset-block-end: 0;
margin-block-start: calc(-1 * var(--cpd-space-6x));
background: linear-gradient(
to top,
var(--cpd-color-bg-canvas-default),
transparent
);
}
File diff suppressed because it is too large Load Diff
+383 -11
View File
@@ -6,7 +6,14 @@ Please see LICENSE in the repository root for full details.
*/
import { describe, expect, test, vi } from "vitest";
import { act, render, screen, type RenderResult } from "@testing-library/react";
import { axe } from "vitest-axe";
import {
act,
render,
screen,
within,
type RenderResult,
} from "@testing-library/react";
import userEvent from "@testing-library/user-event";
import { type JSX, useState, type ReactNode } from "react";
import { TooltipProvider } from "@vector-im/compound-web";
@@ -327,7 +334,7 @@ describe("MediaMuteAndSwitchButton", () => {
expect(onVideoBlurToggle).toHaveBeenCalled();
});
test("renders check icon to mark the selected menu item", async () => {
test("marks the selected menu item as checked", async () => {
const user = userEvent.setup();
const { getByRole } = renderComponent(
<MediaMuteAndSwitchButton
@@ -342,19 +349,384 @@ describe("MediaMuteAndSwitchButton", () => {
/>,
);
// open menu
await user.click(getByRole("button", { name: "Microphone" }));
// The selected item (mic2) renders both an IconOptions SVG and a CheckIcon SVG
const mic1Item = screen.getByRole("menuitemradio", {
name: "Microphone 2",
});
expect(mic1Item.querySelectorAll("svg").length).toBe(2);
screen.getByRole("menuitemradio", { name: "Microphone 2", checked: true });
screen.getByRole("menuitemradio", { name: "Microphone 1", checked: false });
});
// The unselected item (mic1) only renders its IconOptions SVG
const mic2Item = screen.getByRole("menuitemradio", {
test("disables every device while a selection is settling", async () => {
const user = userEvent.setup();
const { promise, resolve } = Promise.withResolvers<void>();
function Wrapper(): JSX.Element {
const [selectedOption, setSelectedOption] = useState("mic1");
return (
<MediaMuteAndSwitchButton
title="Switcher"
iconsAndLabels="audio"
enabled={true}
options={[
{ label: { type: "name", name: "Microphone 1" }, id: "mic1" },
{ label: { type: "name", name: "Microphone 2" }, id: "mic2" },
]}
selectedOption={selectedOption}
onSelect={(id) => {
void promise.then(() => setSelectedOption(id));
}}
outputOptions={[
{ label: { type: "name", name: "Speakers" }, id: "spk1" },
{ label: { type: "name", name: "Headset" }, id: "spk2" },
]}
selectedOutputOption="spk1"
onSelectOutput={vi.fn()}
/>
);
}
const { getByRole } = renderComponent(<Wrapper />);
await user.click(getByRole("button", { name: "Microphone" }));
await user.click(
screen.getByRole("menuitemradio", { name: "Microphone 2" }),
);
// In flight: nothing else can be picked, in either section, so a second
// request cannot overtake the first.
for (const name of ["Microphone 1", "Speakers", "Headset"]) {
expect(screen.getByRole("menuitemradio", { name })).toHaveAttribute(
"aria-disabled",
"true",
);
}
await act(async () => {
resolve();
await promise;
});
// Settled: choosable again.
expect(
screen.getByRole("menuitemradio", { name: "Microphone 1" }),
).not.toHaveAttribute("aria-disabled", "true");
expect(
screen.getByRole("menuitemradio", { name: "Headset" }),
).not.toHaveAttribute("aria-disabled", "true");
});
test("camera menu uses the same selection pattern and keeps the blur toggle", async () => {
const user = userEvent.setup();
const { getByRole } = renderComponent(
<MediaMuteAndSwitchButton
title="Switcher"
iconsAndLabels="video"
enabled={true}
options={[
{ label: { type: "name", name: "Camera 1" }, id: "cam1" },
{ label: { type: "name", name: "Camera 2" }, id: "cam2" },
]}
selectedOption="cam1"
onSelect={vi.fn()}
videoBlurToggleClick={vi.fn()}
/>,
);
await user.click(getByRole("button", { name: "Camera" }));
// Same selection pattern as the microphone menu.
screen.getByRole("menuitemradio", { name: "Camera 1", checked: true });
screen.getByRole("menuitemradio", { name: "Camera 2", checked: false });
// And background blur is still reachable from here.
expect(
screen.getByRole("menuitemcheckbox", { name: "Blur background" }),
).toBeInTheDocument();
});
test("marks focus as keyboard-driven only when the keyboard moved it", async () => {
const user = userEvent.setup();
const { getByRole } = renderComponent(
<MediaMuteAndSwitchButton
title="Switcher"
iconsAndLabels="audio"
enabled={true}
options={[
{ label: { type: "name", name: "Microphone 1" }, id: "mic1" },
{ label: { type: "name", name: "Microphone 2" }, id: "mic2" },
]}
selectedOption="mic1"
onSelect={vi.fn()}
/>,
);
await user.click(getByRole("button", { name: "Microphone" }));
const list = screen
.getByRole("menuitemradio", { name: "Microphone 1" })
.closest("[data-focus-modality]");
// The menu focuses whatever the pointer is over, so focus alone says
// nothing about how someone is navigating.
expect(list).toHaveAttribute("data-focus-modality", "pointer");
await user.keyboard("{ArrowDown}");
expect(list).toHaveAttribute("data-focus-modality", "keyboard");
await user.pointer({
target: screen.getByRole("menuitemradio", { name: "Microphone 2" }),
coords: { clientX: 10, clientY: 10 },
});
expect(list).toHaveAttribute("data-focus-modality", "pointer");
});
test("marks the selected device with the accent fill", async () => {
const user = userEvent.setup();
const { getByRole } = renderComponent(
<MediaMuteAndSwitchButton
title="Switcher"
iconsAndLabels="audio"
enabled={true}
options={[
{ label: { type: "name", name: "Microphone 1" }, id: "mic1" },
{ label: { type: "name", name: "Microphone 2" }, id: "mic2" },
]}
selectedOption="mic2"
onSelect={vi.fn()}
/>,
);
await user.click(getByRole("button", { name: "Microphone" }));
const selected = screen
.getByRole("menuitemradio", { name: "Microphone 2" })
.querySelector("input[type=radio]");
expect(selected).toBeChecked();
// A read-only control is painted muted, which loses the accent fill that
// marks the selection and makes the menu differ from settings.
expect(selected).not.toHaveAttribute("readonly");
});
test("the open menu has no accessibility violations", async () => {
const user = userEvent.setup();
const { getByRole } = renderComponent(
<MediaMuteAndSwitchButton
title="Switcher"
iconsAndLabels="audio"
enabled={true}
options={[
{ label: { type: "name", name: "Microphone 1" }, id: "mic1" },
{ label: { type: "name", name: "Microphone 2" }, id: "mic2" },
]}
selectedOption="mic1"
onSelect={vi.fn()}
outputOptions={[
{ label: { type: "name", name: "Speakers" }, id: "spk1" },
{ label: { type: "name", name: "Headset" }, id: "spk2" },
]}
selectedOutputOption="spk1"
onSelectOutput={vi.fn()}
/>,
);
await user.click(getByRole("button", { name: "Microphone" }));
// Includes the menu's own structure: wrappers between the menu and its
// items break the relationship the roles describe.
const menu = document.querySelector('[role="menu"]');
expect(await axe(menu as HTMLElement)).toHaveNoViolations();
});
test("puts the speaker section above the microphone section", async () => {
const user = userEvent.setup();
const { getByRole } = renderComponent(
<MediaMuteAndSwitchButton
title="Switcher"
iconsAndLabels="audio"
enabled={true}
options={[
{ label: { type: "name", name: "Microphone 1" }, id: "mic1" },
{ label: { type: "name", name: "Microphone 2" }, id: "mic2" },
]}
selectedOption="mic1"
onSelect={vi.fn()}
outputOptions={[
{ label: { type: "name", name: "Speakers" }, id: "spk1" },
{ label: { type: "name", name: "Headset" }, id: "spk2" },
]}
selectedOutputOption="spk1"
onSelectOutput={vi.fn()}
/>,
);
await user.click(getByRole("button", { name: "Microphone" }));
const speakers = screen.getByRole("menuitemradio", { name: "Speakers" });
const microphone = screen.getByRole("menuitemradio", {
name: "Microphone 1",
});
expect(mic2Item.querySelectorAll("svg").length).toBe(1);
expect(
speakers.compareDocumentPosition(microphone) &
Node.DOCUMENT_POSITION_FOLLOWING,
).toBeTruthy();
});
test("lists speaker and microphone sections", async () => {
const user = userEvent.setup();
const { getByRole } = renderComponent(
<MediaMuteAndSwitchButton
title="Switcher"
iconsAndLabels="audio"
enabled={true}
options={[
{ label: { type: "name", name: "Microphone 1" }, id: "mic1" },
{ label: { type: "name", name: "Microphone 2" }, id: "mic2" },
]}
selectedOption="mic1"
onSelect={vi.fn()}
outputOptions={[
{
label: { type: "default", name: "Built-in Output" },
id: "default",
},
{ label: { type: "name", name: "Headset" }, id: "spk2" },
]}
selectedOutputOption="default"
onSelectOutput={vi.fn()}
/>,
);
await user.click(getByRole("button", { name: "Microphone" }));
screen.getByRole("menuitemradio", {
name: "Default (Built-in Output)",
checked: true,
});
screen.getByRole("menuitemradio", { name: "Headset", checked: false });
screen.getByRole("menuitemradio", { name: "Microphone 1", checked: true });
screen.getByRole("menuitemradio", { name: "Microphone 2", checked: false });
});
test("calls the output select callback on speaker click", async () => {
const user = userEvent.setup();
const onSelectOutput = vi.fn();
const { getByRole } = renderComponent(
<MediaMuteAndSwitchButton
title="Switcher"
iconsAndLabels="audio"
enabled={true}
options={[
{ label: { type: "name", name: "Microphone 1" }, id: "mic1" },
{ label: { type: "name", name: "Microphone 2" }, id: "mic2" },
]}
selectedOption="mic1"
outputOptions={[
{ label: { type: "name", name: "Speakers" }, id: "spk1" },
{ label: { type: "name", name: "Headset" }, id: "spk2" },
]}
selectedOutputOption="spk1"
onSelectOutput={onSelectOutput}
/>,
);
await user.click(getByRole("button", { name: "Microphone" }));
await user.click(screen.getByRole("menuitemradio", { name: "Headset" }));
expect(onSelectOutput).toHaveBeenCalledWith("spk2");
});
test("shows a single device entry disabled", async () => {
const user = userEvent.setup();
const onSelect = vi.fn();
const { getByRole } = renderComponent(
<MediaMuteAndSwitchButton
title="Switcher"
iconsAndLabels="audio"
enabled={true}
options={[
{ label: { type: "name", name: "Microphone 1" }, id: "mic1" },
]}
selectedOption="mic1"
onSelect={onSelect}
/>,
);
await user.click(getByRole("button", { name: "Microphone" }));
// Shown rather than hidden, so the menu keeps its shape, but not choosable.
const only = screen.getByRole("menuitemradio", { name: "Microphone 1" });
expect(only).toHaveAttribute("aria-disabled", "true");
});
test("shows a default speaker where the platform lists none", async () => {
const user = userEvent.setup();
const { getByRole } = renderComponent(
<MediaMuteAndSwitchButton
title="Switcher"
iconsAndLabels="audio"
enabled={true}
options={[
{ label: { type: "name", name: "Microphone 1" }, id: "mic1" },
{ label: { type: "name", name: "Microphone 2" }, id: "mic2" },
]}
selectedOption="mic1"
onSelect={vi.fn()}
// Safari enumerates no output devices at all, and offers no way to
// choose one.
outputOptions={[]}
selectedOutputOption={undefined}
onSelectOutput={undefined}
/>,
);
await user.click(getByRole("button", { name: "Microphone" }));
// A heading with nothing under it says the feature is broken. Audio is
// playing somewhere, so the section names that somewhere and disables it.
const speakers = screen
.getAllByRole("group")
.find((group) => group.getAttribute("aria-label") === "Speaker")!;
const entries = within(speakers).getAllByRole("menuitemradio");
expect(entries).toHaveLength(1);
expect(entries[0]).toHaveAccessibleName("Default");
expect(entries[0]).toHaveAttribute("aria-disabled", "true");
// And marked as the selection: it is where audio is going, so an unchecked
// lone entry would read as nothing being chosen at all.
expect(entries[0]).toHaveAttribute("aria-checked", "true");
expect(
within(entries[0]).getByRole("radio", { hidden: true }),
).toBeChecked();
});
test("shows the speaker section disabled when output selection is unsupported", async () => {
const user = userEvent.setup();
const { getByRole } = renderComponent(
<MediaMuteAndSwitchButton
title="Switcher"
iconsAndLabels="audio"
enabled={true}
options={[
{ label: { type: "name", name: "Microphone 1" }, id: "mic1" },
{ label: { type: "name", name: "Microphone 2" }, id: "mic2" },
]}
selectedOption="mic1"
onSelect={vi.fn()}
outputOptions={[
{ label: { type: "name", name: "Speakers" }, id: "spk1" },
{ label: { type: "name", name: "Headset" }, id: "spk2" },
]}
selectedOutputOption="spk1"
// No callback: nothing can be picked here.
onSelectOutput={undefined}
/>,
);
await user.click(getByRole("button", { name: "Microphone" }));
expect(
screen.getByRole("menuitemradio", { name: "Speakers" }),
).toHaveAttribute("aria-disabled", "true");
expect(
screen.getByRole("menuitemradio", { name: "Headset" }),
).toHaveAttribute("aria-disabled", "true");
// The microphone section is unaffected.
expect(
screen.getByRole("menuitemradio", { name: "Microphone 2" }),
).not.toHaveAttribute("aria-disabled", "true");
});
});
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,80 @@
/*
Copyright 2026 Element Creations Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
.meter {
display: flex;
align-items: center;
gap: var(--cpd-space-3x);
padding-block: var(--cpd-space-2x);
/* The icon shares a centre with the radio controls in the device rows above,
which sit one 4x padding in from the menu's edge. Two things pull this row
out of line with them, and both come off the padding: the row is inset by a
border width so that the menu's frame stays visible behind it (see
`.stickyMeter`), and the icon is wider than a radio control, so it has to
start further left for the two to share a centre. Both are
measured in the MeterAlignsWithTheDeviceRows story rather than trusted:
jsdom lays nothing out, so only a real browser can hold this. */
padding-inline-start: calc(
var(--cpd-space-4x) - var(--cpd-border-width-1) - 2px
);
padding-inline-end: calc(var(--cpd-space-4x) * 2 + var(--cpd-space-2x));
}
.icon {
color: var(--cpd-color-icon-secondary);
flex-shrink: 0;
/* Same width as a device row's icon column, so the bars start where the
device names start. */
inline-size: 24px;
}
.segments {
/* A bar and the space beside it are always the same size; what changes with
the width available is how many bars there are. Spreading a fixed number of
bars instead would make the meter a different shape in every place it is
used, and close them up into one solid block wherever the space ran short —
and a meter whose bars touch stops reading as a count, which is what
carries the level for anyone who cannot rely on the colour.
Both sizes are set here and nowhere else: the component reads them back off
the rendered bars to work out how many fit. */
flex: 1;
/* The width decides the count, never the other way round. Containment is
what enforces that: without it the bars are both a floor under the width
and the widest thing in the menu, so the meter would set the menu's width
and the device names — which are what a person is reading — would have to
fit around it. */
contain: inline-size;
min-inline-size: 0;
display: flex;
align-items: center;
/* Half again as wide as a bar, as the design has it: bars set as close as
their own width read as a solid block long before they touch. */
gap: var(--cpd-space-1-5x);
}
.segment {
flex: none;
inline-size: var(--cpd-space-1x);
block-size: var(--cpd-space-4x);
border-radius: var(--cpd-space-1x);
background: var(--cpd-color-bg-subtle-primary);
}
/* Filled segments carry the level by count as well as by colour, so the meter
stays readable without relying on hue. */
.segmentLit {
background: var(--cpd-color-bg-accent-rest);
}
.message {
color: var(--cpd-color-text-secondary);
/* A paragraph brings a margin below it, which in a centred row does not push
the text down but lifts it: the margin box is what gets centred, so the
words end up above the middle and the icon beside them looks low. */
margin-block: 0;
}
@@ -0,0 +1,189 @@
/*
Copyright 2026 Element Creations Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { expect, within } from "storybook/test";
import { type JSX } from "react";
import type { Meta, StoryObj } from "@storybook/react-vite";
import {
MicrophoneLevelMeter,
type MicrophoneLevelMeterProps,
} from "./MicrophoneLevelMeter";
import styles from "./MicrophoneLevelMeter.module.css";
import { METER_SEGMENTS } from "../state/MicrophoneLevel";
/**
* A width to show the meter at, close to the menu it lives in.
*
* Not a copy of the menu's width, and nothing depends on the two agreeing: a
* bar and the gap beside it are a fixed size now, so this only decides how many
* bars there is room for. Without a width at all the stories would shrink-wrap
* to almost nothing and show a meter two bars wide, which is no use to anyone
* looking at them.
*/
const STORY_WIDTH = 256;
const meta = {
component: MicrophoneLevelMeter,
decorators: [
(Story): JSX.Element => (
<div style={{ inlineSize: STORY_WIDTH }}>
<Story />
</div>
),
],
argTypes: {
state: {
description:
"What the selected microphone can say about itself: a level, or a reason there is none.",
},
},
} satisfies Meta<typeof MicrophoneLevelMeter>;
export default meta;
type Story = StoryObj<typeof meta>;
/**
* A quiet room. Nothing is lit: room hiss below the noise floor must not read
* as "it can hear me".
*/
export const Silent: Story = {
args: { state: { type: "level", level: 0 } },
play: async ({ canvasElement }) => {
await expect(litSegments(canvasElement)).toBe(0);
},
};
export const QuietSpeech: Story = {
args: { state: { type: "level", level: 5 } },
};
export const NormalSpeech: Story = {
args: { state: { type: "level", level: 12 } },
play: async ({ canvasElement }) => {
// Shown at something like the width of the menu, so the meter in a story
// reads like the meter in a call rather than like a handful of bars. The
// design's own mock has sixteen of them at this width; a floor rather than
// a count, because the number follows from the bar and gap sizes and those
// are the design's to change.
await expect(
canvasElement.getElementsByClassName(styles.segment).length,
).toBeGreaterThanOrEqual(15);
},
};
export const LoudSpeech: Story = {
args: { state: { type: "level", level: METER_SEGMENTS } },
};
/**
* The three volumes differ by how many bars are lit, so the level survives
* greyscale and a screen reader as well as it survives colour.
*/
export const VolumesAreDistinguishable: Story = {
args: { state: { type: "level", level: 5 } },
play: async ({ canvasElement, mount }) => {
const lit: number[] = [];
for (const level of [5, 12, METER_SEGMENTS]) {
await mount(<MicrophoneLevelMeter state={{ type: "level", level }} />);
lit.push(litSegments(canvasElement));
await expect(within(canvasElement).getByRole("meter")).toHaveAttribute(
"aria-valuenow",
String(level),
);
}
// Three different counts, rising: the level is carried by how many bars
// are lit, not by their colour alone.
await expect(new Set(lit).size).toBe(lit.length);
await expect(lit).toEqual([...lit].sort((a, b) => a - b));
},
};
/**
* Permission refused. A message with a next action, never a still meter that
* reads as silence.
*/
export const PermissionDenied: Story = {
args: { state: { type: "permission-denied" } },
play: async ({ canvasElement }) => {
const canvas = within(canvasElement);
await expect(canvas.queryByRole("meter")).toBeNull();
await expect(
canvas.getByText(/Allow access in your browser settings/),
).toBeVisible();
},
};
/** No input device at all, told apart from a refusal. */
export const NoDevice: Story = {
args: { state: { type: "no-device" } },
play: async ({ canvasElement }) => {
const canvas = within(canvasElement);
await expect(canvas.queryByRole("meter")).toBeNull();
await expect(canvas.getByText(/No microphone found/)).toBeVisible();
// The icon sits on the middle of the words, however many lines they run to.
// A paragraph's own margin would centre its margin box instead, leaving the
// text high and the icon looking low beside it.
const middle = (element: Element): number => {
const box = element.getBoundingClientRect();
return box.top + box.height / 2;
};
const icon = canvasElement.getElementsByClassName(styles.icon)[0];
const words = canvasElement.getElementsByClassName(styles.message)[0];
await expect(Math.abs(middle(icon) - middle(words))).toBeLessThanOrEqual(1);
},
};
/**
* The same meter at two widths.
*
* A bar and the space beside it are always the same size; what changes is how
* many bars there are. Spreading a fixed number of bars instead would make the
* meter a different shape in every place it is used, and close the bars up into
* one block wherever the space ran short — and bars that touch cannot be
* counted, which is what carries the level without colour.
*/
export const ShapeStaysTheSameAtAnyWidth: Story = {
args: { state: { type: "level", level: 12 } },
play: async ({ mount, args }) => {
const narrow = await measureAt(mount, args, 180);
const wide = await measureAt(mount, args, 400);
await expect(narrow.bar).toBe(wide.bar);
await expect(narrow.gap).toBe(wide.gap);
await expect(narrow.count).toBeLessThan(wide.count);
// And the bars are still bars, not one run of colour.
await expect(narrow.gap).toBeGreaterThan(0);
},
};
/** How many bars are painted as carrying level, rather than as empty. */
function litSegments(canvasElement: HTMLElement): number {
return canvasElement.getElementsByClassName(styles.segmentLit).length;
}
/** Renders the meter at one width and reports the shape of its bars. */
async function measureAt(
mount: (ui: JSX.Element) => Promise<unknown>,
args: MicrophoneLevelMeterProps,
width: number,
): Promise<{ count: number; bar: number; gap: number }> {
await mount(
<div style={{ inlineSize: width }}>
<MicrophoneLevelMeter {...args} />
</div>,
);
const bars = Array.from(document.body.getElementsByClassName(styles.segment));
const first = bars[0].getBoundingClientRect();
const second = bars[1].getBoundingClientRect();
return {
count: bars.length,
bar: first.width,
gap: second.left - first.right,
};
}
@@ -0,0 +1,40 @@
/*
Copyright 2026 Element Creations Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { describe, expect, test } from "vitest";
import { render, screen } from "@testing-library/react";
import { MicrophoneLevelMeter } from "./MicrophoneLevelMeter";
import { METER_SEGMENTS } from "../state/MicrophoneLevel";
describe("MicrophoneLevelMeter", () => {
test("announces the level rather than relying on hue", () => {
render(<MicrophoneLevelMeter state={{ type: "level", level: 6 }} />);
const meter = screen.getByRole("meter", { name: "Microphone level" });
expect(meter).toHaveAttribute("aria-valuenow", "6");
expect(meter).toHaveAttribute("aria-valuemax", String(METER_SEGMENTS));
expect(meter).toHaveAttribute("aria-valuetext", `6 of ${METER_SEGMENTS}`);
});
test("shows distinct messages for denied permission and no input device", () => {
const denied = render(
<MicrophoneLevelMeter state={{ type: "permission-denied" }} />,
);
// A next action, not a flat meter that reads as silence.
expect(
denied.getByText(/Allow access in your browser settings/),
).toBeInTheDocument();
expect(denied.queryByRole("meter")).toBeNull();
const missing = render(
<MicrophoneLevelMeter state={{ type: "no-device" }} />,
);
expect(missing.getByText(/No microphone found/)).toBeInTheDocument();
expect(missing.queryByRole("meter")).toBeNull();
});
});
+120
View File
@@ -0,0 +1,120 @@
/*
Copyright 2026 Element Creations Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { useCallback, useState, type FC, type Ref } from "react";
import { Text } from "@vector-im/compound-web";
import { MicOnIcon } from "@vector-im/compound-design-tokens/assets/web/icons";
import classNames from "classnames";
import { useTranslation } from "react-i18next";
import { distinctUntilChanged, map } from "rxjs";
import styles from "./MicrophoneLevelMeter.module.css";
import { METER_SEGMENTS, type MicrophoneState } from "../state/MicrophoneLevel";
import { observeElementSize$ } from "../utils/elementSize";
export interface MicrophoneLevelMeterProps {
state: MicrophoneState;
className?: string;
/**
* The meter's own element. Its height is what a scroll container has to keep
* clear to stop the meter covering the row it has just scrolled to.
*/
ref?: Ref<HTMLDivElement>;
}
/**
* The live input level of the selected microphone, shown beneath it.
*
* It says whether the microphone is picking anything up, which is not the same
* as whether the user is being heard: it keeps moving while muted, and the mute
* control is what says nothing is transmitted.
*/
export const MicrophoneLevelMeter: FC<MicrophoneLevelMeterProps> = ({
state,
className,
ref,
}) => {
const { t } = useTranslation();
// How many bars there is room for. The bars never change size, so this is
// what absorbs a change of width. Starts at the full count so that the first
// paint is a meter rather than a single bar, and so that a renderer with no
// layout at all — jsdom — still draws the whole thing.
const [barCount, setBarCount] = useState(METER_SEGMENTS);
const track = useCallback(
(element: HTMLDivElement | null): (() => void) | undefined => {
if (element === null) return;
const subscription = observeElementSize$(element)
.pipe(
map(({ width }) => barsThatFit(element, width)),
distinctUntilChanged(),
)
.subscribe(setBarCount);
return (): void => subscription.unsubscribe();
},
[],
);
if (state.type !== "level")
return (
<div ref={ref} className={classNames(styles.meter, className)}>
<MicOnIcon width={24} height={24} className={styles.icon} aria-hidden />
<Text size="sm" className={styles.message}>
{state.type === "permission-denied"
? t("microphone_level.permission_denied")
: t("microphone_level.no_device")}
</Text>
</div>
);
return (
<div ref={ref} className={classNames(styles.meter, className)}>
<MicOnIcon width={24} height={24} className={styles.icon} aria-hidden />
<div
ref={track}
className={styles.segments}
role="meter"
aria-label={t("microphone_level.label")}
aria-valuemin={0}
aria-valuemax={METER_SEGMENTS}
aria-valuenow={state.level}
aria-valuetext={t("microphone_level.value", {
level: state.level,
max: METER_SEGMENTS,
})}
>
{Array.from({ length: barCount }, (_, i) => (
<span
key={i}
aria-hidden
className={classNames(styles.segment, {
// The level is a share of the scale, not a number of bars: how
// many bars stand for it depends on how many there are.
[styles.segmentLit]:
i < Math.round((state.level / METER_SEGMENTS) * barCount),
})}
/>
))}
</div>
</div>
);
};
/**
* How many bars fit across `width`, at the size the stylesheet draws them.
*
* Measured off a rendered bar rather than told: the size of a bar and of the
* space beside it are a design question, settled in the stylesheet, and reading
* them back is what keeps them from being settled twice.
*/
function barsThatFit(track: HTMLElement, width: number): number {
const gap = Number.parseFloat(getComputedStyle(track).columnGap);
const bar = track.firstElementChild?.getBoundingClientRect().width ?? 0;
// A renderer that lays nothing out tells us nothing; keep the full count.
if (!(bar > 0) || !(gap >= 0)) return METER_SEGMENTS;
return Math.max(1, Math.floor((width + gap) / (bar + gap)));
}
+54
View File
@@ -0,0 +1,54 @@
/*
Copyright 2026 Element Creations Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { type FC, useEffect, useRef } from "react";
import { facingModeFromLocalTrack, type LocalVideoTrack } from "livekit-client";
import { VideoCallOffIcon } from "@vector-im/compound-design-tokens/assets/web/icons";
import styles from "./MediaMuteAndSwitchButton.module.css";
interface Props {
/** The camera as the pipeline has it, or none while the camera is off. */
track: LocalVideoTrack | null;
}
/**
* The user's own camera, for the camera menu's preview.
*
* Attached to the same track the pipeline is synced to, so it shows whatever
* effect is in force — the processed picture where one has been chosen, and
* the plain one where none ever has — without a capture or a pipeline of its
* own. A second consumer of one track, not a second track.
*/
export const SelfPreview: FC<Props> = ({ track }) => {
const video = useRef<HTMLVideoElement>(null);
useEffect(() => {
const element = video.current;
if (!track || !element) return;
track.attach(element);
return (): void => {
track.detach(element);
};
}, [track]);
// Nothing to show, and saying so: an effect chosen now still applies when
// the camera comes back, so the grid below stays live.
if (!track) return <VideoCallOffIcon width={32} height={32} />;
return (
<video
ref={video}
muted
playsInline
className={
facingModeFromLocalTrack(track).facingMode === "user"
? styles.mirrored
: undefined
}
/>
);
};
@@ -0,0 +1,42 @@
/*
Copyright 2026 Element Creations Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { afterEach, describe, expect, test } from "vitest";
import { renderHook } from "@testing-library/react";
import { useMicrophoneLevel } from "./useMicrophoneLevel";
import { restoreAudioCapture, stubAudioCapture } from "../utils/test";
// The capture itself, and releasing it, belong to observeMicrophoneState$ and
// are covered in src/state/MicrophoneLevel.test.ts. What is left here is the
// bridging: when the hook watches, and what it reports before it has an answer.
describe("useMicrophoneLevel", () => {
afterEach(restoreAudioCapture);
test("holds no capture while the meter is not shown", () => {
const capture = stubAudioCapture();
renderHook(() => useMicrophoneLevel("mic1", false));
expect(capture.getUserMedia).not.toHaveBeenCalled();
});
test("starts from nothing rather than the previous device's level", () => {
const capture = stubAudioCapture();
const { result, rerender } = renderHook(
({ id }: { id: string }) => useMicrophoneLevel(id, true),
{ initialProps: { id: "mic1" } },
);
capture.grant();
rerender({ id: "mic2" });
// No level carried over: the meter reads the new device or nothing at all.
expect(result.current).toEqual({ type: "level", level: 0 });
});
});
+41
View File
@@ -0,0 +1,41 @@
/*
Copyright 2026 Element Creations Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { useEffect, useState } from "react";
import {
type MicrophoneState,
observeMicrophoneState$,
} from "../state/MicrophoneLevel";
const IDLE: MicrophoneState = { type: "level", level: 0 };
/**
* Reads the live input level of a microphone, while `active`.
*
* A bridge and nothing else: the capture, its lifetime and the maths belong to
* {@link observeMicrophoneState$}. Scoped to `active` so the device is held
* only while whatever shows the meter is on screen, rather than for the length
* of a call.
*/
export function useMicrophoneLevel(
deviceId: string | undefined,
active: boolean,
): MicrophoneState {
const [state, setState] = useState<MicrophoneState>(IDLE);
useEffect(() => {
if (!active) return;
// Idle first, so a new device starts from nothing rather than from the
// level the previous one was reading.
setState(IDLE);
const subscription = observeMicrophoneState$(deviceId).subscribe(setState);
return (): void => subscription.unsubscribe();
}, [deviceId, active]);
return state;
}
Binary file not shown.

After

Width:  |  Height:  |  Size: 27 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 22 KiB

@@ -43,13 +43,40 @@ const wasmFileset: WasmFileset = {
};
/**
* Track processor that applies effects such as blurring to a user's background.
* Track processor that applies background effects — blur, or a replacement
* image — to a user's camera.
*
* This is just like LiveKit's prebuilt BackgroundTransformer except that it
* loads the segmentation models from our own bundle rather than as an external
* resource fetched from the public internet.
* resource fetched from the public internet. The library's own `assetPaths`
* option cannot do this: it takes a directory and resolves the WASM through
* MediaPipe's FilesetResolver, which picks its own variants, whereas we want
* only the SIMD ones.
*/
export class BlurBackgroundTransformer extends BackgroundTransformer {
export class BackgroundEffectTransformer extends BackgroundTransformer {
/**
* Called once, when a frame has actually come out of the pipeline.
*
* The promises say nothing useful about when that happens. Measured on four
* devices, attaching resolves in about three seconds and switching in none
* at all, and then a browser on the slow path spends another twelve to
* fifteen seconds before the first frame appears. Only the frame itself
* marks the end of the wait.
*/
public onFirstFrame: (() => void) | undefined;
private produced = false;
public override async transform(
frame: VideoFrame,
controller: TransformStreamDefaultController<VideoFrame>,
): Promise<void> {
await super.transform(frame, controller);
if (!this.produced) {
this.produced = true;
this.onFirstFrame?.();
}
}
public async init({
outputCanvas,
inputElement: inputVideo,
@@ -73,8 +100,16 @@ export class BlurBackgroundTransformer extends BackgroundTransformer {
outputConfidenceMasks: false,
});
if (this.options.blurRadius) {
// BackgroundTransformer applies these at the end of its own init. Because
// we replace init wholesale rather than extending it, we have to repeat
// them, or an effect that was already selected when the pipeline starts is
// silently ignored.
if (this.options.imagePath) {
await this.loadAndSetBackground(this.options.imagePath);
}
if (typeof this.options.blurRadius === "number") {
this.gl?.setBlurRadius(this.options.blurRadius);
}
this.gl?.setBackgroundDisabled(this.options.backgroundDisabled ?? false);
}
}
+401
View File
@@ -0,0 +1,401 @@
/*
Copyright 2026 Element Creations Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { type FC, useCallback, useRef, useState } from "react";
import {
createLocalVideoTrack,
type LocalVideoTrack,
VideoPresets,
} from "livekit-client";
import {
BackgroundProcessorWrapper,
supportsBackgroundProcessors as supportsBackgroundProcessorsLivekitSdk,
supportsModernBackgroundProcessors,
type SwitchBackgroundProcessorOptions,
} from "@livekit/track-processors";
import { logger } from "matrix-js-sdk/lib/logger";
import { BackgroundEffectTransformer } from "./BackgroundEffectTransformer";
import { blurRadius, imagePathFor } from "./backgroundEffects";
import { supportsBackgroundProcessors } from "./backgroundProcessing";
import { platform } from "../Platform";
/**
* Whether background effects can hold their frame rate on this device, asked
* without a call in the way.
*
* The camera and the pipeline only: no SFU, no encoder, no second participant.
* If a device cannot hold a frame rate here it cannot hold one in a call
* either, so this is the cheap half of the question — and it is the half that
* has never been asked. The gate that keeps effects off phones was added from
* reports, not measurements, and nothing in the app will run the pipeline
* there, so there has been nothing to measure.
*
* Deliberately ignores that gate: measuring what it forbids is the point. It
* reports what the gate would have said, and runs anyway.
*
* Exploration only, and never ported: an unlisted route, its text not
* translated, its numbers meant for whoever is holding the phone.
*/
type Effect = "none" | "blur" | "image";
interface Bucket {
/** Seconds since the first frame. */
at: number;
fps: number;
}
interface Run {
effect: Effect;
device: string;
platform: string;
/** Whether the browser has the API that avoids the canvas fallback. */
modernApi: boolean;
/** What the app's own verdict would have said about this device. */
appWouldAllow: boolean;
sdkWouldAllow: boolean;
capture: { width?: number; height?: number; frameRate?: number };
processed: { width?: number; height?: number; frameRate?: number };
/** Milliseconds from asking for the effect until the pipeline reports ready. */
startupMs: number | null;
/**
* Where that time went: attaching builds the segmenter — fetching its
* assets, compiling the WASM, setting up the GPU — while switching only
* tells an already-built one what to draw. Split, because the two have
* different answers: one can be paid ahead of time, the other cannot.
*/
attachMs: number | null;
switchMs: number | null;
/**
* The segmenter's own assets, as the browser saw them. A transfer size of
* zero means the bytes came from cache and no network was involved, which is
* what separates "the first time on this device" from "every time".
*/
assets: {
name: string;
transferredBytes: number;
bytes: number;
ms: number;
}[];
meanFps: number;
/** The worst five-second stretch — where throttling shows up first. */
worstFps: number;
buckets: Bucket[];
/** Worst observed delay on a 100ms timer: the main thread's own stutter. */
worstTimerLagMs: number;
/** The longest the picture simply stopped, in milliseconds. */
worstFrameGapMs: number;
/** From asking for the run to the first frame on screen. */
firstFrameMs: number | null;
frames: number;
seconds: number;
}
const bucketSeconds = 5;
function optionsFor(effect: Effect): SwitchBackgroundProcessorOptions {
switch (effect) {
case "blur":
return { mode: "background-blur", blurRadius };
case "image": {
const imagePath = imagePathFor("indoor");
return imagePath
? { mode: "virtual-background", imagePath }
: { mode: "disabled" };
}
default:
return { mode: "disabled" };
}
}
export const BackgroundEffectsBench: FC = () => {
const video = useRef<HTMLVideoElement | null>(null);
const [seconds, setSeconds] = useState(60);
const [busy, setBusy] = useState(false);
const [status, setStatus] = useState("Ready.");
const [runs, setRuns] = useState<Run[]>([]);
const measure = useCallback(
async (effect: Effect): Promise<void> => {
setBusy(true);
setStatus(`Starting the camera for "${effect}"…`);
let track: LocalVideoTrack | undefined;
try {
// The resolution and frame rate a real call captures at, so the cost
// measured here is the cost a call would pay.
track = await createLocalVideoTrack({
resolution: VideoPresets.h720.resolution,
});
const capture = track.mediaStreamTrack.getSettings();
let startupMs: number | null = null;
let attachMs: number | null = null;
let switchMs: number | null = null;
if (effect !== "none") {
setStatus("Starting the pipeline…");
const askedAt = performance.now();
const pipeline = new BackgroundProcessorWrapper(
new BackgroundEffectTransformer({ backgroundDisabled: true }),
"background-effect-bench",
);
await track.setProcessor(pipeline);
const attachedAt = performance.now();
await pipeline.switchTo(optionsFor(effect));
const readyAt = performance.now();
attachMs = Math.round(attachedAt - askedAt);
switchMs = Math.round(readyAt - attachedAt);
startupMs = Math.round(readyAt - askedAt);
}
const element = video.current!;
track.attach(element);
await element.play().catch(() => undefined);
setStatus(`Measuring "${effect}" for ${seconds}s…`);
const result = await countFrames(element, seconds, (done) =>
setStatus(`Measuring "${effect}": ${done}s of ${seconds}s`),
);
const processed = track.mediaStreamTrack.getSettings();
setRuns((previous) => [
...previous,
{
effect,
device: navigator.userAgent,
platform,
modernApi: supportsModernBackgroundProcessors(),
appWouldAllow: supportsBackgroundProcessors(),
sdkWouldAllow: supportsBackgroundProcessorsLivekitSdk(),
capture: {
width: capture.width,
height: capture.height,
frameRate: capture.frameRate,
},
processed: {
width: processed.width,
height: processed.height,
frameRate: processed.frameRate,
},
startupMs,
attachMs,
switchMs,
assets: segmenterAssets(),
...result,
},
]);
setStatus(`Done: ${effect}.`);
} catch (e) {
logger.error("Background effects bench failed", e);
setStatus(`Failed: ${e instanceof Error ? e.message : String(e)}`);
} finally {
track?.stop();
setBusy(false);
}
},
[seconds],
);
return (
<div style={{ padding: 16, font: "14px system-ui", lineHeight: 1.5 }}>
<h1 style={{ fontSize: 20 }}>Background effects: can this device?</h1>
<p>
The camera and the pipeline, with no call in the way. Run{" "}
<strong>No effect</strong> first — every other number is only meaningful
beside it, on this device, in this light.
</p>
<p>
This device: <code>{platform}</code>, the app would{" "}
<strong>{supportsBackgroundProcessors() ? "allow" : "refuse"}</strong>{" "}
effects here, the browser has{" "}
<strong>
{supportsModernBackgroundProcessors()
? "the fast path"
: "only the canvas fallback"}
</strong>
.
</p>
<label>
Seconds per run:{" "}
<select
value={seconds}
onChange={(e) => setSeconds(Number(e.target.value))}
disabled={busy}
>
<option value={30}>30</option>
<option value={60}>60</option>
<option value={300}>300 (five minutes, for throttling)</option>
<option value={600}>600 (ten minutes)</option>
</select>
</label>
<div style={{ display: "flex", gap: 8, margin: "12px 0" }}>
{(["none", "blur", "image"] as const).map((effect) => (
<button
key={effect}
disabled={busy}
onClick={() => void measure(effect)}
style={{ padding: "8px 12px" }}
>
{effect === "none" ? "No effect" : effect}
</button>
))}
</div>
<p>
<strong>{status}</strong>
</p>
<video
ref={video}
muted
playsInline
style={{ inlineSize: "100%", maxInlineSize: 320, background: "#000" }}
/>
{runs.map((run, i) => (
<div key={i} style={{ marginBlockStart: 12 }}>
<strong>{run.effect}</strong>: mean {run.meanFps.toFixed(1)}fps, worst{" "}
{run.worstFps.toFixed(1)}fps
{run.startupMs !== null && (
<>
, ready in {run.startupMs}ms ({run.attachMs}ms attaching,{" "}
{run.switchMs}ms switching)
</>
)}
, first frame at {run.firstFrameMs ?? "?"}ms, picture stopped for up
to {run.worstFrameGapMs}ms, main thread stalled up to{" "}
{run.worstTimerLagMs}ms
</div>
))}
{runs.length > 0 && (
<textarea
readOnly
value={JSON.stringify(runs, null, 2)}
style={{ inlineSize: "100%", blockSize: 240, marginBlockStart: 12 }}
/>
)}
</div>
);
};
/**
* What the browser did to fetch the segmenter: bytes over the wire, bytes
* total, and how long each took. Same-origin, so the sizes are not hidden.
*/
function segmenterAssets(): Run["assets"] {
return performance
.getEntriesByType("resource")
.filter((e): e is PerformanceResourceTiming =>
/\.(wasm|tflite|task|binarypb)(\?|$)/.test(e.name),
)
.map((e) => ({
name: e.name.slice(e.name.lastIndexOf("/") + 1),
transferredBytes: e.transferSize,
bytes: e.encodedBodySize,
ms: Math.round(e.duration),
}));
}
/**
* Counts frames as the video actually presents them, which is the rate the
* pipeline is really delivering rather than the rate it was asked for.
*/
async function countFrames(
element: HTMLVideoElement,
seconds: number,
onProgress: (done: number) => void,
): Promise<
Pick<
Run,
| "meanFps"
| "worstFps"
| "buckets"
| "worstTimerLagMs"
| "worstFrameGapMs"
| "firstFrameMs"
| "frames"
| "seconds"
>
> {
return new Promise((resolve) => {
const buckets: Bucket[] = [];
const startedAt = performance.now();
let frames = 0;
let inBucket = 0;
let firstFrameMs: number | null = null;
let lastFrameAt = startedAt;
let worstFrameGapMs = 0;
// A phone that is struggling stalls its main thread, and a timer is the
// cheapest witness to that: Safari has no long-task observer.
let worstTimerLagMs = 0;
let lastTick = performance.now();
const ticker = window.setInterval(() => {
const now = performance.now();
worstTimerLagMs = Math.max(
worstTimerLagMs,
Math.round(now - lastTick - 100),
);
lastTick = now;
}, 100);
// On the clock, not on the frames. Closing a bucket only when a frame
// arrives cannot report a stretch with no frames in it at all — the first
// measurement on a phone spent fourteen seconds inside one bucket and
// reported it as a seventh of a frame per second rather than as a freeze.
const bucketer = window.setInterval(() => {
const at = Math.round((performance.now() - startedAt) / 1000);
buckets.push({ at, fps: inBucket / bucketSeconds });
inBucket = 0;
onProgress(at);
}, bucketSeconds * 1000);
const finish = (): void => {
window.clearInterval(ticker);
window.clearInterval(bucketer);
const elapsed = (performance.now() - startedAt) / 1000;
resolve({
meanFps: elapsed > 0 ? frames / elapsed : 0,
worstFps: buckets.length ? Math.min(...buckets.map((b) => b.fps)) : 0,
buckets,
worstTimerLagMs,
worstFrameGapMs: Math.round(worstFrameGapMs),
firstFrameMs: firstFrameMs === null ? null : Math.round(firstFrameMs),
frames,
seconds: Math.round(elapsed),
});
};
const onFrame = (): void => {
const now = performance.now();
firstFrameMs ??= now - startedAt;
worstFrameGapMs = Math.max(worstFrameGapMs, now - lastFrameAt);
lastFrameAt = now;
frames += 1;
inBucket += 1;
if (now - startedAt >= seconds * 1000) return finish();
element.requestVideoFrameCallback(onFrame);
};
if (!("requestVideoFrameCallback" in element)) {
window.clearInterval(ticker);
window.clearInterval(bucketer);
resolve({
meanFps: 0,
worstFps: 0,
buckets: [],
worstTimerLagMs: 0,
worstFrameGapMs: 0,
firstFrameMs: null,
frames: 0,
seconds: 0,
});
return;
}
element.requestVideoFrameCallback(onFrame);
// A freeze can outlast the run: without this, a device that never presents
// another frame never finishes and reports nothing at all.
window.setTimeout(finish, seconds * 1000 + 2000);
});
}
+261 -20
View File
@@ -6,41 +6,122 @@ Please see LICENSE in the repository root for full details.
*/
import {
ProcessorWrapper,
supportsBackgroundProcessors as supportsBackgroundProcessorsLivekitSdk,
BackgroundProcessorWrapper,
type ProcessorWrapper,
type BackgroundOptions,
type SwitchBackgroundProcessorOptions,
} from "@livekit/track-processors";
import {
createContext,
type FC,
type JSX,
use,
useCallback,
useEffect,
useMemo,
useRef,
useState,
} from "react";
import { type LocalVideoTrack } from "livekit-client";
import { logger } from "matrix-js-sdk/lib/logger";
import { combineLatest, map, type Observable } from "rxjs";
import { useObservable } from "observable-hooks";
import { BehaviorSubject, combineLatest, map, type Observable } from "rxjs";
import { useObservable, useObservableEagerState } from "observable-hooks";
import {
backgroundBlur as backgroundBlurSettings,
backgroundEffect as backgroundEffectSetting,
useSetting,
} from "../settings/settings";
import { BlurBackgroundTransformer } from "./BlurBackgroundTransformer";
import { BackgroundEffectTransformer } from "./BackgroundEffectTransformer";
import {
blurRadius,
imagePathFor,
parseEffect,
type BackgroundEffect,
} from "./backgroundEffects";
import { BackgroundImageStore } from "./backgroundImages";
import { supportsBackgroundProcessors } from "./backgroundProcessing";
import { type Behavior } from "../state/Behavior";
import { type ObservableScope } from "../state/ObservableScope";
import { platform } from "../Platform";
//TODO-MULTI-SFU: This is not yet fully there.
// it is a combination of exposing observable and react hooks.
// preferably we should not make this a context anymore and instead just a vm?
/** A background the user added, as the view needs it: an id and a picture. */
export interface AddedBackgroundImage {
id: string;
/** Lives as long as the provider does. */
url: string;
}
/**
* What the publishing side needs: whether effects can run at all, and the
* pipeline to attach. Deliberately not the backgrounds a user has added — the
* publisher has no use for those, and every test that stands in for this would
* have to carry them.
*/
export type ProcessorState = {
supported: boolean | undefined;
processor: undefined | ProcessorWrapper<BackgroundOptions>;
};
/** What the camera menu needs: the backgrounds this device keeps. */
export interface AddedBackgrounds {
/** Oldest first. */
added: AddedBackgroundImage[];
/**
* Keeps a file as a background. Rejects with `UnusableImage` for a file that
* cannot be used, and `RangeError` once the device keeps as many as it will.
*/
addBackground: (file: Blob) => Promise<string>;
removeBackground: (id: string) => Promise<void>;
}
/** What the camera menu needs to know about the pipeline itself. */
export interface BackgroundProcessing {
/**
* Whether the pipeline is still being built. Only ever true once a session,
* and only the first time an effect is chosen: building it fetches, compiles
* and sets up the segmenter, which on some browsers holds the page still for
* long enough that saying nothing looks like a crash.
*/
settling: boolean;
/**
* The camera track the pipeline is synced to: the pre-join preview's before
* joining, the published one after, and none while the camera is off. It
* draws with whatever effect is in force, and plainly when none has ever
* been chosen, so the menu can show someone how they look without building
* anything of its own.
*/
cameraTrack: LocalVideoTrack | null;
}
const BackgroundProcessingContext = createContext<
BackgroundProcessing | undefined
>(undefined);
export function useBackgroundProcessing(): BackgroundProcessing {
const value = use(BackgroundProcessingContext);
if (value === undefined)
throw new Error(
"useBackgroundProcessing must be used within a ProcessorProvider",
);
return value;
}
const AddedBackgroundsContext = createContext<AddedBackgrounds | undefined>(
undefined,
);
export function useAddedBackgrounds(): AddedBackgrounds {
const value = use(AddedBackgroundsContext);
if (value === undefined)
throw new Error(
"useAddedBackgrounds must be used within a ProcessorProvider",
);
return value;
}
const ProcessorContext = createContext<ProcessorState | undefined>(undefined);
export function useTrackProcessor(): ProcessorState {
@@ -97,6 +178,27 @@ export function applyProcessor(
/**
* Updates your video tracks to always use the given processor.
*/
/**
* The camera track the pipeline was last synced to.
*
* Module state rather than context, because one of the two paths that sync a
* track is not React at all: in a call the publisher does it, from an
* observable. There is one camera and one pipeline for the app's lifetime, so
* there is one track to know about; each path reports its own, and clears it
* only if it is still the one reported, so a lobby leaving cannot blank a call
* that has just started.
*/
const syncedCameraTrack$ = new BehaviorSubject<LocalVideoTrack | null>(null);
function reportCameraTrack(track: LocalVideoTrack | null): void {
if (syncedCameraTrack$.value !== track) syncedCameraTrack$.next(track);
}
function withdrawCameraTrack(track: LocalVideoTrack | null): void {
if (track !== null && syncedCameraTrack$.value === track)
syncedCameraTrack$.next(null);
}
export const trackProcessorSync = (
scope: ObservableScope,
videoTrack$: Behavior<LocalVideoTrack | null>,
@@ -109,6 +211,15 @@ export const trackProcessorSync = (
if (!videoTrack) return;
applyProcessor(videoTrack, processorState.processor);
});
let reported: LocalVideoTrack | null = null;
videoTrack$.pipe(scope.bind()).subscribe({
next: (videoTrack) => {
withdrawCameraTrack(reported);
reported = videoTrack;
reportCameraTrack(videoTrack);
},
complete: () => withdrawCameraTrack(reported),
});
};
export const useTrackProcessorSync = (
@@ -119,37 +230,167 @@ export const useTrackProcessorSync = (
if (!videoTrack) return;
applyProcessor(videoTrack, processor);
}, [processor, videoTrack]);
useEffect(() => {
reportCameraTrack(videoTrack);
return (): void => withdrawCameraTrack(videoTrack);
}, [videoTrack]);
};
interface Props {
children: JSX.Element;
}
function supportsBackgroundProcessors(): boolean {
return supportsBackgroundProcessorsLivekitSdk() && platform === "desktop";
/** Translates a chosen effect into the pipeline's own vocabulary. */
function switchOptionsFor(
effect: BackgroundEffect,
added: AddedBackgroundImage[],
): SwitchBackgroundProcessorOptions {
const withImage = (
imagePath: string | undefined,
): SwitchBackgroundProcessorOptions =>
// A background the device no longer has — removed, or storage cleared —
// leaves the user with no effect rather than a pipeline drawing nothing.
imagePath
? { mode: "virtual-background", imagePath }
: { mode: "disabled" };
switch (effect.kind) {
case "blur":
return { mode: "background-blur", blurRadius };
case "shipped":
return withImage(imagePathFor(effect.id));
case "added":
return withImage(added.find((a) => a.id === effect.id)?.url);
default:
return { mode: "disabled" };
}
}
export const ProcessorProvider: FC<Props> = ({ children }) => {
// The setting the user wants to have
const [blurActivated] = useSetting(backgroundBlurSettings);
const [effectRaw] = useSetting(backgroundEffectSetting);
const supported = useMemo(() => supportsBackgroundProcessors(), []);
const blur = useMemo(
() =>
new ProcessorWrapper(
new BlurBackgroundTransformer({ blurRadius: 15 }),
"background-blur",
),
// One pipeline for the lifetime of the app, so the pre-join preview and the
// call share it and its priming frame is spent before anything is published
// (D4).
const transformer = useMemo(
() => new BackgroundEffectTransformer({ backgroundDisabled: true }),
[],
);
const pipeline = useMemo(
() => new BackgroundProcessorWrapper(transformer, "background-effect"),
[transformer],
);
// The backgrounds this device keeps. Their URLs live as long as the provider
// does, and are replaced wholesale whenever the set changes: an object URL
// outlives the blob it names unless it is revoked, and the alternative is
// tracking one lifetime per image.
const store = useMemo(() => new BackgroundImageStore(), []);
const [added, setAdded] = useState<AddedBackgroundImage[]>([]);
const urls = useRef<string[]>([]);
const reread = useCallback(async (): Promise<void> => {
const kept = await store.list();
urls.current.forEach((url) => URL.revokeObjectURL(url));
urls.current = kept.map((background) =>
URL.createObjectURL(background.image),
);
setAdded(kept.map(({ id }, i) => ({ id, url: urls.current[i] })));
}, [store]);
useEffect(() => {
reread().catch((e) => logger.warn("Could not read added backgrounds", e));
const opened = urls;
return (): void => {
opened.current.forEach((url) => URL.revokeObjectURL(url));
opened.current = [];
};
}, [reread]);
const addBackground = useCallback(
async (file: Blob): Promise<string> => {
const kept = await store.add(file);
await reread();
return kept.id;
},
[store, reread],
);
const removeBackground = useCallback(
async (id: string): Promise<void> => {
await store.remove(id);
await reread();
},
[store, reread],
);
// D5: nothing is attached until an effect is first chosen, so a user who
// never chooses one pays neither the segmentation assets nor the time to
// initialise them. Once attached it stays attached, including at no effect,
// so a later change is a switch rather than a fresh attachment and spends no
// further priming frame.
const [attached, setAttached] = useState(
() => parseEffect(effectRaw).kind !== "none",
);
useEffect(() => {
if (parseEffect(effectRaw).kind !== "none") setAttached(true);
}, [effectRaw]);
// D2: switch the running pipeline in place rather than tearing it down, so
// the previous effect stays in force until the new one is live.
const options = useMemo(
() => switchOptionsFor(parseEffect(effectRaw), added),
[effectRaw, added],
);
useEffect(() => {
if (!supported || !attached) return;
pipeline
.switchTo(options)
.catch((e) => logger.warn("Failed to switch background effect", e));
}, [pipeline, supported, attached, options]);
// The wait runs from deciding to attach until a frame actually comes out,
// which is the only thing that marks the end of it: the promises resolve
// while the segmenter is still being built, and on the slow path the page
// then holds still for another twelve to fifteen seconds. Bounded by the
// frame rather than by a timer, so it cannot end early and claim to be ready.
const [producedAFrame, setProducedAFrame] = useState(false);
useEffect(() => {
transformer.onFirstFrame = (): void => setProducedAFrame(true);
return (): void => {
transformer.onFirstFrame = undefined;
};
}, [transformer]);
const settling = supported === true && attached && !producedAFrame;
// This is the actual state exposed through the context
const processorState = useMemo(
() => ({
supported,
processor: supported && blurActivated ? blur : undefined,
processor: supported && attached ? pipeline : undefined,
}),
[supported, blurActivated, blur],
[supported, attached, pipeline],
);
return <ProcessorContext value={processorState}>{children}</ProcessorContext>;
const addedBackgrounds = useMemo(
() => ({ added, addBackground, removeBackground }),
[added, addBackground, removeBackground],
);
const cameraTrack = useObservableEagerState(syncedCameraTrack$);
const backgroundProcessing = useMemo(
() => ({ settling, cameraTrack }),
[settling, cameraTrack],
);
return (
<ProcessorContext value={processorState}>
<AddedBackgroundsContext value={addedBackgrounds}>
<BackgroundProcessingContext value={backgroundProcessing}>
{children}
</BackgroundProcessingContext>
</AddedBackgroundsContext>
</ProcessorContext>
);
};
+92
View File
@@ -0,0 +1,92 @@
/*
Copyright 2026 Element Creations Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { maxAddedBackgrounds } from "./backgroundImages";
import { describe, expect, test } from "vitest";
import {
parseEffect,
serializeEffect,
shippedBackgrounds,
imagePathFor,
} from "./backgroundEffects";
describe("the chosen background effect", () => {
test("round-trips through its stored form", () => {
for (const effect of [
{ kind: "none" } as const,
{ kind: "blur" } as const,
{ kind: "shipped", id: shippedBackgrounds[0].id } as const,
{ kind: "added", id: "a-uuid" } as const,
])
expect(parseEffect(serializeEffect(effect))).toEqual(effect);
});
test("falls back to no effect when the stored form names nothing we ship", () => {
// A background removed between releases, or a value from a future one.
expect(parseEffect("image:a-background-we-no-longer-ship")).toEqual({
kind: "none",
});
expect(parseEffect("")).toEqual({ kind: "none" });
});
// The regression this guards is not hypothetical: the first stand-ins were
// the app's own overlay gradients, which have no opaque pixel in them at all,
// so as backgrounds their dark half was simply missing.
test("ships only opaque backgrounds", async () => {
const { readFile } = await import("node:fs/promises");
for (const background of shippedBackgrounds) {
const name = background.imagePath.split("/").pop()!.split("?")[0];
const bytes = await readFile(`src/graphics/${name}`);
// JPEG, which has no alpha channel to carry a hole in.
expect([bytes[0], bytes[1], bytes[2]]).toEqual([0xff, 0xd8, 0xff]);
}
});
// Three to a row, and the add tile goes at the limit. At four of their own
// the grid stopped at eight and left its last corner empty; the rule is that
// it is whole both at the limit and one short of it, with the add tile.
test("fills the grid at the limit and one short of it", () => {
const fixed = 2 + shippedBackgrounds.length; // no effect, blur, shipped
expect((fixed + maxAddedBackgrounds) % 3).toBe(0);
expect((fixed + maxAddedBackgrounds - 1 + 1) % 3).toBe(0);
});
test("gives every shipped background an image to draw", () => {
for (const background of shippedBackgrounds)
expect(imagePathFor(background.id)).toBeTruthy();
});
});
describe("the blur control in settings", () => {
// The control is a checkbox over a setting that holds more than two states,
// so what it shows and what it writes are both derived. This is that
// derivation, which is what keeps it from disagreeing with the camera menu.
const shows = (stored: string): boolean =>
parseEffect(stored).kind === "blur";
const writes = (on: boolean): string =>
serializeEffect(on ? { kind: "blur" } : { kind: "none" });
test("reads as off while an image background is in force", () => {
expect(shows(serializeEffect({ kind: "shipped", id: "indoor" }))).toBe(
false,
);
expect(shows(serializeEffect({ kind: "added", id: "a-uuid" }))).toBe(false);
});
test("reads as on while blur is in force", () => {
expect(shows("blur")).toBe(true);
});
test("replaces an image background when it is turned on", () => {
expect(writes(true)).toBe("blur");
});
test("leaves no effect behind when it is turned off", () => {
expect(writes(false)).toBe("none");
});
});
+82
View File
@@ -0,0 +1,82 @@
/*
Copyright 2024-2025 New Vector Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import indoorStandIn from "../graphics/background-indoor-standin.jpg?url";
import outdoorStandIn from "../graphics/background-outdoor-standin.jpg?url";
/** How much to blur, when the chosen effect is blur. */
export const blurRadius = 15;
export interface ShippedBackground {
id: string;
imagePath: string;
}
// EXPLORATION SHORTCUT (S1): the two shipped backgrounds, one indoor and one
// outdoor, do not exist yet. These stand-ins are the app's own gradients, laid
// on the canvas colour they are painted over and kept as JPEG.
//
// Not a detail: those gradients are overlay scrims, and measured, not one
// pixel in either is opaque — where they look dark they are transparent. Used
// directly as a background their dark half was simply absent, which is what
// showed on the slow path. Flattening them is what makes them a picture rather
// than a veil, and JPEG cannot carry transparency at all, so it cannot come
// back. Never port this.
export const shippedBackgrounds: ShippedBackground[] = [
{ id: "indoor", imagePath: indoorStandIn },
{ id: "outdoor", imagePath: outdoorStandIn },
];
export type BackgroundEffect =
| { kind: "none" }
| { kind: "blur" }
/** One of the images that come with the application. */
| { kind: "shipped"; id: string }
/** One the user added, kept on this device. */
| { kind: "added"; id: string };
export const noEffect: BackgroundEffect = { kind: "none" };
/**
* Parses the stored form of a chosen effect, falling back to no effect.
*
* A shipped background is checked against what we ship, so a stored one that a
* later release no longer carries falls back rather than leaving the user with
* an effect that cannot be drawn. An added one cannot be checked here — only
* the device knows what it keeps — so that check belongs where it is resolved.
*/
export function parseEffect(raw: string): BackgroundEffect {
if (raw === "blur") return { kind: "blur" };
if (raw.startsWith("image:")) {
const id = raw.slice("image:".length);
if (shippedBackgrounds.some((b) => b.id === id))
return { kind: "shipped", id };
}
if (raw.startsWith("added:")) {
const id = raw.slice("added:".length);
if (id) return { kind: "added", id };
}
return noEffect;
}
/** The stored form of a chosen effect. */
export function serializeEffect(effect: BackgroundEffect): string {
switch (effect.kind) {
case "blur":
return "blur";
case "shipped":
return `image:${effect.id}`;
case "added":
return `added:${effect.id}`;
default:
return "none";
}
}
export function imagePathFor(id: string): string | undefined {
return shippedBackgrounds.find((b) => b.id === id)?.imagePath;
}
+187
View File
@@ -0,0 +1,187 @@
/*
Copyright 2026 Element Creations Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { logger } from "matrix-js-sdk/lib/logger";
/**
* The most backgrounds of their own a device keeps.
*
* Five, because it fills the grid. With no effect, blur and the two shipped
* images that makes nine tiles at the limit, where the add tile goes; and one
* short of it, four and the add tile make nine too. At four the grid stopped
* at eight and left its last corner empty.
*/
export const maxAddedBackgrounds = 5;
/**
* The longest edge an added background is kept at.
*
* Not a display constraint: the pipeline crops and scales the image to the
* camera on every use, so a larger one buys nothing. It is what stops a
* forty-megapixel photograph sitting in storage and being decoded on every
* call.
*/
export const maxStoredEdge = 1920;
/** A background the user added, as it is kept on the device. */
export interface AddedBackground {
id: string;
image: Blob;
addedAt: number;
}
/** Why a file cannot be used as a background. */
export type UnusableReason = "not-an-image" | "animated" | "undecodable";
export class UnusableImage extends Error {
public constructor(public readonly reason: UnusableReason) {
super(reason);
this.name = "UnusableImage";
}
}
/**
* Whether the file holds more than one frame.
*
* Read from the file rather than guessed from its type: an animated WebP and a
* still one are both `image/webp`, and a `.png` may be an APNG. Where the
* browser cannot tell us, the types that are usually animated are refused
* rather than accepted and left to move behind someone's head.
*/
async function isAnimated(file: Blob): Promise<boolean> {
const decoder = (
globalThis as { ImageDecoder?: new (init: unknown) => unknown }
).ImageDecoder;
if (!decoder) return /gif|apng/.test(file.type);
try {
const d = new decoder({ data: await file.arrayBuffer(), type: file.type });
await (d as { completed: Promise<void> }).completed;
const track = (d as { tracks: { selectedTrack?: { frameCount: number } } })
.tracks.selectedTrack;
return (track?.frameCount ?? 1) > 1;
} catch (e) {
logger.debug("Could not read frame count, judging by type", e);
return /gif|apng/.test(file.type);
}
}
/**
* Prepares a file to be kept as a background: refuses what cannot be used, and
* reduces what is larger than we keep.
*
* Does not crop. The pipeline covers the camera's frame with whatever it is
* given, recomputed as the frame changes, so cropping here would bake in one
* shape and lose the rest of the picture for good.
*/
export async function prepareImage(file: Blob): Promise<Blob> {
if (!file.type.startsWith("image/")) throw new UnusableImage("not-an-image");
if (await isAnimated(file)) throw new UnusableImage("animated");
let bitmap: ImageBitmap;
try {
bitmap = await createImageBitmap(file);
} catch (e) {
logger.debug("Could not decode the chosen file", e);
throw new UnusableImage("undecodable");
}
try {
// Every picture goes through the canvas, not only the oversized ones: a
// background covers what is behind it, so it has to be opaque, and a
// picture with transparency in it — a logo, a screenshot with rounded
// corners — would otherwise be stored with its holes and drawn with them.
// Passing small files straight through is what kept them.
const longest = Math.max(bitmap.width, bitmap.height);
const scale = Math.min(1, maxStoredEdge / longest);
const canvas = new OffscreenCanvas(
Math.round(bitmap.width * scale),
Math.round(bitmap.height * scale),
);
const context = canvas.getContext("2d");
if (!context) throw new UnusableImage("undecodable");
// The ground the picture is laid on, so nothing it does not cover is a
// hole. Black rather than white: an unfilled corner reads as the frame's
// own edge rather than as a lamp.
context.fillStyle = "black";
context.fillRect(0, 0, canvas.width, canvas.height);
context.drawImage(bitmap, 0, 0, canvas.width, canvas.height);
return await canvas.convertToBlob({ type: "image/webp", quality: 0.9 });
} finally {
bitmap.close();
}
}
const DB_NAME = "element-call-background-images";
const STORE = "backgrounds";
/**
* The backgrounds a user added, kept on this device.
*
* They never leave it: there is no upload here and no identifier that would
* let anything fetch one.
*/
export class BackgroundImageStore {
private db?: Promise<IDBDatabase>;
/** The factory is injected so a test can hand over its own. */
public constructor(
private readonly indexedDB: IDBFactory = globalThis.indexedDB,
) {}
private async open(): Promise<IDBDatabase> {
return (this.db ??= new Promise((resolve, reject) => {
const request = this.indexedDB.open(DB_NAME, 1);
request.onupgradeneeded = (): void => {
request.result.createObjectStore(STORE, { keyPath: "id" });
};
request.onsuccess = (): void => resolve(request.result);
request.onerror = (): void => reject(request.error);
}));
}
private async run<T>(
mode: IDBTransactionMode,
body: (store: IDBObjectStore) => IDBRequest<T>,
): Promise<T> {
const db = await this.open();
return new Promise((resolve, reject) => {
const request = body(db.transaction(STORE, mode).objectStore(STORE));
request.onsuccess = (): void => resolve(request.result);
request.onerror = (): void => reject(request.error);
});
}
public async list(): Promise<AddedBackground[]> {
const all = await this.run<AddedBackground[]>("readonly", (s) =>
s.getAll(),
);
return all.sort((a, b) => a.addedAt - b.addedAt);
}
/**
* Keeps a file as a background, and answers with it.
*
* Throws {@link UnusableImage} for a file that cannot be used, and
* {@link RangeError} once the device is already keeping as many as it will.
*/
public async add(file: Blob): Promise<AddedBackground> {
const image = await prepareImage(file);
if ((await this.list()).length >= maxAddedBackgrounds)
throw new RangeError("Already keeping the most backgrounds we keep");
const background: AddedBackground = {
id: crypto.randomUUID(),
image,
addedAt: Date.now(),
};
await this.run("readwrite", (s) => s.add(background));
return background;
}
public async remove(id: string): Promise<void> {
await this.run("readwrite", (s) => s.delete(id));
}
}
+54
View File
@@ -0,0 +1,54 @@
/*
Copyright 2026 Element Creations Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import {
supportsBackgroundProcessors as supportsBackgroundProcessorsLivekitSdk,
supportsModernBackgroundProcessors,
} from "@livekit/track-processors";
/**
* Whether this device can run background effects at all.
*
* The one answer, for everything that asks. The controls and the pipeline used
* to work it out separately, and differently: the pipeline required a desktop
* while the controls asked only whether the browser had the APIs. A phone
* browser has them, so it was offered a choice the pipeline then refused to
* honour, and the video simply never changed — the worst of both, since it
* neither worked nor said why.
*
* Anything that offers a background effect must ask this, so that what is
* offered and what can be delivered cannot drift apart again.
*
* It no longer asks for a desktop. Measured, that test sorted devices by the
* wrong thing: a phone on the fast path held 99% of its frame rate and was
* refused, while a desktop on the slow one held 73% and was allowed. What the
* cost tracks is the path, which {@link usesFallbackProcessing} names — and a
* cost is something to tell someone about, not to decide for them, because the
* user is the only one who knows whether they would rather show the room.
*/
export function supportsBackgroundProcessors(): boolean {
return supportsBackgroundProcessorsLivekitSdk();
}
/**
* Whether effects here run the slow way.
*
* Browsers without `MediaStreamTrackProcessor` — Safari and Firefox, on every
* platform — fall back to drawing each frame through a canvas. Measured across
* four of them, that costs between a sixth and a quarter of the frame rate,
* phone or desktop, and on Safari it also blocks the page for twelve to
* fifteen seconds the first time the segmenter is built.
*
* Not a reason to withhold the feature: that cost ships today wherever the
* fallback runs, and it is a cost in smoothness while the feature is about
* privacy. It is a reason to say so.
*/
export function usesFallbackProcessing(): boolean {
return (
supportsBackgroundProcessors() && !supportsModernBackgroundProcessors()
);
}
+10 -8
View File
@@ -170,14 +170,16 @@ function createInCallView(args: CreateInCallViewArgs = {}): RenderResult & {
const renderResult = render(
<BrowserRouter>
<MediaDevicesContext value={mediaDevices}>
<ReactionsSenderProvider
vm={vm}
rtcSession={rtcSession.asMockedSession()}
>
<TooltipProvider>
<RoomContext value={livekitRoom}>{content}</RoomContext>
</TooltipProvider>
</ReactionsSenderProvider>
<ProcessorProvider>
<ReactionsSenderProvider
vm={vm}
rtcSession={rtcSession.asMockedSession()}
>
<TooltipProvider>
<RoomContext value={livekitRoom}>{content}</RoomContext>
</TooltipProvider>
</ReactionsSenderProvider>
</ProcessorProvider>
</MediaDevicesContext>
</BrowserRouter>,
);
+10
View File
@@ -27,6 +27,16 @@ vi.mock("../livekit/TrackProcessorContext", () => ({
processor: undefined,
}),
useTrackProcessorSync: (): void => {},
useBackgroundProcessing: (): { settling: boolean } => ({ settling: false }),
useAddedBackgrounds: (): {
added: [];
addBackground: () => Promise<void>;
removeBackground: () => Promise<void>;
} => ({
added: [],
addBackground: async (): Promise<void> => {},
removeBackground: async (): Promise<void> => {},
}),
}));
vi.mock("react-use-measure", () => ({
+85 -3
View File
@@ -5,8 +5,10 @@ SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { describe, expect, it, vi } from "vitest";
import { render } from "@testing-library/react";
import { afterEach, describe, expect, it, vi } from "vitest";
import { render, screen } from "@testing-library/react";
import userEvent from "@testing-library/user-event";
import { of } from "rxjs";
import { LeaveToHomeProvider } from "../LeaveToHomeContext";
import { TooltipProvider } from "@vector-im/compound-web";
import { type MatrixClient } from "matrix-js-sdk";
@@ -19,6 +21,7 @@ import {
import { LobbyView } from "./LobbyView";
import { E2eeType } from "../e2ee/e2eeType";
import { mockMediaDevices, mockMuteStates } from "../utils/test";
import { type MediaDevices } from "../state/MediaDevices";
import { MediaDevicesContext } from "../MediaDevicesContext";
import { type ProcessorState } from "../livekit/TrackProcessorContext";
import { type EncryptionSystem } from "../e2ee/sharedKeyManagement";
@@ -39,6 +42,16 @@ vi.mock("../livekit/TrackProcessorContext", () => ({
processor: undefined,
}),
useTrackProcessorSync: (): void => {},
useBackgroundProcessing: (): { settling: boolean } => ({ settling: false }),
useAddedBackgrounds: (): {
added: [];
addBackground: () => Promise<void>;
removeBackground: () => Promise<void>;
} => ({
added: [],
addBackground: async (): Promise<void> => {},
removeBackground: async (): Promise<void> => {},
}),
}));
vi.mock("react-use-measure", () => ({
@@ -77,9 +90,10 @@ function renderLobbyView(
props: Partial<Parameters<typeof LobbyView>[0]> = {},
withAppBar = false,
platform = "android",
devices: Partial<MediaDevices> = {},
): ReturnType<typeof render> {
platformMock.mockReturnValue(platform);
const mediaDevices = mockMediaDevices({});
const mediaDevices = mockMediaDevices(devices);
const muteStates = mockMuteStates();
const hideHeader = withAppBar ? true : false;
const lobbyView = (
@@ -178,3 +192,71 @@ describe("LobbyView", () => {
expect(await axe(container)).toHaveNoViolations();
});
});
describe("LobbyView microphone level", () => {
const realMediaDevices = Object.getOwnPropertyDescriptor(
navigator,
"mediaDevices",
);
afterEach(() => {
vi.unstubAllGlobals();
// Put navigator back, or every later test in the run inherits the stub.
if (realMediaDevices === undefined) {
Reflect.deleteProperty(navigator, "mediaDevices");
} else {
Object.defineProperty(navigator, "mediaDevices", realMediaDevices);
}
});
/** Just enough of the Web Audio and capture APIs for the meter to run. */
function stubAudioCapture(): void {
vi.stubGlobal(
"AudioContext",
class {
public readonly state = "running";
public createAnalyser(): object {
return {
fftSize: 1024,
getByteTimeDomainData: (): void => {},
};
}
public createMediaStreamSource(): object {
return { connect: (): void => {} };
}
public close(): void {}
},
);
// Only this property: replacing navigator wholesale drops the getters on
// its prototype, such as userAgent.
Object.defineProperty(navigator, "mediaDevices", {
configurable: true,
value: {
getUserMedia: vi.fn().mockResolvedValue({ getTracks: () => [] }),
},
});
}
it("shows the microphone level meter", async () => {
stubAudioCapture();
const user = userEvent.setup();
const { getByRole } = renderLobbyView({}, false, "desktop", {
requestDeviceNames: (): void => {},
audioInput: {
available$: of(
new Map([["mic1", { type: "name", name: "Microphone 1" }]]),
),
selected$: of({ id: "mic1" }),
select: (): void => {},
},
} as unknown as Partial<MediaDevices>);
// The meter lives with the microphone picker, which the pre-join screen
// reaches through the same chevron as a call in progress.
await user.click(getByRole("button", { name: "Microphone" }));
expect(
await screen.findByRole("meter", { name: "Microphone level" }),
).toBeInTheDocument();
});
});
+10 -3
View File
@@ -22,13 +22,14 @@ import { useHostBridge } from "../HostBridge";
import {
useSetting,
soundEffectVolume as soundEffectVolumeSetting,
backgroundBlur as backgroundBlurSetting,
backgroundEffect as backgroundEffectSetting,
developerMode,
} from "./settings";
import { PreferencesSettingsTab } from "./PreferencesSettingsTab";
import { Slider } from "../Slider";
import { DeviceSelection } from "./DeviceSelection";
import { useTrackProcessor } from "../livekit/TrackProcessorContext";
import { parseEffect, serializeEffect } from "../livekit/backgroundEffects";
import {
DeveloperSettingsTab,
type DeveloperSettingsSnapshot,
@@ -83,7 +84,13 @@ export const SettingsModal: FC<Props> = ({
const BlurCheckbox: React.FC = (): ReactNode => {
const { supported } = useTrackProcessor();
const [blurActive, setBlurActive] = useSetting(backgroundBlurSetting);
// The same setting the camera menu's Background effects grid writes, so
// the two controls cannot disagree. An image background reads here as blur
// off, and turning blur on replaces it.
const [effect, setEffect] = useSetting(backgroundEffectSetting);
const blurActive = parseEffect(effect).kind === "blur";
const setBlurActive = (on: boolean): void =>
setEffect(serializeEffect(on ? { kind: "blur" } : { kind: "none" }));
return (
<>
@@ -97,7 +104,7 @@ export const SettingsModal: FC<Props> = ({
supported ? "" : t("settings.blur_not_supported_by_browser")
}
type="checkbox"
checked={!!blurActive}
checked={blurActive}
onChange={(b): void => setBlurActive(b.target.checked)}
disabled={!supported}
/>
+12
View File
@@ -117,6 +117,18 @@ export const videoInput = new Setting<string | undefined>(
export const backgroundBlur = new Setting<boolean>("background-blur", false);
/**
* The background effect applied to the local camera: "none", "blur", or
* "image:<id>". See backgroundEffects.ts for the stored form.
*/
export const backgroundEffect = new Setting<string>(
"background-effect",
// Defaults to whatever blur the user had already chosen, so someone who
// turned blur on before this existed still has it afterwards. Their own
// choice replaces this the moment they make one.
backgroundBlur.getValue() ? "blur" : "none",
);
export const showHandRaisedTimer = new Setting<boolean>(
"hand-raised-show-timer",
false,
+147 -5
View File
@@ -5,8 +5,7 @@ SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { describe, expect, test, vi } from "vitest";
import { of } from "rxjs";
import { afterEach, describe, expect, test, vi } from "vitest";
const getPlatform = vi.hoisted(() => vi.fn(() => "desktop"));
vi.mock("../Platform", () => ({
@@ -15,9 +14,23 @@ vi.mock("../Platform", () => ({
},
isFirefox: (): boolean => false,
}));
vi.mock("@livekit/components-core", () => ({
createMediaDeviceObserver: () => of([]),
}));
// One observer per device kind, so a test can add and remove hardware.
const observers = vi.hoisted(
() => new Map<string, { next: (devices: unknown[]) => void }>(),
);
vi.mock("@livekit/components-core", async () => {
const { BehaviorSubject: Subject } = await import("rxjs");
return {
createMediaDeviceObserver: (kind: string) => {
let observer = observers.get(kind);
if (observer === undefined) {
observer = new Subject<unknown[]>([]);
observers.set(kind, observer);
}
return observer;
},
};
});
import { AudioOutput, MediaDevices } from "./MediaDevices";
import { AndroidControlledAudioOutput } from "./AndroidControlledAudioOutput";
@@ -57,3 +70,132 @@ describe("MediaDevices audio output", () => {
expect(devices.audioOutput).toBeInstanceOf(IOSControlledAudioOutput);
});
});
function device(deviceId: string, label: string, groupId = deviceId): object {
return { deviceId, label, groupId, kind: "audioinput" };
}
/** Replaces the hardware of one kind, as the browser would report it. */
function setDevices(kind: string, devices: object[]): void {
const observer = observers.get(kind);
if (observer === undefined) throw new Error(`nothing observing ${kind}`);
observer.next(devices);
}
function newMediaDevices(): MediaDevices {
return new MediaDevices(new ObservableScope(), {
controlledAudioDevices: false,
});
}
describe("MediaDevices selection", () => {
afterEach(() => {
localStorage.clear();
for (const kind of observers.keys()) setDevices(kind, []);
});
test("persists the selected device across sessions", () => {
const devices = newMediaDevices();
setDevices("audioinput", [
device("mic1", "Microphone 1"),
device("mic2", "Microphone 2"),
]);
devices.audioInput.select("mic2");
// A later call on the same machine reads the same stored preference.
expect(newMediaDevices().audioInput.selected$.value?.id).toBe("mic2");
});
test("updates the available devices when hardware changes", () => {
const devices = newMediaDevices();
setDevices("audioinput", [device("mic1", "Microphone 1")]);
expect([...devices.audioInput.available$.value.keys()]).toEqual(["mic1"]);
// A headset is plugged in.
setDevices("audioinput", [
device("mic1", "Microphone 1"),
device("mic2", "Headset"),
]);
expect([...devices.audioInput.available$.value.keys()]).toEqual([
"mic1",
"mic2",
]);
// And unplugged again.
setDevices("audioinput", [device("mic1", "Microphone 1")]);
expect([...devices.audioInput.available$.value.keys()]).toEqual(["mic1"]);
});
test("falls back to the default device when the selected device disappears", () => {
const devices = newMediaDevices();
setDevices("audioinput", [
device("mic1", "Microphone 1"),
device("mic2", "Headset"),
]);
devices.audioInput.select("mic2");
expect(devices.audioInput.selected$.value?.id).toBe("mic2");
// The headset is unplugged mid-call.
setDevices("audioinput", [device("mic1", "Microphone 1")]);
expect(devices.audioInput.selected$.value?.id).toBe("mic1");
});
test("falls back when the remembered device is absent", () => {
const devices = newMediaDevices();
setDevices("audioinput", [device("mic1", "Microphone 1")]);
// Remembered from a previous call, on hardware that is not here now.
devices.audioInput.select("a-device-from-last-time");
expect(devices.audioInput.selected$.value?.id).toBe("mic1");
});
test("falls back to numbered labels when labels are unavailable", () => {
const devices = newMediaDevices();
// The browser withholds names until permission has been granted.
setDevices("audioinput", [device("mic1", ""), device("mic2", "")]);
expect([...devices.audioInput.available$.value.values()]).toEqual([
{ type: "number", number: 1 },
{ type: "number", number: 2 },
]);
});
test("lists Default as a distinct entry", () => {
const devices = newMediaDevices();
setDevices("audiooutput", [device("spk1", "Speakers")]);
const available = devices.audioOutput.available$.value;
// Default follows the operating system and re-points when it changes, so
// it is its own choice rather than an alias for the device it resolves to.
// It carries no name of its own precisely because which device it resolves
// to is not knowable from here.
expect(available.get("spk1")).toEqual({ type: "name", name: "Speakers" });
expect(available.get("")).toEqual({ type: "default", name: null });
});
test("selecting one device kind leaves the others unchanged", () => {
const devices = newMediaDevices();
setDevices("audioinput", [
device("mic1", "Microphone 1"),
device("mic2", "Headset"),
]);
setDevices("audiooutput", [
device("spk1", "Speakers"),
device("spk2", "Headset"),
]);
setDevices("videoinput", [device("cam1", "Camera 1")]);
devices.audioOutput.select("spk2");
const audioInputBefore = devices.audioInput.selected$.value?.id;
const videoInputBefore = devices.videoInput.selected$.value?.id;
devices.audioInput.select("mic2");
expect(devices.audioOutput.selected$.value?.id).toBe("spk2");
expect(devices.videoInput.selected$.value?.id).toBe(videoInputBefore);
expect(audioInputBefore).not.toBe("mic2");
expect(devices.audioInput.selected$.value?.id).toBe("mic2");
});
});
+162
View File
@@ -0,0 +1,162 @@
/*
Copyright 2026 Element Creations Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { afterEach, describe, expect, test, vi } from "vitest";
import {
ATTACK_MS,
METER_SEGMENTS,
observeMicrophoneState$,
RELEASE_MS,
segmentsForVolume,
smoothVolume,
} from "./MicrophoneLevel";
import { restoreAudioCapture, stubAudioCapture } from "../utils/test";
describe("segmentsForVolume", () => {
test("shows nothing for silence", () => {
expect(segmentsForVolume(0)).toBe(0);
});
test("shows nothing for the hiss of a quiet room", () => {
// Without a noise floor these light the first segments permanently, which
// reads as "it can hear me" when nobody is speaking.
expect(segmentsForVolume(0.005)).toBe(0);
expect(segmentsForVolume(0.015)).toBe(0);
});
test("distinguishes quiet, normal and loud speech", () => {
const quiet = segmentsForVolume(0.06);
const normal = segmentsForVolume(0.2);
const loud = segmentsForVolume(0.8);
expect(quiet).toBeGreaterThan(0);
expect(normal).toBeGreaterThan(quiet);
expect(loud).toBeGreaterThan(normal);
});
test("moves the meter visibly for normal speech", () => {
// Ordinary speech should reach the middle of the meter, not scrape along
// the floor: a meter that barely moves reads as a broken microphone.
expect(segmentsForVolume(0.2)).toBeGreaterThanOrEqual(METER_SEGMENTS / 4);
});
test("never exceeds the meter", () => {
expect(segmentsForVolume(1)).toBe(METER_SEGMENTS);
expect(segmentsForVolume(4)).toBe(METER_SEGMENTS);
});
test("treats a missing reading as silence", () => {
expect(segmentsForVolume(NaN)).toBe(0);
expect(segmentsForVolume(-1)).toBe(0);
});
});
describe("smoothVolume", () => {
test("rises faster than it falls", () => {
const rise = smoothVolume(0, 1, 50);
const fall = 1 - smoothVolume(1, 0, 50);
expect(rise).toBeGreaterThan(fall);
});
test("registers a syllable as it starts", () => {
// Most of the way there within one attack time constant, so speech does
// not lag the speaker.
expect(smoothVolume(0, 1, ATTACK_MS)).toBeGreaterThan(0.6);
});
test("rides over the gaps between words", () => {
// A pause of a few tens of milliseconds should not collapse the meter, or
// it flickers rather than reading as a level.
expect(smoothVolume(1, 0, 30)).toBeGreaterThan(0.7);
// A real silence still brings it down.
expect(smoothVolume(1, 0, RELEASE_MS * 3)).toBeLessThan(0.1);
});
test("behaves the same whatever the frame rate", () => {
const oneStep = smoothVolume(0, 1, 32);
let twoSteps = smoothVolume(0, 1, 16);
twoSteps = smoothVolume(twoSteps, 1, 16);
expect(twoSteps).toBeCloseTo(oneStep, 5);
});
test("holds still when no time has passed", () => {
expect(smoothVolume(0.5, 1, 0)).toBe(0.5);
});
});
describe("observeMicrophoneState$", () => {
afterEach(restoreAudioCapture);
test("releases a capture the browser grants after nobody is watching", async () => {
const capture = stubAudioCapture();
const subscription = observeMicrophoneState$("mic1").subscribe();
// The user gives up on the permission prompt and closes the menu, and only
// then does the browser hand the microphone over.
subscription.unsubscribe();
capture.grant();
await vi.waitFor(() => expect(capture.track.stop).toHaveBeenCalled());
});
test("releases the capture and the audio context when the subscription ends", async () => {
const capture = stubAudioCapture();
const subscription = observeMicrophoneState$("mic1").subscribe();
capture.grant();
await vi.waitFor(() => expect(capture.contexts).toHaveLength(1));
subscription.unsubscribe();
expect(capture.track.stop).toHaveBeenCalled();
expect(capture.contexts[0].close).toHaveBeenCalled();
});
test("tells denied permission and a missing device apart", async () => {
for (const [name, expected] of [
["NotAllowedError", "permission-denied"],
["NotFoundError", "no-device"],
] as const) {
const capture = stubAudioCapture();
capture.getUserMedia.mockRejectedValue(named(new Error(name), name));
const seen: string[] = [];
const subscription = observeMicrophoneState$("mic1").subscribe((state) =>
seen.push(state.type),
);
await vi.waitFor(() => expect(seen).toContain(expected));
subscription.unsubscribe();
restoreAudioCapture();
}
});
test("says nothing on a frame that did not change the level", async () => {
const capture = stubAudioCapture();
let emissions = 0;
const subscription = observeMicrophoneState$("mic1").subscribe(
() => emissions++,
);
capture.grant();
await vi.waitFor(() => expect(emissions).toBe(1));
// The analyser is read every animation frame, but the meter has only
// METER_SEGMENTS steps: a steady signal must not redraw the meter.
capture.drawFrames(20);
expect(emissions).toBe(1);
subscription.unsubscribe();
});
});
/** An error with the `name` the browser would give it, not just a message. */
function named(error: Error, name: string): Error {
error.name = name;
return error;
}
+192
View File
@@ -0,0 +1,192 @@
/*
Copyright 2026 Element Creations Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { distinctUntilChanged, Observable } from "rxjs";
import { logger } from "matrix-js-sdk/lib/logger";
/**
* What the microphone selector can say about the input, beyond its level.
*
* Silence and a broken microphone look identical on a meter, so the states a
* user has to act on are named rather than drawn as a flat bar.
*/
export type MicrophoneState =
| { type: "level"; level: number }
| { type: "permission-denied" }
| { type: "no-device" };
/**
* Steps the meter is drawn in. The level is quantised to these.
*
* Enough of them that they sit close together across the width of the menu:
* the bars keep a fixed size, so too few leaves visible gaps between them.
*/
export const METER_SEGMENTS = 24;
/**
* Observes what a microphone is picking up, as the meter should show it.
*
* Subscribing opens the device and unsubscribing releases it, so it is held for
* exactly as long as something is watching — the menu, and not the call.
*
* Opens a capture of its own rather than reading the track the call already
* holds. That is a shortcut, recorded as S1 in the feature spec: the pre-join
* screen freezes its audio track to the device selected when it mounted, so a
* meter fed from that track could not follow the picker.
*
* The level says whether the microphone is picking anything up, which is not
* the same as whether the user is being heard: it keeps reading while muted,
* and the mute control is what says nothing is transmitted.
*/
export function observeMicrophoneState$(
deviceId: string | undefined,
): Observable<MicrophoneState> {
return new Observable<MicrophoneState>((subscriber) => {
let stream: MediaStream | undefined;
let context: AudioContext | undefined;
let frame: number | undefined;
// Safe to call more than once: unsubscribing runs it, and `start` runs it
// again for anything the browser handed over after that.
const release = (): void => {
if (frame !== undefined) cancelAnimationFrame(frame);
stream?.getTracks().forEach((track) => track.stop());
void context?.close();
frame = undefined;
stream = undefined;
context = undefined;
};
const start = async (): Promise<void> => {
stream = await navigator.mediaDevices.getUserMedia({
audio:
deviceId === undefined ? true : { deviceId: { exact: deviceId } },
});
// A permission prompt outlives the subscription that asked for it, so by
// now nobody may be watching — and `release` ran while `stream` was still
// undefined. Nothing will call it again, so release here or the device
// stays held, with the indicator lit and no meter on screen.
if (subscriber.closed) return release();
context = new AudioContext();
// Chrome starts the context suspended unless it was created during a
// gesture; opening the menu is one, but resume explicitly so the meter
// cannot silently sit at zero.
if (context.state === "suspended") await context.resume();
if (subscriber.closed) return release();
const analyser = context.createAnalyser();
analyser.fftSize = 1024;
context.createMediaStreamSource(stream).connect(analyser);
const samples = new Uint8Array(analyser.fftSize);
let displayed = 0;
let previousFrame = performance.now();
const read = (): void => {
analyser.getByteTimeDomainData(samples);
// Root mean square of the waveform around its centre, which is the
// loudness a listener perceives rather than the tallest spike.
let sum = 0;
for (const sample of samples) {
const centred = (sample - 128) / 128;
sum += centred * centred;
}
const now = performance.now();
displayed = smoothVolume(
displayed,
Math.sqrt(sum / samples.length),
now - previousFrame,
);
previousFrame = now;
subscriber.next({ type: "level", level: segmentsForVolume(displayed) });
frame = requestAnimationFrame(read);
};
read();
};
start().catch((e: unknown) => subscriber.next(stateForFailure(e)));
return release;
}).pipe(
// Read every animation frame, but quantised to a whole number of segments,
// so most frames say nothing new and should not reach React.
distinctUntilChanged(
(a, b) =>
a.type === b.type &&
(a.type !== "level" || b.type !== "level" || a.level === b.level),
),
);
}
/** What a failure to open the microphone means for the person using it. */
function stateForFailure(e: unknown): MicrophoneState {
const name = e instanceof Error ? e.name : "";
if (name === "NotAllowedError" || name === "SecurityError")
return { type: "permission-denied" };
if (name === "NotFoundError" || name === "OverconstrainedError")
return { type: "no-device" };
logger.error("Could not read the microphone level", e);
return { type: "no-device" };
}
/**
* Loudness below which the microphone is treated as picking up nothing.
*
* A quiet room is never digitally silent, and without a floor that hiss lights
* the first segments permanently — which reads as "it can hear me" when nobody
* is speaking.
*/
const NOISE_FLOOR = 0.02;
/**
* Quantises a 0..1 volume to a whole number of meter segments.
*
* Exported for the tests: the mapping from loudness to segments is the part
* that decides whether quiet, normal and loud speech look different.
*/
export function segmentsForVolume(volume: number): number {
if (!Number.isFinite(volume) || volume <= NOISE_FLOOR) return 0;
// Volume arrives as amplitude, where speech occupies a small part of the top
// of the range. A square root spreads that out, so ordinary speech moves the
// meter through its middle rather than barely leaving the floor.
const aboveFloor = (Math.min(volume, 1) - NOISE_FLOOR) / (1 - NOISE_FLOOR);
return Math.min(
METER_SEGMENTS,
Math.ceil(Math.sqrt(aboveFloor) * METER_SEGMENTS),
);
}
/**
* How quickly the meter follows a rise in loudness, as a time constant in
* milliseconds. Short, so a syllable registers the moment it starts.
*/
export const ATTACK_MS = 50;
/**
* How quickly the meter follows a fall. Longer than the attack: speech is full
* of gaps a few tens of milliseconds long, and a meter that tracked them
* exactly would flicker rather than read as a level.
*/
export const RELEASE_MS = 120;
/**
* Moves a displayed level towards a new reading, fast upwards and slowly
* downwards.
*
* Framed in elapsed time rather than frames, so the meter behaves the same on a
* 60Hz and a 120Hz display, and does not jump when a frame is dropped.
*/
export function smoothVolume(
displayed: number,
reading: number,
elapsedMs: number,
): number {
if (elapsedMs <= 0) return displayed;
const timeConstant = reading > displayed ? ATTACK_MS : RELEASE_MS;
const towards = 1 - Math.exp(-elapsedMs / timeConstant);
return displayed + (reading - displayed) * towards;
}
+90
View File
@@ -8,6 +8,7 @@ import { map, type Observable, of, type SchedulerLike } from "rxjs";
import { type RunHelpers, TestScheduler } from "rxjs/testing";
import {
expect,
type Mock,
type MockedObject,
type MockInstance,
onTestFinished,
@@ -590,3 +591,92 @@ export class MockConnection extends Connection {
public async start(): Promise<void> {}
public async stop(): Promise<void> {}
}
export interface StubbedCapture {
getUserMedia: Mock;
/** Hands the microphone over, as the browser does once permission is given. */
grant: () => void;
track: { stop: Mock };
contexts: { close: Mock }[];
/** Runs the animation frames the level meter reads on, in order. */
drawFrames: (count: number) => void;
}
/**
* Stubs just enough of the capture and Web Audio APIs for a microphone level to
* be read, with the grant held back so a test decides when — or whether — it
* lands, and with animation frames driven by hand rather than by a clock.
*
* Call {@link restoreAudioCapture} afterwards, or every later test in the run
* inherits the stub.
*/
export function stubAudioCapture(): StubbedCapture {
const track = { stop: vi.fn() };
const contexts: { close: Mock }[] = [];
let grant = (): void => {};
const granted = new Promise<MediaStream>((resolve) => {
grant = (): void =>
resolve({ getTracks: () => [track] } as unknown as MediaStream);
});
const frames: FrameRequestCallback[] = [];
vi.stubGlobal("requestAnimationFrame", (callback: FrameRequestCallback) => {
frames.push(callback);
return frames.length;
});
vi.stubGlobal("cancelAnimationFrame", vi.fn());
vi.stubGlobal(
"AudioContext",
class {
public readonly state = "running";
public readonly close = vi.fn();
public constructor() {
contexts.push(this);
}
public createAnalyser(): object {
return {
fftSize: 1024,
// Digital silence, which is the midpoint of the range and not zero:
// a buffer left at zero reads as a full-scale waveform.
getByteTimeDomainData: (samples: Uint8Array): void => {
samples.fill(128);
},
};
}
public createMediaStreamSource(): object {
return { connect: (): void => {} };
}
},
);
// Only this property: replacing navigator wholesale drops the getters on its
// prototype, such as userAgent.
Object.defineProperty(navigator, "mediaDevices", {
configurable: true,
value: { getUserMedia: vi.fn().mockReturnValue(granted) },
});
return {
getUserMedia: navigator.mediaDevices.getUserMedia as unknown as Mock,
grant: () => grant(),
track,
contexts,
drawFrames: (count): void => {
for (let i = 0; i < count; i++) frames.shift()?.(i);
},
};
}
const realMediaDevices = Object.getOwnPropertyDescriptor(
navigator,
"mediaDevices",
);
/** Undoes {@link stubAudioCapture}. */
export function restoreAudioCapture(): void {
vi.unstubAllGlobals();
if (realMediaDevices === undefined) {
Reflect.deleteProperty(navigator, "mediaDevices");
} else {
Object.defineProperty(navigator, "mediaDevices", realMediaDevices);
}
}
+45 -4
View File
@@ -24,6 +24,45 @@ import react, { reactCompilerPreset } from "@vitejs/plugin-react";
import { realpathSync } from "fs";
import * as fs from "node:fs";
/**
* Keeps a plugin's transform off MediaPipe's WASM loader.
*
* nodePolyfills rewrites any module mentioning `process`, and that loader does.
* It is a classic script whose whole job is to set `self.ModuleFactory`;
* rewritten as an ES module it sets nothing on `self`, and background effects
* then fail at runtime with "ModuleFactory not set". A production build is
* unaffected, because there the loader is emitted as a static asset and never
* passes through a transform, which is why this only bites in dev.
*/
function exceptMediaPipeWasm(plugins: PluginOption): PluginOption {
const isLoader = (id: string): boolean =>
id.includes("tasks-vision") && id.includes("wasm");
return (Array.isArray(plugins) ? plugins : [plugins]).map((plugin) => {
if (
!plugin ||
typeof plugin !== "object" ||
!("transform" in plugin) ||
typeof plugin.transform !== "function"
)
return plugin;
const transform = plugin.transform;
return {
...plugin,
transform(this: unknown, code: string, id: string, options: unknown) {
if (isLoader(id)) return null;
return (
transform as (
this: unknown,
c: string,
i: string,
o: unknown,
) => unknown
).call(this, code, id, options);
},
} as PluginOption;
});
}
export const vitePluginsConfig = ({
mode,
html = true,
@@ -42,10 +81,12 @@ export const vitePluginsConfig = ({
}),
react(),
wasm(),
nodePolyfills({
// Enables the 'events' module, which is required by the matrix-js-sdk
include: ["events"],
}),
exceptMediaPipeWasm(
nodePolyfills({
// Enables the 'events' module, which is required by the matrix-js-sdk
include: ["events"],
}),
),
svgrPlugin({
svgrOptions: {
// This enables ref forwarding on SVGR components, which is needed, for