Compare commits

...
Author SHA1 Message Date
Timo K. 84d596d2e4 Add playwright test for widget with sdk rtc data sending 2026-11-13 17:45:58 +01:00
Timo K. 0475f2b2b0 cleanup widget sdk
- dont use the callViewModel anymore (bundle size 6 MB to 12 MB, and
   the entry from 2.1 MB to 1.1 MB)
 - better docs and readme information
2026-11-13 15:50:06 +01:00
Johannes Marbach 413b795459 Merge pull request #4452 from element-hq/johannes/agents-fixes
Compact AGENTS.md and split hand-off out into a skill
2026-11-13 14:22:39 +01:00
renovate[bot] e9c1e25542 Update ghcr.io/element-hq/element-web:develop Docker digest to 2901117 (#4453)
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
2026-11-13 09:06:50 +00:00
Timo f88d8d5830 Merge pull request #4456 from element-hq/toger5/getRTCTrasnport_and_getDomain
Use non deprecated non unstable names for getRTCTransports.
2026-11-12 16:12:02 +01:00
Timo K. 6c66f9a3e7 use non deprecated non unstable names for getRTCTransports.
Also remove getDomain where possible
2026-11-12 16:01:11 +01:00
Johannes Marbach 99f48ff987 Merge pull request #4454 from element-hq/johannes/lk-jwt-service-0.10.0
Bump lk-jwt-service to 0.10.0
2026-11-12 13:01:09 +01:00
Timo 6846c8dd3e Merge pull request #4431 from element-hq/toger5/component-no-store-worker
Keep the store worker out of the component package
2026-11-12 12:27:47 +01:00
Johannes Marbach 28ee707ace Bump lk-jwt-service to 0.10.0 2026-11-12 08:29:26 +01:00
Timo K.andClaude Fable 5.1 3100c1c410 Cover the standalone client's initialisation and session restore
Neither had a test before the move, so the moved lines showed up as
uncovered. The client initialisation is tested through a fake client
and mocked stores: which store it syncs into with and without
IndexedDB, with and without the worker, the fallback to memory when
the IndexedDB store cannot start, clearing the stores only when not
restoring, carrying on without crypto, and counting as ready only
after a sync from the network rather than the one restored from the
store. The session restore is tested for no stored session, a
restored one, one the homeserver no longer knows, and any other
failure.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-11 19:24:21 +01:00
Timo K.andClaude Fable 5.1 fc1e91f54c Keep the store worker out of the component package
The package carried the standalone app's IndexedDB store worker: 11 MB
with a copy of the Matrix SDK and its crypto wasm, referenced by an
absolute path no host could serve, and never used by the component,
which takes the host's client. The bundler emits the worker file as
soon as it loads the `?worker` import, used or not, and the import sat
in the Matrix utilities the component reaches for the room URLs.

The client initialisation moves into a module of its own, with the
worker import, which only the standalone login, registration and
session restore paths import. ClientProvider takes the session loader
through a `loadClient` prop instead of importing it itself, since even
a dynamic import keeps a module in the package: the standalone app
passes a lazy `initSPA` from App.tsx, the component passes nothing,
and given neither a client nor a loader no one is signed in.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-11 19:24:21 +01:00
Johannes Marbach d5f72f464d Compact AGENTS.md and split hand-off out into a skill 2026-11-11 16:33:46 +01:00
renovate[bot] 96949a57da Update ghcr.io/element-hq/element-web:develop Docker digest to 8dc3767 (#4447)
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
2026-11-11 14:47:22 +00:00
Johannes Marbach 2e05ef77a0 Merge pull request #4441 from element-hq/johannes/flakes
Try to fix flaky background tests
2026-11-11 15:37:31 +01:00
Johannes Marbach 73826ec4f9 Reformat 2026-11-11 15:29:03 +01:00
Johannes Marbach 1135c02e6f Merge branch 'main' into johannes/flakes 2026-11-11 13:53:30 +01:00
Johannes MarbachandTimo 9dc425b5f8 Apply batched suggestions from code review
Improve comments

Co-authored-by: Timo <16718859+toger5@users.noreply.github.com>
2026-11-11 13:52:05 +01:00
Johannes Marbach c1b94b62dd Merge pull request #4434 from element-hq/johannes/intent
Stabilise m.call.intent
2026-11-11 11:02:22 +01:00
Johannes Marbach fa6efd9dad Merge branch 'main' into johannes/intent 2026-11-11 10:45:51 +01:00
Johannes Marbach 6906404a64 Pin ubuntu image to 24.04 because 26.04 breaks Playwright, renovate should update it 2026-11-11 10:20:44 +01:00
Johannes Marbach 98e8343d0b Stabilise m.call.intent 2026-11-11 10:10:33 +01:00
Timo 88659d6498 Merge pull request #4444 from element-hq/toger5/better-component-readme
Shorten component README
2026-11-10 16:56:40 +01:00
Timo K. 704cbd65a6 Shorten component readme
- At the end there were not that many unique facts in there. It now
   only lists the relevant bits without much text around it.
2026-11-10 14:31:12 +01:00
Timo 5f59283b4a Merge pull request #4428 from element-hq/toger5/component-assets-as-files
Ship the component's assets as files beside the module
2026-11-10 14:18:30 +01:00
Johannes Marbach 07dc08cd1c Merge pull request #4440 from element-hq/johannes/embedded
Make one site use the embedded build in playwright tests
2026-11-10 07:46:51 +01:00
Johannes Marbach 60d6029184 Delint 2026-11-10 07:38:23 +01:00
Johannes Marbach f41bc2589a Merge branch 'main' into johannes/embedded 2026-11-10 07:36:15 +01:00
Johannes Marbach ceeb26b271 Remove test.slow() 2026-11-10 07:30:57 +01:00
Johannes Marbach e822f967c1 Merge pull request #4442 from element-hq/johannes/shard
Shard playwright tests
2026-11-10 07:22:10 +01:00
renovate[bot] ec46b04b71 Update ghcr.io/element-hq/element-web:develop Docker digest to 37d88b4 (#4443)
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
2026-11-10 03:24:30 +00:00
Johannes Marbach 4f3dc9af75 Shard playwright tests 2026-11-09 13:00:52 +01:00
Johannes Marbach cfd8312e46 Try to fix flaky background tests 2026-11-09 12:20:00 +01:00
renovate[bot] 38edc47f64 Update ghcr.io/element-hq/element-web:develop Docker digest to 0c39f7b (#4439)
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
2026-11-09 09:29:45 +00:00
Johannes Marbach ab4f0f0f77 Make one site use the embedded build in playwright tests 2026-11-09 10:05:04 +01:00
renovate[bot] 9411a56afc Update ghcr.io/element-hq/element-web:develop Docker digest to f30be05 (#4438)
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
2026-11-08 01:46:48 +00:00
Timo K.andClaude Fable 5.1 a5374791e4 Resolve asset references from the chunk that holds them
The runtime path was `./` plus the asset's path from the package root,
which is right for every chunk the build emits today, all of which sit
at that root, and wrong for one emitted under a directory. Vite names
the referencing chunk, so the path is now computed relative to it.

The sound loader kept a branch for sounds inlined as data URLs, with a
comment saying the component build produces them. No build does any
more, and no host would inline a sound of this size, so the branch and
its tests go.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-07 16:06:05 +01:00
renovate[bot] a0601876d8 Update ghcr.io/element-hq/element-web:develop Docker digest to eada111 (#4435)
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
2026-11-07 01:07:59 +00:00
fkwp eef86fc06e Merge pull request #4292 from element-hq/spec/background-effects/12-unbuildable-pipeline
Offer no effects for the session where the pipeline cannot be built
2026-11-06 18:27:12 +01:00
Timo K. cd5c367ce9 fix race in processor setup vs camera start 2026-11-06 16:54:42 +01:00
Timo K.andClaude Fable 5.1 190902eec0 Test the session where the pipeline cannot be built
- The effects state's timeline through preparing, settling and attached,
  and that a failed trial build withholds the effects for the session while
  keeping the choice.
- The footer offers nothing once the pipeline fails to build, and passes
  on the wait as the state's single wait field.
- The call leaves the camera out of the join while the pipeline is
  prepared, publishes it once that is known, and turns on a camera turned
  off before the processor was chosen; the lobby opens no camera meanwhile.
  Both view models are driven by one state subject.
- The transformer's trial build answers whether a segmenter can be made.
- End to end, a browser whose pipeline cannot be built sees the effects
  withheld.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-06 16:54:42 +01:00
Timo K.andClaude Opus 5.5 88f748ec21 Offer no effects for the session where the pipeline cannot be built
- A segmenter that can't be built, with the model unreachable or a GPU
  its delegate refuses, took the camera down with the pipeline: the SDK
  fails the track it was attaching to, and the lobby muted camera and
  microphone, on every load while the choice was kept.
- One is now built on trial when an effect is first wanted, before the
  pipeline is attached to anything, on the CPU where the GPU refuses.
  Where it can't be built, the options are disabled with the existing
  notice for the rest of the session. The choice is kept: a browser
  update may fix what failed.
- The view model takes that verdict from the pipeline's state rather
  than
  asking the SDK, so both say the same, and it no longer builds a WebGL
  context of its own to ask.
- Until the trial is over, the lobby opens no camera and the call makes
  no
  camera track, so neither starts without the effect.

Rebased onto main, where the Publisher is gone and the call view model
publishes the camera through the SDK:
- CallViewModel takes videoProcessorPreparing$. While it is true, the
  camera's mute handler waits before making a track, and the join leaves
  the camera out of its requests (initialPublishRequests); the view
  model
  publishes it once it is known whether the pipeline builds. The
  Publisher's test became CallViewModel tests.

Cherry-picked from spec/background-effects/12-unbuildable-pipeline onto
toger5/background-effect/camera-preview, where the camera menu's preview
takes the camera from the footer's caller rather than from a track
synced
on the background-effects state. The footer factories take the
pipeline's
state again, as this commit needs its verdict and not only the settling
flag; the synced camera and the preview's wrapper type are not brought
along.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-06 16:54:42 +01:00
Timo 9639effd28 Merge pull request #4423 from element-hq/toger5/background-effect/camera-preview
Camera Preview in video mute button context menu
2026-11-06 15:01:06 +00:00
Timo K. f2f67019b4 Show the self preview in the video mute button context menu 2026-11-06 15:35:04 +01:00
Timo K. 51fac690f4 Hand the camera track to the footer view model 2026-11-06 15:35:04 +01:00
Timo K.andClaude Fable 5.1 797a86815b Ship the component's assets as files beside the module
The component module was 18 MB, of which 15 MB was MediaPipe's runtime
as a base64 data URL, with the segmentation model, every sound and the
background pictures inlined the same way: a library build inlines every
asset unless its import says `no-inline`. Every host downloaded all of
it before the first call, whether or not a background effect was ever
switched on, and a host with a Content Security Policy without
`script-src data:` could not run effects at all, since MediaPipe
injects its loader as a script tag.

The imports now say `no-inline`, and the build renders each reference
as `new URL("./assets/…", import.meta.url)`, which webpack 5, Vite and
Rollup follow as an asset of the host's own; the `./` is spelled out
because a bare `assets/…` is read as a package name. The module is
2.3 MB. The runtime is fetched when an effect is first switched on and
cached on its own. MediaPipe's files come through the package's
`exports` rather than a path into node_modules.

MediaPipe's JavaScript (160 kB) stays in the module: LiveKit's track
processors import it statically, so a dynamic import moves nothing.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-06 14:39:25 +01:00
Johannes Marbach 1ae57e9647 Merge pull request #4420 from element-hq/johannes/mark-release-announcements-viewed
Mark known release announcements as viewed
2026-11-06 13:50:54 +01:00
Timo 4b05ab503e Merge pull request #4416 from element-hq/toger5/fix-flickering-device-id-mismatch
fix camera restart due to device id mismatch
2026-11-06 13:45:04 +01:00
Johannes Marbach ea977cecd9 Mark known release announcements as viewed 2026-11-06 11:31:01 +01:00
Johannes Marbach aadbda39b1 Merge pull request #4405 from element-hq/renovate/npm-minimatch-vulnerability
Update dependency minimatch to v11 [SECURITY]
2026-11-06 11:12:09 +01:00
renovate[bot] 6b56c7e683 Update dependency minimatch to v11 [SECURITY] 2026-11-06 09:35:24 +00:00
Timo K. 11e143c8ab fix camera restart due to device id mismatch 2026-11-05 13:21:42 +01:00
123 changed files with 3054 additions and 1225 deletions
+35
View File
@@ -0,0 +1,35 @@
---
name: handoff
description: Implement the first cut of a task and stop for a direction check before the quality pass.
argument-hint: <the task>
disable-model-invocation: true
---
Task: $ARGUMENTS
Stop after the first implementation and hand it back before the quality pass.
## Before the hand-off
- Run the narrowest check that rules out an obviously broken hand-off: the test
file for the touched code, `tsc` on its own, a single story. Not the full
`pnpm lint`, not the whole suite.
- If a wider check is needed before feedback can be useful, keep it to the touched
code and say why it was needed.
- Leave wider refactors, extra tests and documentation polish out. They are
follow-up work, not part of the first hand-off.
## The hand-off
Stop and ask whether the direction is right. The message says:
- what was built and where, in a few lines,
- which check ran and what it returned, verbatim if it failed,
- what was left out on purpose,
- the decisions that could have gone another way.
Do not continue past this point on your own.
## After the user confirms
The skill ends here. Continue with the quality pass as usual.
+1 -1
View File
@@ -14,7 +14,7 @@ permissions:
jobs:
prevent-blocked:
name: Prevent blocked
runs-on: ubuntu-latest
runs-on: ubuntu-24.04
permissions:
pull-requests: read
steps:
@@ -17,7 +17,7 @@ env:
jobs:
build_and_deploy:
name: Build & publish docker
runs-on: ubuntu-latest
runs-on: ubuntu-24.04
permissions:
contents: write # required to upload release asset
packages: write # needed for publishing packages to GHCR
+2 -2
View File
@@ -7,7 +7,7 @@ on:
type: string
package:
type: string # This would ideally be a `choice` type, but that isn't supported yet
description: The package type to be built. Must be one of 'full', 'embedded', or 'sdk'
description: The package type to be built. Must be one of 'full', 'embedded', or 'widget-rtc-sdk'
required: true
build_mode:
type: string # This would ideally be a `choice` type, but that isn't supported yet
@@ -29,7 +29,7 @@ on:
jobs:
build:
name: Build Element Call
runs-on: ubuntu-latest
runs-on: ubuntu-24.04
steps:
- name: Checkout code
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
+6 -8
View File
@@ -25,7 +25,7 @@ jobs:
# Deploy main branch to call.element.dev after build completes
if: github.ref == 'refs/heads/main'
needs: build_full_element_call
runs-on: ubuntu-latest
runs-on: ubuntu-24.04
steps:
- name: Deploy to call.element.dev
uses: actions/github-script@d7906e4ad0b1822421a7e6a35d5ca353c962f410 # v6
@@ -72,14 +72,12 @@ jobs:
SENTRY_AUTH_TOKEN: ${{ secrets.SENTRY_AUTH_TOKEN }}
CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
build_sdk_element_call:
# The 'sdk' package is the CallViewModel bundle that predates the sdk/
# package (build:sdk:production, sdk-target-based-on-call-view-model/), not
# the package itself. The name is shared with pr-deploy.yaml, which always
# runs from main, so it stays until the bundle's consumers have moved over.
build_widget_rtc_sdk:
# The widget wrapper around the sdk/ package (widget-rtc-sdk/), not the
# package itself. Deployed by pr-deploy.yaml as the pr<N>-widget-rtc-sdk preview.
uses: ./.github/workflows/build-element-call.yaml
with:
package: sdk
package: widget-rtc-sdk
vite_app_version: ${{ github.event.release.tag_name || github.sha }}
build_mode: ${{ github.event_name == 'pull_request' && contains(github.event.pull_request.labels.*.name, 'development build') && 'development' || 'production' }}
secrets:
@@ -92,7 +90,7 @@ jobs:
build_storybook:
name: Build Storybook
if: contains(github.event.pull_request.labels.*.name, 'storybook build')
runs-on: ubuntu-latest
runs-on: ubuntu-24.04
steps:
- name: Checkout code
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
+1 -1
View File
@@ -13,7 +13,7 @@ permissions:
jobs:
pr-changelog-label:
runs-on: ubuntu-latest
runs-on: ubuntu-24.04
steps:
- uses: yogevbd/enforce-label-action@a3c219da6b8fa73f6ba62b68ff09c469b3a1c024 # 2.2.2
with:
+1 -1
View File
@@ -13,7 +13,7 @@ jobs:
name: Review
# Only run on labelling and skip forks.
if: github.event.label.name == 'Claude-Review' && github.event.pull_request.head.repo.full_name == github.repository
runs-on: ubuntu-latest
runs-on: ubuntu-24.04
steps:
- name: Checkout code
uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0
+14 -12
View File
@@ -17,7 +17,7 @@ on:
package:
required: true
type: string
description: Which package to deploy - 'full', 'embedded', 'sdk', or 'storybook'
description: Which package to deploy - 'full', 'embedded', 'widget-rtc-sdk', or 'storybook'
artifact_run_id:
required: false
type: string
@@ -32,7 +32,7 @@ on:
jobs:
deploy:
runs-on: ubuntu-latest
runs-on: ubuntu-24.04
permissions:
deployments: write
environment: Netlify
@@ -64,25 +64,27 @@ jobs:
run: curl -s https://raw.githubusercontent.com/element-hq/element-call/main/config/netlify_redirects > webapp/_redirects
- name: Add config file
# Not needed for storybook deployments
if: inputs.package != 'storybook'
run: |
if [ "${INPUTS_PACKAGE}" = "full" ]; then
curl -s "https://raw.githubusercontent.com/${INPUTS_PR_HEAD_FULL_NAME}/${INPUTS_PR_HEAD_REF}/config/config_netlify_preview.json" > webapp/config.json
else
curl -s "https://raw.githubusercontent.com/${INPUTS_PR_HEAD_FULL_NAME}/${INPUTS_PR_HEAD_REF}/config/config_netlify_preview_sdk.json" > webapp/config.json
fi
# Only the app reads a config.json beside it. A widget gets everything
# about the homeserver from its host.
if: inputs.package == 'full'
run: curl -s "https://raw.githubusercontent.com/${INPUTS_PR_HEAD_FULL_NAME}/${INPUTS_PR_HEAD_REF}/config/config_netlify_preview.json" > webapp/config.json
env:
INPUTS_PACKAGE: ${{ inputs.package }}
INPUTS_PR_HEAD_FULL_NAME: ${{ inputs.pr_head_full_name }}
INPUTS_PR_HEAD_REF: ${{ inputs.pr_head_ref }}
- name: Add headers file
# A widget page on another origin imports the widget RTC SDK bundle as a
# module, which the browser only allows with CORS
if: inputs.package == 'widget-rtc-sdk'
run: |
printf '/*\n Access-Control-Allow-Origin: *\n' > webapp/_headers
- name: ☁️ Deploy to Netlify
id: netlify
uses: nwtgck/actions-netlify@4cbaf4c08f1a7bfa537d6113472ef4424e4eb654 # v3.0
with:
publish-dir: webapp
deploy-message: "Deploy from GitHub Actions"
alias: ${{ inputs.package == 'sdk' && format('pr{0}-sdk', inputs.pr_number) || inputs.package == 'storybook' && format('pr{0}-storybook', inputs.pr_number) || format('pr{0}', inputs.pr_number) }}
alias: ${{ inputs.package == 'widget-rtc-sdk' && format('pr{0}-widget-rtc-sdk', inputs.pr_number) || inputs.package == 'storybook' && format('pr{0}-storybook', inputs.pr_number) || format('pr{0}', inputs.pr_number) }}
env:
NETLIFY_AUTH_TOKEN: ${{ secrets.NETLIFY_AUTH_TOKEN }}
NETLIFY_SITE_ID: ${{ secrets.NETLIFY_SITE_ID }}
+1 -1
View File
@@ -10,7 +10,7 @@ on:
jobs:
lint:
name: Lint, format & type check
runs-on: ubuntu-latest
runs-on: ubuntu-24.04
steps:
- name: Checkout code
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
+3 -3
View File
@@ -17,7 +17,7 @@ jobs:
github.event.workflow_run.conclusion == 'success' &&
github.event.workflow_run.event == 'pull_request' &&
github.event.workflow_run.head_repository.full_name == github.repository
runs-on: ubuntu-latest
runs-on: ubuntu-24.04
outputs:
pr_number: ${{ steps.prdetails.outputs.pr_id }}
pr_data_json: ${{ steps.prdetails.outputs.data }}
@@ -46,7 +46,7 @@ jobs:
NETLIFY_AUTH_TOKEN: ${{ secrets.NETLIFY_AUTH_TOKEN }}
NETLIFY_SITE_ID: ${{ secrets.NETLIFY_SITE_ID }}
netlify-sdk:
netlify-widget-rtc-sdk:
needs: prdetails
permissions:
deployments: write
@@ -57,7 +57,7 @@ jobs:
pr_head_full_name: ${{ github.event.workflow_run.head_repository.full_name }}
pr_head_ref: ${{ needs.prdetails.outputs.pr_data_json && fromJSON(needs.prdetails.outputs.pr_data_json).head.ref }}
deployment_ref: ${{ needs.prdetails.outputs.pr_data_json && fromJSON(needs.prdetails.outputs.pr_data_json).head.sha || github.ref || github.head_ref }}
package: sdk
package: widget-rtc-sdk
secrets:
ELEMENT_BOT_TOKEN: ${{ secrets.ELEMENT_BOT_TOKEN }}
NETLIFY_AUTH_TOKEN: ${{ secrets.NETLIFY_AUTH_TOKEN }}
+3 -3
View File
@@ -17,7 +17,7 @@ permissions: {}
jobs:
versioning:
name: Versioning
runs-on: ubuntu-latest
runs-on: ubuntu-24.04
outputs:
DRY_RUN: ${{ steps.dry_run.outputs.DRY_RUN }}
PREFIXED_VERSION: ${{ steps.prefixed_version.outputs.PREFIXED_VERSION }}
@@ -59,7 +59,7 @@ jobs:
publish_npm:
needs: versioning
name: Publish NPM
runs-on: ubuntu-latest
runs-on: ubuntu-24.04
outputs:
ARTIFACT_VERSION: ${{ steps.artifact_version.outputs.ARTIFACT_VERSION }}
permissions:
@@ -111,7 +111,7 @@ jobs:
needs: [versioning, publish_npm]
if: ${{ needs.versioning.outputs.DRY_RUN == 'false' }}
name: Update release notes
runs-on: ubuntu-latest
runs-on: ubuntu-24.04
# Shared with publish-embedded-packages.yaml, which appends to the same release: two
# appends at once would each write back the body without the other's section
concurrency:
@@ -14,7 +14,7 @@ on:
jobs:
versioning:
name: Versioning
runs-on: ubuntu-latest
runs-on: ubuntu-24.04
outputs:
DRY_RUN: ${{ steps.dry_run.outputs.DRY_RUN }}
PREFIXED_VERSION: ${{ steps.prefixed_version.outputs.PREFIXED_VERSION }}
@@ -76,7 +76,7 @@ jobs:
needs: [build_element_call, versioning]
if: always()
name: Publish tarball
runs-on: ubuntu-latest
runs-on: ubuntu-24.04
permissions:
contents: write # required to upload release asset
steps:
@@ -107,7 +107,7 @@ jobs:
needs: [build_element_call, versioning]
if: always()
name: Publish NPM
runs-on: ubuntu-latest
runs-on: ubuntu-24.04
outputs:
ARTIFACT_VERSION: ${{ steps.artifact_version.outputs.ARTIFACT_VERSION }}
permissions:
@@ -152,7 +152,7 @@ jobs:
needs: [build_element_call, versioning]
if: always()
name: Publish Android AAR
runs-on: ubuntu-latest
runs-on: ubuntu-24.04
outputs:
ARTIFACT_VERSION: ${{ steps.artifact_version.outputs.ARTIFACT_VERSION }}
permissions:
@@ -212,7 +212,7 @@ jobs:
needs: [build_element_call, versioning]
if: always()
name: Publish SwiftPM Library
runs-on: ubuntu-latest
runs-on: ubuntu-24.04
outputs:
ARTIFACT_VERSION: ${{ steps.artifact_version.outputs.ARTIFACT_VERSION }}
permissions:
@@ -282,7 +282,7 @@ jobs:
needs: [versioning, publish_npm, publish_android, publish_ios]
if: always()
name: Update release notes
runs-on: ubuntu-latest
runs-on: ubuntu-24.04
# Shared with publish-component.yaml, which appends to the same release: two appends
# at once would each write back the body without the other's section
concurrency:
+2 -2
View File
@@ -23,7 +23,7 @@ jobs:
needs: build_element_call
if: always()
name: Publish tarball
runs-on: ubuntu-latest
runs-on: ubuntu-24.04
permissions:
contents: write # required to upload release asset
packages: write
@@ -68,7 +68,7 @@ jobs:
add_docker_release_note:
needs: publish_docker
name: Add docker release note
runs-on: ubuntu-latest
runs-on: ubuntu-24.04
steps:
- name: Add release note
uses: softprops/action-gh-release@3bb12739c298aeb8a4eeaf626c5b8d85266b0e65 # v2
+68 -4
View File
@@ -25,7 +25,7 @@ on:
jobs:
vitest:
name: Run unit tests
runs-on: ubuntu-latest
runs-on: ubuntu-24.04
steps:
- name: Checkout code
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
@@ -71,10 +71,27 @@ jobs:
flags: unittests
fail_ci_if_error: true
playwright:
name: Run end-to-end tests
name: Run end-to-end tests (${{ matrix.part }})
if: ${{ !inputs.disable_e2e_tests }}
timeout-minutes: 60
runs-on: ubuntu-latest
runs-on: ubuntu-24.04
# We manually shard the tests across runners. Playwright's own shards cut
# the list of tests into equal counts in file order, which would leave the
# widget suite, the slowest tests by far, together at the end.
strategy:
fail-fast: false
matrix:
include:
- part: chromium-1
args: --project=chromium --shard=1/2
- part: chromium-2
args: --project=chromium --shard=2/2
- part: widget-1
args: --project=widget --shard=1/2
- part: widget-2
args: --project=widget --shard=2/2
- part: firefox-mobile
args: --project=firefox --project=mobile
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
with:
@@ -107,17 +124,64 @@ jobs:
run: pnpm playwright install --with-deps
- name: Run backend components
run: |
# Make sure the directory for the embedded build exists as Docker might
# create it with wrong access rights otherwise.
mkdir -p embedded/web/dist
docker compose -f docker-compose-dev.yml -f docker-compose-playwright.yml pull
docker compose -f docker-compose-dev.yml -f docker-compose-playwright.yml up -d
docker ps
- name: Run Playwright tests
env:
USE_DOCKER: 1
run: pnpm exec playwright test
ARGS: ${{ matrix.args }}
# Unquoted on purpose: the part's arguments are several words
run: pnpm exec playwright test $ARGS
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
if: ${{ !cancelled() }}
with:
name: blob-report-${{ matrix.part }}
path: blob-report
if-no-files-found: error
# Only read by the merge below.
retention-days: 1
e2e_report:
# Combines the blobs into a single HTML report and blocks PRs on failure.
name: Run end-to-end tests
needs: playwright
if: ${{ always() && !inputs.disable_e2e_tests }}
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1
with:
# Inside a called workflow, github.repository is the caller so we need to make sure to still check out the EC repo here.
repository: ${{ inputs.matrix-js-sdk-sha && 'element-hq/element-call' || github.repository }}
persist-credentials: false
- name: Enable Corepack
run: corepack enable
- uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4
with:
cache: "pnpm"
node-version-file: ".node-version"
- name: Install dependencies
run: scripts/layered.sh --ignore-pnpmfile
env:
PR_HEAD_REPO_OWNER: ${{ github.event.pull_request.head.repo.owner.login }}
JS_SDK_GITHUB_BASE_REF: ${{ inputs.matrix-js-sdk-sha }}
- name: Download the parts' reports
uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4
with:
pattern: blob-report-*
path: all-blob-reports
merge-multiple: true
- name: Merge into one HTML report
run: pnpm exec playwright merge-reports --reporter html ./all-blob-reports
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: html-report
path: playwright-report
if-no-files-found: error
retention-days: 4
- name: Fail on a failed part
if: ${{ needs.playwright.result != 'success' }}
run: exit 1
+1 -1
View File
@@ -7,7 +7,7 @@ on:
jobs:
download:
runs-on: ubuntu-latest
runs-on: ubuntu-24.04
permissions:
contents: read
+1 -1
View File
@@ -8,7 +8,7 @@ on:
jobs:
upload:
runs-on: ubuntu-latest
runs-on: ubuntu-24.04
permissions:
contents: read
+1 -1
View File
@@ -10,7 +10,7 @@ permissions: {}
jobs:
zizmor:
name: Run zizmor 🌈
runs-on: ubuntu-latest
runs-on: ubuntu-24.04
permissions:
security-events: write # Required for upload-sarif (used by zizmor-action) to upload SARIF files.
steps:
+5
View File
@@ -2,6 +2,7 @@ node_modules
.DS_Store
.env
dist
/dist-widget-rtc-sdk-example
dist-ssr
*.local
*.bkp
@@ -44,3 +45,7 @@ storybook-static
# Nested checkout made by scripts/layered.sh
/matrix-js-sdk/
# Claude
/.claude/*
!/.claude/skills/
+25 -4
View File
@@ -16,19 +16,35 @@ host's page. It is the MatrixRTC reference implementation.
`@media`, global `i18next` or the `widget` global — take it from a provider.
- The change works standalone, as a widget and as a component. Say what you checked.
- Every gate below is green before you push.
- Hand off after the first implementation, before the quality pass. Then commit on
the user's word where the setup allows it, or hand them the message.
## Read before you
| … | … |
| -------------------------------- | ---------------------------------------------------------- |
| start a task of any size | [docs/agents/workflow.md](docs/agents/workflow.md) |
| touch call logic or a view model | [docs/agents/architecture.md](docs/agents/architecture.md) |
| write any code | [docs/agents/code-style.md](docs/agents/code-style.md) |
| write a test, story or e2e spec | [docs/agents/testing.md](docs/agents/testing.md) |
| open a PR | [CONTRIBUTING.md](CONTRIBUTING.md) |
| implement from a feature spec | [FEATURES_SPEC/AGENTS.md](FEATURES_SPEC/AGENTS.md) |
## Scratch files live in `agent-workspace/`
Git-ignored. One kebab-case subfolder per task, matching the branch topic:
`agent-workspace/<slug>/`.
| File | Purpose |
| --------------------------- | -------------------------------------------------------------------------------- |
| `plan.md` | The concrete approach — files, symbols, edit order — once direction is confirmed |
| `commit-msg.txt` | `git commit -F agent-workspace/<slug>/commit-msg.txt` |
| `pr-body.md` | `gh pr create --body-file agent-workspace/<slug>/pr-body.md` |
| `implementation-summary.md` | What was built, decisions, trade-offs |
| `NN-prompt.md` | Raw prompt text, numbered, when worth keeping |
- Nothing here is durable. The folder is ignored, so anything worth keeping is
promoted into the PR body or a doc before the task ends.
- Never write scratch files to the repo root. Stray `load_test_summary.md` and
`config.json_` files are what this folder prevents.
- Start `pr-body.md` from `.github/PULL_REQUEST_TEMPLATE.md` and fill every
section.
## Gates
@@ -42,6 +58,11 @@ pnpm i18n:check
`knip` fails on dead code; a file deliberately inert ahead of its consumer goes in
`knip.ts` `ignoreFiles` with a reason.
Ready means every gate is green and the change is covered at the layers
[testing.md](docs/agents/testing.md) asks for. Read the diff against
[code-style.md](docs/agents/code-style.md) first. Re-run the whole checklist after
any fix.
## A good PR
- One slice, under ~400 changed lines, green on its own, linked to its issue.
+48 -99
View File
@@ -221,119 +221,68 @@ See also:
- [Developing with linked packages](./docs/linking.md)
#### Element Call as a component (experimental)
#### Element Call as a component
Element Call can also be embedded directly into another React application
rather than being loaded in an iframe as a widget. `pnpm build:component`
builds it as a library into `component/dist` (the bundle, its stylesheet and
type declarations), and
Element Call can be rendered inside another React app instead of an iframe.
The host supplies its own Matrix client, and the call stays inside the element
it is mounted in.
```sh
pnpm dev:component
```
serves a harness on port 3001 that stands in for such an application: it signs
in twice against the development backend and shows two calls side by side, in
resizable boxes, with page furniture of its own around them. Use it to see how
Element Call behaves when it does not own the page — the size it is given,
whether it stays inside its container, and what it says to its host, which is
logged along the bottom. The harness is served with the same development
certificate as the app, so unless the development CA is trusted, the browser
needs a certificate exception for `https://localhost:3001` as well (see the
note under [Backend](#backend)). It reads the same `public/config.json` as
`pnpm dev` if one exists, and runs with Element Call's defaults otherwise.
The call lays itself out for the size of the element it is mounted in, not the
window: a host that shrinks the container to a corner of its page gets the
picture-in-picture layout, just as a host that shrank the whole iframe used to.
The breakpoints in the stylesheets the component uses are
`@container element-call` queries against its root element for the same reason;
for the standalone app the root is the page, so they mean what the media queries
they replaced did. (The standalone-only views, such as the home and login pages,
still use plain media queries, since the component never shows them.)
The component's stylesheet is confined to the element it is mounted in: the
build rewrites every selector so that it matches only Element Call's root or
what is inside it, with `html`, `body` and `:root` standing for that root (see
`component/build/scopeStylesToRoot.ts`). A host's own page keeps its styles,
and Element Call brings its own fonts and design tokens along.
The component speaks every language the app does. English is bundled in; the
other locales are split into chunks the host's bundler loads the first time
they are needed. It starts in the browser's language, and follows the host's
own language setting through the `language` prop (`supportedLanguages` lists
the tags it accepts, and anything else falls back to its base language or to
English). The `theme` prop works the same way and takes the same values as the
widget's `theme` URL parameter: `light`, `dark`, `light-high-contrast` or
`dark-high-contrast`. Both can change while a call is running without
disturbing it.
A host must call and await `initializeElementCall(config)` once before
rendering the component: it loads the `Intl` polyfills, applies the
deployment-wide `config.json`-style configuration and sets up translations.
The component itself takes the host's `client` and the `roomId` to call in, an
`intent` saying what the user asked for (which decides whether to show the
lobby, ring, and so on), an optional `config` overriding what the intent
implies, and an optional `hostBridge` through which Element Call tells the host
that the user has joined or hung up, that it wants to stay on screen, and so
on. The host makes its own requests (`join`, `hangUp`, `setDeviceMute`) through
the handle exposed on `ref`. The full API is documented in the type declarations
(`component/index.tsx` and `component/host.ts`).
A few things differ from the widget on purpose: the component draws a solid
background rather than a gradient unless told otherwise, never offers to edit
the user's profile (the account is the host's), scopes its keyboard shortcuts to
its own root element so that several instances can share a page, and only shows
its own post-call and error screens when the host has not supplied a `close()`
callback; with one, it asks the host to unmount it instead. The
[global JS controls](./docs/controls.md) on `window` are unchanged and remain
page-wide, so with several instances on one page they apply to all of them.
Each release publishes the package to npm, with the same version as the release:
##### Install
```sh
npm install @element-hq/element-call-component
```
To use an unreleased commit, a host can instead install it as a git dependency
on the `component` directory of this repository,
Releases publish the package with the same version as Element Call. For an
unreleased commit, depend on this repository directly:
```json
"@element-hq/element-call-component": "github:element-hq/element-call#main&path:/component"
```
whose `prepare` script runs the build on install. That build needs pnpm (via
Corepack) on the host's machine, runs a full `pnpm install` of this repository,
since it inherits the `--max-old-space-size` setting of
the app build; the host's pnpm also has to allow it to run at all
(`allowBuilds` in its `pnpm-workspace.yaml`). Note that `component/` is a pnpm
project of its own for this reason, so pnpm commands run from inside that
directory target it rather than the repository; run them from the repository
root. The host imports the component from
`@element-hq/element-call-component` and the stylesheet from
`@element-hq/element-call-component/style.css`, and has to provide `react`,
`react-dom`, `matrix-js-sdk`, `@vector-im/compound-web` and
`@vector-im/compound-design-tokens` itself, since the bundle leaves them
external.
Compound's stylesheet are the host's to load too: the component shares
the host's copy of Compound, (required for example for tooltips)
This builds on install, so the host needs pnpm (via Corepack) and must allow
the build (`allowBuilds` in its `pnpm-workspace.yaml`).
The component is large, and a host will usually load it lazily, only once a
call is shown. Everything a host needs in order to talk about a call before
then is also exported
from `@element-hq/element-call-component/api` (a small entry point that does not
load the component).
##### Use
- props and the types they are made of
- the handle
- the host bridge
- `UserIntent`
- `HeaderStyle`
- `BackgroundStyle` enums
- `configurationForIntent`, the defaults each intent implies
Import from it wherever a value such as
`BackgroundStyle.Solid` is needed on a path that must stay light; the main
entry point re-exports all of it too.
1. Provide the externals yourself: `react`, `react-dom`, `matrix-js-sdk`,
`@vector-im/compound-web`, `@vector-im/compound-design-tokens`, and
Compound's stylesheet.
2. Import `@element-hq/element-call-component/style.css`.
3. Await `initializeElementCall(config)` once before rendering.
4. Render the component with `client`, `roomId` and an `intent`; optionally
pass `config`, `hostBridge`, `language` and `theme` (both can change
mid-call). Control it through the `ref` handle (`join`, `hangUp`,
`setDeviceMute`).
Load the component lazily. Types, enums and `configurationForIntent` are
available without it from `@element-hq/element-call-component/api`. The full
API is in `component/index.tsx`, `component/api.ts` and `component/host.ts`.
##### How it differs from the widget
- It lays out for its container, not the window: a small container gets the
picture-in-picture layout.
- Its styles and keyboard shortcuts are scoped to its root, so the host page is
untouched and several instances can share a page. The
[global JS controls](./docs/controls.md) stay page-wide.
- It draws a solid background by default and never offers profile editing.
- If the host bridge implements `close()`, it asks to be unmounted instead of
showing its own post-call and error screens.
- Sounds, images and the MediaPipe runtime are separate files the host's
bundler copies (webpack 5, Vite, Rollup); MediaPipe is fetched only when a
background effect is first used.
##### Develop
- `pnpm build:component` builds the library into `component/dist`.
- `pnpm dev:component` serves a test host on `https://localhost:3001`: two
calls side by side in resizable boxes, with host messages logged at the
bottom. It uses the development backend, `public/config.json` if present,
and needs the same certificate exception as `pnpm dev` (see
[Backend](#backend)).
- Run pnpm commands from the repository root; `component/` is its own pnpm
project.
### Backend
+5 -3
View File
@@ -26,8 +26,7 @@
"features": {
"feature_element_call_video_rooms": true,
"feature_video_rooms": true,
"feature_group_calls": true,
"feature_release_announcement": false
"feature_group_calls": true
},
"default_federate": true,
"default_theme": "light",
@@ -40,7 +39,10 @@
},
"setting_defaults": {
"breadcrumbs": true,
"feature_group_calls": true
"feature_group_calls": true,
"releaseAnnouncementData": {
"room_list_section": true
}
},
"jitsi": {
"preferred_domain": "meet.element.io"
+5 -3
View File
@@ -26,8 +26,7 @@
"features": {
"feature_element_call_video_rooms": true,
"feature_video_rooms": true,
"feature_group_calls": true,
"feature_release_announcement": false
"feature_group_calls": true
},
"default_federate": true,
"default_theme": "light",
@@ -40,7 +39,10 @@
},
"setting_defaults": {
"breadcrumbs": true,
"feature_group_calls": true
"feature_group_calls": true,
"releaseAnnouncementData": {
"room_list_section": true
}
},
"jitsi": {
"preferred_domain": "meet.element.io"
-13
View File
@@ -1,13 +0,0 @@
{
"default_server_config": {
"m.homeserver": {
"base_url": "https://call-unstable.ems.host",
"server_name": "call-unstable.ems.host"
}
},
"ssla": "https://static.element.io/legal/element-software-and-services-license-agreement-uk-1.pdf",
"matrix_rtc_session": {
"membership_event_expiry_ms": 180000000,
"network_error_retry_ms": 100
}
}
+2 -2
View File
@@ -3,7 +3,7 @@ networks:
services:
auth-service:
image: ghcr.io/element-hq/lk-jwt-service:0.8.0
image: ghcr.io/element-hq/lk-jwt-service:0.10.0
pull_policy: always
hostname: auth-server
environment:
@@ -33,7 +33,7 @@ services:
- ecbackend
auth-service-1:
image: ghcr.io/element-hq/lk-jwt-service:0.8.0
image: ghcr.io/element-hq/lk-jwt-service:0.10.0
pull_policy: always
hostname: auth-server-1
environment:
+7 -2
View File
@@ -19,7 +19,12 @@ services:
- ./backend/playwright_homeserver-othersite.yaml:/data/cfg/homeserver.yaml:Z
element-web:
# Pin to a SHA so that upstream cannot break our tests. Renovate handles regular updates.
image: ghcr.io/element-hq/element-web:develop@sha256:e063472b6bdf6d0bfa1d43dfa4ef7effee1efd2c9684daee665d1bedfe746a47
image: ghcr.io/element-hq/element-web:develop@sha256:fd1c291a228374aff127ceb6eea3c20291b233623bbbd96dcae53eca679fd9d7
element-web-1:
# Pin to a SHA so that upstream cannot break our tests. Renovate handles regular updates.
image: ghcr.io/element-hq/element-web:develop@sha256:e063472b6bdf6d0bfa1d43dfa4ef7effee1efd2c9684daee665d1bedfe746a47
image: ghcr.io/element-hq/element-web:develop@sha256:fd1c291a228374aff127ceb6eea3c20291b233623bbbd96dcae53eca679fd9d7
volumes:
# The other site opens the embedded package under test in place of the
# copy its image bundles (the first site opens the dev server's app.
# Built by the Playwright run (see build.setup.ts).
- ./embedded/web/dist:/app/widgets/element-call:ro,Z
+5 -1
View File
@@ -14,7 +14,11 @@ This folder contains documentation for setup, usage, and development of Element
Entry point: [AGENTS.md](../AGENTS.md).
- [Agent workflow](./agents/workflow.md)
- [Architecture](./agents/architecture.md)
- [Code style](./agents/code-style.md)
- [Testing](./agents/testing.md)
Opt-in skills in [`.claude/skills/`](../.claude/skills/):
- `/handoff <task>` — get the first implementation back for a direction check
before lint, the full suite and polish.
+5 -5
View File
@@ -76,11 +76,11 @@ Code that builds in only one is a bug.
Packaged exactly like the component (own pnpm project, run pnpm from the repo
root, types from `sdk/tsconfig.build.json`, externals checked by
`pnpm lint:externals`). Interface only so far; the design is `sdk-plan.md`.
- `build:sdk-target-based-on-call-view-model` — the SDK demo that predates the
package, one bundle of the `CallViewModel`; entry
`sdk-target-based-on-call-view-model/main.ts`. CI's `sdk` package
(`build:sdk:production` / `build:sdk:development`, the Netlify `-sdk` preview)
is still this bundle, not `build:sdk`.
- `build:widget-rtc-sdk` — a single bundle for widget pages that wraps the
`sdk/` package and the widget API, nothing of `src/`; entry
`widget-rtc-sdk/main.ts`. CI builds it as the `widget-rtc-sdk` package
(`build:widget-rtc-sdk:production` / `:development`) and deploys it as the
`pr<N>-widget-rtc-sdk` Netlify preview.
- `build:component` — `@element-hq/element-call-component`, sources in `component/`
(its own pnpm project; run pnpm from the repo root). Host API is in the README;
`pnpm lint:externals` rejects an import of a `react` / `react-dom` /
+7
View File
@@ -34,6 +34,10 @@ translations. The browser is Playwright's, so a fresh clone needs
## End-to-end
The Element Web instance `HOST1` opens the app the dev server serves. The instance
`HOST2` (the other site) opens the embedded package as a host ships it. This package
is built by the `build-embedded` setup project and mounted in `docker-compose-playwright.yml`.
- `playwright/*.spec.ts` — standalone, on Chromium and Firefox.
- `playwright/widget/` — widget mode, where cross-context bugs surface.
- `playwright/component/` — the component in a host page, via the harness on port
@@ -48,10 +52,13 @@ Test what a user observes: the peer sees the change, it survives a reconnect, it
right after a reload. Reuse `playwright/spa-helpers.ts`,
`playwright/widget/test-helpers.ts`, `playwright/fixtures/`.
To mark new release announcements as viewed, add them in `backend/ew.test*.config.json`.
```sh
pnpm test # unit + storybook
pnpm backend # Synapse + LiveKit, required for e2e
pnpm test:playwright # or :open
pnpm dev:component # component harness, port 3001
pnpm dev:sdk # SDK harness, port 3002
pnpm dev:widget-rtc-sdk # widget RTC SDK example widget, port 3003
```
-50
View File
@@ -1,50 +0,0 @@
# Agent workflow
## Scratch files live in `agent-workspace/`
Git-ignored. One kebab-case subfolder per task, matching the branch topic:
`agent-workspace/<slug>/`.
| File | Purpose |
| --------------------------- | -------------------------------------------------------------------------------- |
| `plan.md` | The concrete approach — files, symbols, edit order — once direction is confirmed |
| `commit-msg.txt` | `git commit -F agent-workspace/<slug>/commit-msg.txt` |
| `pr-body.md` | `gh pr create --body-file agent-workspace/<slug>/pr-body.md` |
| `implementation-summary.md` | What was built, decisions, trade-offs |
| `NN-prompt.md` | Raw prompt text, numbered, when worth keeping |
- Nothing here is durable. The folder is ignored, so anything worth keeping is
promoted into the PR body, a doc, or a feature spec before the task ends.
- Never write scratch files to the repo root. Stray `load_test_summary.md` and
`config.json_` files are what this folder prevents.
- Start `pr-body.md` from `.github/PULL_REQUEST_TEMPLATE.md` and fill every
section.
- Commit subjects are plain imperative English. No conventional-commits prefixes.
- A `plan.md` and a feature spec sit at different altitudes; neither replaces the
other. A spec in `FEATURES_SPEC/` is durable and deliberately abstract —
behaviour, decisions, acceptance criteria — so the feature can be rebuilt against
a `main` nobody has seen yet. A plan is one slice of it landing on today's
`main`: the paths, symbols and edit order the spec's decisions and criteria must
not name. `FEATURES_SPEC/AGENTS.md` wins wherever the two genuinely overlap.
## Hand off before the quality pass
- Implement the change, run the narrowest check that rules out an obviously broken
handoff, then stop and ask whether the direction is right.
- Full `pnpm lint`, the whole suite, coverage and benchmarks come after the
direction is confirmed.
- Wider refactors, extra tests and documentation polish are follow-up work, not
part of the first handoff.
- If a check is needed before feedback, keep it to the touched code and say why.
## Commit and PR readiness
- Commit once the user confirms direction, or asks for one. Not before. Where the
setup has no git identity, a sandbox included, write the message to
`commit-msg.txt` and hand it over instead.
- Ready means every gate in [AGENTS.md](../../AGENTS.md#gates) is green and the
change is covered at the layers [testing.md](./testing.md) asks for. Read the diff
against [code-style.md](./code-style.md) first.
- Re-run the whole checklist after any fix. Only on green, either way.
- Once a human has started reviewing, fix forward — never force-push a
regeneration over a review in progress.
+2 -1
View File
@@ -14,7 +14,8 @@ export default {
"vite-embedded.config.ts",
"vite-sdk.config.ts",
"vite-sdk-dev.config.ts",
"vite-sdk-target-based-on-call-view-model.config.ts",
"vite-widget-rtc-sdk.config.ts",
"vite-widget-rtc-sdk-dev.config.ts",
"vite-component.config.ts",
"vite-component-dev.config.ts",
],
+5 -4
View File
@@ -8,6 +8,7 @@
"dev:embedded": "vite --config vite-embedded.config.js",
"dev:component": "vite --config vite-component-dev.config.ts",
"dev:sdk": "vite --config vite-sdk-dev.config.ts",
"dev:widget-rtc-sdk": "vite --config vite-widget-rtc-sdk-dev.config.ts",
"build": "pnpm build:full",
"build:full": "node --max-old-space-size=16384 node_modules/vite/bin/vite.js build",
"build:full:production": "pnpm build:full",
@@ -18,9 +19,9 @@
"build:sdk": "pnpm build:sdk:js && pnpm build:sdk:types",
"build:sdk:js": "pnpm build:full --config vite-sdk.config.js",
"build:sdk:types": "tsc -p sdk/tsconfig.build.json",
"build:sdk-target-based-on-call-view-model": "pnpm build:full --config vite-sdk-target-based-on-call-view-model.config.js",
"build:sdk:production": "pnpm build:sdk-target-based-on-call-view-model",
"build:sdk:development": "pnpm build:sdk-target-based-on-call-view-model --mode development",
"build:widget-rtc-sdk": "pnpm build:full --config vite-widget-rtc-sdk.config.js",
"build:widget-rtc-sdk:production": "pnpm build:widget-rtc-sdk",
"build:widget-rtc-sdk:development": "pnpm build:widget-rtc-sdk --mode development",
"build:component": "pnpm build:component:js && pnpm build:component:types",
"build:component:js": "pnpm build:full --config vite-component.config.js",
"build:component:types": "tsc -p component/tsconfig.build.json",
@@ -107,7 +108,7 @@
"livekit-client": "^2.18.1",
"lodash-es": "^4.17.21",
"loglevel": "^1.9.1",
"matrix-js-sdk": "github:matrix-org/matrix-js-sdk#a9ef09fbf2f1644c8663d0aeaf6a921cd16e7b01",
"matrix-js-sdk": "github:matrix-org/matrix-js-sdk#10e4faabbb49baca655492d523b58faa9eea04b9",
"matrix-widget-api": "^1.18.0",
"node-stdlib-browser": "^1.3.1",
"normalize.css": "^8.0.1",
+49 -21
View File
@@ -13,6 +13,7 @@ import { fileURLToPath } from "node:url";
import { COMPONENT_HARNESS_URL } from "./playwright/component/harness.ts";
import { SDK_HARNESS_URL } from "./playwright/sdk/harness.ts";
import { WIDGET_RTC_SDK_URL } from "./playwright/widget/test-helpers.ts";
const baseURL = process.env.USE_DOCKER
? "http://localhost:8080"
@@ -26,6 +27,19 @@ process.env.NODE_EXTRA_CA_CERTS = join(
"backend/dev_tls_local-ca.crt",
);
const desktopChrome = {
...devices["Desktop Chrome"],
permissions: ["clipboard-write", "clipboard-read", "microphone", "camera"],
ignoreHTTPSErrors: true,
launchOptions: {
args: [
"--use-fake-ui-for-media-stream",
"--use-fake-device-for-media-stream",
"--mute-audio",
],
},
};
/**
* See https://playwright.dev/docs/test-configuration.
*/
@@ -43,11 +57,11 @@ export default defineConfig({
* Reporter to use. See https://playwright.dev/docs/test-reporters
*
* On CI, `list` prints a line per test as it finishes, which keeps the job
* log moving; `github` annotates failures on the PR; `html` is uploaded as
* an artifact and must never try to open a browser there.
* log moving; `github` annotates failures on the PR; the `blob` format is
* uploaded by sharded CI runs and combined into a HTML report by the workflow.
*/
reporter: process.env.CI
? [["list"], ["github"], ["html", { open: "never" }]]
? [["list"], ["github"], ["blob"]]
: [["list"], ["html"]],
/* Stop a broken run early on CI instead of waiting out every retry. */
maxFailures: process.env.CI ? 5 : undefined,
@@ -63,29 +77,28 @@ export default defineConfig({
},
/* Configure projects for major browsers */
projects: [
{
// Builds the embedded package into the directory the other site's
// Element Web container serves it from (see docker-compose-playwright.yml).
name: "build-embedded",
testMatch: "**/widget/embedded/build.setup.ts",
},
{
name: "chromium",
testIgnore: "**/mobile/**",
use: {
...devices["Desktop Chrome"],
permissions: [
"clipboard-write",
"clipboard-read",
"microphone",
"camera",
],
ignoreHTTPSErrors: true,
launchOptions: {
args: [
"--use-fake-ui-for-media-stream",
"--use-fake-device-for-media-stream",
"--mute-audio",
],
},
},
testIgnore: ["**/mobile/**", "**/widget/**"],
use: desktopChrome,
},
{
// Element Call as a widget in Element Web, on Chromium. This uses its
// own project so that it can be sharded on CI.
name: "widget",
dependencies: ["build-embedded"],
testMatch: "**/widget/**/*.@(spec|test).ts",
use: desktopChrome,
},
{
name: "firefox",
dependencies: ["build-embedded"],
testIgnore: "**/mobile/**",
use: {
...devices["Desktop Firefox"],
@@ -171,5 +184,20 @@ export default defineConfig({
timeout: 500,
},
},
{
// The widget RTC SDK's example widget, which Element Web opens as a
// widget. Bundled rather than from the dev server: Firefox now and then
// stalls on the dev server's hundreds of module requests.
command:
"pnpm exec vite build --config vite-widget-rtc-sdk-dev.config.ts && " +
"pnpm exec vite preview --config vite-widget-rtc-sdk-dev.config.ts",
url: WIDGET_RTC_SDK_URL,
reuseExistingServer: !process.env.CI,
ignoreHTTPSErrors: true,
gracefulShutdown: {
signal: "SIGTERM",
timeout: 500,
},
},
],
});
+100
View File
@@ -106,6 +106,66 @@ async function drawsPerSecond(page: Page): Promise<number> {
});
}
test.describe("the camera menu's preview", () => {
test("shows the camera with the effect in force", async ({ page }) => {
test.setTimeout(120_000);
await page.goto("/");
await SpaHelpers.createCall(page, "Preview user", "Preview in the menu");
const camera = await cameraColour(page.locator("video").first());
await page.getByRole("button", { name: "Camera", exact: true }).click();
const preview = page.getByRole("menu").locator("video");
await expect(preview).toBeVisible();
await expect
.poll(async () => distance(await averageColour(preview), camera), {
timeout: 20_000,
})
.toBeLessThan(30);
const tile = page.getByRole("menuitemradio", { name: "Background 1" });
await tile.click();
const picture = await averageColour(tile.locator("img"));
await expect
.poll(async () => distance(await averageColour(preview), picture), {
timeout: 60_000,
})
.toBeLessThan(distance(camera, picture) / 2);
});
test("shows the call's own camera once joined", async ({ page }) => {
test.setTimeout(120_000);
await page.goto("/");
await SpaHelpers.createCall(
page,
"Call preview user",
"Preview in the call's menu",
true,
);
await expect(page.getByTestId("incall_videomute")).toBeEnabled({
timeout: 10_000,
});
const camera = await cameraColour(page.locator("video").first());
await page.getByRole("button", { name: "Camera", exact: true }).click();
const preview = page.getByRole("menu").locator("video");
await expect(preview).toBeVisible();
await expect
.poll(async () => distance(await averageColour(preview), camera), {
timeout: 20_000,
})
.toBeLessThan(30);
const tile = page.getByRole("menuitemradio", { name: "Background 1" });
await tile.click();
const picture = await averageColour(tile.locator("img"));
await expect
.poll(async () => distance(await averageColour(preview), picture), {
timeout: 60_000,
})
.toBeLessThan(distance(camera, picture) / 2);
});
});
test.describe("background effects section", () => {
test("section stays within the call area at every size", async ({ page }) => {
test.setTimeout(120_000);
@@ -151,6 +211,46 @@ test.describe("background effects section", () => {
});
});
test.describe("a pipeline that fails to build", () => {
test("offers no effects for the session, and keeps the choice", async ({
page,
}) => {
test.setTimeout(120_000);
await page.goto("/");
await SpaHelpers.createCall(page, "Failing user", "Pipeline fails");
const preview = page.locator("video").first();
const camera = await cameraColour(preview);
await page.route("**/*.tflite*", async (route) => route.abort());
await page.getByRole("button", { name: "Camera", exact: true }).click();
const tile = page.getByRole("menuitemradio", { name: "Background 1" });
await tile.click();
await expect(
page.getByText("Background effects are not supported on this platform."),
).toBeVisible({ timeout: 30_000 });
await expect(tile).toBeDisabled();
await page.keyboard.press("Escape");
// The camera and microphone carry on without it.
expect(distance(await cameraColour(preview), camera)).toBeLessThan(30);
await expect(page.getByTestId("incall_mute")).toHaveAttribute(
"aria-checked",
"true",
);
await page.unroute("**/*.tflite*");
await page.reload();
await expect(page.getByTestId("lobby_joinCall")).toBeVisible();
await page.getByRole("button", { name: "Camera", exact: true }).click();
await expect(tile).toHaveAttribute("aria-checked", "true");
const picture = await averageColour(tile.locator("img"));
await expect
.poll(async () => distance(await averageColour(preview), picture), {
timeout: 60_000,
})
.toBeLessThan(distance(camera, picture) / 2);
});
});
test.describe("the chosen effect", () => {
test("chosen effect survives leaving and rejoining", async ({ page }) => {
test.setTimeout(120_000);
+8 -42
View File
@@ -29,49 +29,14 @@ export interface MyFixtures {
addUser: (username: string, host: string) => Promise<UserBaseFixture>;
}
// Minimal config.json for the local element-web instance
const CONFIG_JSON = {
default_server_config: {
"m.homeserver": {
base_url: "https://synapse.m.localhost",
server_name: "synapse.m.localhost",
},
},
element_call: {
participant_limit: 8,
brand: "Element Call",
},
// The default language is set here for test consistency
setting_defaults: {
language: "en-GB",
feature_group_calls: true,
},
// the location tests want a map style url.
map_style_url:
"https://api.maptiler.com/maps/streets/style.json?key=fU3vlMsMn4Jb6dnEIFsx",
features: {
// We don't want to go through the feature announcement during the e2e test
feature_release_announcement: false,
feature_element_call_video_rooms: true,
feature_video_rooms: true,
feature_group_calls: true,
},
};
export const widgetTest = test.extend<MyFixtures>({
// allow per-test override: `widgetTest.use({ callType: "dm" })`
callType: ["room", { option: true }],
asWidget: async ({ browser, context, callType }, pUse) => {
await context.route(`http://localhost:8081/config.json*`, async (route) => {
await route.fulfill({ json: CONFIG_JSON });
});
const brooksDisplayName = `brooks_${Date.now()}`;
const whistlerDisplayName = `whistler_${Date.now()}`;
asWidget: async ({ browser, callType }, pUse, testInfo) => {
// Tests in parallel workers can reach this in the same millisecond.
const uniqueSuffix = `${Date.now()}_${testInfo.parallelIndex}`;
const brooksDisplayName = `brooks_${uniqueSuffix}`;
const whistlerDisplayName = `whistler_${uniqueSuffix}`;
// Register users
const {
@@ -164,13 +129,14 @@ export const widgetTest = test.extend<MyFixtures>({
* Provide a way to add additional users within a test.
* The returned user will be registered on the default homeserver, the name will be made unique by appending a timestamp.
*/
addUser: async ({ browser }, use) => {
addUser: async ({ browser }, use, testInfo) => {
await use(
async (
username: string,
host: string = HOST1,
): Promise<UserBaseFixture> => {
const uniqueSuffix = Date.now();
// Tests in parallel workers can reach this in the same millisecond.
const uniqueSuffix = `${Date.now()}_${testInfo.parallelIndex}`;
const { page, clientHandle, mxId } = await TestHelpers.registerUser(
browser,
`${username.toLowerCase()}_${uniqueSuffix}`,
+20
View File
@@ -0,0 +1,20 @@
/*
Copyright 2026 Element Creations Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { test as setup } from "@playwright/test";
import { execFileSync } from "node:child_process";
import { mkdirSync } from "node:fs";
const EMBEDDED_DIST = "embedded/web/dist";
setup("build the embedded package", () => {
setup.setTimeout(10 * 60 * 1000);
mkdirSync(EMBEDDED_DIST, { recursive: true });
execFileSync("pnpm", ["build:embedded", "--outDir", EMBEDDED_DIST], {
stdio: "inherit",
});
});
@@ -0,0 +1,56 @@
/*
Copyright 2026 Element Creations Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { expect, type Route } from "@playwright/test";
import { widgetTest } from "../../fixtures/widget-user.ts";
import { HOST2, TestHelpers } from "../test-helpers.ts";
/** Where the other site's Element Web serves the embedded package from */
const CONFIG_JSON = "**/widgets/element-call/config.json";
const HOST_ANSWERS: Record<string, (route: Route) => Promise<void>> = {
"a 404": async (route) => route.fulfill({ status: 404 }),
"a failed fetch": async (route) => route.abort("failed"),
"an HTML page": async (route) =>
route.fulfill({
status: 200,
contentType: "text/html",
body: "<!doctype html><title>Element</title>",
}),
};
widgetTest.skip(
({ browserName }) => browserName === "firefox",
"The widget suite does not run on Firefox, see the other specs here",
);
for (const [answer, respond] of Object.entries(HOST_ANSWERS)) {
widgetTest(
`reaches the lobby when config.json gets ${answer}`,
async ({ addUser }) => {
const { page } = await addUser("Brooks", HOST2);
await TestHelpers.createRoom("Config Room", page);
let answered = 0;
await page.route(CONFIG_JSON, async (route) => {
answered++;
await respond(route);
});
await TestHelpers.startCallInCurrentRoom(page);
await expect(
page
.locator('iframe[title="Element Call"]')
.contentFrame()
.getByTestId("lobby_joinCall"),
).toBeVisible({ timeout: 15000 });
expect(answered).toBeGreaterThan(0);
},
);
}
+21 -35
View File
@@ -22,6 +22,12 @@ const PASSWORD = "foobarbaz1!";
export const HOST1 = "https://app.m.localhost/#/welcome";
export const HOST2 = "https://app.othersite.m.localhost/#/welcome";
/**
* Where the widget RTC SDK's example widget is served (`pnpm dev:widget-rtc-sdk`):
* `widget-rtc-sdk/index.html`, running the bundle from source.
*/
export const WIDGET_RTC_SDK_URL = "https://localhost:3003";
export type RtcMode = "compat" | "2_0";
export class TestHelpers {
@@ -144,7 +150,11 @@ export class TestHelpers {
await this.dismissStartupToasts(page);
await TestHelpers.setDevToolElementCallDevUrl(page);
// The other site's Element Web opens the embedded package it serves itself
// (see docker-compose-playwright.yml), not the dev server's app.
if (host !== HOST2) {
await TestHelpers.setDevToolElementCallDevUrl(page);
}
const clientHandle = await page.evaluateHandle(() =>
window.mxMatrixClientPeg.get(),
@@ -153,9 +163,10 @@ export class TestHelpers {
return { page, clientHandle, mxId };
}
// Dismisses any toasts that appear on startup, such as "Failed to load service worker" or "Back up your chats".
// Toast can be stacked, and only the top one can be dismiss, so just look at what is on top and
// dismiss (if part of expected toats)
// Dismisses the toasts a fresh device gets after sign-in.
//
// Toasts stack and only the top one can be dismissed, so look
// at what is on top each time.
public static async dismissStartupToasts(page: Page): Promise<void> {
const expectedToasts: {
title: string;
@@ -163,6 +174,7 @@ export class TestHelpers {
confirm?: string;
}[] = [
{ title: "Failed to load service worker", button: "OK" },
{ title: "Verify this device", button: "Later" },
{ title: "Back up your chats", button: "Not now" },
{
title: "Turn on key storage",
@@ -213,30 +225,6 @@ export class TestHelpers {
}
}
/**
* Clears what Element Web stacks over the room list after sign-in. The
* release announcement is held back while any toast is up, so it is closed
* after the toasts, not before.
*/
public static async clearRoomListOverlays(page: Page): Promise<void> {
await TestHelpers.dismissStartupToasts(page);
await TestHelpers.closeReleaseAnnouncement(page, "Introducing Sections");
}
public static async closeReleaseAnnouncement(
page: Page,
name: string,
): Promise<void> {
try {
await page
.getByRole("dialog", { name })
.getByRole("button", { name: "OK" })
.click({ timeout: 2000 });
} catch {
// Announcement not shown; nothing to do
}
}
public static async expandAllSections(page: Page): Promise<void> {
try {
await page
@@ -282,15 +270,14 @@ export class TestHelpers {
/**
* Opens the room list's "New" menu and picks the given item.
*
* While the sections announcement is anchored to the "New" button, the
* button has no menu at all, so the overlays go first. Driven by keyboard,
* as a toast arriving afterwards sits over the button and swallows clicks.
* Driven by keyboard, as a toast arriving late sits over the button and
* swallows clicks.
*/
public static async chooseFromRoomListNewMenu(
page: Page,
itemName: string,
): Promise<void> {
await TestHelpers.clearRoomListOverlays(page);
await TestHelpers.dismissStartupToasts(page);
await page
.getByRole("navigation", { name: "Room list" })
.getByRole("button", { name: "New", exact: true })
@@ -427,9 +414,8 @@ export class TestHelpers {
name: "Toggle Invites section",
});
await expect(invitesSection).toBeVisible({ timeout: 10000 });
// Toasts and the announcement sit over the top of the room list, where
// this header is
await TestHelpers.clearRoomListOverlays(page);
// Toasts sit over the top of the room list, where this header is.
await TestHelpers.dismissStartupToasts(page);
if ((await invitesSection.getAttribute("aria-expanded")) !== "true") {
await invitesSection.click();
}
+129
View File
@@ -0,0 +1,129 @@
/*
Copyright 2026 Element Creations Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { expect, type FrameLocator, type Page } from "@playwright/test";
import { widgetTest } from "../fixtures/widget-user.ts";
import { WIDGET_RTC_SDK_URL } from "./test-helpers.ts";
/**
* The widget RTC SDK's example widget, `widget-rtc-sdk/index.html`, added to a
* room as a custom widget and opened in two Element Webs: the bundle's whole
* public API, from the widget API to the data channel.
*/
widgetTest.describe.configure({ timeout: 300_000 });
const WIDGET_ID = "widget-rtc-sdk-example";
const WIDGET_NAME = "Widget RTC SDK";
/** Accepts the prompts Element Web shows before it runs a widget, and finds it. */
async function openWidget(page: Page): Promise<FrameLocator> {
// Only for a widget someone else added
const loadPrompt = page.getByRole("button", { name: "Continue" });
const capabilitiesPrompt = page.getByRole("button", { name: "Approve" });
await expect(loadPrompt.or(capabilitiesPrompt)).toBeVisible({
timeout: 30_000,
});
if (await loadPrompt.isVisible()) await loadPrompt.click();
await capabilitiesPrompt.click({ timeout: 30_000 });
// The SDK authenticates with the media transport by an OpenID token, and
// asks for more than one, so the answer has to be remembered
const openIdPrompt = page.getByRole("dialog", {
name: "Allow this widget to verify your identity",
});
await openIdPrompt.getByText("Remember this").click({ timeout: 30_000 });
await openIdPrompt.getByRole("button", { name: "Continue" }).click();
return page.frameLocator(`iframe[title="${WIDGET_NAME}"]`);
}
widgetTest(
"data sent from one widget arrives in the other",
async ({ asWidget }) => {
const { brooks, whistler } = asWidget;
// Brooks adds the example widget to the room and pins it to the top, so
// that both Element Webs open it without anyone looking for it
await brooks.clientHandle.evaluate(
async (client, { widgetId, name, url }) => {
const roomId = client
.getRooms()
.find((room) => room.name === "Welcome Room")!.roomId;
await client.sendStateEvent(
roomId,
// @ts-expect-error not in the js-sdk's state event map
"im.vector.modular.widgets",
{
id: widgetId,
type: "m.custom",
name,
url,
data: {},
creatorUserId: client.getUserId(),
},
widgetId,
);
await client.sendStateEvent(
roomId,
// @ts-expect-error not in the js-sdk's state event map
"io.element.widgets.layout",
{
widgets: {
[widgetId]: {
container: "top",
index: 0,
width: 100,
height: 50,
},
},
},
"",
);
},
{
widgetId: WIDGET_ID,
name: WIDGET_NAME,
// Element Web fills the $ variables in and adds widgetId and parentUrl
url:
`${WIDGET_RTC_SDK_URL}/?widgetId=$matrix_widget_id&roomId=$matrix_room_id` +
"&userId=$matrix_user_id&deviceId=$org.matrix.msc3819.matrix_device_id" +
"&baseUrl=$org.matrix.msc4039.matrix_base_url",
},
);
const brooksWidget = await openWidget(brooks.page);
const whistlerWidget = await openWidget(whistler.page);
// The example joins on load; each sees the other as a member
for (const [widget, other] of [
[brooksWidget, whistler],
[whistlerWidget, brooks],
] as const) {
await expect(widget.locator("#connect_status")).toHaveText("Connected", {
timeout: 60_000,
});
await expect(widget.locator("#members")).toContainText(other.mxId, {
timeout: 60_000,
});
}
// Data is not queued: the receiver drops what arrives before it has matched
// the sender on the media transport, which the example does not show. So
// send until it arrives.
await expect(async () => {
await brooksWidget.getByRole("button", { name: "Send Text" }).click();
await expect(whistlerWidget.locator("#data")).toContainText(
"Hello, world!",
{ timeout: 2_000 },
);
}).toPass({ timeout: 60_000 });
// Our own data is not echoed back
await expect(brooksWidget.locator("#data")).not.toContainText(
"Hello, world!",
);
},
);
+44 -31
View File
@@ -8,7 +8,7 @@ overrides:
'@livekit/components-core>rxjs': ^7.8.1
'@livekit/track-processors>@mediapipe/tasks-vision': ^0.10.18
tar: ^7.5.11
minimatch: ^10.2.3
minimatch: ^11.0.0
glob: ^10.5.0
qs: ^6.15.2
js-yaml: ^4.2.0
@@ -191,8 +191,8 @@ importers:
specifier: ^1.9.1
version: 1.9.2
matrix-js-sdk:
specifier: github:matrix-org/matrix-js-sdk#a9ef09fbf2f1644c8663d0aeaf6a921cd16e7b01
version: https://codeload.github.com/matrix-org/matrix-js-sdk/tar.gz/a9ef09fbf2f1644c8663d0aeaf6a921cd16e7b01
specifier: github:matrix-org/matrix-js-sdk#10e4faabbb49baca655492d523b58faa9eea04b9
version: https://codeload.github.com/matrix-org/matrix-js-sdk/tar.gz/10e4faabbb49baca655492d523b58faa9eea04b9
matrix-widget-api:
specifier: ^1.18.0
version: 1.20.0
@@ -1145,6 +1145,9 @@ packages:
resolution: {integrity: sha512-O8jcjabXaleOG9DQ0+ARXWZBTfnP4WNAqzuiJK7ll44AmxGKv/J2M4TPjxjY3znBCfvBXFzucm1twdyFybFqEA==}
engines: {node: '>=12'}
'@isaacs/nonrecursive-glob-match@1.0.3':
resolution: {integrity: sha512-SI7kyKHq4XXDRozw+r9GlRZkia1ZkFrpLW98r2IC2Bpr8YkFGE3fvjzmxCG0x78A++69HmDNrDvSMYBZmHGpKA==}
'@joshwooding/vite-plugin-react-docgen-typescript@0.7.0':
resolution: {integrity: sha512-qvsTEwEFefhdirGOPnu9Wp6ChfIwy2dBCRuETU3uE+4cC+PFoxMSiiEhxk4lOluA34eARHA0OxqsEUYDqRMgeQ==}
peerDependencies:
@@ -1192,8 +1195,8 @@ packages:
'@types/dom-mediacapture-transform': ^0.1.9
livekit-client: ^1.12.0 || ^2.1.0
'@matrix-org/matrix-sdk-crypto-wasm@18.9.0':
resolution: {integrity: sha512-NtLzNDTEfNASBjNCTGZHuk7N0jrQhS6Zx2CM857rCHZYYh/rpbKGQNrJ7azktm4S1P5Jy3WU3nwJTGohGw26qQ==}
'@matrix-org/matrix-sdk-crypto-wasm@18.9.2':
resolution: {integrity: sha512-GHeuSeb5Fj6Wg2+NE/WoLJJs9U3nOzttO+2sj7lOaS5gNBpGIksc48N/Q/kK/Z+Lxh7YMhopB7M8b6HyASXnGQ==}
engines: {node: '>= 18'}
'@mdx-js/react@3.1.1':
@@ -3530,8 +3533,8 @@ packages:
boolbase@1.0.0:
resolution: {integrity: sha512-JZOSA7Mo9sNGB8+UjSgzdLtokWAky1zbztM3WRLCbZ70/3cTANmQmOdR7y2g+J0e2WXywy1yS468tY+IruqEww==}
brace-expansion@5.0.9:
resolution: {integrity: sha512-ScQ4IuvIEF1TMlP7Zt+vjJ//9zlPb2SDcxWxM3bk8s6t6GGdJ7KO1dCcTidOPJKePW30LE/2cT7wCyPho9/Wxg==}
brace-expansion@5.0.12:
resolution: {integrity: sha512-YovQ3rzhaLMIrDjNDMkNS01tea93qhEhG5xy8f6+R0l+dw3Ki+5sCoIoI942iuLZTHWogWktgwVDhU09iNEimQ==}
engines: {node: 20 || >=22}
braces@3.0.3:
@@ -4731,9 +4734,9 @@ packages:
matrix-events-sdk@0.0.1:
resolution: {integrity: sha512-1QEOsXO+bhyCroIe2/A5OwaxHvBm7EsSQ46DEDn8RBIfQwN5HWBpFvyWWR4QY0KHPPnnJdI99wgRiAl7Ad5qaA==}
matrix-js-sdk@https://codeload.github.com/matrix-org/matrix-js-sdk/tar.gz/a9ef09fbf2f1644c8663d0aeaf6a921cd16e7b01:
resolution: {gitHosted: true, integrity: sha512-ZAhn/zB44MgrXqUzi9ffSFcmZY3+2nx7bEgfEJR7chYDFyT/XmlmfCJx7Ar4rOTAYZJRVnMxrEwiEHGDapUi0Q==, tarball: https://codeload.github.com/matrix-org/matrix-js-sdk/tar.gz/a9ef09fbf2f1644c8663d0aeaf6a921cd16e7b01}
version: 45.0.0
matrix-js-sdk@https://codeload.github.com/matrix-org/matrix-js-sdk/tar.gz/10e4faabbb49baca655492d523b58faa9eea04b9:
resolution: {gitHosted: true, integrity: sha512-7cbPN96wGt93B73jZieW2HBtTnasgKJ7wzTgiV0SjwkNyb50EK3vDa17LdSJpyOwEjFP5edyPRprk865HQJqww==, tarball: https://codeload.github.com/matrix-org/matrix-js-sdk/tar.gz/10e4faabbb49baca655492d523b58faa9eea04b9}
version: 45.1.0
engines: {node: '>=22.0.0'}
matrix-widget-api@1.20.0:
@@ -4768,9 +4771,9 @@ packages:
minimalistic-crypto-utils@1.0.1:
resolution: {integrity: sha512-JIYlbt6g8i5jKfJ3xz7rF0LXmv2TkDxBLUkiBeZ7bAx4GnnNMr8xFpGnOxn6GhTEHx3SjRrZEoU+j04prX1ktg==}
minimatch@10.2.6:
resolution: {integrity: sha512-vpLQEs+VLCr1nU0BXS07maYoFwlDAH0gngQuuttxIwutDFEMHq2blX+8vpgxDdK3J1PwjCJiep77OitTZ4Ll1A==}
engines: {node: 18 || 20 || >=22}
minimatch@11.0.4:
resolution: {integrity: sha512-/UJ1dEoVQTjPNDh6SGlUo+ueLMmoHTas4q+8yaAR/5t6nLAIiBDJtoYUh20IYleix7ehiQkgczb84PE3uzHkxg==}
engines: {node: 20 || >=22}
minimist@1.2.8:
resolution: {integrity: sha512-2yyAR8qBkN3YuheJanUpWC5U3bb5osDywNB8RzDVlDwDHbocAJveqqj1u8+SVD7jkWT4yvsHCpWqqWqAxb0zCA==}
@@ -4943,8 +4946,8 @@ packages:
resolution: {integrity: sha512-LaNjtRWUBY++zB5nE/NwcaoMylSPk+S+ZHNB1TzdbMJMny6dynpAGt7X/tl/QYq3TIeE6nxHppbo2LGymrG5Pw==}
engines: {node: '>=10'}
p-retry@8.0.1:
resolution: {integrity: sha512-NAigyu8hfvJe7MsS18mnc4is0MZtau3QMdBklaJKK23oBQb6occO3UPM3vHmTckW8KhfyjMZaUOqdTFEYliHgg==}
p-retry@9.0.0:
resolution: {integrity: sha512-HuEats8gYn/UUYa5uyveyQGv75mQmwdFtuMgnZlS12dj/7bdnJSO6keaTjgcZUhvkANTj5EO3lACUmV3xmgj0Q==}
engines: {node: '>=22'}
p-try@2.2.0:
@@ -5315,6 +5318,10 @@ packages:
randomfill@1.0.4:
resolution: {integrity: sha512-87lcbR8+MhcWcUiQ+9e+Rwx8MyR2P7qnt15ynUlbm3TU/fjbgz4GsvfSUDTemtCCtVCqb4ZcEFlyPNTh9bBTLw==}
re2js@2.9.1:
resolution: {integrity: sha512-JIT3bEEUZTOR2YS1walNEIfH4cF/cQeZcBUFq7ldqO6lR6g0X5CBq/S8rsa9omzZcsNf8qc36jCkWpKKtncrZg==}
engines: {node: '>=20.0.0'}
react-docgen-typescript@2.4.0:
resolution: {integrity: sha512-ZtAp5XTO5HRzQctjPU0ybY0RRCQO19X/8fxn3w7y2VVTUbGHDKULPTL4ky3vB05euSgG5NpALhEhDPvQ56wvXg==}
peerDependencies:
@@ -6902,7 +6909,7 @@ snapshots:
ignore: 5.3.2
import-fresh: 3.3.1
js-yaml: 4.3.1
minimatch: 10.2.6
minimatch: 11.0.4
strip-json-comments: 3.1.1
transitivePeerDependencies:
- supports-color
@@ -6981,7 +6988,7 @@ snapshots:
dependencies:
'@humanwhocodes/object-schema': 2.0.3
debug: 4.4.3(supports-color@7.2.0)
minimatch: 10.2.6
minimatch: 11.0.4
transitivePeerDependencies:
- supports-color
@@ -7117,6 +7124,8 @@ snapshots:
wrap-ansi: 8.1.0
wrap-ansi-cjs: wrap-ansi@7.0.0
'@isaacs/nonrecursive-glob-match@1.0.3': {}
'@joshwooding/vite-plugin-react-docgen-typescript@0.7.0(typescript@5.9.3)(vite@8.2.4(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(sass@1.103.1)(terser@5.46.1)(yaml@2.9.0))':
dependencies:
glob: 10.5.0
@@ -7169,7 +7178,7 @@ snapshots:
'@types/dom-mediacapture-transform': 0.1.11
livekit-client: 2.23.0(@types/dom-mediacapture-record@1.0.22)
'@matrix-org/matrix-sdk-crypto-wasm@18.9.0': {}
'@matrix-org/matrix-sdk-crypto-wasm@18.9.2': {}
'@mdx-js/react@3.1.1(@types/react@19.2.18)(react@19.3.1)':
dependencies:
@@ -8776,7 +8785,7 @@ snapshots:
'@typescript-eslint/types': 8.69.0
'@typescript-eslint/visitor-keys': 8.69.0
debug: 4.4.3(supports-color@7.2.0)
minimatch: 10.2.6
minimatch: 11.0.4
semver: 7.8.5
tinyglobby: 0.2.17
ts-api-utils: 2.5.0(typescript@5.9.3)
@@ -8791,7 +8800,7 @@ snapshots:
'@typescript-eslint/types': 8.72.0
'@typescript-eslint/visitor-keys': 8.72.0
debug: 4.4.3(supports-color@7.2.0)
minimatch: 10.2.6
minimatch: 11.0.4
semver: 7.8.5
tinyglobby: 0.2.17
ts-api-utils: 2.5.0(typescript@5.9.3)
@@ -9116,7 +9125,7 @@ snapshots:
boolbase@1.0.0: {}
brace-expansion@5.0.9:
brace-expansion@5.0.12:
dependencies:
balanced-match: 4.0.4
@@ -9661,7 +9670,7 @@ snapshots:
json-stable-stringify-without-jsonify: 1.0.1
levn: 0.4.1
lodash.merge: 4.6.2
minimatch: 10.2.6
minimatch: 11.0.4
natural-compare: 1.4.0
optionator: 0.9.4
strip-ansi: 6.0.1
@@ -9781,7 +9790,7 @@ snapshots:
filelist@1.0.6:
dependencies:
minimatch: 10.2.6
minimatch: 11.0.4
fill-range@7.1.1:
dependencies:
@@ -9881,7 +9890,7 @@ snapshots:
dependencies:
foreground-child: 3.3.1
jackspeak: 3.4.3
minimatch: 10.2.6
minimatch: 11.0.4
minipass: 7.1.3
package-json-from-dist: 1.0.1
path-scurry: 1.11.1
@@ -10006,7 +10015,7 @@ snapshots:
jiti: 2.7.0
jsonc-parser: 3.3.1
magic-string: 1.2.3
minimatch: 10.2.6
minimatch: 11.0.4
ora: 9.4.1
react: 19.2.8
react-i18next: 17.0.13(i18next@26.4.1(typescript@5.9.3))(react-dom@19.2.8(react@19.2.8))(react@19.2.8)(typescript@5.9.3)
@@ -10404,17 +10413,18 @@ snapshots:
matrix-events-sdk@0.0.1: {}
matrix-js-sdk@https://codeload.github.com/matrix-org/matrix-js-sdk/tar.gz/a9ef09fbf2f1644c8663d0aeaf6a921cd16e7b01:
matrix-js-sdk@https://codeload.github.com/matrix-org/matrix-js-sdk/tar.gz/10e4faabbb49baca655492d523b58faa9eea04b9:
dependencies:
'@babel/runtime': 8.0.8
'@matrix-org/matrix-sdk-crypto-wasm': 18.9.0
'@matrix-org/matrix-sdk-crypto-wasm': 18.9.2
another-json: 0.2.0
bs58: 6.0.0
content-type: 3.1.1
loglevel: 1.9.2
matrix-events-sdk: 0.0.1
matrix-widget-api: 1.20.0
p-retry: 8.0.1
p-retry: 9.0.0
re2js: 2.9.1
sdp-transform: 3.0.0
unhomoglyph: 1.0.6
@@ -10449,9 +10459,10 @@ snapshots:
minimalistic-crypto-utils@1.0.1: {}
minimatch@10.2.6:
minimatch@11.0.4:
dependencies:
brace-expansion: 5.0.9
'@isaacs/nonrecursive-glob-match': 1.0.3
brace-expansion: 5.0.12
minimist@1.2.8: {}
@@ -10735,7 +10746,7 @@ snapshots:
dependencies:
p-limit: 3.1.0
p-retry@8.0.1:
p-retry@9.0.0:
dependencies:
is-network-error: 1.3.2
@@ -11159,6 +11170,8 @@ snapshots:
randombytes: 2.1.0
safe-buffer: 5.2.1
re2js@2.9.1: {}
react-docgen-typescript@2.4.0(typescript@5.9.3):
dependencies:
typescript: 5.9.3
+1 -1
View File
@@ -27,7 +27,7 @@ overrides:
# Security fixes:
# - https://github.com/advisories/GHSA-7r86-cg39-jmmj
# - https://github.com/advisories/GHSA-23c5-xmqv-rm74
"minimatch": "^10.2.3"
"minimatch": "^11.0.0"
# Security fix: https://github.com/element-hq/element-call/security/dependabot/109
"glob": "^10.5.0"
# Security fixes:
@@ -1,68 +0,0 @@
# SDK mode (LEGACY)
> Superseded by the `sdk/` package (`@element-hq/matrixrtc-sdk`) and its harness in
> `sdk/dev`; see `sdk/SdkArchitecture.md`. This bundle is in production use, so it
> keeps building as a thin layer over that package until its consumers have moved
> to the package directly. It will then be removed. Its members are the package's
> `RTCMember`s now, in place of its own wrapper around the raw membership.
EC can be build in sdk mode. This will result in a compiled js file that can be imported in very simple webapps.
It allows to use matrixRTC in combination with livekit without relying on element call.
This is done by instantiating the call view model and exposing some useful behaviors (observables) and methods.
This folder contains an example index.html file that showcases the sdk in use (hosted on localhost:8123 with a webserver allowing cors (for example `npx serve -l 81234 --cors`)) as a godot engine HTML export template.
## Getting started
To get started run
```
pnpm install
pnpm build:sdk-target-based-on-call-view-model
```
in the repository root. (`pnpm build:sdk` builds the `sdk/` package instead.)
CI's `sdk` package (`build:sdk:production`, `build:sdk:development`, the
`build-output-sdk` artifact and the `pr<N>-sdk` Netlify preview) is still this
bundle, not the `sdk/` package, until its consumers have moved over.
It will create `dist` containing the compiled js file, `matrixrtc-sdk.js`.
This file needs to be hosted. Locally (via `npx serve -l 81234 --cors`) or on a remote server.
Now you just need to add the widget to element web via:
```
/addwidget http://localhost:3000?widgetId=$matrix_widget_id&userId=$matrix_user_id&deviceId=$org.matrix.msc3819.matrix_device_id&baseUrl=$org.matrix.msc4039.matrix_base_url&roomId=$matrix_room_id
```
## Widgets
The sdk mode is particularly interesting to be used in widgets. In widgets you do not need to pay attention to matrix login/cs api ...
To create a widget see the example `index.html` file in this folder. And add it to EW via:
`/addwidget <widgetUrl>` (see **url parameters** for more details on `<widgetUrl>`)
### url parameters
The url parameters are needed to pass initial data to the widget. They will automatically be used
by the matrixRTCSdk to start the postmessage widget api (communication between the client (e.g. Element Web) and the widget)
```
widgetId = $matrix_widget_id
userId = $matrix_user_id
deviceId = $org.matrix.msc3819.matrix_device_id
baseUrl = $org.matrix.msc4039.matrix_base_url
```
`parentUrl = // will be inserted automatically`
Full template use as `<widgetUrl>`:
```
http://localhost:3000?widgetId=$matrix_widget_id&userId=$matrix_user_id&deviceId=$org.matrix.msc3819.matrix_device_id&baseUrl=$org.matrix.msc4039.matrix_base_url&roomId=$matrix_room_id
```
the `$` prefixed variables will be replaced by EW on widget instantiation. (e.g. `$matrix_user_id` -> `@user:example.com` (url encoding will also be applied automatically by EW) -> `%40user%3Aexample.com`)
-191
View File
@@ -1,191 +0,0 @@
/*
Copyright 2025-2026 Element Creations Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
/**
* LEGACY, in production use
*
* The entrypoint of `pnpm build:sdk-target-based-on-call-view-model`, a
* bundle for widgets that exposes `createMatrixRTCSdk`: join a slot and
* exchange realtime data, with the `@element-hq/matrixrtc-sdk` package and
* Element Call's call view model doing the work underneath. New consumers use
* the package directly; this bundle goes away once the existing ones have.
*/
import { logger as rootLogger } from "matrix-js-sdk/lib/logger";
import { EMPTY, filter, map, type Observable, of, switchMap, take } from "rxjs";
import {
type Behavior,
createRTCSlot,
E2eeType,
type LocalRTCMember,
ObservableScope,
type RemoteRTCMember,
type RTCParticipation,
} from "@element-hq/matrixrtc-sdk";
import {
callViewModelOptionsFromParams,
type CallViewModel,
createCallViewModel$,
initialPublishRequests,
} from "../src/state/CallViewModel/CallViewModel";
import {
captureSettings,
rtcParticipationOptions,
rtcSlotOptions,
} from "../src/room/InCallView";
import { createWidgetHostBridge } from "../src/HostBridge";
import { MediaDevices } from "../src/state/MediaDevices";
import { MuteStates } from "../src/state/MuteStates";
import { getUrlParams } from "../src/UrlParams";
import { observeElementSize$ } from "../src/utils/elementSize";
import { initializeWidget } from "../src/widget";
/** The data channel topic this bundle has always used, so that old and new builds talk. */
const TOPIC = "matrixRTC";
export interface MatrixRTCSdk {
/** Joins the slot; `connected$` tracks the progress. A second call does nothing. */
join: () => void;
leave: () => void;
/** Ends everything. The sdk cannot be restarted; create a new one. */
stop: () => void;
connected$: Behavior<boolean>;
/** Our own member, null until our membership has been seen in the room. */
localMember$: Behavior<LocalRTCMember | null>;
/** The other members of the slot, with the media the participation carries for them. */
remoteMembers$: Behavior<RemoteRTCMember[]>;
/** What other members sent with `sendData`. */
data$: Observable<{ rtcBackendIdentity: string; data: string }>;
/** Sends to every member on our transport. Rejects while not joined. */
sendData: (data: unknown) => Promise<void>;
sendRoomMessage: (message: string) => Promise<void>;
}
export async function createMatrixRTCSdk(
application = "m.call",
id = "",
sticky = false,
): Promise<MatrixRTCSdk> {
const logger = rootLogger.getChild("[MatrixRTCSdk]");
const scope = new ObservableScope();
const widget = initializeWidget(application, true);
if (!widget) throw new Error("No widget: this bundle only runs as a widget");
const client = await widget.client;
const hostBridge = createWidgetHostBridge(widget);
const urlParams = getUrlParams();
const { roomId, controlledAudioDevices, callIntent } = urlParams;
const room = roomId === null ? null : client.getRoom(roomId);
if (room === null) throw new Error("No room: the url needs a joined roomId");
const mediaDevices = new MediaDevices(scope, {
controlledAudioDevices,
callIntent,
});
const muteStates = new MuteStates(
scope,
mediaDevices,
{ audioEnabled: false, videoEnabled: false },
hostBridge,
);
const encryptionSystem = { kind: E2eeType.PER_PARTICIPANT } as const;
const capture = captureSettings();
const slot = createRTCSlot(scope, client, room, {
...rtcSlotOptions(encryptionSystem),
application,
id,
});
let vm: CallViewModel | undefined;
const failed =
(what: string) =>
(e: unknown): void =>
logger.error(what, e);
const over = <T>(
pick: (participation: RTCParticipation) => Observable<T>,
fallback: T,
): Behavior<T> =>
scope.behavior(
slot.participation$.pipe(
switchMap((p) => (p === null ? of(fallback) : pick(p))),
),
);
return {
join: (): void => {
if (slot.participation$.value !== null) return;
if (sticky)
widget.api
.setAlwaysOnScreen(true)
.catch(failed("Failed to make the widget sticky"));
const participation = slot.join({
...rtcParticipationOptions(urlParams),
publish: initialPublishRequests(muteStates, mediaDevices, capture),
});
vm = createCallViewModel$(
scope,
participation,
room,
mediaDevices,
muteStates,
{
...callViewModelOptionsFromParams(urlParams),
encryptionSystem,
capture,
hostBridge,
// The bundle owns its page, so the body is the space it has
windowSize$: scope.behavior(observeElementSize$(document.body)),
},
of({}),
of({}),
);
// Once the hang-up has gone through, unstick and ask the host to close us
vm.leave$
.pipe(
switchMap(() =>
participation.state$.pipe(
filter((state) => state.kind === "left"),
take(1),
),
),
scope.bind(),
)
.subscribe(() => {
hostBridge
.setAlwaysOnScreen(false)
.catch(failed("Failed to unstick the widget"))
.then(() => hostBridge.close?.())
.catch(failed("Failed to ask the host to close"));
});
},
leave: (): void => vm?.leave(),
stop: (): void => scope.end(),
connected$: over(
(p) => p.state$.pipe(map((state) => state.kind === "connected")),
false,
),
localMember$: over((p) => p.localMember$, null),
remoteMembers$: over((p) => p.remoteMembers$, []),
// Attested members only: the SDK has matched the sender
data$: slot.participation$.pipe(
switchMap((p) => p?.data$ ?? EMPTY),
filter(({ topic }) => topic === TOPIC),
map(({ member, text }) => ({
rtcBackendIdentity: member.rtcBackendIdentity,
data: text,
})),
),
sendData: async (data): Promise<void> => {
const participation = slot.participation$.value;
if (participation === null) throw new Error("Not joined");
await participation.sendData(TOPIC, JSON.stringify(data));
},
sendRoomMessage: async (message): Promise<void> => {
await client.sendTextMessage(room.roomId, message);
},
};
}
+1 -2
View File
@@ -5,8 +5,7 @@ it joins a MatrixRTC slot in a Matrix room, writes and watches the memberships,
connects to the media transport (LiveKit today), publishes the local media and
exposes every member and their tracks as observables. It has no UI. Element
Call's own `CallViewModel` is one consumer of it; any host can build a different
view on top. The design is in [`SdkArchitecture.md`](./SdkArchitecture.md), what
moved out of Element Call's `src/` in [`SdkMigration.md`](./SdkMigration.md).
view on top. The design is in [`SdkArchitecture.md`](./SdkArchitecture.md).
**Status:** first implementation. `createRTCSlot` and `slot.join()` join the
session, connect to the transport, publish the local media and expose every
+1 -3
View File
@@ -10,8 +10,7 @@ interface.
Element Call becomes one consumer of it: `CallViewModel` keeps what makes a session
a _call_ (ringing, auto-leave, layout, sounds, reactions, settings, the host
bridge) and takes everything else from the client. The dependency direction is
therefore `Element Call → sdk`, never the reverse. What moved out of Element
Call's `src/` to get there is listed in [`SdkMigration.md`](./SdkMigration.md).
therefore `Element Call → sdk`, never the reverse.
## The architecture
@@ -777,7 +776,6 @@ sdk/
src/utils/ mapScoped, network retry, display names, test helpers
dev/ the harness (below)
SdkArchitecture.md this document
SdkMigration.md what moved from Element Call's src/ into the SDK
```
The lint rule `element-call/sdk-import-boundary` keeps it that way: nothing under
-53
View File
@@ -1,53 +0,0 @@
# What moved from `src/` into the SDK
Element Call's `CallViewModel` used to do everything; now it consumes the SDK and
keeps only what makes a session a call. This is the inventory of what moved out
of `src/` on the way, so that a reader of either side knows where to look. It
names source files, so it ages with the code; the shape they make is in
[`SdkArchitecture.md`](./SdkArchitecture.md).
Everything in the table is gone from `src/`. Each module came with its tests;
`sdk/src/utils/test.ts` and `test-fixtures.ts` are the subset of Element Call's
test helpers they need. `Behavior`, `ObservableScope`, the observable operators,
`E2eeType`, `EncryptionSystem` and `MatrixRTCMode` are re-exported from the SDK
by their old `src/` paths, so the rest of Element Call did not have to change.
| Was in `src/` | Is in `sdk/src/` | What changed on the way |
| ----------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------- |
| `state/Behavior.ts`, `state/ObservableScope.ts`, `utils/observable.ts` | `reactive/` | as is |
| `state/SessionBehaviors.ts` | `matrixrtc/JsRtcSessionBehaviors.ts` | the memberships and key rotation behaviors only |
| `state/CallViewModel/localMember/HomeserverConnected.ts` | `matrixrtc/HomeserverConnected.ts` | the grace period defaults to `defaultSessionTimings` instead of `Config.get()` |
| `state/CallViewModel/localMember/LocalMember.ts` | `matrixrtc/LocalMembership.ts`, `media-backend/livekit/LocalMedia.ts`, `participationState.ts` | split into the Matrix side, the media side and the state machine that combines them |
| `state/CallViewModel/localMember/LocalTransport.ts`, `RtcTransportAutoDiscovery.ts` | `matrixrtc/LocalTransport.ts`, `TransportAutoDiscovery.ts` | the override and the fallback are transports from the options, not a setting and a resolved config |
| `state/CallViewModel/remoteMembers/MatrixLivekitMembers.ts` | `media-backend/livekit/LivekitBackend.ts` (`mediaFor`) | matches participants to members inside the backend; the SDK hands out `RTCMember`s |
| `state/CallViewModel/remoteMembers/MatrixMemberMetadata.ts`, `utils/displayname.ts` | `matrixrtc/MatrixMemberMetadata.ts`, `utils/displayname.ts` | as is |
| `state/CallViewModel/remoteMembers/Connection.ts` | `media-backend/livekit/Connection.ts` | throws `MatrixRTCError`s; `ConnectionState` is a discriminated union; fetches its own token for its role |
| `state/CallViewModel/remoteMembers/ConnectionFactory.ts` | `media-backend/livekit/ConnectionFactory.ts` | `LivekitConnectionFactory` builds the room options from the publish requests and `mediaQuality`, not from settings |
| `state/CallViewModel/remoteMembers/ConnectionManager.ts` | `media-backend/livekit/ConnectionMap.ts` | keyed by `TransportLocator`; the publisher role is the local transport's |
| `state/CallViewModel/localMember/Publisher.ts` | `media-backend/livekit/Publisher.ts` | rewritten on publish requests instead of `MuteStates` |
| `livekit/openIDSFU.ts`, later `livekit/auth/` | `media-backend/livekit/auth/` | the flow is chosen by the transport's shape and our role; the delegated leave timeout is a parameter |
| `livekit/options.ts` | `media-backend/livekit/livekitOptions.ts` | `buildLiveKitOptions(mediaQuality?)` only; the `Config` singleton is gone |
| `e2ee/matrixKeyProvider.ts`, the key provider selection | `media-backend/livekit/MatrixKeyProvider.ts`, `KeyProvider.ts` | as is; selection keyed off `encryptionSystem` |
| `e2ee/e2eeType.ts`, `EncryptionSystem` from `e2ee/sharedKeyManagement.ts` | `encryption.ts` | the React hooks stayed behind |
| `MatrixRTCMode`, the timings and media quality types from `config/ConfigOptions.ts` | `config.ts` | `defaultSessionTimings` and `defaultMediaQuality` replace `DEFAULT_CONFIG` |
| `utils/errors.ts` | `errors.ts` | `MatrixRTCError` with `code` and `category` replaces `ElementCallError`; messages are plain English, the host translates by code |
| `doNetworkOperationWithRetry` from `utils/matrix.ts` | `utils/network.ts` | takes a `shouldRetry` predicate |
| the join in `CallViewModel.ts` (`enterRTCSession`) | `matrixrtc/joinJsSdkSession.ts` | the timings and the delayed leave variant are parameters |
| `livekit/MatrixAudioRenderer.tsx`, the participant reads in `state/media/` | `media-backend/livekit/LivekitMediaTrack.ts`, `LivekitMemberMedia.ts` | the view models get `MediaTrack`s with `attach`; nothing in `src/` imports a LiveKit participant any more |
## What still binds Element Call to LiveKit
Each is a gap in the SDK's API rather than something to rename; the import goes
away when the SDK grows the piece. The names Element Call owes to the outside
stay as they are: the `livekit.livekit_service_url` key in `config.json`, the
persisted setting keys `extended-livekit-logs` and `custom-transport`, the
Posthog fields `callBackend` and `callReconnectingCountLivekit`, the rageshake
field `call_backend`, and the translated error strings.
| Gap | Where it shows |
| ----------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------ |
| Preview before `join()`: local tracks for the lobby, handed out as `MediaTrack`s | `LobbyView` (`usePreviewTracks`), `VideoPreview` (`LocalVideoTrack`, `facingModeFromLocalTrack`) |
| Device enumeration without LiveKit's helpers | `MediaDevices` (`createMediaDeviceObserver`), `CallView` (`Room.getLocalDevices`) |
| A `logging` option: log level and a sink for the transport's logs | `initializer.tsx`, the `extended-livekit-logs` setting |
| A `capabilities` field: output selection and background processors, asked of the SDK rather than of LiveKit | `CallFooterViewModel` (`supportsAudioOutputSelection`, `supportsBackgroundProcessors`) |
| Element Call's error and analytics types on the SDK's | `PosthogEvents` (`DisconnectReason`), `errors.ts` (`ConnectionError`, `LivekitConnectionError`) |
+1
View File
@@ -162,6 +162,7 @@ describe("createLocalMedia", () => {
media.release("camera");
expect(trackBehind(camera).stop).toHaveBeenCalled();
expect(media.tracks$.value).toEqual([microphone]);
expect(media.requests.map((r) => r.source)).toEqual(["microphone"]);
media.stop();
expect(trackBehind(microphone).stop).toHaveBeenCalled();
+1
View File
@@ -107,6 +107,7 @@ export function createLocalMedia(
if (source !== "microphone" && source !== "camera") return;
capturedTracks.get(source)?.stop();
capturedTracks.delete(source);
requests.delete(source);
publishCapturedTracks();
};
+1 -1
View File
@@ -140,7 +140,7 @@ function setup(
getDeviceId: () => deviceId,
getDomain: () => "example.org",
getSyncState: () => SyncState.Syncing,
_unstable_getRTCTransports: vi.fn().mockResolvedValue(homeserverTransports),
getRTCTransports: vi.fn().mockResolvedValue(homeserverTransports),
_unstable_getLivekitToken: vi.fn().mockResolvedValue({ jwt: publisherJWT }),
} as unknown as MatrixClient;
const members$ = new BehaviorSubject<RTCMember[]>([]);
+1
View File
@@ -184,6 +184,7 @@ export function createRTCParticipation(
encryptMedia: encryptionSystem.kind !== E2eeType.NONE,
matrixRTCMode,
sendNotificationType: options.sendNotificationType,
callIntent: options.callIntent,
applicationData: options.applicationData,
timings,
delayedLeave,
+8 -4
View File
@@ -12,6 +12,7 @@ Please see LICENSE in the repository root for full details.
*/
import {
type RTCCallIntent,
type RTCNotificationType,
type Transport,
} from "matrix-js-sdk/lib/matrixrtc";
@@ -103,11 +104,14 @@ export interface RTCParticipationOptions {
* to a notification (ringing, timeouts, declines) is the application's job.
*/
sendNotificationType?: RTCNotificationType;
/**
* The kind of call the local member intends to take part in: `"m.audio"`
* or `"m.video"`. Carried by the membership as a field of its own.
*/
callIntent?: RTCCallIntent;
/**
* Whatever the application wants to say about itself in the membership,
* under namespaced keys. Opaque to the SDK, apart from `m.call.intent`
* (`"audio"` or `"video"`), which the membership format carries as a
* field of its own.
* under namespaced keys. Opaque to the SDK.
*/
applicationData?: Record<string, unknown>;
/** Timings the host has configured; the defaults otherwise. */
@@ -287,7 +291,7 @@ export interface RTCMember {
memberId: string;
/** Which transport this member is on; undefined when the membership has none. */
transport$: Behavior<TransportMetadata | undefined>;
/** What the member's application says about itself, e.g. `m.call.intent`. */
/** What the member's application says about itself, e.g. `intent`. */
applicationData$: Behavior<Record<string, unknown>>;
displayName$: Behavior<string>;
+2 -2
View File
@@ -31,7 +31,7 @@ function client(
): Parameters<typeof discoverLocalTransport>[0] {
return {
getDomain: () => "example.org",
_unstable_getRTCTransports: vi.fn().mockResolvedValue(advertised),
getRTCTransports: vi.fn().mockResolvedValue(advertised),
};
}
@@ -54,7 +54,7 @@ describe("discoverLocalTransport", () => {
logger,
),
).resolves.toBe(hostTransport);
expect(c._unstable_getRTCTransports).not.toHaveBeenCalled();
expect(c.getRTCTransports).not.toHaveBeenCalled();
});
it("prefers the homeserver's transport over the host's fallback", async () => {
+1 -1
View File
@@ -22,7 +22,7 @@ import { TransportAutoDiscovery } from "./TransportAutoDiscovery";
* @throws {MatrixRTCTransportMissingError} where none of these exists.
*/
export async function discoverLocalTransport(
client: Pick<MatrixClient, "getDomain" | "_unstable_getRTCTransports">,
client: Pick<MatrixClient, "getDomain" | "getRTCTransports">,
transportTypes: readonly string[],
{
transport: override,
@@ -36,11 +36,7 @@ const configTransport: Transport = {
function makeClient(): MockedObject<DiscoveryClient> {
return {
getDomain: vi.fn().mockReturnValue("example.org"),
baseUrl: "https://matrix.example.org",
_unstable_getRTCTransports: vi.fn().mockResolvedValue([]),
getOpenIdToken: vi.fn(),
getDeviceId: vi.fn(),
getRTCTransports: vi.fn().mockResolvedValue([]),
} as unknown as MockedObject<DiscoveryClient>;
}
@@ -57,7 +53,7 @@ describe("TransportAutoDiscovery", () => {
"prefers backend transport other app config $transports",
async ({ transports }) => {
const client = makeClient();
client._unstable_getRTCTransports.mockResolvedValue(transports);
client.getRTCTransports.mockResolvedValue(transports);
const discovery = new TransportAutoDiscovery({
client,
@@ -71,7 +67,7 @@ describe("TransportAutoDiscovery", () => {
expect(discoveredTransport).toStrictEqual(backendTransport);
expect(discoveredTransport).not.toStrictEqual(configTransport);
expect(client._unstable_getRTCTransports).toHaveBeenCalledTimes(1);
expect(client.getRTCTransports).toHaveBeenCalledTimes(1);
},
);
@@ -81,10 +77,7 @@ describe("TransportAutoDiscovery", () => {
url: "wss://sfu.example.org",
};
const client = makeClient();
client._unstable_getRTCTransports.mockResolvedValue([
backendTransport,
mLivekit,
]);
client.getRTCTransports.mockResolvedValue([backendTransport, mLivekit]);
const discovery = new TransportAutoDiscovery({
client,
@@ -100,7 +93,7 @@ describe("TransportAutoDiscovery", () => {
it("Retries limit_exceeded backend transport", async () => {
const client = makeClient();
client._unstable_getRTCTransports
client.getRTCTransports
.mockRejectedValueOnce(
new MatrixError(
{
@@ -124,7 +117,7 @@ describe("TransportAutoDiscovery", () => {
backendTransport,
);
expect(client._unstable_getRTCTransports).toHaveBeenCalledTimes(2);
expect(client.getRTCTransports).toHaveBeenCalledTimes(2);
});
const INVALID_TEST_CASES: Array<{ transports: Transport[] }> = [
@@ -135,7 +128,7 @@ describe("TransportAutoDiscovery", () => {
"falls back to config when backend has no (valid) livekit transports $transports",
async ({ transports }) => {
const client = makeClient();
client._unstable_getRTCTransports.mockResolvedValue(transports);
client.getRTCTransports.mockResolvedValue(transports);
const discovery = new TransportAutoDiscovery({
client,
@@ -152,7 +145,7 @@ describe("TransportAutoDiscovery", () => {
it("returns null when backend and config are all unavailable", async () => {
const client = makeClient();
client._unstable_getRTCTransports.mockResolvedValue([]);
client.getRTCTransports.mockResolvedValue([]);
const discovery = new TransportAutoDiscovery({
client,
+2 -5
View File
@@ -10,10 +10,7 @@ import { type Logger } from "matrix-js-sdk/lib/logger";
import { doNetworkOperationWithRetry } from "../utils/network";
type TransportDiscoveryClient = Pick<
MatrixClient,
"getDomain" | "_unstable_getRTCTransports"
>;
type TransportDiscoveryClient = Pick<MatrixClient, "getRTCTransports">;
export interface TransportAutoDiscoveryProps {
client: TransportDiscoveryClient;
@@ -74,7 +71,7 @@ export class TransportAutoDiscovery {
this.logger.info("First try to use getRTCTransports end point ...");
try {
const transportList = await doNetworkOperationWithRetry(async () =>
client._unstable_getRTCTransports(),
client.getRTCTransports(),
);
for (const type of this.transportTypes) {
const first = transportList.find(
+8 -6
View File
@@ -60,7 +60,7 @@ describe("joinJsSdkSession", () => {
expect(s.join).toHaveBeenCalledWith(ownMemberMock, [transportWithAlias], {
notificationType: "ring",
callIntent: undefined,
applicationData: {},
applicationData: undefined,
manageMediaKeys: true,
delayedLeaveEventRestartMs: 300_000,
delayedLeaveEventDelayMs: 3_600_000,
@@ -95,13 +95,14 @@ describe("joinJsSdkSession", () => {
);
});
it("lifts the call intent out of the application data and sends sticky events in Matrix 2.0 mode", () => {
it("passes the call intent and application data through and sends sticky events in Matrix 2.0 mode", () => {
const s = session();
joinJsSdkSession("!room:example.org", s, ownMemberMock, transport, {
encryptMedia: true,
matrixRTCMode: MatrixRTCMode.Matrix_2_0,
applicationData: { "m.call.intent": "audio", custom: 1 },
callIntent: "m.audio",
applicationData: { custom: 1 },
timings: defaultSessionTimings,
delayedLeave: defaultSessionTimings.delayedLeave,
});
@@ -110,7 +111,7 @@ describe("joinJsSdkSession", () => {
ownMemberMock,
[transportWithAlias],
expect.objectContaining({
callIntent: "audio",
callIntent: "m.audio",
applicationData: { custom: 1 },
unstableSendStickyEvents: true,
}),
@@ -124,7 +125,8 @@ describe("joinJsSdkSession", () => {
joinJsSdkSession("!room:example.org", s, ownMemberMock, transport, {
encryptMedia: true,
matrixRTCMode: MatrixRTCMode.Matrix_2_0,
applicationData: { "m.call.intent": "audio", custom: 1 },
callIntent: "m.audio",
applicationData: { custom: 1 },
timings: defaultSessionTimings,
delayedLeave: defaultSessionTimings.delayedLeave,
});
@@ -133,7 +135,7 @@ describe("joinJsSdkSession", () => {
ownMemberMock,
[transport],
expect.objectContaining({
callIntent: "audio",
callIntent: "m.audio",
applicationData: { custom: 1 },
unstableSendStickyEvents: true,
}),
+3 -5
View File
@@ -23,6 +23,7 @@ interface Options {
encryptMedia: boolean;
matrixRTCMode: MatrixRTCMode;
sendNotificationType?: RTCNotificationType;
callIntent?: RTCCallIntent;
applicationData?: Record<string, unknown>;
timings: SessionTimings;
/** The delayed leave timings to use: delegated to the SFU or not. */
@@ -43,15 +44,12 @@ export function joinJsSdkSession(
encryptMedia,
matrixRTCMode,
sendNotificationType,
callIntent,
applicationData,
timings,
delayedLeave,
}: Options,
): void {
// The membership format carries the intent as a field of its own
const { "m.call.intent": intent, ...rest } = applicationData ?? {};
const callIntent =
typeof intent === "string" ? (intent as RTCCallIntent) : undefined;
// Give up on the network as soon as either the sync has been down for the
// grace period or the delayed leave has probably been sent, whichever is
// sooner
@@ -78,7 +76,7 @@ export function joinJsSdkSession(
{
notificationType: sendNotificationType,
callIntent,
applicationData: rest,
applicationData,
manageMediaKeys: encryptMedia,
delayedLeaveEventRestartMs: delayedLeave.restart_ms,
delayedLeaveEventDelayMs: delayedLeave.delay_ms,
@@ -225,6 +225,51 @@ describe("createLivekitMediaTrack", () => {
);
});
// LiveKit's media stream track is the processed one while a processor runs,
// and that one names no device; the camera behind it does
it.each([
{ deviceId: "cam-1", switches: false },
{ deviceId: "cam-2", switches: true },
])(
"leaves a processed camera alone unless its device changes: $deviceId",
async ({ deviceId, switches }) => {
vi.stubGlobal("MediaStreamTrack", class {});
onTestFinished(() => {
vi.unstubAllGlobals();
});
const camera = Object.assign(Object.create(LocalVideoTrack.prototype), {
source: Track.Source.Camera,
_mediaStreamTrack: {
readyState: "live",
label: "",
getSettings: () => ({ deviceId: "cam-1" }),
},
// What LiveKit shows as the track, with no device of its own
processor: {
processedTrack: { label: "", getSettings: () => ({}) },
},
}) as LocalVideoTrack;
const switchActiveDevice = vi.fn().mockResolvedValue(true);
const { participant, publication, room } = fakes({
track: camera as unknown as FakeTrack,
});
const track = createLocalLivekitMediaTrack(
scope,
Object.assign(participant, {
isLocal: true,
}) as unknown as LocalParticipant,
publication as unknown as TrackPublication,
Object.assign(room, { switchActiveDevice }),
vi.fn(),
);
await track.setDevice(deviceId);
expect(switchActiveDevice.mock.calls).toEqual(
switches ? [["videoinput", deviceId, false]] : [],
);
},
);
// LiveKit stops a camera that is turned off; the processor is built when it
// is turned on again
it.each([
@@ -199,11 +199,13 @@ export function createLocalLivekitMediaTrack(
throw new MatrixRTCError(`A ${base.source} track has no device`);
// LiveKit decides whether to restart by the constraints it keeps, which
// a browser that does not echo the device id back (Firefox) leaves
// without one; a track already on the device is left alone here
// without one; a track already on the device is left alone here. Asked
// of the camera track itself: with a processor on, LiveKit's media
// stream track is the processed one, which names no device
const current = publication.track;
if (
current instanceof LocalTrack &&
current.mediaStreamTrack.getSettings().deviceId === deviceId
(await current.getDeviceId(false)) === deviceId
)
return;
await room.switchActiveDevice(
+12 -2
View File
@@ -30,7 +30,7 @@ import { HomePage } from "./home/HomePage";
import { LoginPage } from "./auth/LoginPage";
import { RegisterPage } from "./auth/RegisterPage";
import { RoomPage } from "./room/RoomPage";
import { ClientProvider } from "./ClientContext";
import { type ClientLoader, ClientProvider } from "./ClientContext";
import { ErrorPage, LoadingPage } from "./FullScreenView";
import { Initializer } from "./initializer";
import { type WidgetHelpers } from "./widget";
@@ -57,6 +57,16 @@ import { LeaveToHomeProvider } from "./LeaveToHomeContext";
const SentryRoute = Sentry.withSentryReactRouterV7Routing(Route);
// Element Call's own session, for when no host supplies a client.
// - Passed in here, not imported by ClientProvider: this module is only in the
// app builds, so the client setup and the store worker behind it stay out of
// the component package (see the `loadClient` prop).
// - Imported on demand, as it was before, so that a widget, which gets its
// client from the host, never downloads it.
// - Module-level, so that the provider's effect sees one stable loader.
const loadStandaloneClient: ClientLoader = async (...args) =>
(await import("./utils/spa")).initSPA(...args);
interface SimpleProviderProps {
children: JSX.Element;
}
@@ -146,7 +156,7 @@ export const App: FC<Props> = ({ vm, widget }) => {
const content =
loaded && clientReady ? (
<ClientProvider client={widgetClient}>
<ClientProvider client={widgetClient} loadClient={loadStandaloneClient}>
<MediaDevicesContext value={vm.mediaDevices}>
<BackgroundEffectsProvider effects={vm.backgroundEffects}>
<Sentry.ErrorBoundary
+25 -6
View File
@@ -11,7 +11,11 @@ import { BrowserRouter } from "react-router-dom";
import { type MatrixClient } from "matrix-js-sdk";
import { type FC } from "react";
import { ClientProvider, useClientState } from "./ClientContext";
import {
type ClientLoader,
ClientProvider,
useClientState,
} from "./ClientContext";
const mockClient = (userId = "@alice:example.org"): MatrixClient =>
({
@@ -95,13 +99,13 @@ test("follows the client when the host swaps it", () => {
test("finds a client of its own when the host supplies none", async () => {
const client = mockClient();
vi.doMock("./utils/spa", () => ({
initSPA: vi.fn().mockResolvedValue({ client, passwordlessUser: true }),
}));
const loadClient = vi
.fn<ClientLoader>()
.mockResolvedValue({ client, passwordlessUser: true });
const { container } = render(
<BrowserRouter>
<ClientProvider>
<ClientProvider loadClient={loadClient}>
<ShowClientState />
</ClientProvider>
</BrowserRouter>,
@@ -110,6 +114,21 @@ test("finds a client of its own when the host supplies none", async () => {
// Nothing to show until a session has been restored or created
expect(container.textContent).toBe("loading");
await waitFor(() => expect(container.textContent).toBe("@alice:example.org"));
// Given the session store to restore from and to clear on a stale session
expect(loadClient).toHaveBeenCalledWith(
expect.any(Function),
expect.any(Function),
);
});
vi.doUnmock("./utils/spa");
test("has no one signed in when given neither a client nor a way to load one", async () => {
const { container } = render(
<BrowserRouter>
<ClientProvider>
<ShowClientState />
</ClientProvider>
</BrowserRouter>,
);
await waitFor(() => expect(container.textContent).toBe("unauthenticated"));
});
+32 -12
View File
@@ -143,9 +143,34 @@ interface Props {
* own.
*/
client?: MatrixClient;
/**
* How Element Call finds a client when none is supplied: the standalone app
* restores or creates a session of its own. Stable, as a hook dependency.
* Left out along with the client, no one is signed in.
*
* A prop, rather than this module importing `utils/spa` itself, because:
* - This module is in the component build, and so is everything it imports,
* dynamically or not: a dynamic import becomes a chunk in the package,
* not a module left out of it.
* - Behind `utils/spa` sits `utils/initClient` and the client store worker,
* which the bundler emits as soon as it loads the import (see there).
* - `App.tsx`, which the component never imports, is where the standalone
* app passes the loader in; the component passes nothing, having the
* host's client.
*/
loadClient?: ClientLoader;
}
export const ClientProvider: FC<Props> = ({ children, client }) => {
/**
* Restores or creates a session of Element Call's own, kept where the given
* functions keep it. Resolves to null when no one is signed in.
*/
export type ClientLoader = (
loadSession: () => Session | undefined,
clearSession: () => void,
) => Promise<InitResult | null>;
export const ClientProvider: FC<Props> = ({ children, client, loadClient }) => {
const leaveToHome = useLeaveToHome();
const hostBridge = useHostBridge();
@@ -179,9 +204,13 @@ export const ClientProvider: FC<Props> = ({ children, client }) => {
// React does in strict mode), we need to make sure not to doubly initialize
// the client.
if (initializing.current) return;
if (loadClient === undefined) {
setInitClientState(null);
return;
}
initializing.current = true;
loadClient()
loadClient(loadSession, clearSession)
.then((initResult) => {
setInitClientState(initResult);
if (PosthogAnalytics.instance.isEnabled())
@@ -189,7 +218,7 @@ export const ClientProvider: FC<Props> = ({ children, client }) => {
})
.catch((err) => logger.error(err))
.finally(() => (initializing.current = false));
}, [client]);
}, [client, loadClient]);
const changePassword = useCallback(
async (password: string) => {
@@ -360,15 +389,6 @@ export type InitResult = {
passwordlessUser: boolean;
};
/**
* Restores or creates a session of Element Call's own. Only reached when no
* client was supplied for it to use.
*/
async function loadClient(): Promise<InitResult | null> {
const { initSPA } = await import("./utils/spa");
return initSPA(loadSession, clearSession);
}
export interface Session {
user_id: string;
device_id: string;
+10 -8
View File
@@ -18,6 +18,8 @@ Please see LICENSE in the repository root for full details.
*/
import {
RTC_CALL_INTENT_AUDIO,
RTC_CALL_INTENT_VIDEO,
type RTCCallIntent,
type RTCNotificationType,
} from "matrix-js-sdk/lib/matrixrtc";
@@ -179,40 +181,40 @@ export function configurationForIntent(intent: UserIntent): UrlConfiguration {
switch (intent) {
case UserIntent.StartNewCall:
preset.skipLobby = false;
preset.callIntent = "video";
preset.callIntent = RTC_CALL_INTENT_VIDEO;
break;
case UserIntent.JoinExistingCall:
// On desktop this will be overridden based on which button was used to join the call
preset.skipLobby = false;
preset.callIntent = "video";
preset.callIntent = RTC_CALL_INTENT_VIDEO;
break;
case UserIntent.StartNewCallVoice:
preset.skipLobby = false;
preset.callIntent = "audio";
preset.callIntent = RTC_CALL_INTENT_AUDIO;
break;
case UserIntent.JoinExistingCallVoice:
// On desktop this will be overridden based on which button was used to join the call
preset.skipLobby = false;
preset.callIntent = "audio";
preset.callIntent = RTC_CALL_INTENT_AUDIO;
break;
case UserIntent.StartNewCallDMVoice:
preset.callIntent = "audio";
preset.callIntent = RTC_CALL_INTENT_AUDIO;
// Fall through
case UserIntent.StartNewCallDM:
preset.skipLobby = true;
preset.sendNotificationType = "ring";
preset.autoLeaveWhenOthersLeft = true;
preset.waitForCallPickup = true;
preset.callIntent = preset.callIntent ?? "video";
preset.callIntent = preset.callIntent ?? RTC_CALL_INTENT_VIDEO;
break;
case UserIntent.JoinExistingCallDMVoice:
preset.callIntent = "audio";
preset.callIntent = RTC_CALL_INTENT_AUDIO;
// Fall through
case UserIntent.JoinExistingCallDM:
// On desktop this will be overridden based on which button was used to join the call
preset.skipLobby = true;
preset.autoLeaveWhenOthersLeft = true;
preset.callIntent = preset.callIntent ?? "video";
preset.callIntent = preset.callIntent ?? RTC_CALL_INTENT_VIDEO;
break;
// Non widget usecase defaults
default:
+10 -6
View File
@@ -21,6 +21,10 @@ import {
UserIntent,
} from "../src/UrlParams";
import { mockConfig } from "./utils/test";
import {
RTC_CALL_INTENT_AUDIO,
RTC_CALL_INTENT_VIDEO,
} from "matrix-js-sdk/lib/matrixrtc";
const ROOM_NAME = "roomNameHere";
const ROOM_ID = "!d45f138fsd";
@@ -275,7 +279,7 @@ describe("UrlParams", () => {
).toMatchObject({
...startNewCallDefaults("desktop"),
skipLobby: false,
callIntent: "video",
callIntent: RTC_CALL_INTENT_VIDEO,
});
});
@@ -322,7 +326,7 @@ describe("UrlParams", () => {
).toMatchObject({
...startNewCallDefaults("desktop"),
skipLobby: false,
callIntent: "audio",
callIntent: RTC_CALL_INTENT_AUDIO,
});
});
@@ -333,7 +337,7 @@ describe("UrlParams", () => {
),
).toMatchObject({
...joinExistingCallDefaults("desktop"),
callIntent: "audio",
callIntent: RTC_CALL_INTENT_AUDIO,
});
});
@@ -348,7 +352,7 @@ describe("UrlParams", () => {
sendNotificationType: "ring",
autoLeaveWhenOthersLeft: true,
waitForCallPickup: true,
callIntent: "audio",
callIntent: RTC_CALL_INTENT_AUDIO,
});
});
@@ -363,7 +367,7 @@ describe("UrlParams", () => {
skipLobby: true,
autoLeaveWhenOthersLeft: true,
waitForCallPickup: false,
callIntent: "video",
callIntent: RTC_CALL_INTENT_VIDEO,
});
});
@@ -377,7 +381,7 @@ describe("UrlParams", () => {
skipLobby: true,
autoLeaveWhenOthersLeft: true,
waitForCallPickup: false,
callIntent: "audio",
callIntent: RTC_CALL_INTENT_AUDIO,
});
});
});
+1 -1
View File
@@ -13,7 +13,7 @@ import {
type MatrixClient,
} from "matrix-js-sdk";
import { initClient } from "../utils/matrix";
import { initClient } from "../utils/initClient";
import { type Session } from "../ClientContext";
/**
* This provides the login method to login using user credentials.
+1 -1
View File
@@ -14,7 +14,7 @@ import {
} from "matrix-js-sdk";
import { logger } from "matrix-js-sdk/lib/logger";
import { initClient } from "../utils/matrix";
import { initClient } from "../utils/initClient";
import { type Session } from "../ClientContext";
import { Config } from "../config/Config";
import { useUrlParams } from "../UrlParams";
+1
View File
@@ -141,6 +141,7 @@ export const Default: Story = {
backgroundImageRefusal: undefined,
addBackgroundImage: fn(),
removeBackgroundEffect: fn(),
cameraTrack: null,
hangup: fn(),
buttonSize: "lg",
showFooter: true,
+6
View File
@@ -33,7 +33,9 @@ import { useBehavior } from "../useBehavior";
import { type LayoutSwitchViewModel } from "../state/LayoutSwitchViewModel";
import { LayoutSwitch } from "../room/LayoutSwitch";
import { type BackgroundEffectGridItem } from "./BackgroundEffectGrid";
import { SelfPreview } from "./SelfPreview";
import { type UnusableReason } from "../tracks/backgroundImages";
import { type LocalVideoMediaTrack } from "@element-hq/matrixrtc-sdk";
export interface BackgroundImageRefusal {
reason: UnusableReason | "not-kept";
@@ -92,6 +94,8 @@ export interface FooterState {
backgroundEffectSettling: boolean;
/** Why the last file offered couldn't be kept, if it couldn't. */
backgroundImageRefusal: BackgroundImageRefusal | undefined;
/** The camera to preview, with the effect in force; null while it is off. */
cameraTrack: LocalVideoMediaTrack | null;
showFooter: boolean;
/* This is needed for WindowMode = "flat" */
@@ -181,6 +185,7 @@ export const CallFooter: FC<FooterProps> = ({
const backgroundEffectSettling = useBehavior(vm.backgroundEffectSettling$);
const addBackgroundImage = useBehavior(vm.addBackgroundImage$);
const removeBackgroundEffect = useBehavior(vm.removeBackgroundEffect$);
const cameraTrack = useBehavior(vm.cameraTrack$);
const refusal = useBehavior(vm.backgroundImageRefusal$);
const refusalMessage = useMemo(() => {
switch (refusal?.reason) {
@@ -264,6 +269,7 @@ export const CallFooter: FC<FooterProps> = ({
backgroundEffectSettling={backgroundEffectSettling}
onAddBackgroundImage={addBackgroundImage}
onRemoveBackgroundEffect={removeBackgroundEffect}
selfPreview={<SelfPreview track={cameraTrack} />}
backgroundImageRefusal={refusalMessage}
backgroundEffectNotice={
backgroundEffectNotice === "unavailable"
+94 -10
View File
@@ -7,14 +7,17 @@ Please see LICENSE in the repository root for full details.
import { afterEach, describe, expect, it, vi } from "vitest";
import { BehaviorSubject } from "rxjs";
import { type LocalVideoMediaTrack } from "@element-hq/matrixrtc-sdk";
import {
flushPromises,
testScope,
mockMuteStates,
mockMediaDevices,
mockVideoTrack,
} from "../utils/test";
import { constant } from "../state/Behavior";
import { type Behavior, constant } from "../state/Behavior";
import { type BackgroundEffectsState } from "../state/BackgroundEffects";
import type { CallViewModel } from "../state/CallViewModel/CallViewModel";
import type { Layout } from "../state/layout-types";
import type { SpotlightTileViewModel } from "../state/TileViewModel";
@@ -78,6 +81,7 @@ vi.mock("livekit-client", () => ({
function buildMinimalCallViewModel(layout: Layout): CallViewModel {
return {
layout$: constant(layout),
camera$: constant(undefined),
edgeToEdge$: constant(false),
showHeader$: constant(false),
hangup: (): void => {},
@@ -109,6 +113,16 @@ const pipLayout: Layout = {
spotlight: {} as SpotlightTileViewModel,
};
const noPipeline = constant<BackgroundEffectsState>({
supported: false,
processor: undefined,
});
/** The provider's verdict, which the SDK's check stands for here. */
function pipelineAsTheSdkSays(): Behavior<BackgroundEffectsState> {
return constant({ supported: sdkSupportMock(), processor: undefined });
}
const twoMicsAndOneCamMediaDevices = mockMediaDevices({
audioInput: {
available$: constant(
@@ -140,7 +154,7 @@ describe("createCallFooterViewModel", () => {
buildMinimalCallViewModel(gridLayout),
mockMuteStates(),
twoMicsAndOneCamMediaDevices,
constant(false),
noPipeline,
/* reactionIdentifier */ undefined,
{ showControls: true, header: HeaderStyle.Standard },
);
@@ -183,7 +197,7 @@ describe("createCallFooterViewModel", () => {
select: vi.fn(),
},
}),
constant(false),
noPipeline,
/* reactionIdentifier */ undefined,
{ showControls: true, header: HeaderStyle.Standard },
);
@@ -202,7 +216,7 @@ describe("createCallFooterViewModel", () => {
buildMinimalCallViewModel(layout),
mockMuteStates(),
twoMicsAndOneCamMediaDevices,
constant(false),
noPipeline,
/* reactionIdentifier */ undefined,
{ showControls: true, header: HeaderStyle.Standard },
);
@@ -225,7 +239,7 @@ describe("createCallFooterViewModel", () => {
buildMinimalCallViewModel(gridLayout),
mockMuteStates(),
twoMicsAndOneCamMediaDevices,
constant(false),
noPipeline,
/* reactionIdentifier */ undefined,
{ showControls: true, header: HeaderStyle.Standard },
);
@@ -271,7 +285,8 @@ describe("createCallFooterViewModel", () => {
testScope(),
mockMuteStates(),
twoMicsAndOneCamMediaDevices,
constant(false),
pipelineAsTheSdkSays(),
/* camera$ */ constant(null),
/* openSettings */ undefined,
/* hangup */ undefined,
/* showLogo */ false,
@@ -356,21 +371,52 @@ describe("createCallFooterViewModel", () => {
it("passes on the wait for the first effect", () => {
platformMock.mockReturnValue("desktop");
const settling$ = new BehaviorSubject(true);
const pipeline$ = new BehaviorSubject<BackgroundEffectsState>({
supported: true,
processor: undefined,
wait: "settling",
});
const vm = createLobbyFooterViewModel(
testScope(),
mockMuteStates(),
twoMicsAndOneCamMediaDevices,
settling$,
pipeline$,
/* camera$ */ constant(null),
/* openSettings */ undefined,
/* hangup */ undefined,
/* showLogo */ false,
);
expect(vm.backgroundEffectSettling$.value).toBe(true);
settling$.next(false);
pipeline$.next({ ...pipeline$.value, wait: undefined });
expect(vm.backgroundEffectSettling$.value).toBe(false);
});
it("offers nothing once the pipeline fails to build, keeping the choice", () => {
platformMock.mockReturnValue("desktop");
backgroundEffectSetting.setValue("blur");
const pipeline$ = new BehaviorSubject<BackgroundEffectsState>({
supported: true,
processor: undefined,
});
const vm = createLobbyFooterViewModel(
testScope(),
mockMuteStates(),
twoMicsAndOneCamMediaDevices,
pipeline$,
/* camera$ */ constant(null),
/* openSettings */ undefined,
/* hangup */ undefined,
/* showLogo */ false,
);
expect(vm.backgroundEffect$.value).toBe("blur");
pipeline$.next({ supported: false, processor: undefined });
expect(vm.selectBackgroundEffect$.value).toBeUndefined();
expect(vm.addBackgroundImage$.value).toBeUndefined();
expect(vm.backgroundEffect$.value).toBe("none");
expect(backgroundEffectSetting.getValue()).toBe("blur");
});
it("adding is unavailable at the limit", () => {
sdkSupportMock.mockReturnValue(true);
const added$ = addedBackgrounds.added$ as BehaviorSubject<
@@ -427,6 +473,44 @@ describe("createCallFooterViewModel", () => {
expect(store.remove).toHaveBeenCalledWith("spare");
});
it("previews the lobby's camera while there is one", () => {
const track = mockVideoTrack({});
const camera$ = new BehaviorSubject<LocalVideoMediaTrack | null>(track);
const vm = createLobbyFooterViewModel(
testScope(),
mockMuteStates(),
twoMicsAndOneCamMediaDevices,
noPipeline,
camera$,
/* openSettings */ undefined,
/* hangup */ undefined,
/* showLogo */ false,
);
expect(vm.cameraTrack$.value).toBe(track);
// The lobby releases the camera when it is turned off
camera$.next(null);
expect(vm.cameraTrack$.value).toBeNull();
});
it("previews the call's camera only while it is not muted", () => {
const muted$ = new BehaviorSubject(false);
const track = mockVideoTrack({ muted$ });
const vm = createCallFooterViewModel(
testScope(),
{ ...buildMinimalCallViewModel(gridLayout), camera$: constant(track) },
mockMuteStates(),
twoMicsAndOneCamMediaDevices,
noPipeline,
/* reactionIdentifier */ undefined,
{ showControls: true, header: HeaderStyle.Standard },
);
expect(vm.cameraTrack$.value).toBe(track);
muted$.next(true);
expect(vm.cameraTrack$.value).toBeNull();
muted$.next(false);
expect(vm.cameraTrack$.value).toBe(track);
});
it("availability is the same before and during a call", () => {
for (const supported of [true, false]) {
sdkSupportMock.mockReturnValue(supported);
@@ -436,7 +520,7 @@ describe("createCallFooterViewModel", () => {
buildMinimalCallViewModel(gridLayout),
mockMuteStates(),
twoMicsAndOneCamMediaDevices,
constant(false),
pipelineAsTheSdkSays(),
/* reactionIdentifier */ undefined,
{ showControls: true, header: HeaderStyle.Standard },
);
+46 -19
View File
@@ -11,13 +11,12 @@ import {
distinctUntilChanged,
map,
type Observable,
of,
switchMap,
} from "rxjs";
import { supportsAudioOutputSelection } from "livekit-client";
import {
supportsBackgroundProcessors,
supportsModernBackgroundProcessors,
} from "@livekit/track-processors";
import { type LocalVideoMediaTrack } from "@element-hq/matrixrtc-sdk";
import { supportsModernBackgroundProcessors } from "@livekit/track-processors";
import { logger } from "matrix-js-sdk/lib/logger";
import {
@@ -33,6 +32,7 @@ import {
} from "../tracks/backgroundImages";
import { type CallViewModel } from "../state/CallViewModel/CallViewModel";
import { type BackgroundEffectsState } from "../state/BackgroundEffects";
import { type MenuOptions } from "./MediaMuteAndSwitchButton";
import { type MediaDevices } from "../state/MediaDevices";
import {
@@ -87,7 +87,9 @@ function buildDeviceBehaviors(
mediaDevices: MediaDevices,
/** return empty arrays for audioOptions and videoOptions*/
disableSwitcher$: Behavior<boolean>,
backgroundEffectSettling$: Observable<boolean>,
backgroundEffectsState$: Observable<BackgroundEffectsState>,
/** The camera as the pipeline has it, for the menu's preview. */
camera$: Observable<LocalVideoMediaTrack | null | undefined>,
): Pick<
ViewModel<FooterSnapshot>,
| "audioOptions$"
@@ -107,6 +109,7 @@ function buildDeviceBehaviors(
| "addBackgroundImage$"
| "backgroundImageRefusal$"
| "removeBackgroundEffect$"
| "cameraTrack$"
> {
const options$ = (
available$: Behavior<Map<string, MenuOptions["label"]>>,
@@ -123,10 +126,14 @@ function buildDeviceBehaviors(
),
);
const supported = supportsBackgroundProcessors();
// Also false once the pipeline has failed to build this session.
const supported$ = backgroundEffectsState$.pipe(
map(({ supported }) => supported === true),
distinctUntilChanged(),
);
const slow = !supportsModernBackgroundProcessors();
const offered$ = disableSwitcher$.pipe(
map((switcherDisabled) => !switcherDisabled && supported),
const offered$ = combineLatest([disableSwitcher$, supported$]).pipe(
map(([switcherDisabled, supported]) => !switcherDisabled && supported),
);
// A new object for each refusal, so the same one twice is shown twice.
const refusal$ = new BehaviorSubject<BackgroundImageRefusal | undefined>(
@@ -157,8 +164,10 @@ function buildDeviceBehaviors(
),
selectVideoButtonOption$: constant(mediaDevices.videoInput.select),
backgroundEffect$: scope.behavior(
backgroundEffectSetting.value$.pipe(
map((raw) => (supported ? serializeEffect(parseEffect(raw)) : "none")),
combineLatest([backgroundEffectSetting.value$, supported$]).pipe(
map(([raw, supported]) =>
supported ? serializeEffect(parseEffect(raw)) : "none",
),
),
),
selectBackgroundEffect$: scope.behavior(
@@ -186,7 +195,10 @@ function buildDeviceBehaviors(
),
),
backgroundEffectSettling$: scope.behavior(
backgroundEffectSettling$.pipe(distinctUntilChanged()),
backgroundEffectsState$.pipe(
map(({ wait }) => wait !== undefined),
distinctUntilChanged(),
),
false,
),
// Kept and offered, not put on: that waits for the user to choose it.
@@ -208,6 +220,17 @@ function buildDeviceBehaviors(
),
),
backgroundImageRefusal$: refusal$,
// None while muted: the call mutes its track rather than releasing it.
cameraTrack$: scope.behavior(
camera$.pipe(
switchMap((camera) =>
camera
? camera.muted$.pipe(map((muted) => (muted ? null : camera)))
: of(null),
),
),
null,
),
// Never the one in force: taking it away would leave nothing chosen.
removeBackgroundEffect$: scope.behavior(
offered$.pipe(
@@ -237,8 +260,8 @@ function buildDeviceBehaviors(
* @param callModel - The root CallViewModel; provides layout, grid mode, reactions, etc.
* @param muteStates - Audio and video mute state + toggles.
* @param mediaDevices - Available and selected input devices.
* @param backgroundEffectSettling$ - Whether the first effect chosen is still
* being prepared.
* @param backgroundEffectsState$ - The background pipeline's state: whether
* effects run here, and the wait for the first.
* @param reactionIdentifier - The local user's reaction identifier string, or
* undefined when reactions are not supported (hides the reaction button).
* @param options - `showControls`: whether the call controls should be shown.
@@ -249,7 +272,7 @@ export function createCallFooterViewModel(
callModel: CallViewModel,
muteStates: MuteStates,
mediaDevices: MediaDevices,
backgroundEffectSettling$: Observable<boolean>,
backgroundEffectsState$: Observable<BackgroundEffectsState>,
reactionIdentifier: string | undefined,
options: { showControls: boolean; header: HeaderStyle },
): ViewModel<FooterSnapshot> {
@@ -268,7 +291,8 @@ export function createCallFooterViewModel(
scope,
mediaDevices,
disableDeviceSwitcher$,
backgroundEffectSettling$,
backgroundEffectsState$,
callModel.camera$,
),
// candidat to move into the FooterViewModel
showFooter$: callModel.showFooter$,
@@ -331,8 +355,9 @@ export function createCallFooterViewModel(
* @param scope - ObservableScope that bounds the lifetime of derived behaviors.
* @param muteStates - Audio and video mute state + toggles.
* @param mediaDevices - Available and selected input devices.
* @param backgroundEffectSettling$ - Whether the first effect chosen is still
* being prepared.
* @param backgroundEffectsState$ - The background pipeline's state: whether
* effects run here, and the wait for the first.
* @param camera$ - The lobby's camera, for the camera menu's preview.
* @param openSettings - Callback to open the settings modal, or undefined.
* @param hangup - Callback to leave/cancel, or undefined (hides the button).
* @param showLogo - Whether to show the Element Call logo.
@@ -341,7 +366,8 @@ export function createLobbyFooterViewModel(
scope: ObservableScope,
muteStates: MuteStates,
mediaDevices: MediaDevices,
backgroundEffectSettling$: Observable<boolean>,
backgroundEffectsState$: Observable<BackgroundEffectsState>,
camera$: Behavior<LocalVideoMediaTrack | null>,
openSettings: (() => void) | undefined,
hangup: (() => void) | undefined,
showLogo: boolean,
@@ -389,7 +415,8 @@ export function createLobbyFooterViewModel(
scope,
mediaDevices,
constant(false),
backgroundEffectSettling$,
backgroundEffectsState$,
camera$,
),
};
}
@@ -97,3 +97,35 @@ Please see LICENSE in the repository root for full details.
.refusal {
margin: var(--cpd-space-2x) var(--cpd-space-4x) 0;
}
.selfPreview {
position: relative;
display: flex;
align-items: center;
justify-content: center;
flex-shrink: 0;
aspect-ratio: 16 / 9;
margin-block-end: var(--cpd-space-2x);
overflow: clip;
border-start-start-radius: var(--cpd-space-3x);
border-start-end-radius: var(--cpd-space-3x);
background: var(--cpd-color-bg-subtle-secondary);
color: var(--cpd-color-icon-secondary);
}
.selfPreview > video {
position: absolute;
inset: 0;
inline-size: 100%;
block-size: 100%;
object-fit: cover;
/* Rounded itself too: a video can be drawn on a layer of its own, which the
frame's rounded clip doesn't always reach. */
border-radius: inherit;
}
/* Up to the menu's frame, over the padding at its top. */
.selfPreviewPinned,
.deviceListLeadsWithPreview {
margin-block-start: calc(-1 * var(--cpd-space-5x));
}
@@ -12,6 +12,7 @@ import type { Meta, StoryObj } from "@storybook/react-vite";
import { MediaMuteAndSwitchButton } from "./MediaMuteAndSwitchButton";
import styles from "./MediaMuteAndSwitchButton.module.css";
import { type BackgroundEffectGridItem } from "./BackgroundEffectGrid";
import { SelfPreview } from "./SelfPreview";
import { shippedBackgrounds } from "../tracks/backgroundEffectCatalog";
import meterStyles from "./MicrophoneLevelMeter.module.css";
import { MediaDevicesContext } from "../MediaDevicesContext";
@@ -60,6 +61,50 @@ const WithAMicrophone: FC<{ children: ReactNode }> = ({ children }) => {
return <>{children}</>;
};
/** A camera for the preview: a figure against a room, drawn to a stream. */
const FakeCamera: FC = () => {
const [stream, setStream] = useState<MediaStream>();
useEffect(() => {
const canvas = document.createElement("canvas");
canvas.width = 640;
canvas.height = 360;
const context = canvas.getContext("2d")!;
const draw = (): void => {
context.fillStyle = "#6b7f8e";
context.fillRect(0, 0, 640, 360);
context.fillStyle = "#c9a27e";
context.beginPath();
context.arc(320, 150, 62, 0, 2 * Math.PI);
context.fill();
context.fillStyle = "#34495e";
context.beginPath();
context.ellipse(320, 380, 150, 140, 0, Math.PI, 2 * Math.PI);
context.fill();
};
draw();
// Redrawn, so the stream keeps delivering frames.
const timer = setInterval(draw, 200);
const captured = canvas.captureStream(10);
setStream(captured);
return (): void => {
clearInterval(timer);
captured.getTracks().forEach((track) => track.stop());
};
}, []);
return (
<video
data-testid="preview-camera"
autoPlay
muted
playsInline
ref={(video): void => {
if (video && stream && video.srcObject !== stream)
video.srcObject = stream;
}}
/>
);
};
/** Supplies a call-sized root. Without one the list is bounded by the whole Storybook frame. */
const WithACallArea: FC<{ children: ReactNode; blockSize?: number }> = ({
children,
@@ -171,8 +216,8 @@ export const VideoMute: Story = {
{ label: { type: "name", name: "Camera 1" }, id: "1" },
{ label: { type: "name", name: "Camera 2" }, id: "2" },
],
selectedOption: "1",
selfPreview: <SelfPreview track={null} />,
},
};
@@ -199,6 +244,8 @@ export const VideoUnmute: Story = {
selectedBackgroundEffect: "none",
onSelectBackgroundEffect: fn(),
onAddBackgroundImage: fn(),
// As the footer always gives the camera menu one.
selfPreview: <FakeCamera />,
},
};
@@ -344,6 +391,9 @@ export const BackgroundEffectsSettling: Story = {
checked: false,
}))
await expect(other).toHaveAttribute("aria-busy", "false");
await expect(menu.queryByTestId("preview-camera")).toBeNull();
const frames = document.body.querySelectorAll(`.${styles.selfPreview} svg`);
await expect(frames).toHaveLength(1);
},
};
@@ -501,6 +551,113 @@ export const RemovingWithLiveSelection: Story = {
},
};
/** Where the call leaves room, the camera holds the top while the options scroll. */
export const BackgroundEffectsWithPreview: Story = {
args: {
...VideoUnmute.args,
// Enough that the list fills its share, so the preview's cost counts.
options: Array.from({ length: 8 }, (_, i) => ({
label: { type: "name" as const, name: `Camera ${i + 1}` },
id: `${i + 1}`,
})),
},
decorators: [
(Story): JSX.Element => (
<WithACallArea blockSize={640}>
<Story />
</WithACallArea>
),
],
play: async ({ canvasElement }) => {
const canvas = within(canvasElement);
await userEvent.click(canvas.getByRole("button", { name: "Camera" }));
const picture = await within(document.body).findByTestId("preview-camera");
const preview = picture.parentElement!;
const list = document.body.querySelector<HTMLElement>(
`.${styles.deviceList}`,
)!;
const frame = document.body
.querySelector("[role='menu']")!
.getBoundingClientRect();
const box = preview.getBoundingClientRect();
await expect(list.contains(preview)).toBe(false);
await expect(box.top - frame.top).toBeLessThanOrEqual(2);
await expect(frame.width - box.width).toBeLessThanOrEqual(2);
await expect(box.width / box.height).toBeCloseTo(16 / 9, 1);
await expect(box.bottom).toBeLessThanOrEqual(
list.getBoundingClientRect().top,
);
const listShare = Math.round(640 * 0.6);
// The list runs on through the menu's padding at its foot, which isn't
// part of its share.
const listEnd =
list.getBoundingClientRect().bottom -
Number.parseFloat(getComputedStyle(list).paddingBlockEnd);
await expect(listEnd - box.top).toBeLessThanOrEqual(listShare + 1);
await expect(list.scrollHeight).toBeGreaterThan(list.clientHeight);
list.scrollTop = list.scrollHeight;
await expect(preview.getBoundingClientRect().top).toBe(box.top);
},
};
/** With the camera off the preview says so, and effects can still be chosen. */
export const BackgroundEffectsPreviewWithTheCameraOff: Story = {
args: {
...VideoUnmute.args,
enabled: false,
selfPreview: <SelfPreview track={null} />,
},
play: async ({ args, canvasElement }) => {
const canvas = within(canvasElement);
await userEvent.click(canvas.getByRole("button", { name: "Camera" }));
const menu = within(document.body);
await menu.findByRole("group", { name: "Background effects" });
const preview = document.body.querySelector(`.${styles.selfPreview}`)!;
await expect(preview.querySelector("video")).toBeNull();
await expect(preview.querySelector("svg")).not.toBeNull();
await userEvent.click(menu.getByRole("menuitemradio", { name: "Blur" }));
await expect(args.onSelectBackgroundEffect).toHaveBeenCalledWith("blur");
},
};
/** In a call too short to hold it, the camera leads the options and scrolls away. */
export const BackgroundEffectsPreviewScrollsWithTheList: Story = {
args: VideoUnmute.args,
decorators: [
(Story): JSX.Element => (
<WithACallArea blockSize={300}>
<Story />
</WithACallArea>
),
],
play: async ({ canvasElement }) => {
const canvas = within(canvasElement);
await userEvent.click(canvas.getByRole("button", { name: "Camera" }));
const picture = await within(document.body).findByTestId("preview-camera");
const preview = picture.parentElement!;
const list = document.body.querySelector<HTMLElement>(
`.${styles.deviceList}`,
)!;
await expect(list.contains(preview)).toBe(true);
const cameras = within(document.body).getByRole("group", {
name: "Camera",
});
await expect(
preview.compareDocumentPosition(cameras) &
Node.DOCUMENT_POSITION_FOLLOWING,
).toBeTruthy();
list.scrollTop = list.scrollHeight;
await expect(preview.getBoundingClientRect().bottom).toBeLessThan(
list.getBoundingClientRect().top,
);
},
};
/** In a short call the effects scroll into view with the list. */
export const BackgroundEffectsScrollWhenTheyDoNotFit: Story = {
args: {
+54 -6
View File
@@ -14,10 +14,12 @@ import {
type FC,
useEffect,
type ReactElement,
type ReactNode,
} from "react";
import {
Alert,
Button,
InlineSpinner,
Menu,
MenuItem,
MenuScrollArea,
@@ -87,6 +89,8 @@ export interface MediaMuteAndSwitchButtonProps {
onRemoveBackgroundEffect?: (id: string) => void;
/** Why the last file offered couldn't be used; a new object each time. */
backgroundImageRefusal?: { text: string };
/** The user's own camera, shown at the top of the camera menu. */
selfPreview?: ReactNode;
/**
* For any toggle and option this method will be called.
* So toggles need to be implemented by listening here and setting the right toggle item to `enabled`
@@ -103,9 +107,16 @@ const LIST_SHARE_OF_CALL = 0.6;
/** Smallest device list height in px, so a short call still shows more than one device. */
const MIN_LIST_HEIGHT = 160;
/** The menu's width as its stylesheet sets it, to place the preview before
the menu is drawn. */
const MENU_WIDTH = 296;
/** Space kept between the menu and the call area's sides. */
const MENU_MARGIN = 16;
/** Below the preview, as its stylesheet draws it. */
const PREVIEW_GAP = 12;
export const MediaMuteAndSwitchButton: FC<MediaMuteAndSwitchButtonProps> = ({
enabled,
busy,
@@ -124,6 +135,7 @@ export const MediaMuteAndSwitchButton: FC<MediaMuteAndSwitchButtonProps> = ({
onAddBackgroundImage,
onRemoveBackgroundEffect,
backgroundImageRefusal,
selfPreview,
onSelect,
}) => {
// Requested but not yet selected. Keyed by kind too, since Chrome uses
@@ -202,21 +214,53 @@ export const MediaMuteAndSwitchButton: FC<MediaMuteAndSwitchButtonProps> = ({
// Measured on the call area: CSS can't size the portalled menu against it.
const rootElement = useRootElement();
const [listMaxHeight, setListMaxHeight] = useState<number>();
const [listShare, setListShare] = useState<number>();
const [previewBlock, setPreviewBlock] = useState(0);
useEffect(() => {
if (!menuOpen) return;
// Followed, since a host can resize the call while the menu is open.
const subscription = observeElementSize$(rootElement)
.pipe(
map(({ height }) =>
Math.max(MIN_LIST_HEIGHT, Math.round(height * LIST_SHARE_OF_CALL)),
map(({ width, height }) => ({
height: Math.round(height * LIST_SHARE_OF_CALL),
width: Math.min(MENU_WIDTH, Math.round(width - 2 * MENU_MARGIN)),
})),
distinctUntilChanged(
(a, b) => a.height === b.height && a.width === b.width,
),
distinctUntilChanged(),
)
.subscribe(setListMaxHeight);
.subscribe(({ height, width }) => {
setListShare(height);
setPreviewBlock(Math.round((width * 9) / 16) + PREVIEW_GAP);
});
return (): void => subscription.unsubscribe();
}, [menuOpen, rootElement]);
// Paid for out of the list's share, so a preview makes the menu no taller.
const hasPreview =
iconsAndLabels === "video" &&
selfPreview !== undefined &&
listShare !== undefined;
const previewPinned =
hasPreview && listShare - previewBlock >= MIN_LIST_HEIGHT;
const listMaxHeight =
listShare === undefined
? undefined
: Math.max(
MIN_LIST_HEIGHT,
listShare - (previewPinned ? previewBlock : 0),
);
const preview = hasPreview && (
<div
aria-hidden
className={classNames(styles.selfPreview, {
[styles.selfPreviewPinned]: previewPinned,
})}
>
{backgroundEffectSettling ? <InlineSpinner size={32} /> : selfPreview}
</div>
);
useEffect(() => {
if (menuOpen) devices.requestDeviceNames(); // No-op after the first call
}, [menuOpen, devices]);
@@ -407,9 +451,12 @@ export const MediaMuteAndSwitchButton: FC<MediaMuteAndSwitchButtonProps> = ({
/>
}
>
{previewPinned && preview}
<MenuScrollArea
ref={trackFocusSource}
className={styles.deviceList}
className={classNames(styles.deviceList, {
[styles.deviceListLeadsWithPreview]: hasPreview && !previewPinned,
})}
data-testid="device-list"
style={
{
@@ -418,6 +465,7 @@ export const MediaMuteAndSwitchButton: FC<MediaMuteAndSwitchButtonProps> = ({
} as CSSProperties
}
>
{hasPreview && !previewPinned && preview}
{audioMeter && speakerOptions && (
<MenuScrollSection title={t("settings.devices.speaker")}>
{deviceItems(
+10
View File
@@ -0,0 +1,10 @@
/*
Copyright 2026 Element Creations Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
.mirrored {
transform: scaleX(-1);
}
+47
View File
@@ -0,0 +1,47 @@
/*
Copyright 2026 Element Creations Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { act, render } from "@testing-library/react";
import { describe, expect, it } from "vitest";
import { BehaviorSubject } from "rxjs";
import { SelfPreview } from "./SelfPreview";
import styles from "./SelfPreview.module.css";
import { mockVideoTrack } from "../utils/test";
describe("SelfPreview", () => {
it("detaches only its own element from a camera shown twice", () => {
const track = mockVideoTrack({});
const first = render(<SelfPreview track={track} />);
const second = render(<SelfPreview track={track} />);
const [a, b] = [first, second].map((r) =>
r.container.querySelector("video"),
);
expect(track.attach).toHaveBeenCalledWith(a);
expect(track.attach).toHaveBeenCalledWith(b);
first.unmount();
expect(track.detach).toHaveBeenCalledExactlyOnceWith(a);
});
it("mirrors the camera while it faces the user, and only then", () => {
const facingMode$ = new BehaviorSubject<"user" | "environment">("user");
const { container } = render(
<SelfPreview track={mockVideoTrack({ facingMode$ })} />,
);
const video = (): HTMLVideoElement => container.querySelector("video")!;
expect(video().classList.contains(styles.mirrored)).toBe(true);
act(() => facingMode$.next("environment"));
expect(video().classList.contains(styles.mirrored)).toBe(false);
});
it("says the camera is off while there is none", () => {
const { container } = render(<SelfPreview track={null} />);
expect(container.querySelector("video")).toBeNull();
expect(container.querySelector("svg")).not.toBeNull();
});
});
+48
View File
@@ -0,0 +1,48 @@
/*
Copyright 2026 Element Creations Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { type FC, useEffect, useRef } from "react";
import { constant, type LocalVideoMediaTrack } from "@element-hq/matrixrtc-sdk";
import { VideoCallOffIcon } from "@vector-im/compound-design-tokens/assets/web/icons";
import styles from "./SelfPreview.module.css";
import { useBehavior } from "../useBehavior";
const noFacingMode = constant(undefined);
interface Props {
/** The camera as the pipeline has it, or none while the camera is off. */
track: LocalVideoMediaTrack | null;
}
/**
* The user's own camera with the effect in force: the pipeline's own track,
* shown a second time rather than captured again.
*/
export const SelfPreview: FC<Props> = ({ track }) => {
const videoRef = useRef<HTMLVideoElement | null>(null);
useEffect(() => {
const element = videoRef.current;
if (!track || !element) return;
track.attach(element);
// This element only: the pre-join preview may be showing the same camera
return (): void => {
track.detach(element);
};
}, [track]);
const facingMode = useBehavior(track?.facingMode$ ?? noFacingMode);
if (!track) return <VideoCallOffIcon width={32} height={32} />;
return (
<video
ref={videoRef}
muted
playsInline
className={facingMode === "user" ? styles.mirrored : undefined}
/>
);
};
+16
View File
@@ -108,6 +108,22 @@ describe("Config.initWith", () => {
expect(Config.get().media_quality).toEqual(DEFAULT_CONFIG.media_quality);
});
it("falls back to the defaults when config.json is not JSON", async () => {
vi.spyOn(globalThis, "fetch").mockResolvedValue(
new Response("<!doctype html><title>Element</title>", {
status: 200,
headers: { "content-type": "text/html" },
}),
);
const warnSpy = vi.spyOn(logger, "warn").mockImplementation(() => {});
Reflect.set(Config, "internalInstance", undefined);
await expect(Config.init()).resolves.toBeUndefined();
expect(Config.get().media_quality).toEqual(DEFAULT_CONFIG.media_quality);
expect(warnSpy).toHaveBeenCalledTimes(1);
});
it("replaces a config initialized earlier", () => {
Config.initWith({ ssla: "https://first.invalid/ssla" });
Config.initWith({ ssla: "https://second.invalid/ssla" });
+6 -1
View File
@@ -156,5 +156,10 @@ async function downloadConfig(fetchTarget: string): Promise<ConfigOptions> {
return {};
}
return response.json();
try {
return await response.json();
} catch (e) {
logger.warn(`Config at ${fetchTarget} is not JSON, using defaults`, e);
return {};
}
}
+24 -24
View File
@@ -7,30 +7,30 @@ Please see LICENSE in the repository root for full details.
import { type RelationType } from "matrix-js-sdk";
import catSoundOgg from "../sound/reactions/cat.ogg?url";
import catSoundMp3 from "../sound/reactions/cat.mp3?url";
import clapSoundOgg from "../sound/reactions/clap.ogg?url";
import clapSoundMp3 from "../sound/reactions/clap.mp3?url";
import cricketsSoundOgg from "../sound/reactions/crickets.ogg?url";
import cricketsSoundMp3 from "../sound/reactions/crickets.mp3?url";
import dogSoundOgg from "../sound/reactions/dog.ogg?url";
import dogSoundMp3 from "../sound/reactions/dog.mp3?url";
import genericSoundOgg from "../sound/reactions/generic.ogg?url";
import genericSoundMp3 from "../sound/reactions/generic.mp3?url";
import lightbulbSoundOgg from "../sound/reactions/lightbulb.ogg?url";
import lightbulbSoundMp3 from "../sound/reactions/lightbulb.mp3?url";
import partySoundOgg from "../sound/reactions/party.ogg?url";
import partySoundMp3 from "../sound/reactions/party.mp3?url";
import deerSoundOgg from "../sound/reactions/deer.ogg?url";
import deerSoundMp3 from "../sound/reactions/deer.mp3?url";
import rockSoundOgg from "../sound/reactions/rock.ogg?url";
import rockSoundMp3 from "../sound/reactions/rock.mp3?url";
import waveSoundOgg from "../sound/reactions/wave.ogg?url";
import waveSoundMp3 from "../sound/reactions/wave.mp3?url";
import baduntssSoundOgg from "../sound/reactions/baduntss.ogg?url";
import baduntssSoundMp3 from "../sound/reactions/baduntss.mp3?url";
import bellSoundOgg from "../sound/reactions/bell.ogg?url";
import bellSoundMp3 from "../sound/reactions/bell.mp3?url";
import catSoundOgg from "../sound/reactions/cat.ogg?url&no-inline";
import catSoundMp3 from "../sound/reactions/cat.mp3?url&no-inline";
import clapSoundOgg from "../sound/reactions/clap.ogg?url&no-inline";
import clapSoundMp3 from "../sound/reactions/clap.mp3?url&no-inline";
import cricketsSoundOgg from "../sound/reactions/crickets.ogg?url&no-inline";
import cricketsSoundMp3 from "../sound/reactions/crickets.mp3?url&no-inline";
import dogSoundOgg from "../sound/reactions/dog.ogg?url&no-inline";
import dogSoundMp3 from "../sound/reactions/dog.mp3?url&no-inline";
import genericSoundOgg from "../sound/reactions/generic.ogg?url&no-inline";
import genericSoundMp3 from "../sound/reactions/generic.mp3?url&no-inline";
import lightbulbSoundOgg from "../sound/reactions/lightbulb.ogg?url&no-inline";
import lightbulbSoundMp3 from "../sound/reactions/lightbulb.mp3?url&no-inline";
import partySoundOgg from "../sound/reactions/party.ogg?url&no-inline";
import partySoundMp3 from "../sound/reactions/party.mp3?url&no-inline";
import deerSoundOgg from "../sound/reactions/deer.ogg?url&no-inline";
import deerSoundMp3 from "../sound/reactions/deer.mp3?url&no-inline";
import rockSoundOgg from "../sound/reactions/rock.ogg?url&no-inline";
import rockSoundMp3 from "../sound/reactions/rock.mp3?url&no-inline";
import waveSoundOgg from "../sound/reactions/wave.ogg?url&no-inline";
import waveSoundMp3 from "../sound/reactions/wave.mp3?url&no-inline";
import baduntssSoundOgg from "../sound/reactions/baduntss.ogg?url&no-inline";
import baduntssSoundMp3 from "../sound/reactions/baduntss.mp3?url&no-inline";
import bellSoundOgg from "../sound/reactions/bell.ogg?url&no-inline";
import bellSoundMp3 from "../sound/reactions/bell.mp3?url&no-inline";
export const ElementCallReactionEventType = "io.element.call.reaction";
+12 -12
View File
@@ -8,18 +8,18 @@ Please see LICENSE in the repository root for full details.
import { type ReactNode, useEffect } from "react";
import { type CallViewModel } from "../state/CallViewModel/CallViewModel";
import joinCallSoundMp3 from "../sound/join_call.mp3";
import joinCallSoundOgg from "../sound/join_call.ogg";
import leftCallSoundMp3 from "../sound/left_call.mp3";
import leftCallSoundOgg from "../sound/left_call.ogg";
import handSoundOgg from "../sound/raise_hand.ogg";
import handSoundMp3 from "../sound/raise_hand.mp3";
import screenShareStartedOgg from "../sound/screen_share_started.ogg";
import screenShareStartedMp3 from "../sound/screen_share_started.mp3";
import declineMp3 from "../sound/call_declined.mp3?url";
import declineOgg from "../sound/call_declined.ogg?url";
import timeoutMp3 from "../sound/call_timeout.mp3?url";
import timeoutOgg from "../sound/call_timeout.ogg?url";
import joinCallSoundMp3 from "../sound/join_call.mp3?url&no-inline";
import joinCallSoundOgg from "../sound/join_call.ogg?url&no-inline";
import leftCallSoundMp3 from "../sound/left_call.mp3?url&no-inline";
import leftCallSoundOgg from "../sound/left_call.ogg?url&no-inline";
import handSoundOgg from "../sound/raise_hand.ogg?url&no-inline";
import handSoundMp3 from "../sound/raise_hand.mp3?url&no-inline";
import screenShareStartedOgg from "../sound/screen_share_started.ogg?url&no-inline";
import screenShareStartedMp3 from "../sound/screen_share_started.mp3?url&no-inline";
import declineMp3 from "../sound/call_declined.mp3?url&no-inline";
import declineOgg from "../sound/call_declined.ogg?url&no-inline";
import timeoutMp3 from "../sound/call_timeout.mp3?url&no-inline";
import timeoutOgg from "../sound/call_timeout.ogg?url&no-inline";
import { useAudioContext } from "../useAudioContext";
import { prefetchSounds } from "../soundUtils";
import { useLatest } from "../useLatest";
+6 -13
View File
@@ -75,7 +75,6 @@ import {
type RTCSlotOptions,
constant,
} from "@element-hq/matrixrtc-sdk";
import { map } from "rxjs";
import {
advancedCamera,
advancedScreenShare,
@@ -181,7 +180,7 @@ export const ActiveCall: FC<ActiveCallProps> = (props) => {
props.muteStates,
mediaDevices,
capture,
backgroundEffectsState$.value.processor,
backgroundEffectsState$.value,
),
});
@@ -195,9 +194,7 @@ export const ActiveCall: FC<ActiveCallProps> = (props) => {
...callViewModelOptionsFromParams(urlParams),
encryptionSystem: props.e2eeSystem,
capture,
videoProcessor$: scope.behavior(
backgroundEffectsState$.pipe(map((state) => state.processor)),
),
backgroundEffectsState$,
hostBridge,
autoLeaveWhenOthersLeft,
waitForCallPickup: waitForCallPickup && sendNotificationType === "ring",
@@ -241,7 +238,7 @@ export const ActiveCall: FC<ActiveCallProps> = (props) => {
vm,
props.muteStates,
mediaDevices,
backgroundEffectsState$.pipe(map(({ settling }) => settling ?? false)),
backgroundEffectsState$,
`${props.client.getUserId()}:${props.client.getDeviceId()}`,
{ showControls: urlParams.showControls, header: urlParams.header },
);
@@ -751,8 +748,7 @@ export const InCallView: FC<InCallViewProps> = ({
/**
* How the tracks are captured and encoded, from Element Call's settings and
* configuration. Shared with the legacy SDK bundle so that a widget built from
* it behaves like the app.
* configuration.
*/
export function captureSettings(): CaptureSettings {
const config = Config.get();
@@ -801,8 +797,7 @@ export function captureSettings(): CaptureSettings {
/**
* What the client needs from Element Call's configuration, settings and URL:
* the SDK reads none of them itself. What to publish depends on the mute
* switches and the devices, so the caller adds that. Shared with the legacy
* SDK bundle so that a widget built from it behaves like the app.
* switches and the devices, so the caller adds that.
*/
export function rtcSlotOptions(
encryptionSystem: EncryptionSystem,
@@ -825,9 +820,7 @@ export function rtcParticipationOptions(
const session = config.matrix_rtc_session;
return {
sendNotificationType: urlParams.sendNotificationType,
applicationData: urlParams.callIntent
? { "m.call.intent": urlParams.callIntent }
: undefined,
callIntent: urlParams.callIntent,
timings: {
syncDisconnectGracePeriodMs: config.sync_disconnect_grace_period_ms,
networkErrorRetryMs: session.network_error_retry_ms,
+5 -5
View File
@@ -16,7 +16,6 @@ import {
type LocalMedia,
type LocalVideoMediaTrack,
} from "@element-hq/matrixrtc-sdk";
import { map } from "rxjs";
import inCallStyles from "./InCallView.module.css";
import styles from "./LobbyView.module.css";
@@ -134,9 +133,7 @@ export const LobbyView: FC<Props> = ({
muteStates,
devices,
captureSettings(),
mediaScope.behavior(
backgroundEffectsState$.pipe(map((state) => state.processor)),
),
backgroundEffectsState$,
),
);
return (): void => {
@@ -158,13 +155,15 @@ export const LobbyView: FC<Props> = ({
null,
);
useEffect(() => {
if (mediaVm === null) return;
const footerScope = new ObservableScope();
setFooterVm(
createLobbyFooterViewModel(
footerScope,
muteStates,
devices,
backgroundEffectsState$.pipe(map(({ settling }) => settling ?? false)),
backgroundEffectsState$,
mediaVm.videoTrack$,
openSettings,
hangup,
// Logo and header are connected: only show the logo in SPA with header.
@@ -179,6 +178,7 @@ export const LobbyView: FC<Props> = ({
backgroundEffectsState$,
hangup,
hideHeader,
mediaVm,
muteStates,
openSettings,
]);
+2 -1
View File
@@ -8,6 +8,7 @@ Please see LICENSE in the repository root for full details.
import { expect, type MockedFunction, test, vi } from "vitest";
import { act, render } from "@testing-library/react";
import { BehaviorSubject } from "rxjs";
import { RTC_CALL_INTENT_AUDIO } from "matrix-js-sdk/lib/matrixrtc";
import { useAudioContext } from "../useAudioContext";
import { createRingingMedia } from "../state/media/RingingMediaViewModel";
@@ -39,7 +40,7 @@ test("ringtone plays on loop while ringing", () => {
userId: alice.userId,
displayName$: constant("Alice"),
mxcAvatarUrl$: constant(undefined),
intent: "audio",
intent: RTC_CALL_INTENT_AUDIO,
pickupState$,
});
+2 -2
View File
@@ -12,8 +12,8 @@ import { type RingingMediaViewModel } from "../state/media/RingingMediaViewModel
import { useBehavior } from "../useBehavior";
import { useInitial } from "../useInitial";
import { prefetchSounds } from "../soundUtils";
import ringtoneMp3 from "../sound/ringtone.mp3?url";
import ringtoneOgg from "../sound/ringtone.ogg?url";
import ringtoneMp3 from "../sound/ringtone.mp3?url&no-inline";
import ringtoneOgg from "../sound/ringtone.ogg?url&no-inline";
import { type UseAudioContext, useAudioContext } from "../useAudioContext";
import { useLatest } from "../useLatest";
+7 -26
View File
@@ -34,23 +34,6 @@ describe("prefetchSounds", () => {
expect(bytesOf(sounds.beep)).toEqual([1, 2, 3]);
});
it("decodes base64 data: URLs without fetching them", async () => {
// "OggS" — the bytes 0x4f 0x67 0x67 0x53
const sounds = await prefetchSounds({
beep: { ogg: "data:audio/ogg;base64,T2dnUw==" },
});
expect(fetchSpy).not.toHaveBeenCalled();
expect(bytesOf(sounds.beep)).toEqual([0x4f, 0x67, 0x67, 0x53]);
});
it("leaves out a sound with a percent-encoded data: URL", async () => {
const sounds = await prefetchSounds({
beep: { ogg: "data:audio/ogg,Ogg%53" },
});
expect(fetchSpy).not.toHaveBeenCalled();
expect(sounds).toEqual({});
});
it("leaves out a sound whose response is not okay", async () => {
fetchSpy.mockResolvedValue(new Response(null, { status: 404 }));
const sounds = await prefetchSounds({
@@ -60,18 +43,16 @@ describe("prefetchSounds", () => {
});
it("leaves out a sound that fails to load, keeping the others", async () => {
fetchSpy.mockRejectedValue(new TypeError("Failed to fetch"));
// Fetched in the order given
fetchSpy
.mockRejectedValueOnce(new TypeError("Failed to fetch"))
.mockResolvedValueOnce(
new Response(new Uint8Array([1, 2, 3]), { status: 200 }),
);
const sounds = await prefetchSounds({
beep: { ogg: "https://example.org/beep.ogg" },
boop: { ogg: "data:audio/ogg;base64,T2dnUw==" },
boop: { ogg: "https://example.org/boop.ogg" },
});
expect(Object.keys(sounds)).toEqual(["boop"]);
});
it("leaves out a sound with a malformed data: URL", async () => {
const sounds = await prefetchSounds({
beep: { ogg: "data:audio/ogg;base64" },
});
expect(sounds).toEqual({});
});
});
+1 -27
View File
@@ -32,38 +32,12 @@ function getPreferredAudioFormat(): "ogg" | "mp3" {
const preferredFormat = getPreferredAudioFormat();
/**
* Decode a `data:` URL into the bytes it carries.
*
* The component build inlines the sound files as `data:` URLs. Those could be
* `fetch`ed like any other URL, but that fetch is subject to the host page's
* Content Security Policy, and `connect-src` lists rarely allow `data:` (a `*`
* source does not cover it). Reading the bytes out of the URL directly needs no
* network access, so no policy can stop it.
* @param url The `data:` URL to decode.
* @returns The decoded bytes.
*/
function decodeDataUrl(url: string): ArrayBuffer {
const separator = url.indexOf(",");
if (!url.startsWith("data:") || separator === -1)
throw new Error("Not a valid data: URL");
const header = url.slice("data:".length, separator);
// Vite only percent-encodes SVGs; every other inlined asset is base64.
if (!header.split(";").includes("base64"))
throw new Error("Only base64 data: URLs are supported");
// `decoded` is a binary string: one char per byte
const decoded = atob(url.slice(separator + 1));
const bytes = Uint8Array.from(decoded, (c) => c.charCodeAt(0));
return bytes.buffer;
}
/**
* Load the bytes of a single sound file.
* @param url Where the sound is: an ordinary URL to fetch, or a `data:` URL.
* @param url Where the sound is.
* @returns The bytes, or null if the sound could not be loaded.
*/
async function loadSound(url: string): Promise<ArrayBuffer | null> {
if (url.startsWith("data:")) return decodeDataUrl(url);
const response = await fetch(url);
if (isFailure(response)) return null;
return await response.arrayBuffer();
+18 -12
View File
@@ -7,7 +7,11 @@ Please see LICENSE in the repository root for full details.
import { it, vi, expect, beforeEach, afterEach, describe } from "vitest";
import { firstValueFrom, of, Subject, take, toArray } from "rxjs";
import { type RTCCallIntent } from "matrix-js-sdk/lib/matrixrtc";
import {
RTC_CALL_INTENT_AUDIO,
RTC_CALL_INTENT_VIDEO,
type RTCCallIntent,
} from "matrix-js-sdk/lib/matrixrtc";
import { AndroidControlledAudioOutput } from "./AndroidControlledAudioOutput.ts";
import type { Controls, OutputDevice } from "../controls";
@@ -117,7 +121,7 @@ describe("Default selection", () => {
const controlledAudioOutput = new AndroidControlledAudioOutput(
of(BASE_DEVICE_LIST),
testScope,
"video",
RTC_CALL_INTENT_VIDEO,
mockControls,
);
@@ -142,7 +146,7 @@ describe("Default selection", () => {
const controlledAudioOutput = new AndroidControlledAudioOutput(
of(BASE_DEVICE_LIST),
testScope,
"audio",
RTC_CALL_INTENT_AUDIO,
mockControls,
);
@@ -163,7 +167,7 @@ describe("Default selection", () => {
});
});
["audio", "video"].forEach((callIntent) => {
[RTC_CALL_INTENT_AUDIO, RTC_CALL_INTENT_VIDEO].forEach((callIntent) => {
it(`Default to BT headset for ${callIntent} calls if present`, async () => {
const controlledAudioOutput = new AndroidControlledAudioOutput(
of(BT_HEADSET_BASE_DEVICE_LIST),
@@ -190,7 +194,7 @@ describe("Default selection", () => {
});
});
["audio", "video"].forEach((callIntent) => {
[RTC_CALL_INTENT_AUDIO, RTC_CALL_INTENT_VIDEO].forEach((callIntent) => {
it(`Default to wired headset for ${callIntent} calls if present`, async () => {
const controlledAudioOutput = new AndroidControlledAudioOutput(
of(WIRED_HEADSET_BASE_DEVICE_LIST),
@@ -300,7 +304,7 @@ describe("Test select a device", () => {
b: BT_HEADSET_BASE_DEVICE_LIST,
}),
testScope,
"audio",
RTC_CALL_INTENT_AUDIO,
mockControls,
);
@@ -340,7 +344,7 @@ describe("Test select a device", () => {
c: BASE_DEVICE_LIST,
}),
testScope,
"audio",
RTC_CALL_INTENT_AUDIO,
mockControls,
);
@@ -393,7 +397,7 @@ describe("Available device changes", () => {
});
it("When a BT headset is added, control should switch to use it", () => {
createAudioControlledOutput("video");
createAudioControlledOutput(RTC_CALL_INTENT_VIDEO);
// Emit the base device list, the speaker should be selected
availableSource$.next(BASE_DEVICE_LIST);
@@ -419,7 +423,9 @@ describe("Available device changes", () => {
// Android does not set `isExternalHeadset` to true for wired headphones, so we can't test this case.'
it.skip("When a wired headset is added, control should switch to use it", async () => {
const controlledAudioOutput = createAudioControlledOutput("video");
const controlledAudioOutput = createAudioControlledOutput(
RTC_CALL_INTENT_VIDEO,
);
// Emit the base device list, the speaker should be selected
availableSource$.next(BASE_DEVICE_LIST);
@@ -446,7 +452,7 @@ describe("Available device changes", () => {
});
it("When the active bt headset is removed on audio call, control should switch to earpiece", () => {
createAudioControlledOutput("audio");
createAudioControlledOutput(RTC_CALL_INTENT_AUDIO);
// Emit the BT headset device list, the BT headset should be selected
availableSource$.next(BT_HEADSET_BASE_DEVICE_LIST);
@@ -472,7 +478,7 @@ describe("Available device changes", () => {
});
it("When the active bt headset is removed on video call, control should switch to speaker", () => {
createAudioControlledOutput("video");
createAudioControlledOutput(RTC_CALL_INTENT_VIDEO);
availableSource$.next(BT_HEADSET_BASE_DEVICE_LIST);
@@ -497,7 +503,7 @@ describe("Available device changes", () => {
});
it("Do not repeatidly set the same device", () => {
createAudioControlledOutput("video");
createAudioControlledOutput(RTC_CALL_INTENT_VIDEO);
availableSource$.next(BT_HEADSET_BASE_DEVICE_LIST);
availableSource$.next(BT_HEADSET_BASE_DEVICE_LIST);
+6 -3
View File
@@ -22,7 +22,10 @@ import {
type SelectedAudioOutputDevice,
} from "./MediaDevices.ts";
import type { ObservableScope } from "./ObservableScope.ts";
import type { RTCCallIntent } from "matrix-js-sdk/lib/matrixrtc";
import {
RTC_CALL_INTENT_AUDIO,
type RTCCallIntent,
} from "matrix-js-sdk/lib/matrixrtc";
import { type Controls, type OutputDevice } from "../controls.ts";
import { type Behavior } from "./Behavior.ts";
@@ -102,7 +105,7 @@ export class AndroidControlledAudioOutput implements MediaDevice<
* @constructor
* @param controlledDevices$ - The list of available output devices coming from the hosting application, ordered by preference order (most preferred first).
* @param scope - The ObservableScope to create the Behaviors in.
* @param initialIntent - The initial call intent (e.g. "audio" or "video") that can be used to determine the default audio routing (e.g. default to earpiece for audio calls and speaker for video calls).
* @param initialIntent - The initial call intent (e.g. "m.audio" or "m.video") that can be used to determine the default audio routing (e.g. default to earpiece for audio calls and speaker for video calls).
* @param controls - The controls provided by the hosting application to control the audio routing and notify of user actions.
*/
public constructor(
@@ -316,7 +319,7 @@ export class AndroidControlledAudioOutput implements MediaDevice<
this.logger.debug(
`Android routing logic intent: ${this.initialIntent} finding best default...`,
);
if (this.initialIntent === "audio") {
if (this.initialIntent === RTC_CALL_INTENT_AUDIO) {
const systemProposed = available[0];
// If no headset is connected, android will route to the speaker by default,
// but for a voice call we want to route to the earpiece instead,
+41 -19
View File
@@ -6,7 +6,7 @@ Please see LICENSE in the repository root for full details.
*/
import { beforeEach, describe, expect, it, vi } from "vitest";
import { BehaviorSubject, distinctUntilChanged, map } from "rxjs";
import { BehaviorSubject, distinctUntilChanged, map, of } from "rxjs";
import { type BackgroundProcessorWrapper } from "@livekit/track-processors";
import {
@@ -47,23 +47,28 @@ function fakePipeline(): {
};
}
/** One letter per state: idle, waiting for a frame, attached. */
/** One letter per state: idle, preparing, waiting for a frame, attached. */
function letter(state: BackgroundEffectsState): string {
if (state.supported === false) return "x";
if (state.wait === "preparing") return "p";
if (state.processor === undefined) return "i";
return state.settling ? "w" : "a";
return state.wait === "settling" ? "w" : "a";
}
describe("the pipeline's state", () => {
function testState({
effect,
answer = "-y",
firstFrame = "",
expected,
}: {
effect: string;
/** When the trial build answers, from being asked. */
answer?: string;
firstFrame?: string;
expected: string;
}): void {
withTestScheduler(({ behavior, schedule, expectObservable }) => {
withTestScheduler(({ behavior, cold, schedule, expectObservable }) => {
const transformer = {
onFirstFrame: undefined as (() => void) | undefined,
};
@@ -72,6 +77,7 @@ describe("the pipeline's state", () => {
effect$: behavior(effect, { n: "none", b: "blur" }),
setEffect: vi.fn(),
added$: new BehaviorSubject<AddedBackground[] | undefined>([]),
canSegment: () => cold(answer, { y: true, n: false }),
pipeline: fakePipeline().pipeline,
transformer,
});
@@ -84,15 +90,21 @@ describe("the pipeline's state", () => {
it("defaults to no effect", () => testState({ effect: "n", expected: "i" }));
it("attaches only once it is known the pipeline builds", () =>
testState({ effect: "nb", answer: "--y", expected: "ip-w" }));
it("offers none where the pipeline fails to build", () =>
testState({ effect: "nb", answer: "-n", expected: "ipx" }));
it("waits from the first effect until a frame carries it", () =>
testState({
effect: " nb-n-b",
firstFrame: " --f",
expected: " iwa",
effect: " nb---n-b",
firstFrame: "---f",
expected: " ipwa",
}));
it("attaches on first use and stays attached", () =>
testState({ effect: "nb-n", firstFrame: "--f", expected: "iwa" }));
testState({ effect: "nb--n", firstFrame: "---f", expected: "ipwa" }));
});
describe("background effects", () => {
@@ -109,6 +121,7 @@ describe("background effects", () => {
effect$,
setEffect,
added$,
canSegment: () => of(true),
pipeline: fake.pipeline,
transformer: { onFirstFrame: undefined },
...options,
@@ -118,8 +131,6 @@ describe("background effects", () => {
effect$.next(raw);
await flushPromises();
};
const blur = async (on: boolean): Promise<void> =>
choose(on ? "blur" : "none");
beforeEach(() => {
effect$ = new BehaviorSubject("none");
@@ -128,6 +139,17 @@ describe("background effects", () => {
fake = fakePipeline();
});
it("offers none for the session where the pipeline fails to build", async () => {
const effects = build({ canSegment: () => of(false) });
await choose("blur");
expect(effects.state$.value.supported).toBe(false);
expect(effects.state$.value.processor).toBeUndefined();
expect(effects.state$.value.wait).toBeUndefined();
// A browser update may fix it, so the next session asks again.
expect(effect$.value).toBe("blur");
});
it("puts a shipped background on as that picture", async () => {
build();
await choose(`image:${shippedBackgrounds[0].id}`);
@@ -141,10 +163,10 @@ describe("background effects", () => {
it("switches in place rather than reattaching", async () => {
const effects = build();
await blur(true);
await choose("blur");
const pipeline = effects.state$.value.processor;
await blur(false);
await blur(true);
await choose("none");
await choose("blur");
expect(effects.state$.value.processor).toBe(pipeline);
expect(fake.switches).toEqual([
@@ -157,18 +179,18 @@ describe("background effects", () => {
it("keeps switching after a switch fails", async () => {
vi.mocked(fake.pipeline.switchTo).mockRejectedValueOnce(new Error("lost"));
build();
await blur(true);
await blur(false);
await choose("blur");
await choose("none");
expect(fake.switches).toEqual([{ mode: "disabled" }]);
});
it("switches one at a time, skipping those overtaken", async () => {
const finished = fake.holdNext();
build();
await blur(true);
await blur(false);
await blur(true);
await blur(false);
await choose("blur");
await choose("none");
await choose("blur");
await choose("none");
expect(fake.switches).toHaveLength(1);
finished();
+45 -11
View File
@@ -8,9 +8,15 @@ Please see LICENSE in the repository root for full details.
import {
combineLatest,
distinctUntilChanged,
first,
from,
map,
Observable,
type ObservableInput,
of,
scan,
startWith,
switchMap,
} from "rxjs";
import {
type BackgroundProcessorWrapper,
@@ -38,8 +44,12 @@ const blurRadius = 15;
export interface BackgroundEffectsState {
supported: boolean | undefined;
processor: undefined | VideoProcessor;
/** From the first effect chosen until a frame carrying it is drawn. */
settling?: boolean;
/**
* The effect is not ready:
* preparing, asking whether the pipeline can be built.
* settling, once it is attached until a frame carrying the effect
*/
wait?: "preparing" | "settling";
}
export interface BackgroundEffectsOptions {
@@ -51,6 +61,8 @@ export interface BackgroundEffectsOptions {
setEffect: (id: EffectId) => void;
/** The backgrounds the device keeps, undefined until it has said. */
added$: Behavior<AddedBackground[] | undefined>;
/** Builds a segmenter on trial, and answers whether it could. */
canSegment: () => ObservableInput<boolean>;
/** The background processor pipeline to be switched between effects. */
pipeline: BackgroundProcessorWrapper;
/** Tells, once, that a frame carrying an effect has been drawn. */
@@ -70,6 +82,7 @@ export function createBackgroundEffects(
effect$,
setEffect,
added$,
canSegment,
pipeline,
transformer,
}: BackgroundEffectsOptions,
@@ -77,9 +90,11 @@ export function createBackgroundEffects(
// Wrapped once, so the call and the preview see one processor throughout.
const processor = fromLivekitProcessor(pipeline);
const choice$ = effect$.pipe(map(parseEffect));
const wanted$ = choice$.pipe(
map((effect) => effect.kind !== "none"),
distinctUntilChanged(),
const wanted$ = scope.behavior(
choice$.pipe(
map((effect) => effect.kind !== "none"),
distinctUntilChanged(),
),
);
combineLatest([choice$, added$])
@@ -95,6 +110,18 @@ export function createBackgroundEffects(
setEffect("none");
});
// Asked once an effect is first wanted. A browser update may fix what
// fails, so the choice is kept for the next session rather than cleared.
const buildable$ = scope.behavior<boolean | undefined>(
supported
? wanted$.pipe(
first(Boolean),
switchMap(() => from(canSegment())),
startWith(undefined),
)
: of(undefined),
);
const drewAFrame$ = scope.behavior(
new Observable<boolean>((subscriber) => {
subscriber.next(false);
@@ -105,18 +132,25 @@ export function createBackgroundEffects(
}),
);
// In one step, so that nothing sees the pipeline neither preparing nor
// attached.
const state$ = scope.behavior(
combineLatest([wanted$, drewAFrame$]).pipe(
scan<[boolean, boolean], BackgroundEffectsState>(
(previous, [wanted, drewAFrame]) => {
combineLatest([wanted$, buildable$, drewAFrame$]).pipe(
scan<[boolean, boolean | undefined, boolean], BackgroundEffectsState>(
(previous, [wanted, buildable, drewAFrame]) => {
// Attached the first time an effect is wanted and never detached
// after, so someone who never turns one on pays for none of it.
const enable =
previous.processor !== undefined || (supported && wanted);
previous.processor !== undefined || (wanted && buildable === true);
const preparing = supported && wanted && buildable === undefined;
return {
supported,
supported: supported && buildable !== false,
processor: enable ? processor : undefined,
settling: enable && !drewAFrame,
wait: preparing
? "preparing"
: enable && !drewAFrame
? "settling"
: undefined,
};
},
{ supported, processor: undefined },
@@ -12,7 +12,11 @@ import {
MatrixEvent,
type Room,
} from "matrix-js-sdk";
import { type RTCCallIntent } from "matrix-js-sdk/lib/matrixrtc";
import {
RTC_CALL_INTENT_AUDIO,
RTC_CALL_INTENT_VIDEO,
type RTCCallIntent,
} from "matrix-js-sdk/lib/matrixrtc";
import { map, mergeMap, NEVER, type Observable, startWith } from "rxjs";
import { withTestScheduler } from "../../utils/test";
@@ -33,10 +37,12 @@ function mockRingEvent(
eventId: string,
lifetimeMs: number | undefined,
sender = local.userId,
legacyIntent?: string,
): CallNotificationWrapper {
return {
event_id: eventId,
...(lifetimeMs === undefined ? {} : { lifetime: lifetimeMs }),
...(legacyIntent === undefined ? {} : { "m.call.intent": legacyIntent }),
notification_type: "ring",
sender,
} as unknown as CallNotificationWrapper;
@@ -128,7 +134,25 @@ test("ring attempt times out after nobody joins", () => {
});
expectObservable(summarizeRingAttempts$(ringAttempts$)).toBe("-a 29ms A", {
a: { intent: "audio", recipient: alice.userId },
a: { intent: RTC_CALL_INTENT_AUDIO, recipient: alice.userId },
A: { outcome: "timeout" },
});
});
});
test("ring attempt maps the legacy intent on the notification to its stable value", () => {
withTestScheduler(({ scope, expectObservable, hot }) => {
const { ringAttempts$ } = createCallNotificationLifecycle$({
scope,
...defaultProps,
memberUserIds$: constant<string[]>([]),
sentCallNotification$: hot("-a", {
a: mockRingEvent("$notif1", 30, local.userId, "video"),
}),
});
expectObservable(summarizeRingAttempts$(ringAttempts$)).toBe("-a 29ms A", {
a: { intent: RTC_CALL_INTENT_VIDEO, recipient: alice.userId },
A: { outcome: "timeout" },
});
});
@@ -148,7 +172,7 @@ test("ring attempt is accepted once recipient joins", () => {
});
expectObservable(summarizeRingAttempts$(ringAttempts$)).toBe("-aA", {
a: { intent: "audio", recipient: alice.userId },
a: { intent: RTC_CALL_INTENT_AUDIO, recipient: alice.userId },
A: { outcome: "accept" },
});
});
@@ -166,7 +190,7 @@ test("ring attempt is immediately accepted if recipient is already joined", () =
});
expectObservable(summarizeRingAttempts$(ringAttempts$)).toBe("-(aA)", {
a: { intent: "audio", recipient: alice.userId },
a: { intent: RTC_CALL_INTENT_AUDIO, recipient: alice.userId },
A: { outcome: "accept" },
});
});
@@ -201,7 +225,7 @@ test("ring attempt can be declined", () => {
});
expectObservable(summarizeRingAttempts$(ringAttempts$)).toBe("-aA", {
a: { intent: "audio", recipient: alice.userId },
a: { intent: RTC_CALL_INTENT_AUDIO, recipient: alice.userId },
A: { outcome: "decline" },
});
});
@@ -236,7 +260,7 @@ test("ring attempt times out if recipient declines too late", () => {
});
expectObservable(summarizeRingAttempts$(ringAttempts$)).toBe("-a 29ms A", {
a: { intent: "audio", recipient: alice.userId },
a: { intent: RTC_CALL_INTENT_AUDIO, recipient: alice.userId },
A: { outcome: "timeout" },
});
});
@@ -271,7 +295,7 @@ test("decline event relating to wrong event is ignored (times out)", () => {
});
expectObservable(summarizeRingAttempts$(ringAttempts$)).toBe("-a 29ms A", {
a: { intent: "audio", recipient: alice.userId },
a: { intent: RTC_CALL_INTENT_AUDIO, recipient: alice.userId },
A: { outcome: "timeout" },
});
});
@@ -306,7 +330,7 @@ test("decline event from wrong sender is ignored (times out)", () => {
});
expectObservable(summarizeRingAttempts$(ringAttempts$)).toBe("-a 29ms A", {
a: { intent: "audio", recipient: alice.userId },
a: { intent: RTC_CALL_INTENT_AUDIO, recipient: alice.userId },
A: { outcome: "timeout" },
});
});
@@ -9,7 +9,9 @@ import {
type IRTCNotificationContent,
type MatrixRTCSession,
MatrixRTCSessionEvent,
RTC_CALL_INTENT_AUDIO,
type RTCCallIntent,
toStableCallIntent,
} from "matrix-js-sdk/lib/matrixrtc";
import {
filter,
@@ -155,7 +157,10 @@ export function createCallNotificationLifecycle$({
);
return of({
intent: notificationEvent["m.call.intent"] ?? "audio",
// The notification still carries the legacy intent values
intent:
toStableCallIntent(notificationEvent["m.call.intent"]) ??
RTC_CALL_INTENT_AUDIO,
recipient,
outcome$: race(timeout$, accept$, decline$).pipe(
take(1),
+96 -8
View File
@@ -21,6 +21,7 @@ import {
type CallMembership,
type IRTCNotificationContent,
MatrixRTCSessionEvent,
RTC_CALL_INTENT_AUDIO,
} from "matrix-js-sdk/lib/matrixrtc";
import { deepCompare } from "matrix-js-sdk/lib/utils";
@@ -70,9 +71,11 @@ import {
} from "./CallViewModel.ts";
import { type Handler, MuteState } from "../MuteStates.ts";
import {
type LocalVideoMediaTrack,
type RTCParticipation,
type VideoProcessor,
} from "@element-hq/matrixrtc-sdk";
import { type BackgroundEffectsState } from "../BackgroundEffects";
initializeWidget();
@@ -1557,7 +1560,7 @@ describe.each(modes)("CallViewModel (%s mode)", (mode) => {
b: expect.objectContaining({
type: "ringing",
userId: alice.userId,
intent: "audio",
intent: RTC_CALL_INTENT_AUDIO,
}),
});
// Layout should show placeholder media for the participant we're
@@ -1603,7 +1606,7 @@ describe.each(modes)("CallViewModel (%s mode)", (mode) => {
b: expect.objectContaining({
type: "ringing",
userId: alice.userId,
intent: "audio",
intent: RTC_CALL_INTENT_AUDIO,
}),
});
// Layout should show placeholder media for the participant we're
@@ -1692,28 +1695,113 @@ describe("a camera with a video processor", () => {
muteStates,
mockMediaDevices({}),
{},
processor,
{ supported: true, processor },
).find((request) => request.source === "camera"),
).toMatchObject({ processor });
});
it("leaves the camera out of the join while the processor is prepared", () => {
const muteStates = mockMuteStates();
vi.spyOn(muteStates.video.enabled$, "value", "get").mockReturnValue(true);
const sources = (wait?: "preparing"): string[] =>
initialPublishRequests(
muteStates,
mockMediaDevices({}),
{},
{
supported: true,
processor: undefined,
wait,
},
).map((request) => request.source);
expect(sources("preparing")).not.toContain("camera");
expect(sources()).toContain("camera");
});
it("waits to make a camera track until it is known the processor builds", async () => {
const setHandler = vi.spyOn(MuteState.prototype, "setHandler");
onTestFinished(() => setHandler.mockRestore());
const effects$ = new BehaviorSubject<BackgroundEffectsState>({
supported: true,
processor: undefined,
wait: "preparing",
});
let participation: RTCParticipation | undefined;
withCallViewModel(
{},
(_vm, _session, { rtcParticipation }) => {
participation = rtcParticipation;
},
{ backgroundEffectsState$: effects$ },
);
const camera = participation!.localMember$.value?.tracks$.value?.find(
(track) => track.source === "camera",
) as LocalVideoMediaTrack;
const turnedOn = cameraHandler(setHandler)(true);
await flushPromises();
expect(camera.setEnabled).not.toHaveBeenCalledWith(true);
effects$.next({ supported: true, processor });
await turnedOn;
expect(camera.setEnabled).toHaveBeenCalledWith(true);
expect(participation!.publish).not.toHaveBeenCalled();
});
it("turns the camera on only once its processor is on the track", async () => {
const setHandler = vi.spyOn(MuteState.prototype, "setHandler");
onTestFinished(() => setHandler.mockRestore());
const effects$ = new BehaviorSubject<BackgroundEffectsState>({
supported: true,
processor: undefined,
wait: "preparing",
});
let participation: RTCParticipation | undefined;
withCallViewModel(
{},
(_vm, _session, { rtcParticipation }) => {
participation = rtcParticipation;
},
{ backgroundEffectsState$: effects$ },
);
const camera = participation!.localMember$.value?.tracks$.value?.find(
(track) => track.source === "camera",
) as LocalVideoMediaTrack;
// Attaching takes its time, as the pipeline is built on the track
let attached!: () => void;
vi.mocked(camera.setProcessor).mockReturnValueOnce(
new Promise<void>((resolve) => {
attached = resolve;
}),
);
const turnedOn = cameraHandler(setHandler)(true);
effects$.next({ supported: true, processor });
await flushPromises();
expect(camera.setProcessor).toHaveBeenCalledWith(processor);
expect(camera.setEnabled).not.toHaveBeenCalledWith(true);
attached();
await turnedOn;
expect(camera.setEnabled).toHaveBeenCalledWith(true);
});
// The SDK builds the processor on the camera's restart, before the room
// sees a frame, so the track is unmuted rather than published anew
it("turns on a camera turned off before the processor was chosen", async () => {
const setHandler = vi.spyOn(MuteState.prototype, "setHandler");
onTestFinished(() => setHandler.mockRestore());
const videoProcessor$ = new BehaviorSubject<VideoProcessor | undefined>(
undefined,
);
const effects$ = new BehaviorSubject<BackgroundEffectsState>({
supported: true,
processor: undefined,
});
let participation: RTCParticipation | undefined;
withCallViewModel(
{},
(_vm, _session, { rtcParticipation }) => {
participation = rtcParticipation;
},
{ videoProcessor$ },
{ backgroundEffectsState$: effects$ },
);
videoProcessor$.next(processor);
effects$.next({ supported: true, processor });
await cameraHandler(setHandler)(true);
await flushPromises();
+65 -8
View File
@@ -37,6 +37,7 @@ import {
combineLatest,
distinctUntilChanged,
filter,
firstValueFrom,
fromEvent,
map,
merge,
@@ -77,6 +78,7 @@ import {
type ReactionInfo,
type ReactionOption,
} from "../../reactions";
import { type BackgroundEffectsState } from "../BackgroundEffects";
import { shallowEquals as arrayShallowEquals } from "../../utils/array";
import { type MediaDevices } from "../MediaDevices";
import { type MuteStates } from "../MuteStates";
@@ -170,13 +172,16 @@ export function initialPublishRequests(
muteStates: MuteStates,
mediaDevices: MediaDevices,
capture: CaptureSettings,
processor?: VideoProcessor,
effects?: BackgroundEffectsState,
): PublishRequest[] {
const requests: PublishRequest[] = [];
if (muteStates.audio.enabled$.value)
requests.push(publishRequest("microphone", mediaDevices, capture));
if (muteStates.video.enabled$.value)
requests.push(publishRequest("camera", mediaDevices, capture, processor));
// Left out while the processor is prepared; the call publishes it after
if (muteStates.video.enabled$.value && effects?.wait !== "preparing")
requests.push(
publishRequest("camera", mediaDevices, capture, effects?.processor),
);
return requests;
}
@@ -187,8 +192,8 @@ export interface CallViewModelOptions {
* defaults otherwise.
*/
capture?: CaptureSettings;
/** Background blur and the like, applied to the camera as it changes. */
videoProcessor$?: Behavior<VideoProcessor | undefined>;
/** The background effects state (preparing, supported...) including the processor itself. */
backgroundEffectsState$?: Behavior<BackgroundEffectsState>;
/**
* The application hosting Element Call, which can ask it to hang up and wants
* to know when the user joins or leaves. Defaults to no host.
@@ -363,6 +368,8 @@ export interface CallViewModel {
keyRotationSuppressed$: Behavior<boolean>;
/** The local member, once our membership has been seen in the room. */
localMember$: Behavior<LocalRTCMember | null>;
/** Our own camera track, while one is published. */
camera$: Behavior<LocalVideoMediaTrack | undefined>;
remoteMembers$: Behavior<RemoteRTCMember[]>;
/** The transports the client holds a connection to, for the developer panel. */
connectedTransports$: Behavior<TransportMetadata[]>;
@@ -488,7 +495,21 @@ export function createCallViewModel$(
sentCallNotification$ = of(null),
} = options;
const { capture = {}, videoProcessor$ = constant(undefined) } = options;
const {
capture = {},
backgroundEffectsState$ = constant<BackgroundEffectsState>({
supported: false,
processor: undefined,
}),
} = options;
const videoProcessor$ = scope.behavior(
backgroundEffectsState$.pipe(map(({ processor }) => processor)),
);
// Resolves once it is known whether the processor builds
const processorKnown = async (): Promise<unknown> =>
firstValueFrom(
backgroundEffectsState$.pipe(filter(({ wait }) => wait !== "preparing")),
);
const localTracks$ = scope.behavior<
(LocalAudioMediaTrack | LocalVideoMediaTrack)[]
>(
@@ -512,6 +533,19 @@ export function createCallViewModel$(
"screenShare",
) as Behavior<LocalVideoMediaTrack | undefined>;
const processorAttached = followProcessor(
scope,
camera$,
videoProcessor$,
logger,
);
// A camera turned on waits for its processor to be known, then to be on
// the track: LiveKit restarts the track with whatever processor it has then.
const cameraReady = async (): Promise<void> => {
await processorKnown();
await processorAttached();
};
const supply = (source: "microphone" | "camera"): TrackSupply => ({
publish: async () =>
rtcParticipation.publish(
@@ -522,6 +556,7 @@ export function createCallViewModel$(
source === "camera" ? videoProcessor$.value : undefined,
),
),
ready: source === "camera" ? cameraReady : undefined,
});
followMuteState(
scope,
@@ -548,6 +583,29 @@ export function createCallViewModel$(
);
followDevice(scope, camera$, mediaDevices.videoInput, "camera", logger);
// A camera wanted at the join was left out of it while the processor was
// prepared (see initialPublishRequests), and is published once that is known
if (
backgroundEffectsState$.value.wait === "preparing" &&
muteStates.video.enabled$.value
)
processorKnown()
.then(async () => {
if (camera$.value !== undefined || !muteStates.video.enabled$.value)
return;
await rtcParticipation.publish(
publishRequest(
"camera",
mediaDevices,
capture,
videoProcessor$.value,
),
);
})
.catch((e) => {
logger.error("Failed to publish the camera", e);
});
// The output device follows the selection onto the client the same way. A
// host that routes audio itself picks the output outside the browser, so the
// client is left alone
@@ -561,8 +619,6 @@ export function createCallViewModel$(
});
});
followProcessor(scope, camera$, videoProcessor$, logger);
const members$ = scope.behavior<(LocalRTCMember | RemoteRTCMember)[]>(
combineLatest(
[rtcParticipation.localMember$, rtcParticipation.remoteMembers$],
@@ -1508,6 +1564,7 @@ export function createCallViewModel$(
leave,
toggleScreenSharing: toggleScreenSharing,
sharingScreen$: sharingScreen$,
camera$,
tapScreen: (): void => screenTap$.next(),
tapControls: (): void => controlsTap$.next(),
+9 -5
View File
@@ -21,6 +21,10 @@ import {
EARPIECE_CONFIG_ID,
IOSControlledAudioOutput,
} from "./IOSControlledAudioOutput";
import {
RTC_CALL_INTENT_AUDIO,
RTC_CALL_INTENT_VIDEO,
} from "matrix-js-sdk/lib/matrixrtc";
// `vi.mock` calls are hoisted above all imports, so the static imports below
// already see these mocks. Force the iOS platform so that the virtual earpiece
@@ -81,7 +85,7 @@ describe("Default selection", () => {
const output = new IOSControlledAudioOutput(
constant(false),
testScope,
"audio",
RTC_CALL_INTENT_AUDIO,
);
const selected = latestSelection(output);
@@ -100,7 +104,7 @@ describe("Default selection", () => {
const output = new IOSControlledAudioOutput(
constant(false),
testScope,
"video",
RTC_CALL_INTENT_VIDEO,
);
const selected = latestSelection(output);
@@ -113,7 +117,7 @@ describe("Default selection", () => {
const output = new IOSControlledAudioOutput(
constant(false),
testScope,
"audio",
RTC_CALL_INTENT_AUDIO,
);
const selected = latestSelection(output);
@@ -130,7 +134,7 @@ describe("Explicit selection", () => {
const output = new IOSControlledAudioOutput(
constant(false),
testScope,
"audio",
RTC_CALL_INTENT_AUDIO,
);
const selected = latestSelection(output);
@@ -150,7 +154,7 @@ describe("Explicit selection", () => {
const output = new IOSControlledAudioOutput(
constant(false),
testScope,
"audio",
RTC_CALL_INTENT_AUDIO,
);
const selected = latestSelection(output);
+6 -3
View File
@@ -8,7 +8,10 @@ Please see LICENSE in the repository root for full details.
import { logger as rootLogger } from "matrix-js-sdk/lib/logger";
import { combineLatest, merge, startWith, Subject, tap } from "rxjs";
import type { RTCCallIntent } from "matrix-js-sdk/lib/matrixrtc";
import {
RTC_CALL_INTENT_AUDIO,
type RTCCallIntent,
} from "matrix-js-sdk/lib/matrixrtc";
import {
availableOutputDevices$ as controlledAvailableOutputDevices$,
outputDevice$ as controlledOutputSelection$,
@@ -111,7 +114,7 @@ export class IOSControlledAudioOutput implements MediaDevice<
* Chooses the default output device when no explicit selection (from the user
* or the hosting application) has been made yet.
*
* For voice calls (`initialIntent === "audio"`) we want to start on the
* For voice calls (`initialIntent === "m.audio"`) we want to start on the
* earpiece rather than the speaker, like a regular phone call. We only
* override when the device that would otherwise be the default is the
* speaker: if the host already routed to a headset (e.g. Bluetooth) — which
@@ -122,7 +125,7 @@ export class IOSControlledAudioOutput implements MediaDevice<
available: Map<string, AudioOutputDeviceLabel>,
): string | undefined {
const firstId = available.keys().next().value;
if (this.initialIntent === "audio") {
if (this.initialIntent === RTC_CALL_INTENT_AUDIO) {
const firstLabel =
firstId !== undefined ? available.get(firstId) : undefined;
if (firstLabel?.type === "speaker") {
+2 -1
View File
@@ -21,6 +21,7 @@ import { type MediaDevices } from "./MediaDevices";
import { MuteStates } from "./MuteStates";
import { nullHostBridge } from "../HostBridge";
import { flushPromises, mockMediaDevices, testScope } from "../utils/test";
import { type BackgroundEffectsState } from "./BackgroundEffects";
describe("LobbyMediaViewModel", () => {
it("opens what starts enabled in one capture and follows the devices", async () => {
@@ -135,7 +136,7 @@ function setup(
muteStates,
mediaDevices,
{},
constant(processor),
constant<BackgroundEffectsState>({ supported: true, processor }),
);
return { media, vm, muteStates, mediaDevices };
}
+14 -4
View File
@@ -13,7 +13,6 @@ import {
type ObservableScope,
type PublishRequest,
trackBySource$,
type VideoProcessor,
} from "@element-hq/matrixrtc-sdk";
import { logger as rootLogger } from "matrix-js-sdk/lib/logger";
import { combineLatest, debounceTime, map } from "rxjs";
@@ -25,6 +24,7 @@ import {
import { type MediaDevices } from "./MediaDevices";
import { type MuteStates } from "./MuteStates";
import { followDevice, followProcessor } from "./localMediaControls";
import { type BackgroundEffectsState } from "./BackgroundEffects";
export interface LobbyMediaViewModel {
/** The camera, once open, for the preview. */
@@ -42,9 +42,13 @@ export function createLobbyMediaViewModel(
muteStates: MuteStates,
mediaDevices: MediaDevices,
capture: CaptureSettings,
videoProcessor$: Behavior<VideoProcessor | undefined>,
/** No camera is opened while its pipeline is prepared: it would show the room without the processor. */
backgroundEffectsState$: Behavior<BackgroundEffectsState>,
): LobbyMediaViewModel {
const logger = rootLogger.getChild("[LobbyMedia]");
const videoProcessor$ = scope.behavior(
backgroundEffectsState$.pipe(map(({ processor }) => processor)),
);
const microphone$ = trackBySource$(
scope,
media.tracks$,
@@ -64,13 +68,19 @@ export function createLobbyMediaViewModel(
muteStates.video.enabled$,
microphone$,
camera$,
backgroundEffectsState$,
])
.pipe(debounceTime(0), scope.bind())
.subscribe(([audio, video, microphone, camera]) => {
.subscribe(([audio, video, microphone, camera, { wait }]) => {
const open: PublishRequest[] = [];
if (audio && microphone === undefined && !opening.has("microphone"))
open.push(publishRequest("microphone", mediaDevices, capture));
if (video && camera === undefined && !opening.has("camera"))
if (
video &&
camera === undefined &&
!opening.has("camera") &&
wait !== "preparing"
)
open.push(
publishRequest(
"camera",
+6 -2
View File
@@ -35,6 +35,10 @@ import { AudioOutput, MediaDevices } from "./MediaDevices";
import { AndroidControlledAudioOutput } from "./AndroidControlledAudioOutput";
import { IOSControlledAudioOutput } from "./IOSControlledAudioOutput";
import { ObservableScope } from "./ObservableScope";
import {
RTC_CALL_INTENT_AUDIO,
RTC_CALL_INTENT_VIDEO,
} from "matrix-js-sdk/lib/matrixrtc";
// Which audio output implementation is used is decided by what the app hosting
// Element Call told it, rather than being discovered from the environment.
@@ -52,7 +56,7 @@ describe("MediaDevices audio output", () => {
const devices = new MediaDevices(new ObservableScope(), {
controlledAudioDevices: true,
callIntent: "audio",
callIntent: RTC_CALL_INTENT_AUDIO,
});
expect(devices.audioOutput).toBeInstanceOf(AndroidControlledAudioOutput);
@@ -63,7 +67,7 @@ describe("MediaDevices audio output", () => {
const devices = new MediaDevices(new ObservableScope(), {
controlledAudioDevices: true,
callIntent: "video",
callIntent: RTC_CALL_INTENT_VIDEO,
});
expect(devices.audioOutput).toBeInstanceOf(IOSControlledAudioOutput);
+15 -11
View File
@@ -6,7 +6,11 @@ Please see LICENSE in the repository root for full details.
*/
import { test, expect } from "vitest";
import { type RTCCallIntent } from "matrix-js-sdk/lib/matrixrtc";
import {
RTC_CALL_INTENT_AUDIO,
RTC_CALL_INTENT_VIDEO,
type RTCCallIntent,
} from "matrix-js-sdk/lib/matrixrtc";
import { calculateInitialMuteState } from "./initialMuteState";
@@ -14,10 +18,10 @@ test.each<{
callIntent: RTCCallIntent;
allowJoinUnmutedViaIntent: boolean;
}>([
{ callIntent: "audio", allowJoinUnmutedViaIntent: false },
{ callIntent: "audio", allowJoinUnmutedViaIntent: true },
{ callIntent: "video", allowJoinUnmutedViaIntent: false },
{ callIntent: "video", allowJoinUnmutedViaIntent: true },
{ callIntent: RTC_CALL_INTENT_AUDIO, allowJoinUnmutedViaIntent: false },
{ callIntent: RTC_CALL_INTENT_AUDIO, allowJoinUnmutedViaIntent: true },
{ callIntent: RTC_CALL_INTENT_VIDEO, allowJoinUnmutedViaIntent: false },
{ callIntent: RTC_CALL_INTENT_VIDEO, allowJoinUnmutedViaIntent: true },
{ callIntent: "unknown", allowJoinUnmutedViaIntent: false },
{ callIntent: "unknown", allowJoinUnmutedViaIntent: true },
])(
@@ -29,15 +33,15 @@ test.each<{
allowJoinUnmutedViaIntent,
);
expect(audioEnabled).toBe(true);
expect(videoEnabled).toBe(callIntent !== "audio");
expect(videoEnabled).toBe(callIntent !== RTC_CALL_INTENT_AUDIO);
},
);
test.each<{
callIntent: RTCCallIntent;
}>([
{ callIntent: "audio" },
{ callIntent: "video" },
{ callIntent: RTC_CALL_INTENT_AUDIO },
{ callIntent: RTC_CALL_INTENT_VIDEO },
{ callIntent: "unknown" },
])(
"Should always mute on start if skipping lobby and the host does not vouch for the intent (callIntent: $callIntent)",
@@ -55,8 +59,8 @@ test.each<{
test.each<{
callIntent: RTCCallIntent;
}>([
{ callIntent: "audio" },
{ callIntent: "video" },
{ callIntent: RTC_CALL_INTENT_AUDIO },
{ callIntent: RTC_CALL_INTENT_VIDEO },
{ callIntent: "unknown" },
])(
"Can start unmuted if skipping lobby and the host vouches for the intent (callIntent: $callIntent)",
@@ -67,6 +71,6 @@ test.each<{
true,
);
expect(audioEnabled).toBe(true);
expect(videoEnabled).toBe(callIntent !== "audio");
expect(videoEnabled).toBe(callIntent !== RTC_CALL_INTENT_AUDIO);
},
);
+6 -3
View File
@@ -6,7 +6,10 @@ Please see LICENSE in the repository root for full details.
*/
import { logger } from "matrix-js-sdk/lib/logger";
import { type RTCCallIntent } from "matrix-js-sdk/lib/matrixrtc";
import {
RTC_CALL_INTENT_AUDIO,
type RTCCallIntent,
} from "matrix-js-sdk/lib/matrixrtc";
/**
* Calculates the initial mute state for media devices based on configuration.
@@ -40,10 +43,10 @@ export function calculateInitialMuteState(
// A host that vouches for the intent is a trusted environment, so it allows
// unmuted by default. Same for when showing a lobby, as users can adjust
// their settings there. Additionally, if the call intent is "audio", we
// their settings there. Additionally, if the call intent is "m.audio", we
// disable video by default.
return {
audioEnabled: true,
videoEnabled: callIntent != "audio",
videoEnabled: callIntent != RTC_CALL_INTENT_AUDIO,
};
}

Some files were not shown because too many files have changed in this diff Show More