Compare commits

...

150 Commits

Author SHA1 Message Date
Valere
80e760ca55 review 2025-12-16 13:40:06 +01:00
Valere
190cdfcb60 comment now dead state variant 2025-12-12 17:03:16 +01:00
Valere
8f2055b4f4 eslint fix 2025-12-12 14:46:13 +01:00
Valere
93da69983d post merge: partial mapping of tracks/publish states 2025-12-12 14:40:45 +01:00
Valere
b3b76d8b3d post merge 2025-12-12 11:54:43 +01:00
Valere
d052d55547 Merge branch 'livekit' into valere/double_pub_problems_highlevel 2025-12-12 11:54:35 +01:00
Valere
610af439a8 cleaning: just use LocalTrackPublished event to pause/unpause 2025-12-12 10:37:37 +01:00
Valere
f8310b4611 publisher: only use highlevel participant APIs 2025-12-12 10:31:08 +01:00
Timo
cf453baf5d Merge pull request #3611 from element-hq/toger5/connection-state-refactor
Unify LiveKit and Matrix connection states
2025-12-11 18:23:34 +01:00
Timo K
7edc97b917 remove continue button things 2025-12-11 17:24:35 +01:00
Timo K
8225e4f260 remove page.pause 2025-12-11 17:22:02 +01:00
Timo K
207b161b3b fix logger and dismiss button presses 2025-12-11 17:17:56 +01:00
Timo K
9a7e797af4 fix lint 2025-12-11 16:17:45 +01:00
Timo K
08306d663a remove duplicated connecting state and update Test setup 2025-12-11 16:04:12 +01:00
Timo K
328cc7133a update playwright so that we do not even need the dismiss anymore. 2025-12-11 11:32:28 +01:00
Timo K
170a38c0ba fix playwright incompatible browser toast 2025-12-11 11:30:14 +01:00
Timo K
aabd76044b fix lint 2025-12-10 21:25:35 +01:00
Timo K
e626698fda fix connection tests 2025-12-10 21:22:55 +01:00
Timo K
8dac0366b6 fix lints 2025-12-10 21:17:33 +01:00
Timo K
b380532d30 lots of error logging and fixing playwright 2025-12-10 21:14:13 +01:00
Timo K
667a3d0e3d fix test not checking for livekit connection state anymore. 2025-12-10 19:18:16 +01:00
Timo K
1941fc9ca1 fix tests. 2025-12-10 19:12:52 +01:00
Timo K
6efce232f8 fix playwright tests 2025-12-10 18:50:19 +01:00
Timo K
ef2f53c38a Merge branch 'livekit' into toger5/connection-state-refactor 2025-12-10 13:49:06 +01:00
Timo K
0ebc6078dd Update LocalMember.ts 2025-12-10 12:08:59 +01:00
Timo K
7af89b4216 fix lint 2025-12-09 17:36:56 +01:00
Robin
ee2b0c6a5d Merge pull request #3625 from robintown/misc
Miscellaneous clean-up
2025-12-09 10:01:11 -05:00
Timo K
bf801364a6 cleanup and tests 2025-12-09 15:23:30 +01:00
Robin
6ee3ef2795 Edit a misleading log line
The factory function is called once per item to construct the item. It is not called on future updates to the item's data.
2025-12-08 23:38:54 -05:00
Robin
2f3f9f95eb Use more compact optional chaining and coalescing notation 2025-12-08 23:38:15 -05:00
Robin
9481dc401c Remove extraneous 'scope running' check
Semantically, behaviors are only meaningful for as long as their scope is running. Setting a behavior's value to an empty array once its scope ends is not guaranteed to work (as it depends on execution order of how the scope is ended), and subscribers should be robust enough to handle clean-up of all connections at the end of the scope either way.
2025-12-08 23:34:42 -05:00
Robin
d8b9568400 Stop publisher in a less brittle way 2025-12-08 23:33:41 -05:00
Timo K
7c40b0e177 ideas 2025-12-05 19:48:02 +01:00
Valere Fedronic
e4404e5bb1 Merge pull request #3608 from robintown/initial-layout
Fix the wrong layout being used until window size changes
2025-12-04 16:44:50 +01:00
Valere Fedronic
f5d4017981 Merge pull request #3606 from element-hq/valere/fix_transiant_earpiece_overlay
fix: The handset mode overlay is visible a split second for every call
2025-12-04 11:05:26 +01:00
Valere
71bf55f358 also test that video is muted when earpiece overlay is on 2025-12-04 10:37:14 +01:00
Valere
7a2c1af44b review: use simple transition instead of keyframe 2025-12-04 10:36:57 +01:00
Valere
940c787040 review: quick renaming 2025-12-04 10:06:45 +01:00
Valere
5d55f83e90 Merge branch 'livekit' into valere/fix_transiant_earpiece_overlay 2025-12-04 10:05:06 +01:00
Valere Fedronic
9e3e359445 Merge pull request #3605 from element-hq/valere/layout_switch
Fix: Allow to manually switch back to grid mode when there is a screenshare
2025-12-04 09:50:56 +01:00
Valere
be7407ea3d review: quick renaming 2025-12-04 09:37:07 +01:00
Valere Fedronic
0a3483729f Merge pull request #3615 from element-hq/valere/fix_lk_server_1.9.4
keep livekit 1.9.4 as the latest is breaking the dev backend
2025-12-04 09:28:36 +01:00
Robin
f599e22e94 Merge pull request #3614 from robintown/simplify-analytics-id
Simplify computation of analytics ID
2025-12-03 13:59:01 -05:00
Valere
fdc66a1d62 fix: existing screenshare switching twice 2025-12-03 18:36:51 +01:00
Valere
c7491c3e97 move fixture to correct folder 2025-12-03 17:25:32 +01:00
Valere
af08c1830e keep livekit 1.9.4 as the latest is breaking the dev backend 2025-12-03 17:18:35 +01:00
Valere
bcb2b36888 keep livekit 1.9.4 as the latest is breaking the dev backend 2025-12-03 17:17:58 +01:00
Valere Fedronic
b9e346d64c Merge pull request #3596 from element-hq/valere/bugfix_earpiece_mute_video
Fix: Camera is not muted when the earpiece mode is enabled
2025-12-03 17:12:10 +01:00
Valere
0ed7194d87 fix: earpiece overlay not showing + playwright test! 2025-12-03 17:10:57 +01:00
Robin
bbd92f666b Simplify computation of analytics ID
Since we now bundle a trusted Element Call widget with our messenger applications and this widget reports analytics to an endpoint determined by the messenger app, there is no longer any reason to compute a different analytics ID from the one used by the messenger app.
2025-12-03 10:42:04 -05:00
Timo K
88721be952 cleanup 2025-12-03 10:04:22 +01:00
Valere
6e7bedbaec Merge branch 'livekit' into valere/layout_switch 2025-12-03 09:17:31 +01:00
Valere
a93ceeae4b review: Keep previous behavior for now to always auto switch 2025-12-03 09:01:26 +01:00
Valere
b85f36598c fix: mistake in file name 2025-12-03 08:54:52 +01:00
Timo K
2e646bfac1 Unify LiveKit and Matrix connection states 2025-12-02 19:49:13 +01:00
Valere Fedronic
e0bf51b63b Merge pull request #3590 from element-hq/valere/noise_cancellation
Config: UrlParams to control noiseSuppression and echoCancellation
2025-12-02 18:36:32 +01:00
Valere
f6a3a371cb fix lint 2025-12-02 17:52:27 +01:00
Valere
ac9acc0158 review: refactor convert params to object for generateRoomOption 2025-12-02 17:47:00 +01:00
Valere
91f3b8b114 Merge branch 'livekit' into valere/noise_cancellation 2025-12-02 17:44:06 +01:00
Valere
be0c7eb365 review: fix mock import module 2025-12-02 17:43:58 +01:00
Valere
44980a2744 review: rename deviceConnected to canControlDevices 2025-12-02 16:39:26 +01:00
Robin
83ea154e1a Fix the wrong layout being used until window size changes
While looking into what had regressed https://github.com/element-hq/element-call/issues/3588, I found that 28047217b8 had filled in a couple of behaviors with non-reactive default values, the "natural window mode" behavior being among them. This meant that the app would no longer determine the correct window mode upon joining a call, instead always guessing "normal" as the value. This change restores its reactivity.
2025-12-02 10:36:53 -05:00
Valere
0e04fd9433 fix: The handset mode overlay is visible a split second for every call 2025-12-02 14:14:28 +01:00
Valere
b5d3f3c72a fix: Allow switching back to grid when there is a screenshare 2025-12-02 11:18:50 +01:00
Valere
60bc6f1e93 refactor: Extract layout mode switch + test 2025-12-02 11:00:32 +01:00
Timo
f05d4b158e Merge pull request #3565 from element-hq/renovate/matrix-widget-api
Update dependency matrix-widget-api to v1.15.0
2025-12-01 19:14:52 +01:00
Robin
0240087afe Unpin matrix-js-sdk so it can be upgraded 2025-12-01 13:09:10 -05:00
Timo K
4ec1b19fe2 fix js-sdk tests 2025-12-01 18:54:57 +01:00
Timo K
5f6a45f0e9 also bump js-sdk 2025-12-01 18:45:47 +01:00
renovate[bot]
e2916d9093 Update dependency matrix-widget-api to v1.14.0 2025-12-01 18:23:35 +01:00
Timo
90d4b6a6ce Merge pull request #3598 from element-hq/renovate/github-actions
Update GitHub Actions
2025-12-01 17:42:05 +01:00
Timo
e3eb2ea12b Merge pull request #3599 from element-hq/renovate/embedded-package-dependencies
Update embedded package dependencies
2025-12-01 17:40:29 +01:00
Timo
4f45825cf2 Merge pull request #3591 from element-hq/toger5/fix-local-transport-update
Fix reactivity based on localTransport updates (Cleanup for LocalMembership.ts)
2025-12-01 17:39:55 +01:00
Timo K
8cb8357398 make logs just debug. (info -> debug) 2025-12-01 17:34:37 +01:00
Timo K
63cd4f79dd fix playwright test! (It caught an actual bug!!! so the right wording
would be: fix implementation thanks to the playwright test!
2025-12-01 17:29:35 +01:00
Valere Fedronic
7fd4424e7b Merge pull request #3601 from element-hq/valere/fix_switch_spotlight_screenshare
fix: Automatically switch to spotlight if remote screenshare
2025-12-01 15:30:24 +01:00
Valere
b7fc86ee06 review: move setGridMode close to the value it sets 2025-12-01 15:25:17 +01:00
Timo K
f26aa8f970 fix tests 2025-12-01 15:10:38 +01:00
Valere
1ff479a315 fix: Automatically switch to spotlight if remote screenshare 2025-12-01 15:06:17 +01:00
Timo K
47c6a17d1e review 2025-12-01 14:58:06 +01:00
Timo K
24ed43ce13 Merge branch 'livekit' into toger5/fix-local-transport-update 2025-12-01 14:23:11 +01:00
renovate[bot]
04bbf83ac5 Update GitHub Actions 2025-12-01 11:00:52 +00:00
renovate[bot]
5aa82295fd Update embedded package dependencies 2025-12-01 01:28:55 +00:00
Timo
3331af1108 Merge pull request #3592 from element-hq/valere/fix_flaky_test
test: use `setSystemTime` for better test stability
2025-11-28 22:00:24 +01:00
Timo K
66dece98a5 add more test for publisher 2025-11-28 21:50:22 +01:00
Timo K
2011aef116 skip "Should show error screen if call creation is restricted" on ff 2025-11-28 18:00:04 +01:00
Valere Fedronic
f5c422c559 Merge pull request #3597 from element-hq/valere/test_reconcile
test: Add simple tests for scope.reconcile
2025-11-28 10:00:06 +01:00
Valere
c6bfda94cf test: additional reconcile test 2025-11-28 09:53:18 +01:00
Valere
cdd391b3a2 test: Add simple tests for scope.reconcile 2025-11-28 09:12:05 +01:00
Valere
59d21d2421 Merge branch 'livekit' into valere/bugfix_earpiece_mute_video 2025-11-27 18:47:45 +01:00
Valere
149f3d02ae fix: The force mute state was not synced to the handler 2025-11-27 18:47:33 +01:00
Valere Fedronic
5bc7b31dcd Merge pull request #3593 from element-hq/valere/quality_gate
test: Enable quality gate and touch a file
2025-11-27 18:28:30 +01:00
Timo K
c0913b6546 fix playwright test 2025-11-27 18:02:46 +01:00
Timo K
46f8fe4ec7 fix test errors 2025-11-27 16:43:28 +01:00
Valere
c00e332dd8 Revert "vitest: Add threshold on vitest to check locally"
This reverts commit e5509ba49b.
2025-11-27 16:43:12 +01:00
Valere
e5509ba49b vitest: Add threshold on vitest to check locally 2025-11-27 16:39:45 +01:00
Valere
4c2c7d51cf revert test changes on ObservableScope 2025-11-27 16:38:51 +01:00
Valere
fc39e82666 Fix: Camera is not muted when the earpiece mode is enabled 2025-11-27 15:52:25 +01:00
Timo K
4b0f6e76c4 revert complete behavior check 2025-11-27 15:38:31 +01:00
Timo K
e5117f705d More testing and cleaning up 2025-11-27 14:42:23 +01:00
Valere
794585514a Revert "add untested function"
This reverts commit c078dd87fa.
2025-11-27 10:04:09 +01:00
Valere
c078dd87fa add untested function 2025-11-26 19:28:44 +01:00
Valere
2b3e2f6188 test: Enable quality gate and touch a file 2025-11-26 19:13:07 +01:00
Valere
acd5dde6d8 test: use setSystemTime for better test stability 2025-11-26 16:27:54 +01:00
Timo K
d22d7460fe Another larger refactor to fix sfu switches and in general proper
cleanup.
2025-11-25 20:18:34 +01:00
Valere
7f3596845c fix formatting 2025-11-25 11:40:38 +01:00
Valere
e2607d6399 Config: UrlParams to control noiseSuppression and echoCancellation 2025-11-25 11:10:53 +01:00
Timo
3491a68362 Merge pull request #3576 from element-hq/toger5/fix-skipped-tests
Refactor local transport testing and local membership initialization
2025-11-24 11:36:27 +01:00
Valere
294b90b6dc fix fixture following web change 2025-11-24 11:29:03 +01:00
Timo K
ddd015d696 fix dev EditInPlace 2025-11-24 10:04:01 +01:00
Timo K
7532f72c90 use edit in place (WIP) This is WIP since it will submite -> reload the
page onSave
2025-11-21 18:49:30 +01:00
Timo K
e60bc9e98f add developer tab snapshot test 2025-11-21 17:44:35 +01:00
Timo
1aa1af2ade Merge pull request #3585 from element-hq/toger5/add-log-screenshare
Add log to screenshare
2025-11-21 16:58:04 +01:00
Timo K
ccadc17406 Add log to screenshare. 2025-11-21 16:19:19 +01:00
Timo K
a731981388 Cleanup transport creation and local store hack. 2025-11-21 14:02:15 +01:00
Timo K
4099c4383d move HomeserverConnected 2025-11-21 13:04:28 +01:00
Timo K
d34775fc47 fix linter 2025-11-20 16:40:06 +01:00
Timo K
a29f0162ad remove widget log 2025-11-20 16:34:53 +01:00
Timo K
a69a50d30c add log to error boudnary 2025-11-20 16:32:40 +01:00
Timo K
89a1bfac2d remove unused test helper 2025-11-20 15:25:53 +01:00
Timo K
1fd9ac93c9 refactor local transport testing and local memberhsip initialization 2025-11-20 15:19:00 +01:00
Timo
f6ef872242 Merge pull request #3575 from element-hq/hs/lint-src-imports
Restrict importing src/lib in most cases for matrix- libs
2025-11-20 14:55:41 +01:00
Half-Shot
07de8483d5 Restrict importing src/lib in most cases for matrix- libs 2025-11-20 13:32:09 +00:00
Valere Fedronic
42b4c8301f Merge pull request #3547 from element-hq/valere/fix_blank_widget_auto_leave
fix: Send close widget action on auto-leave
2025-11-20 11:41:43 +01:00
Valere
c2f541ff8c split test in 2 one testing sound and one to just leave 2025-11-20 11:28:24 +01:00
Valere
c1049b6088 fix import causing lint errors in node_modules 2025-11-20 10:46:24 +01:00
Valere
6577db16ce Merge branch 'livekit' into valere/fix_blank_widget_auto_leave 2025-11-20 10:41:31 +01:00
renovate[bot]
cdc3c2b216 Update LiveKit components (#3573)
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
2025-11-19 20:18:18 +01:00
renovate[bot]
47324b52f8 Update dependency livekit-client to v2.16.0 (#3574)
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
2025-11-19 20:14:37 +01:00
alutarius
82706cca4f Correct Caddy reverse proxy config example (#3572) 2025-11-18 18:46:00 +01:00
Timo
665a6be484 Merge pull request #3560 from element-hq/voip-team/call-viewmodel-refactor
`CallViewmodel` refactor
2025-11-18 13:48:03 +01:00
Valere
0076b54e52 minor comments/TODO cleaning 2025-11-18 12:14:17 +01:00
Valere
614762b5f8 fixup headers 2025-11-18 10:13:10 +01:00
Timo
222982c52c Merge pull request #3571 from element-hq/toger5/createCallViewModelFunction$
`createCallViewModel$` function refactor
2025-11-17 19:46:06 +01:00
Robin
171c289f2d Merge pull request #3564 from element-hq/renovate/livekit-client
Update dependency livekit-client to v2.15.15
2025-11-17 13:41:44 -05:00
renovate[bot]
9ad029d4ff Update dependency livekit-client to v2.15.15 2025-11-17 02:42:33 +00:00
Ben Banfield-Zanin
a3950869fb Use the dedicated bug report sub-domain (#3557)
* Update docs for 'rageshake.submit_url' to use the dedicated subdomain

* Update Netlify preview deployment to use the dedicated subdomain for 'rageshake.submit_url'
2025-11-13 15:33:59 +00:00
Robin
5afc3c2598 Merge pull request #3556 from element-hq/renovate/node-24.x
Update Node.js to v24
2025-11-05 13:55:24 -05:00
Robin
8ad0cb1955 Merge pull request #3551 from element-hq/renovate/embedded-package-dependencies
Update embedded package dependencies to v8.13.0
2025-11-05 13:53:27 -05:00
Robin
9bd464a7f1 Merge pull request #3550 from element-hq/renovate/github-actions
Update softprops/action-gh-release digest to 6da8fa9
2025-11-05 13:53:08 -05:00
Robin
f0adae41ef Merge pull request #3552 from element-hq/renovate/livekit-client
Update dependency livekit-client to v2.15.14
2025-11-05 13:49:44 -05:00
renovate[bot]
f9cc930155 Update Node.js to v24 2025-11-05 18:49:21 +00:00
Robin
eebcd133f3 Merge pull request #3553 from element-hq/robin/trusted-publishing
Remove use of npm access token now that trusted publishing is set up
2025-11-05 13:45:52 -05:00
Robin
79bd458dc2 Remove use of npm access token now that trusted publishing is set up
npm has recently limited the lifetime of all access tokens to 90 days (https://gh.io/npm-token-changes), so it would be a bit inconvenient to stick to our current access token-based method of publishing releases. Meanwhile npm has implemented a more secure publishing method based on OIDC in which you tell the registry that a particular GitHub Actions workflow should be a "trusted publisher" for a given package, and then the CLI will authenticate automatically. (https://docs.npmjs.com/trusted-publishers)

I've already set trusted publishing up on the registry side, and since we're already granting the job permission to generate ID tokens for provenance, there should be no additional lines of config needed to make it work. Let's take away the access token and see how this goes next time we release.
2025-11-03 17:13:03 -05:00
renovate[bot]
a4b1ce52cd Update dependency livekit-client to v2.15.14 2025-11-03 01:02:47 +00:00
renovate[bot]
b22a937e7c Update softprops/action-gh-release digest to 6da8fa9 2025-11-01 02:02:21 +00:00
Valere
0309ceb6aa fix lint 2025-10-27 15:59:01 +01:00
Valere
3ecffccd9e fix prettier 2025-10-27 15:41:16 +01:00
Valere
3dd0b156cf fix: Send close widget action on auto-leave 2025-10-27 14:31:23 +01:00
renovate[bot]
214bb75d27 Update embedded package dependencies to v8.13.0 2025-10-01 02:29:37 +00:00
76 changed files with 4864 additions and 1538 deletions

View File

@@ -43,6 +43,29 @@ module.exports = {
// To encourage good usage of RxJS:
"rxjs/no-exposed-subjects": "error",
"rxjs/finnish": ["error", { names: { "^this$": false } }],
"no-restricted-imports": [
"error",
{
paths: ["matrix-widget-api", "matrix-js-sdk"].flatMap((lib) =>
["src", "src/", "src/index", "lib", "lib/", "lib/index"]
.map((path) => `${lib}/${path}`)
.map((name) => ({ name, message: `Please use ${lib} instead` })),
),
patterns: [
...["matrix-widget-api"].map((lib) => ({
group: ["src", "src/", "src/**", "lib", "lib/", "lib/**"].map(
(path) => `${lib}/${path}`,
),
message: `Please use ${lib} instead`,
})),
// XXX: We use /lib in lots of places, so allow for now.
...["matrix-js-sdk"].map((lib) => ({
group: ["src", "src/", "src/**"].map((path) => `${lib}/${path}`),
message: `Please use ${lib} instead`,
})),
],
},
],
},
overrides: [
{

View File

@@ -23,7 +23,7 @@ jobs:
packages: write
steps:
- name: Check it out
uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
- name: 📥 Download artifact
uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4
@@ -42,7 +42,7 @@ jobs:
- name: Extract metadata (tags, labels) for Docker
id: meta
uses: docker/metadata-action@c1e51972afc2121e065aed6d45c65596fe445f3f # v5.8.0
uses: docker/metadata-action@318604b99e75e41977312d83839a89be02ca4893 # v5.9.0
with:
images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
tags: ${{ inputs.docker_tags}}

View File

@@ -32,7 +32,7 @@ jobs:
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
- name: Enable Corepack
run: corepack enable
- name: Yarn cache

View File

@@ -7,7 +7,7 @@ jobs:
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
- name: Enable Corepack
run: corepack enable
- name: Yarn cache

View File

@@ -85,7 +85,7 @@ jobs:
run: find ${{ env.FILENAME_PREFIX }} -type f -print0 | sort -z | xargs -0 sha256sum | tee ${{ env.FILENAME_PREFIX }}.sha256
- name: Upload
if: ${{ needs.versioning.outputs.DRY_RUN == 'false' }}
uses: softprops/action-gh-release@aec2ec56f94eb8180ceec724245f64ef008b89f5 # v2
uses: softprops/action-gh-release@a06a81a03ee405af7f2048a818ed3f03bbf83c7b # v2
with:
files: |
${{ env.FILENAME_PREFIX }}.tar.gz
@@ -100,10 +100,10 @@ jobs:
ARTIFACT_VERSION: ${{ steps.artifact_version.outputs.ARTIFACT_VERSION }}
permissions:
contents: read
id-token: write # required for the provenance flag on npm publish
id-token: write # Allow npm to authenticate as a trusted publisher
steps:
- name: Checkout
uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
- name: 📥 Download built element-call artifact
uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4
@@ -126,8 +126,6 @@ jobs:
npm version ${{ needs.versioning.outputs.PREFIXED_VERSION }} --no-git-tag-version
echo "ARTIFACT_VERSION=$(jq '.version' --raw-output package.json)" >> "$GITHUB_ENV"
npm publish --provenance --access public --tag ${{ needs.versioning.outputs.TAG }} ${{ needs.versioning.outputs.DRY_RUN == 'true' && '--dry-run' || '' }}
env:
NODE_AUTH_TOKEN: ${{ secrets.NPM_RELEASE_TOKEN }}
- id: artifact_version
name: Output artifact version
@@ -144,7 +142,7 @@ jobs:
contents: read
steps:
- name: Checkout
uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
- name: 📥 Download built element-call artifact
uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4
@@ -199,7 +197,7 @@ jobs:
contents: read
steps:
- name: Checkout
uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
with:
path: element-call
@@ -212,7 +210,7 @@ jobs:
path: element-call/embedded/ios/Sources/dist
- name: Checkout element-call-swift
uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
with:
repository: element-hq/element-call-swift
path: element-call-swift
@@ -264,7 +262,7 @@ jobs:
echo "iOS: ${{ needs.publish_ios.outputs.ARTIFACT_VERSION }}"
- name: Add release notes
if: ${{ needs.versioning.outputs.DRY_RUN == 'false' }}
uses: softprops/action-gh-release@aec2ec56f94eb8180ceec724245f64ef008b89f5 # v2
uses: softprops/action-gh-release@a06a81a03ee405af7f2048a818ed3f03bbf83c7b # v2
with:
append_body: true
body: |

View File

@@ -42,7 +42,7 @@ jobs:
- name: Create Checksum
run: find ${{ env.FILENAME_PREFIX }} -type f -print0 | sort -z | xargs -0 sha256sum | tee ${{ env.FILENAME_PREFIX }}.sha256
- name: Upload
uses: softprops/action-gh-release@aec2ec56f94eb8180ceec724245f64ef008b89f5 # v2
uses: softprops/action-gh-release@a06a81a03ee405af7f2048a818ed3f03bbf83c7b # v2
with:
files: |
${{ env.FILENAME_PREFIX }}.tar.gz
@@ -68,7 +68,7 @@ jobs:
runs-on: ubuntu-latest
steps:
- name: Add release note
uses: softprops/action-gh-release@aec2ec56f94eb8180ceec724245f64ef008b89f5 # v2
uses: softprops/action-gh-release@a06a81a03ee405af7f2048a818ed3f03bbf83c7b # v2
with:
append_body: true
body: |

View File

@@ -9,7 +9,7 @@ jobs:
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
- name: Enable Corepack
run: corepack enable
- name: Yarn cache
@@ -33,7 +33,7 @@ jobs:
timeout-minutes: 60
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
- name: Enable Corepack
run: corepack enable
- uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4

View File

@@ -13,7 +13,7 @@ jobs:
steps:
- name: Checkout the code
uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
- name: Enable Corepack
run: corepack enable
@@ -42,7 +42,7 @@ jobs:
- name: Create Pull Request
id: cpr
uses: peter-evans/create-pull-request@271a8d0340265f705b14b6d32b9829c1cb33d45e # v7.0.8
uses: peter-evans/create-pull-request@84ae59a2cdc2258d6fa0732dd66352dddae2a412 # v7.0.9
with:
token: ${{ secrets.ELEMENT_BOT_TOKEN }}
branch: actions/localazy-download

View File

@@ -14,7 +14,7 @@ jobs:
steps:
- name: Checkout the code
uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
- name: Upload
uses: localazy/upload@27e6b5c0fddf4551596b42226b1c24124335d24a # v1

View File

@@ -1 +1 @@
22
24

View File

@@ -13,7 +13,6 @@ coverage:
informational: true
patch:
default:
# Encourage (but don't enforce) 80% coverage on all lines that a PR
# Enforce 80% coverage on all lines that a PR
# touches
target: 80%
informational: true

View File

@@ -18,7 +18,7 @@
"api_host": "https://posthog-element-call.element.io"
},
"rageshake": {
"submit_url": "https://element.io/bugreports/submit"
"submit_url": "https://rageshakes.element.io/api/submit"
},
"sentry": {
"environment": "netlify-pr-preview",

View File

@@ -47,7 +47,7 @@ services:
- ecbackend
livekit:
image: livekit/livekit-server:latest
image: livekit/livekit-server:v1.9.4
pull_policy: always
hostname: livekit-sfu
command: --dev --config /etc/livekit.yaml
@@ -67,7 +67,7 @@ services:
- ecbackend
livekit-1:
image: livekit/livekit-server:latest
image: livekit/livekit-server:v1.9.4
pull_policy: always
hostname: livekit-sfu-1
command: --dev --config /etc/livekit.yaml

View File

@@ -136,8 +136,8 @@ handle @jwt_service {
reverse_proxy http://[::1]:8080 {
header_up Host {host}
header_up X-Forwarded-Server {host}
header_up X-Real-IP {remote_addr}
header_up X-Forwarded-For {remote_addr}
header_up X-Real-IP {remote_host}
header_up X-Forwarded-For {remote_host}
}
}
@@ -146,8 +146,8 @@ handle {
reverse_proxy http://localhost:7880 {
header_up Host {host}
header_up X-Forwarded-Server {host}
header_up X-Real-IP {remote_addr}
header_up X-Forwarded-For {remote_addr}
header_up X-Real-IP {remote_host}
header_up X-Forwarded-For {remote_host}
}
}
```

View File

@@ -96,6 +96,6 @@ These parameters are only supported in the [embedded](./embedded-standalone.md)
| -------------------- | -------------------------------------------------------------------------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------- |
| `posthogApiHost` | Posthog server URL | No | e.g. `https://posthog-element-call.element.io`. Only supported in embedded package. In full package the value from config is used. |
| `posthogApiKey` | Posthog project API key | No | Only supported in embedded package. In full package the value from config is used. |
| `rageshakeSubmitUrl` | Rageshake server URL endpoint | No | e.g. `https://element.io/bugreports/submit`. In full package the value from config is used. |
| `rageshakeSubmitUrl` | Rageshake server URL endpoint | No | e.g. `https://rageshakes.element.io/api/submit`. In full package the value from config is used. |
| `sentryDsn` | Sentry [DSN](https://docs.sentry.io/concepts/key-terms/dsn-explainer/) | No | In full package the value from config is used. |
| `sentryEnvironment` | Sentry [environment](https://docs.sentry.io/concepts/key-terms/key-terms/) | No | In full package the value from config is used. |

View File

@@ -2,11 +2,11 @@
# https://docs.gradle.org/current/userguide/platforms.html#sub::toml-dependencies-format
[versions]
android_gradle_plugin = "8.11.1"
android_gradle_plugin = "8.13.1"
[libraries]
android_gradle_plugin = { module = "com.android.tools.build:gradle", version.ref = "android_gradle_plugin" }
[plugins]
android_library = { id = "com.android.library", version.ref = "android_gradle_plugin" }
maven_publish = { id = "com.vanniktech.maven.publish", version = "0.34.0" }
maven_publish = { id = "com.vanniktech.maven.publish", version = "0.35.0" }

View File

@@ -64,6 +64,14 @@
"developer_mode": {
"always_show_iphone_earpiece": "Show iPhone earpiece option on all platforms",
"crypto_version": "Crypto version: {{version}}",
"custom_livekit_url": {
"current_url": "Currently set to: ",
"from_config": "Currently, no overwrite is set. Url from well-known or config is used.",
"label": "Custom Livekit-url",
"reset": "Reset overwrite",
"save": "Save",
"saving": "Saving..."
},
"debug_tile_layout_label": "Debug tile layout",
"device_id": "Device ID: {{id}}",
"duplicate_tiles_label": "Number of additional tile copies per participant",
@@ -100,11 +108,14 @@
"connection_lost_description": "You were disconnected from the call.",
"e2ee_unsupported": "Incompatible browser",
"e2ee_unsupported_description": "Your web browser does not support encrypted calls. Supported browsers include Chrome, Safari, and Firefox 117+.",
"failed_to_start_livekit": "Failed to start Livekit connection",
"generic": "Something went wrong",
"generic_description": "Submitting debug logs will help us track down the problem.",
"insufficient_capacity": "Insufficient capacity",
"insufficient_capacity_description": "The server has reached its maximum capacity and you cannot join the call at this time. Try again later, or contact your server admin if the problem persists.",
"matrix_rtc_transport_missing": "The server is not configured to work with {{brand}}. Please contact your server admin (Domain: {{domain}}, Error Code: {{ errorCode }}).",
"membership_manager": "Membership Manager Error",
"membership_manager_description": "The Membership Manager had to shut down. This is caused by many consequtive failed network requests.",
"open_elsewhere": "Opened in another tab",
"open_elsewhere_description": "{{brand}} has been opened in another tab. If that doesn't sound right, try reloading the page.",
"room_creation_restricted": "Failed to create call",

View File

@@ -54,7 +54,7 @@
"@opentelemetry/sdk-trace-base": "^2.0.0",
"@opentelemetry/sdk-trace-web": "^2.0.0",
"@opentelemetry/semantic-conventions": "^1.25.1",
"@playwright/test": "^1.56.1",
"@playwright/test": "^1.57.0",
"@radix-ui/react-dialog": "^1.0.4",
"@radix-ui/react-slider": "^1.1.2",
"@radix-ui/react-visually-hidden": "^1.0.3",
@@ -71,7 +71,7 @@
"@types/grecaptcha": "^3.0.9",
"@types/jsdom": "^21.1.7",
"@types/lodash-es": "^4.17.12",
"@types/node": "^22.0.0",
"@types/node": "^24.0.0",
"@types/pako": "^2.0.3",
"@types/qrcode": "^1.5.5",
"@types/react": "^19.0.0",
@@ -109,8 +109,8 @@
"livekit-client": "^2.13.0",
"lodash-es": "^4.17.21",
"loglevel": "^1.9.1",
"matrix-js-sdk": "github:matrix-org/matrix-js-sdk#head=toger5/sticky-events&commit=e7f5bec51b6f70501a025b79fe5021c933385b21",
"matrix-widget-api": "^1.13.0",
"matrix-js-sdk": "^39.2.0",
"matrix-widget-api": "^1.14.0",
"normalize.css": "^8.0.1",
"observable-hooks": "^4.2.3",
"pako": "^2.0.4",

View File

@@ -38,6 +38,7 @@ export default defineConfig({
projects: [
{
name: "chromium",
testIgnore: "**/mobile/**",
use: {
...devices["Desktop Chrome"],
permissions: [
@@ -56,9 +57,9 @@ export default defineConfig({
},
},
},
{
name: "firefox",
testIgnore: "**/mobile/**",
use: {
...devices["Desktop Firefox"],
ignoreHTTPSErrors: true,
@@ -70,6 +71,27 @@ export default defineConfig({
},
},
},
{
name: "mobile",
testMatch: "**/mobile/**",
use: {
...devices["Pixel 7"],
ignoreHTTPSErrors: true,
permissions: [
"clipboard-write",
"clipboard-read",
"microphone",
"camera",
],
launchOptions: {
args: [
"--use-fake-ui-for-media-stream",
"--use-fake-device-for-media-stream",
"--mute-audio",
],
},
},
},
// No safari for now, until I find a solution to fix `Not allowed to request resource` due to calling
// clear http to the homeserver

View File

@@ -75,7 +75,12 @@ test("Should automatically retry non fatal JWT errors", async ({
test("Should show error screen if call creation is restricted", async ({
page,
browserName,
}) => {
test.skip(
browserName === "firefox",
"The is test is not working on firefox CI environment.",
);
await page.goto("/");
// We need the socket connection to fail, but this cannot be done by using the websocket route.

View File

@@ -0,0 +1,73 @@
/*
Copyright 2025 New Vector Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { type Browser, type Page, test, expect } from "@playwright/test";
export interface MobileCreateFixtures {
asMobile: {
creatorPage: Page;
inviteLink: string;
};
}
export const mobileTest = test.extend<MobileCreateFixtures>({
asMobile: async ({ browser }, pUse) => {
const fixtures = await createCallAndInvite(browser);
await pUse({
creatorPage: fixtures.page,
inviteLink: fixtures.inviteLink,
});
},
});
/**
* Create a call and generate an invite link
*/
async function createCallAndInvite(
browser: Browser,
): Promise<{ page: Page; inviteLink: string }> {
const creatorContext = await browser.newContext({ reducedMotion: "reduce" });
const creatorPage = await creatorContext.newPage();
await creatorPage.goto("/");
// ========
// ARRANGE: The first user creates a call as guest, join it, then click the invite button to copy the invite link
// ========
await creatorPage.getByTestId("home_callName").click();
await creatorPage.getByTestId("home_callName").fill("Welcome");
await creatorPage.getByTestId("home_displayName").click();
await creatorPage.getByTestId("home_displayName").fill("Inviter");
await creatorPage.getByTestId("home_go").click();
await expect(creatorPage.locator("video")).toBeVisible();
await creatorPage
.getByRole("button", { name: "Continue in browser" })
.click();
// join
await creatorPage.getByTestId("lobby_joinCall").click();
// Get the invite link
await creatorPage.getByRole("button", { name: "Invite" }).click();
await expect(
creatorPage.getByRole("heading", { name: "Invite to this call" }),
).toBeVisible();
await expect(creatorPage.getByRole("img", { name: "QR Code" })).toBeVisible();
await expect(creatorPage.getByTestId("modal_inviteLink")).toBeVisible();
await expect(creatorPage.getByTestId("modal_inviteLink")).toBeVisible();
await creatorPage.getByTestId("modal_inviteLink").click();
const inviteLink = (await creatorPage.evaluate(
"navigator.clipboard.readText()",
)) as string;
expect(inviteLink).toContain("room/#/");
return {
page: creatorPage,
inviteLink,
};
}

View File

@@ -111,19 +111,27 @@ async function registerUser(
await page.getByRole("textbox", { name: "Confirm password" }).click();
await page.getByRole("textbox", { name: "Confirm password" }).fill(PASSWORD);
await page.getByRole("button", { name: "Register" }).click();
const continueButton = page.getByRole("button", { name: "Continue" });
try {
await expect(continueButton).toBeVisible({ timeout: 5000 });
await page
.getByRole("textbox", { name: "Password", exact: true })
.fill(PASSWORD);
await continueButton.click();
} catch {
// continueButton not visible, continue as normal
}
await expect(
page.getByRole("heading", { name: `Welcome ${username}` }),
).toBeVisible();
const browserUnsupportedToast = page
.getByText("Element does not support this browser")
.locator("..")
.locator("..");
// Dismiss incompatible browser toast
const dismissButton = browserUnsupportedToast.getByRole("button", {
name: "Dismiss",
});
try {
await expect(dismissButton).toBeVisible({ timeout: 700 });
await dismissButton.click();
} catch {
// dismissButton not visible, continue as normal
}
await setDevToolElementCallDevUrl(page);
const clientHandle = await page.evaluateHandle(() =>
@@ -159,7 +167,11 @@ export const widgetTest = test.extend<MyFixtures>({
} = await registerUser(browser, userB);
// Invite the second user
await ewPage1.getByRole("button", { name: "Add", exact: true }).click();
await ewPage1
.getByRole("navigation", { name: "Room list" })
.getByRole("button", { name: "New conversation" })
.click();
await ewPage1.getByRole("menuitem", { name: "New Room" }).click();
await ewPage1.getByRole("textbox", { name: "Name" }).fill("Welcome Room");
await ewPage1.getByRole("button", { name: "Create room" }).click();

View File

@@ -0,0 +1,115 @@
/*
Copyright 2025 Element Creations Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { expect, test } from "@playwright/test";
import { mobileTest } from "../fixtures/fixture-mobile-create.ts";
test("@mobile Start a new call then leave and show the feedback screen", async ({
page,
}) => {
await page.goto("/");
await page.getByTestId("home_callName").click();
await page.getByTestId("home_callName").fill("HelloCall");
await page.getByTestId("home_displayName").click();
await page.getByTestId("home_displayName").fill("John Doe");
await page.getByTestId("home_go").click();
// await page.pause();
await expect(page.locator("video")).toBeVisible();
await expect(page.getByTestId("lobby_joinCall")).toBeVisible();
await page.getByRole("button", { name: "Continue in browser" }).click();
// Join the call
await page.getByTestId("lobby_joinCall").click();
// Ensure that the call is connected
await page
.locator("div")
.filter({ hasText: /^HelloCall$/ })
.click();
// Check the number of participants
await expect(page.locator("div").filter({ hasText: /^1$/ })).toBeVisible();
// The tooltip with the name should be visible
await expect(page.getByTestId("name_tag")).toContainText("John Doe");
// leave the call
await page.getByTestId("incall_leave").click();
await expect(page.getByRole("heading")).toContainText(
"John Doe, your call has ended. How did it go?",
);
await expect(page.getByRole("main")).toContainText(
"Why not finish by setting up a password to keep your account?",
);
await expect(
page.getByRole("link", { name: "Not now, return to home screen" }),
).toBeVisible();
});
mobileTest(
"Test earpiece overlay in controlledAudioDevices mode",
async ({ asMobile, browser }) => {
test.slow(); // Triples the timeout
const { creatorPage, inviteLink } = asMobile;
// ========
// ACT: The other user use the invite link to join the call as a guest
// ========
const guestInviteeContext = await browser.newContext({
reducedMotion: "reduce",
});
const guestPage = await guestInviteeContext.newPage();
await guestPage.goto(inviteLink + "&controlledAudioDevices=true");
await guestPage
.getByRole("button", { name: "Continue in browser" })
.click();
await guestPage.getByTestId("joincall_displayName").fill("Invitee");
await expect(guestPage.getByTestId("joincall_joincall")).toBeVisible();
await guestPage.getByTestId("joincall_joincall").click();
await guestPage.getByTestId("lobby_joinCall").click();
// ========
// ASSERT: check that there are two members in the call
// ========
// There should be two participants now
await expect(
guestPage.getByTestId("roomHeader_participants_count"),
).toContainText("2");
expect(await guestPage.getByTestId("videoTile").count()).toBe(2);
// Same in creator page
await expect(
creatorPage.getByTestId("roomHeader_participants_count"),
).toContainText("2");
expect(await creatorPage.getByTestId("videoTile").count()).toBe(2);
// TEST: control audio devices from the invitee page
await guestPage.evaluate(() => {
window.controls.setAvailableAudioDevices([
{ id: "speaker", name: "Speaker", isSpeaker: true },
{ id: "earpiece", name: "Handset", isEarpiece: true },
{ id: "headphones", name: "Headphones" },
]);
window.controls.setAudioDevice("earpiece");
});
await expect(
guestPage.getByRole("heading", { name: "Handset Mode" }),
).toBeVisible();
await expect(
guestPage.getByRole("button", { name: "Back to Speaker Mode" }),
).toBeVisible();
// Should auto-mute the video when earpiece is selected
await expect(guestPage.getByTestId("incall_videomute")).toBeDisabled();
},
);

View File

@@ -332,6 +332,42 @@ describe("UrlParams", () => {
expect(computeUrlParams("?intent=join_existing").skipLobby).toBe(false);
});
});
describe("noiseSuppression", () => {
it("defaults to true", () => {
expect(computeUrlParams().noiseSuppression).toBe(true);
});
it("is parsed", () => {
expect(
computeUrlParams("?intent=start_call&noiseSuppression=true")
.noiseSuppression,
).toBe(true);
expect(
computeUrlParams("?intent=start_call&noiseSuppression&bar=foo")
.noiseSuppression,
).toBe(true);
expect(computeUrlParams("?noiseSuppression=false").noiseSuppression).toBe(
false,
);
});
});
describe("echoCancellation", () => {
it("defaults to true", () => {
expect(computeUrlParams().echoCancellation).toBe(true);
});
it("is parsed", () => {
expect(computeUrlParams("?echoCancellation=true").echoCancellation).toBe(
true,
);
expect(computeUrlParams("?echoCancellation=false").echoCancellation).toBe(
false,
);
});
});
describe("header", () => {
it("uses header if provided", () => {
expect(computeUrlParams("?header=app_bar&hideHeader=true").header).toBe(

View File

@@ -233,6 +233,17 @@ export interface UrlConfiguration {
*/
waitForCallPickup: boolean;
/**
* Whether to enable echo cancellation for audio capture.
* Defaults to true.
*/
echoCancellation?: boolean;
/**
* Whether to enable noise suppression for audio capture.
* Defaults to true.
*/
noiseSuppression?: boolean;
callIntent?: RTCCallIntent;
}
interface IntentAndPlatformDerivedConfiguration {
@@ -525,6 +536,8 @@ export const computeUrlParams = (search = "", hash = ""): UrlParams => {
]),
waitForCallPickup: parser.getFlag("waitForCallPickup"),
autoLeaveWhenOthersLeft: parser.getFlag("autoLeave"),
noiseSuppression: parser.getFlagParam("noiseSuppression", true),
echoCancellation: parser.getFlagParam("echoCancellation", true),
};
// Log the final configuration for debugging purposes.

View File

@@ -247,9 +247,8 @@ export class PosthogAnalytics {
// wins, and the first writer will send tracking with an ID that doesn't match the one on the server
// until the next time account data is refreshed and this function is called (most likely on next
// page load). This will happen pretty infrequently, so we can tolerate the possibility.
const accountDataAnalyticsId = analyticsIdGenerator();
await this.setAccountAnalyticsId(accountDataAnalyticsId);
analyticsID = await this.hashedEcAnalyticsId(accountDataAnalyticsId);
analyticsID = analyticsIdGenerator();
await this.setAccountAnalyticsId(analyticsID);
}
} catch (e) {
// The above could fail due to network requests, but not essential to starting the application,
@@ -270,37 +269,14 @@ export class PosthogAnalytics {
private async getAnalyticsId(): Promise<string | null> {
const client: MatrixClient = window.matrixclient;
let accountAnalyticsId: string | null;
if (widget) {
accountAnalyticsId = getUrlParams().posthogUserId;
return getUrlParams().posthogUserId;
} else {
const accountData = await client.getAccountDataFromServer(
PosthogAnalytics.ANALYTICS_EVENT_TYPE,
);
accountAnalyticsId = accountData?.id ?? null;
return accountData?.id ?? null;
}
if (accountAnalyticsId) {
// we dont just use the element web analytics ID because that would allow to associate
// users between the two posthog instances. By using a hash from the username and the element web analytics id
// it is not possible to conclude the element web posthog user id from the element call user id and vice versa.
return await this.hashedEcAnalyticsId(accountAnalyticsId);
}
return null;
}
private async hashedEcAnalyticsId(
accountAnalyticsId: string,
): Promise<string> {
const client: MatrixClient = window.matrixclient;
const posthogIdMaterial = "ec" + accountAnalyticsId + client.getUserId();
const bufferForPosthogId = await crypto.subtle.digest(
"sha-256",
new TextEncoder().encode(posthogIdMaterial),
);
const view = new Int32Array(bufferForPosthogId);
return Array.from(view)
.map((b) => Math.abs(b).toString(16).padStart(2, "0"))
.join("");
}
private async setAccountAnalyticsId(analyticsID: string): Promise<void> {

View File

@@ -122,7 +122,7 @@ export interface ConfigOptions {
delayed_leave_event_delay_ms?: number;
/**
* The time (in milliseconds) after which a we consider a delayed event restart http request to have failed.
* The time (in milliseconds) after which we consider a delayed event restart http request to have failed.
* Setting this to a lower value will result in more frequent retries but also a higher chance of failiour.
*
* In the presence of network packet loss (hurting TCP connections), the custom delayedEventRestartLocalTimeoutMs

View File

@@ -21,7 +21,16 @@ export type OpenIDClientParts = Pick<
MatrixClient,
"getOpenIdToken" | "getDeviceId"
>;
/**
* Gets a bearer token from the homeserver and then use it to authenticate
* to the matrix RTC backend in order to get acces to the SFU.
* It has built-in retry for calls to the homeserver with a backoff policy.
* @param client
* @param serviceUrl
* @param matrixRoomId
* @returns Object containing the token information
* @throws FailToGetOpenIdToken
*/
export async function getSFUConfigWithOpenID(
client: OpenIDClientParts,
serviceUrl: string,

View File

@@ -5,7 +5,7 @@ SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { describe, expect, test } from "vitest";
import { beforeEach, describe, expect, test, vi } from "vitest";
import { render, configure } from "@testing-library/react";
import { RaisedHandIndicator } from "./RaisedHandIndicator";
@@ -15,6 +15,13 @@ configure({
});
describe("RaisedHandIndicator", () => {
const fixedTime = new Date("2025-01-01T12:00:00.000Z");
beforeEach(() => {
vi.useFakeTimers();
vi.setSystemTime(fixedTime);
});
test("renders nothing when no hand has been raised", () => {
const { container } = render(<RaisedHandIndicator />);
expect(container.firstChild).toBeNull();

View File

@@ -15,7 +15,7 @@ exports[`RaisedHandIndicator > renders a smaller indicator when miniature is spe
</span>
</div>
<p>
00:01
00:00
</p>
</div>
`;
@@ -35,7 +35,7 @@ exports[`RaisedHandIndicator > renders an indicator when a hand has been raised
</span>
</div>
<p>
00:01
00:00
</p>
</div>
`;
@@ -55,7 +55,7 @@ exports[`RaisedHandIndicator > renders an indicator when a hand has been raised
</span>
</div>
<p>
01:01
01:00
</p>
</div>
`;

View File

@@ -7,34 +7,17 @@
align-items: center;
justify-content: center;
gap: var(--cpd-space-2x);
}
@keyframes fade-in {
from {
opacity: 0;
}
to {
opacity: 1;
}
transition: opacity 200ms;
}
.overlay[data-show="true"] {
animation: fade-in 200ms;
}
@keyframes fade-out {
from {
opacity: 1;
}
to {
opacity: 0;
display: none;
}
opacity: 1;
}
.overlay[data-show="false"] {
animation: fade-out 130ms forwards;
opacity: 0;
pointer-events: none;
transition-duration: 130ms;
}
.overlay::before {

View File

@@ -22,6 +22,7 @@ import {
WebBrowserIcon,
} from "@vector-im/compound-design-tokens/assets/web/icons";
import { Button } from "@vector-im/compound-web";
import { logger } from "matrix-js-sdk/lib/logger";
import {
ConnectionLostError,
@@ -53,7 +54,7 @@ const ErrorPage: FC<ErrorPageProps> = ({
widget,
}: ErrorPageProps): ReactElement => {
const { t } = useTranslation();
logger.error("Error boundary caught:", error);
let icon: ComponentType<SVGAttributes<SVGElement>>;
switch (error.category) {
case ErrorCategory.CONFIGURATION_ISSUE:

View File

@@ -29,6 +29,7 @@ import userEvent from "@testing-library/user-event";
import { type RelationsContainer } from "matrix-js-sdk/lib/models/relations-container";
import { useState } from "react";
import { TooltipProvider } from "@vector-im/compound-web";
import { type ITransport } from "matrix-widget-api";
import { prefetchSounds } from "../soundUtils";
import { useAudioContext } from "../useAudioContext";
@@ -43,7 +44,7 @@ import {
MockRTCSession,
} from "../utils/test";
import { GroupCallView } from "./GroupCallView";
import { type WidgetHelpers } from "../widget";
import { ElementWidgetActions, type WidgetHelpers } from "../widget";
import { LazyEventEmitter } from "../LazyEventEmitter";
import { MatrixRTCTransportMissingError } from "../utils/errors";
import { ProcessorProvider } from "../livekit/TrackProcessorContext";
@@ -112,6 +113,10 @@ beforeEach(() => {
return (
<div>
<button onClick={() => onLeave("user")}>Leave</button>
<button onClick={() => onLeave("allOthersLeft")}>
SimulateOtherLeft
</button>
<button onClick={() => onLeave("error")}>SimulateErrorLeft</button>
</div>
);
},
@@ -243,6 +248,112 @@ test.skip("GroupCallView plays a leave sound synchronously in widget mode", asyn
expect(leaveRTCSession).toHaveBeenCalledOnce();
});
test.skip("Should close widget when all other left and have time to play a sound", async () => {
const user = userEvent.setup();
const widgetClosedCalled = Promise.withResolvers<void>();
const widgetSendMock = vi.fn().mockImplementation((action: string) => {
if (action === ElementWidgetActions.Close) {
widgetClosedCalled.resolve();
}
});
const widgetStopMock = vi.fn().mockResolvedValue(undefined);
const widget = {
api: {
setAlwaysOnScreen: vi.fn().mockResolvedValue(true),
transport: {
send: widgetSendMock,
reply: vi.fn().mockResolvedValue(undefined),
stop: widgetStopMock,
} as unknown as ITransport,
} as Partial<WidgetHelpers["api"]>,
lazyActions: new LazyEventEmitter(),
};
const resolvePlaySound = Promise.withResolvers<void>();
playSound = vi.fn().mockReturnValue(resolvePlaySound);
(useAudioContext as MockedFunction<typeof useAudioContext>).mockReturnValue({
playSound,
playSoundLooping: vitest.fn(),
soundDuration: {},
});
const { getByText } = createGroupCallView(widget as WidgetHelpers);
const leaveButton = getByText("SimulateOtherLeft");
await user.click(leaveButton);
await flushPromises();
expect(widgetSendMock).not.toHaveBeenCalled();
resolvePlaySound.resolve();
await flushPromises();
expect(playSound).toHaveBeenCalledWith("left");
await widgetClosedCalled.promise;
await flushPromises();
expect(widgetStopMock).toHaveBeenCalledOnce();
});
test("Should close widget when all other left", async () => {
const user = userEvent.setup();
const widgetClosedCalled = Promise.withResolvers<void>();
const widgetSendMock = vi.fn().mockImplementation((action: string) => {
if (action === ElementWidgetActions.Close) {
widgetClosedCalled.resolve();
}
});
const widgetStopMock = vi.fn().mockResolvedValue(undefined);
const widget = {
api: {
setAlwaysOnScreen: vi.fn().mockResolvedValue(true),
transport: {
send: widgetSendMock,
reply: vi.fn().mockResolvedValue(undefined),
stop: widgetStopMock,
} as unknown as ITransport,
} as Partial<WidgetHelpers["api"]>,
lazyActions: new LazyEventEmitter(),
};
const { getByText } = createGroupCallView(widget as WidgetHelpers);
const leaveButton = getByText("SimulateOtherLeft");
await user.click(leaveButton);
await flushPromises();
await widgetClosedCalled.promise;
await flushPromises();
expect(widgetStopMock).toHaveBeenCalledOnce();
});
test("Should not close widget when auto leave due to error", async () => {
const user = userEvent.setup();
const widgetStopMock = vi.fn().mockResolvedValue(undefined);
const widgetSendMock = vi.fn().mockResolvedValue(undefined);
const widget = {
api: {
setAlwaysOnScreen: vi.fn().mockResolvedValue(true),
transport: {
send: widgetSendMock,
reply: vi.fn().mockResolvedValue(undefined),
stop: widgetStopMock,
} as unknown as ITransport,
} as Partial<WidgetHelpers["api"]>,
lazyActions: new LazyEventEmitter(),
};
const alwaysOnScreenSpy = vi.spyOn(widget.api, "setAlwaysOnScreen");
const { getByText } = createGroupCallView(widget as WidgetHelpers);
const leaveButton = getByText("SimulateErrorLeft");
await user.click(leaveButton);
await flushPromises();
// When onLeft is called, we first set always on screen to false
await waitFor(() => expect(alwaysOnScreenSpy).toHaveBeenCalledWith(false));
await flushPromises();
// But then we do not close the widget automatically
expect(widgetStopMock).not.toHaveBeenCalledOnce();
expect(widgetSendMock).not.toHaveBeenCalledOnce();
});
test.skip("GroupCallView leaves the session when an error occurs", async () => {
(ActiveCall as MockedFunction<typeof ActiveCall>).mockImplementation(() => {
const [error, setError] = useState<Error | null>(null);

View File

@@ -159,6 +159,8 @@ export const GroupCallView: FC<Props> = ({
};
}, [rtcSession]);
// TODO move this into the callViewModel LocalMembership.ts
// We might actually not need this at all. Since we get into fatalError on those errors already?
useTypedEventEmitter(
rtcSession,
MatrixRTCSessionEvent.MembershipManagerError,
@@ -312,8 +314,11 @@ export const GroupCallView: FC<Props> = ({
const navigate = useNavigate();
// TODO split this into leave and onDisconnect
const onLeft = useCallback(
(reason: "timeout" | "user" | "allOthersLeft" | "decline"): void => {
(
reason: "timeout" | "user" | "allOthersLeft" | "decline" | "error",
): void => {
let playSound: CallEventSounds = "left";
if (reason === "timeout" || reason === "decline") playSound = reason;
@@ -366,7 +371,7 @@ export const GroupCallView: FC<Props> = ({
}
// On a normal user hangup we can shut down and close the widget. But if an
// error occurs we should keep the widget open until the user reads it.
if (reason === "user" && !getUrlParams().returnToLobby) {
if (reason != "error" && !getUrlParams().returnToLobby) {
try {
await widget.api.transport.send(ElementWidgetActions.Close, {});
} catch (e) {
@@ -518,8 +523,7 @@ export const GroupCallView: FC<Props> = ({
}}
onError={
(/**error*/) => {
// TODO this should not be "user". It needs a new case
if (rtcSession.isJoined()) onLeft("user");
if (rtcSession.isJoined()) onLeft("error");
}
}
>

View File

@@ -24,7 +24,7 @@ import { type MatrixRTCSession } from "matrix-js-sdk/lib/matrixrtc";
import classNames from "classnames";
import { BehaviorSubject, map } from "rxjs";
import { useObservable } from "observable-hooks";
import { logger } from "matrix-js-sdk/lib/logger";
import { logger as rootLogger } from "matrix-js-sdk/lib/logger";
import {
VoiceCallSolidIcon,
VolumeOnSolidIcon,
@@ -109,13 +109,17 @@ import { useTrackProcessorObservable$ } from "../livekit/TrackProcessorContext.t
import { type Layout } from "../state/layout-types.ts";
import { ObservableScope } from "../state/ObservableScope.ts";
const logger = rootLogger.getChild("[InCallView]");
const maxTapDurationMs = 400;
export interface ActiveCallProps
extends Omit<InCallViewProps, "vm" | "livekitRoom" | "connState"> {
e2eeSystem: EncryptionSystem;
// TODO refactor those reasons into an enum
onLeft: (reason: "user" | "timeout" | "decline" | "allOthersLeft") => void;
onLeft: (
reason: "user" | "timeout" | "decline" | "allOthersLeft" | "error",
) => void;
}
export const ActiveCall: FC<ActiveCallProps> = (props) => {
@@ -125,6 +129,7 @@ export const ActiveCall: FC<ActiveCallProps> = (props) => {
const mediaDevices = useMediaDevices();
const trackProcessorState$ = useTrackProcessorObservable$();
useEffect(() => {
logger.info("START CALL VIEW SCOPE");
const scope = new ObservableScope();
const reactionsReader = new ReactionsReader(scope, props.rtcSession);
const { autoLeaveWhenOthersLeft, waitForCallPickup, sendNotificationType } =
@@ -144,10 +149,14 @@ export const ActiveCall: FC<ActiveCallProps> = (props) => {
reactionsReader.reactions$,
scope.behavior(trackProcessorState$),
);
// TODO move this somewhere else once we use the callViewModel in the lobby as well!
vm.join();
setVm(vm);
vm.leave$.pipe(scope.bind()).subscribe(props.onLeft);
return (): void => {
logger.info("END CALL VIEW SCOPE");
scope.end();
};
}, [
@@ -264,9 +273,12 @@ export const InCallView: FC<InCallViewProps> = ({
const sharingScreen = useBehavior(vm.sharingScreen$);
const ringOverlay = useBehavior(vm.ringOverlay$);
const fatalCallError = useBehavior(vm.configError$);
const fatalCallError = useBehavior(vm.fatalError$);
// Stop the rendering and throw for the error boundary
if (fatalCallError) throw fatalCallError;
if (fatalCallError) {
logger.debug("fatalCallError stop rendering", fatalCallError);
throw fatalCallError;
}
// We need to set the proper timings on the animation based upon the sound length.
const ringDuration = pickupPhaseAudio?.soundDuration["waiting"] ?? 1;

View File

@@ -79,9 +79,9 @@ export const LobbyView: FC<Props> = ({
waitingForInvite,
}) => {
useEffect(() => {
logger.info("[Lifecycle] GroupCallView Component mounted");
logger.info("[Lifecycle] LobbyView Component mounted");
return (): void => {
logger.info("[Lifecycle] GroupCallView Component unmounted");
logger.info("[Lifecycle] LobbyView Component unmounted");
};
}, []);

View File

@@ -0,0 +1,95 @@
/*
Copyright 2025 New Vector Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { describe, expect, it, vi } from "vitest";
import { render, waitFor } from "@testing-library/react";
import type { MatrixClient } from "matrix-js-sdk";
import type { Room as LivekitRoom } from "livekit-client";
import { DeveloperSettingsTab } from "./DeveloperSettingsTab";
// Mock url params hook to avoid environment-dependent snapshot churn.
vi.mock("../UrlParams", () => ({
useUrlParams: (): { mocked: boolean; answer: number } => ({
mocked: true,
answer: 42,
}),
}));
// Provide a minimal mock of a Livekit Room structure used by the component.
function createMockLivekitRoom(
wsUrl: string,
serverInfo: object,
metadata: string,
): { isLocal: boolean; url: string; room: LivekitRoom } {
const mockRoom = {
serverInfo,
metadata,
engine: { client: { ws: { url: wsUrl } } },
} as unknown as LivekitRoom;
return {
isLocal: true,
url: wsUrl,
room: mockRoom,
};
}
// Minimal MatrixClient mock with only the methods used by the component.
function createMockMatrixClient(): MatrixClient {
return {
doesServerSupportUnstableFeature: vi.fn().mockResolvedValue(true), // ensure stickyEventsSupported eventually becomes true
getCrypto: (): { getVersion: () => string } | undefined => ({
getVersion: () => "crypto-1.0.0",
}),
getUserId: () => "@alice:example.org",
getDeviceId: () => "DEVICE123",
} as unknown as MatrixClient;
}
describe("DeveloperSettingsTab", () => {
it("renders and matches snapshot", async () => {
const client = createMockMatrixClient();
const livekitRooms: {
room: LivekitRoom;
url: string;
isLocal?: boolean;
}[] = [
createMockLivekitRoom(
"wss://local-sfu.example.org",
{ region: "local", version: "1.2.3" },
"local-metadata",
),
{
isLocal: false,
url: "wss://remote-sfu.example.org",
room: {
serverInfo: { region: "remote", version: "4.5.6" },
metadata: "remote-metadata",
engine: { client: { ws: { url: "wss://remote-sfu.example.org" } } },
} as unknown as LivekitRoom,
},
];
const { container } = render(
<DeveloperSettingsTab
client={client}
livekitRooms={livekitRooms}
env={{ MY_MOCK_ENV: 10, ENV: "test" } as unknown as ImportMetaEnv}
/>,
);
// Wait for the async sticky events feature check to resolve so the final UI
// (e.g. enabled Matrix_2_0 radio button) appears deterministically.
await waitFor(() =>
expect(client.doesServerSupportUnstableFeature).toHaveBeenCalled(),
);
expect(container).toMatchSnapshot();
});
});

View File

@@ -11,8 +11,8 @@ import {
useCallback,
useEffect,
useMemo,
useState,
useId,
useState,
} from "react";
import { useTranslation } from "react-i18next";
import {
@@ -21,6 +21,7 @@ import {
} from "matrix-js-sdk";
import { logger } from "matrix-js-sdk/lib/logger";
import {
EditInPlace,
Root as Form,
Heading,
HelpMessage,
@@ -38,6 +39,7 @@ import {
muteAllAudio as muteAllAudioSetting,
alwaysShowIphoneEarpiece as alwaysShowIphoneEarpieceSetting,
matrixRTCMode as matrixRTCModeSetting,
customLivekitUrl as customLivekitUrlSetting,
MatrixRTCMode,
} from "./settings";
import type { Room as LivekitRoom } from "livekit-client";
@@ -47,9 +49,14 @@ import { useUrlParams } from "../UrlParams";
interface Props {
client: MatrixClient;
livekitRooms?: { room: LivekitRoom; url: string; isLocal?: boolean }[];
env: ImportMetaEnv;
}
export const DeveloperSettingsTab: FC<Props> = ({ client, livekitRooms }) => {
export const DeveloperSettingsTab: FC<Props> = ({
client,
livekitRooms,
env,
}) => {
const { t } = useTranslation();
const [duplicateTiles, setDuplicateTiles] = useSetting(duplicateTilesSetting);
const [debugTileLayout, setDebugTileLayout] = useSetting(
@@ -85,6 +92,15 @@ export const DeveloperSettingsTab: FC<Props> = ({ client, livekitRooms }) => {
alwaysShowIphoneEarpieceSetting,
);
const [customLivekitUrl, setCustomLivekitUrl] = useSetting(
customLivekitUrlSetting,
);
const [customLivekitUrlTextBuffer, setCustomLivekitUrlTextBuffer] =
useState(customLivekitUrl);
useEffect(() => {
setCustomLivekitUrlTextBuffer(customLivekitUrl);
}, [customLivekitUrl]);
const [muteAllAudio, setMuteAllAudio] = useSetting(muteAllAudioSetting);
const urlParams = useUrlParams();
@@ -101,7 +117,7 @@ export const DeveloperSettingsTab: FC<Props> = ({ client, livekitRooms }) => {
}, [livekitRooms]);
return (
<Form>
<>
<p>
{t("developer_mode.hostname", {
hostname: window.location.hostname || "unknown",
@@ -200,55 +216,93 @@ export const DeveloperSettingsTab: FC<Props> = ({ client, livekitRooms }) => {
)}
/>{" "}
</FieldRow>
<EditInPlace
onSubmit={(e) => e.preventDefault()}
helpLabel={
customLivekitUrl === null
? t("developer_mode.custom_livekit_url.from_config")
: t("developer_mode.custom_livekit_url.current_url") +
customLivekitUrl
}
label={t("developer_mode.custom_livekit_url.label")}
saveButtonLabel={t("developer_mode.custom_livekit_url.save")}
savingLabel={t("developer_mode.custom_livekit_url.saving")}
cancelButtonLabel={t("developer_mode.custom_livekit_url.reset")}
onSave={useCallback(
(e: React.FormEvent<HTMLFormElement>) => {
setCustomLivekitUrl(
customLivekitUrlTextBuffer === ""
? null
: customLivekitUrlTextBuffer,
);
},
[setCustomLivekitUrl, customLivekitUrlTextBuffer],
)}
value={customLivekitUrlTextBuffer ?? ""}
onChange={useCallback(
(event: ChangeEvent<HTMLInputElement>): void => {
setCustomLivekitUrlTextBuffer(event.target.value);
},
[setCustomLivekitUrlTextBuffer],
)}
onCancel={useCallback(
(e: React.FormEvent<HTMLFormElement>) => {
setCustomLivekitUrl(null);
},
[setCustomLivekitUrl],
)}
/>
<Heading as="h3" type="body" weight="semibold" size="lg">
{t("developer_mode.matrixRTCMode.title")}
</Heading>
<InlineField
name={matrixRTCModeRadioGroup}
control={
<RadioControl
checked={matrixRTCMode === MatrixRTCMode.Legacy}
value={MatrixRTCMode.Legacy}
onChange={onMatrixRTCModeChange}
/>
}
>
<Label>{t("developer_mode.matrixRTCMode.Legacy.label")}</Label>
<HelpMessage>
{t("developer_mode.matrixRTCMode.Legacy.description")}
</HelpMessage>
</InlineField>
<InlineField
name={matrixRTCModeRadioGroup}
control={
<RadioControl
checked={matrixRTCMode === MatrixRTCMode.Compatibil}
value={MatrixRTCMode.Compatibil}
onChange={onMatrixRTCModeChange}
/>
}
>
<Label>{t("developer_mode.matrixRTCMode.Comptibility.label")}</Label>
<HelpMessage>
{t("developer_mode.matrixRTCMode.Comptibility.description")}
</HelpMessage>
</InlineField>
<InlineField
name={matrixRTCModeRadioGroup}
control={
<RadioControl
checked={matrixRTCMode === MatrixRTCMode.Matrix_2_0}
value={MatrixRTCMode.Matrix_2_0}
disabled={!stickyEventsSupported}
onChange={onMatrixRTCModeChange}
/>
}
>
<Label>{t("developer_mode.matrixRTCMode.Matrix_2_0.label")}</Label>
<HelpMessage>
{t("developer_mode.matrixRTCMode.Matrix_2_0.description")}
</HelpMessage>
</InlineField>
<Form>
<InlineField
name={matrixRTCModeRadioGroup}
control={
<RadioControl
checked={matrixRTCMode === MatrixRTCMode.Legacy}
value={MatrixRTCMode.Legacy}
onChange={onMatrixRTCModeChange}
/>
}
>
<Label>{t("developer_mode.matrixRTCMode.Legacy.label")}</Label>
<HelpMessage>
{t("developer_mode.matrixRTCMode.Legacy.description")}
</HelpMessage>
</InlineField>
<InlineField
name={matrixRTCModeRadioGroup}
control={
<RadioControl
checked={matrixRTCMode === MatrixRTCMode.Compatibil}
value={MatrixRTCMode.Compatibil}
onChange={onMatrixRTCModeChange}
/>
}
>
<Label>{t("developer_mode.matrixRTCMode.Comptibility.label")}</Label>
<HelpMessage>
{t("developer_mode.matrixRTCMode.Comptibility.description")}
</HelpMessage>
</InlineField>
<InlineField
name={matrixRTCModeRadioGroup}
control={
<RadioControl
checked={matrixRTCMode === MatrixRTCMode.Matrix_2_0}
value={MatrixRTCMode.Matrix_2_0}
disabled={!stickyEventsSupported}
onChange={onMatrixRTCModeChange}
/>
}
>
<Label>{t("developer_mode.matrixRTCMode.Matrix_2_0.label")}</Label>
<HelpMessage>
{t("developer_mode.matrixRTCMode.Matrix_2_0.description")}
</HelpMessage>
</InlineField>
</Form>
{livekitRooms?.map((livekitRoom) => (
<>
<h3>
@@ -270,9 +324,9 @@ export const DeveloperSettingsTab: FC<Props> = ({ client, livekitRooms }) => {
</>
))}
<p>{t("developer_mode.environment_variables")}</p>
<pre>{JSON.stringify(import.meta.env, null, 2)}</pre>
<pre>{JSON.stringify(env, null, 2)}</pre>
<p>{t("developer_mode.url_params")}</p>
<pre>{JSON.stringify(urlParams, null, 2)}</pre>
</Form>
</>
);
};

View File

@@ -209,7 +209,11 @@ export const SettingsModal: FC<Props> = ({
key: "developer",
name: t("settings.developer_tab_title"),
content: (
<DeveloperSettingsTab client={client} livekitRooms={livekitRooms} />
<DeveloperSettingsTab
env={import.meta.env}
client={client}
livekitRooms={livekitRooms}
/>
),
};

View File

@@ -0,0 +1,411 @@
// Vitest Snapshot v1, https://vitest.dev/guide/snapshot.html
exports[`DeveloperSettingsTab > renders and matches snapshot 1`] = `
<div>
<p>
Hostname: localhost
</p>
<p>
Element Call version: dev
</p>
<p>
Crypto version: crypto-1.0.0
</p>
<p>
Matrix ID: @alice:example.org
</p>
<p>
Device ID: DEVICE123
</p>
<div
class="fieldRow"
>
<div
class="field inputField"
>
<input
aria-describedby="«r1»"
id="duplicateTiles"
min="0"
type="number"
value="0"
/>
<label
for="duplicateTiles"
>
Number of additional tile copies per participant
</label>
</div>
</div>
<div
class="fieldRow"
>
<div
class="field checkboxField"
>
<input
aria-describedby="«r2»"
id="debugTileLayout"
type="checkbox"
/>
<label
for="debugTileLayout"
>
<div
class="checkbox"
>
<svg
fill="none"
height="24"
stroke="#000"
stroke-linecap="round"
stroke-linejoin="round"
stroke-width="2"
viewBox="0 0 24 24"
width="24"
xmlns="http://www.w3.org/2000/svg"
>
<path
d="m20 6-11 11-5-5"
/>
</svg>
</div>
Debug tile layout
</label>
</div>
</div>
<div
class="fieldRow"
>
<div
class="field checkboxField"
>
<input
aria-describedby="«r3»"
id="showConnectionStats"
type="checkbox"
/>
<label
for="showConnectionStats"
>
<div
class="checkbox"
>
<svg
fill="none"
height="24"
stroke="#000"
stroke-linecap="round"
stroke-linejoin="round"
stroke-width="2"
viewBox="0 0 24 24"
width="24"
xmlns="http://www.w3.org/2000/svg"
>
<path
d="m20 6-11 11-5-5"
/>
</svg>
</div>
Show connection statistics
</label>
</div>
</div>
<div
class="fieldRow"
>
<div
class="field checkboxField"
>
<input
aria-describedby="«r4»"
id="muteAllAudio"
type="checkbox"
/>
<label
for="muteAllAudio"
>
<div
class="checkbox"
>
<svg
fill="none"
height="24"
stroke="#000"
stroke-linecap="round"
stroke-linejoin="round"
stroke-width="2"
viewBox="0 0 24 24"
width="24"
xmlns="http://www.w3.org/2000/svg"
>
<path
d="m20 6-11 11-5-5"
/>
</svg>
</div>
Mute all audio (participants, reactions, join sounds)
</label>
</div>
</div>
<div
class="fieldRow"
>
<div
class="field checkboxField"
>
<input
aria-describedby="«r5»"
id="alwaysShowIphoneEarpiece"
type="checkbox"
/>
<label
for="alwaysShowIphoneEarpiece"
>
<div
class="checkbox"
>
<svg
fill="none"
height="24"
stroke="#000"
stroke-linecap="round"
stroke-linejoin="round"
stroke-width="2"
viewBox="0 0 24 24"
width="24"
xmlns="http://www.w3.org/2000/svg"
>
<path
d="m20 6-11 11-5-5"
/>
</svg>
</div>
Show iPhone earpiece option on all platforms
</label>
</div>
</div>
<form
class="_root_19upo_16"
>
<div
class="_field_19upo_26"
>
<label
class="_label_19upo_59"
for="radix-«r6»"
>
Custom Livekit-url
</label>
<div
class="_controls_17lij_8"
>
<input
aria-describedby="radix-«r7»"
class="_control_sqdq4_10"
id="radix-«r6»"
name="input"
title=""
value=""
/>
</div>
<span
class="_message_19upo_85 _help-message_19upo_91"
id="radix-«r7»"
>
Currently, no overwrite is set. Url from well-known or config is used.
</span>
</div>
</form>
<h3
class="_typography_6v6n8_153 _font-body-lg-semibold_6v6n8_74"
>
MatrixRTC mode
</h3>
<form
class="_root_19upo_16"
>
<div
class="_inline-field_19upo_32"
>
<div
class="_inline-field-control_19upo_44"
>
<div
class="_container_1e0uz_10"
>
<input
aria-describedby="radix-«r9» radix-«rb» radix-«rd»"
checked=""
class="_input_1e0uz_18"
id="radix-«r8»"
name="«r0»"
title=""
type="radio"
value="legacy"
/>
<div
class="_ui_1e0uz_19"
/>
</div>
</div>
<div
class="_inline-field-body_19upo_38"
>
<label
class="_label_19upo_59"
for="radix-«r8»"
>
Legacy: state events & oldest membership SFU
</label>
<span
class="_message_19upo_85 _help-message_19upo_91"
id="radix-«r9»"
>
Compatible with old versions of EC that do not support multi SFU
</span>
</div>
</div>
<div
class="_inline-field_19upo_32"
>
<div
class="_inline-field-control_19upo_44"
>
<div
class="_container_1e0uz_10"
>
<input
aria-describedby="radix-«r9» radix-«rb» radix-«rd»"
class="_input_1e0uz_18"
id="radix-«ra»"
name="«r0»"
title=""
type="radio"
value="compatibil"
/>
<div
class="_ui_1e0uz_19"
/>
</div>
</div>
<div
class="_inline-field-body_19upo_38"
>
<label
class="_label_19upo_59"
for="radix-«ra»"
>
Compatibility: state events & multi SFU
</label>
<span
class="_message_19upo_85 _help-message_19upo_91"
id="radix-«rb»"
>
Compatible with homeservers that do not support sticky events (but all other EC clients are v0.17.0 or later)
</span>
</div>
</div>
<div
class="_inline-field_19upo_32"
>
<div
class="_inline-field-control_19upo_44"
>
<div
class="_container_1e0uz_10"
>
<input
aria-describedby="radix-«r9» radix-«rb» radix-«rd»"
class="_input_1e0uz_18"
id="radix-«rc»"
name="«r0»"
title=""
type="radio"
value="matrix_2_0"
/>
<div
class="_ui_1e0uz_19"
/>
</div>
</div>
<div
class="_inline-field-body_19upo_38"
>
<label
class="_label_19upo_59"
for="radix-«rc»"
>
Matrix 2.0: sticky events & multi SFU
</label>
<span
class="_message_19upo_85 _help-message_19upo_91"
id="radix-«rd»"
>
Compatible only with homservers supporting sticky events and all EC clients v0.17.0 or later
</span>
</div>
</div>
</form>
<h3>
LiveKit SFU: wss://local-sfu.example.org
</h3>
<p>
ws-url:
wss://local-sfu.example.org/
</p>
<p>
LiveKit Server Info
(
local
)
</p>
<pre
class="pre"
>
{
"region": "local",
"version": "1.2.3"
}
local-metadata
</pre>
<h3>
LiveKit SFU: wss://remote-sfu.example.org
</h3>
<p>
LiveKit Server Info
(
remote
)
</p>
<pre
class="pre"
>
{
"region": "remote",
"version": "4.5.6"
}
remote-metadata
</pre>
<p>
Environment variables
</p>
<pre>
{
"MY_MOCK_ENV": 10,
"ENV": "test"
}
</pre>
<p>
URL parameters
</p>
<pre>
{
"mocked": true,
"answer": 42
}
</pre>
</div>
`;

View File

@@ -134,3 +134,8 @@ export const matrixRTCMode = new Setting<MatrixRTCMode>(
"matrix-rtc-mode",
MatrixRTCMode.Legacy,
);
export const customLivekitUrl = new Setting<string | null>(
"custom-livekit-url",
null,
);

View File

@@ -16,7 +16,10 @@ import { BehaviorSubject } from "rxjs";
* distinction between Behaviors and Observables, see
* https://monoid.dk/post/behaviors-and-streams-why-both/.
*/
export type Behavior<T> = Omit<BehaviorSubject<T>, "next" | "observers">;
export type Behavior<T> = Omit<
BehaviorSubject<T>,
"next" | "observers" | "error"
>;
/**
* Creates a Behavior which never changes in value.

View File

@@ -1,5 +1,5 @@
/*
Copyright 2025 New Vector Ltd.
Copyright 2025 Element Creations Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.

View File

@@ -1,5 +1,5 @@
/*
Copyright 2025 New Vector Ltd.
Copyright 2025 Element Creations Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.

View File

@@ -1,12 +1,12 @@
/*
Copyright 2025 Element Creations Ltd.
Copyright 2024 New Vector Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { test, vi, onTestFinished, it, describe, expect } from "vitest";
import EventEmitter from "events";
import { test, vi, onTestFinished, it, describe } from "vitest";
import {
BehaviorSubject,
combineLatest,
@@ -18,12 +18,11 @@ import {
of,
switchMap,
} from "rxjs";
import { SyncState, type MatrixClient } from "matrix-js-sdk";
import { SyncState } from "matrix-js-sdk";
import {
ConnectionState,
type LocalTrackPublication,
type RemoteParticipant,
type Room as LivekitRoom,
} from "livekit-client";
import * as ComponentsCore from "@livekit/components-core";
import {
@@ -35,27 +34,18 @@ import {
type LivekitTransport,
} from "matrix-js-sdk/lib/matrixrtc";
import { deepCompare } from "matrix-js-sdk/lib/utils";
import { AutoDiscovery } from "matrix-js-sdk/lib/autodiscovery";
import { createCallViewModel$ } from "./CallViewModel";
import { type Layout } from "../layout-types.ts";
import {
mockLocalParticipant,
mockMatrixRoom,
mockMatrixRoomMember,
mockRemoteParticipant,
withTestScheduler,
mockRtcMembership,
MockRTCSession,
mockMediaDevices,
mockMuteStates,
mockConfig,
testScope,
mockLivekitRoom,
exampleTransport,
} from "../../utils/test.ts";
import { E2eeType } from "../../e2ee/e2eeType.ts";
import type { RaisedHandInfo, ReactionInfo } from "../../reactions/index.ts";
import {
aliceId,
aliceParticipant,
@@ -70,10 +60,6 @@ import {
import { MediaDevices } from "../MediaDevices.ts";
import { getValue } from "../../utils/observable.ts";
import { type Behavior, constant } from "../Behavior.ts";
import {
type ElementCallError,
MatrixRTCTransportMissingError,
} from "../../utils/errors.ts";
import { withCallViewModel } from "./CallViewModelTestUtils.ts";
vi.mock("rxjs", async (importOriginal) => ({
@@ -244,71 +230,6 @@ function mockRingEvent(
const mockLegacyRingEvent = {} as { event_id: string } & ICallNotifyContent;
describe("CallViewModel", () => {
// TODO: Restore this test. It requires makeTransport to not be mocked, unlike
// the rest of the tests in this file… what do we do?
it.skip("test missing RTC config error", async () => {
const rtcMemberships$ = new BehaviorSubject<CallMembership[]>([]);
const emitter = new EventEmitter();
const client = vi.mocked<MatrixClient>({
on: emitter.on.bind(emitter),
off: emitter.off.bind(emitter),
getSyncState: vi.fn().mockReturnValue(SyncState.Syncing),
getUserId: vi.fn().mockReturnValue("@user:localhost"),
getUser: vi.fn().mockReturnValue(null),
getDeviceId: vi.fn().mockReturnValue("DEVICE"),
credentials: {
userId: "@user:localhost",
},
getCrypto: vi.fn().mockReturnValue(undefined),
getDomain: vi.fn().mockReturnValue("example.org"),
} as unknown as MatrixClient);
const matrixRoom = mockMatrixRoom({
roomId: "!myRoomId:example.com",
client,
getMember: vi.fn().mockReturnValue(undefined),
});
const fakeRtcSession = new MockRTCSession(matrixRoom).withMemberships(
rtcMemberships$,
);
mockConfig({});
vi.spyOn(AutoDiscovery, "getRawClientConfig").mockResolvedValue({});
const callVM = createCallViewModel$(
testScope(),
fakeRtcSession.asMockedSession(),
matrixRoom,
mockMediaDevices({}),
mockMuteStates(),
{
encryptionSystem: { kind: E2eeType.PER_PARTICIPANT },
autoLeaveWhenOthersLeft: false,
livekitRoomFactory: (): LivekitRoom =>
mockLivekitRoom({
localParticipant,
disconnect: async () => Promise.resolve(),
setE2EEEnabled: async () => Promise.resolve(),
}),
},
new BehaviorSubject({} as Record<string, RaisedHandInfo>),
new BehaviorSubject({} as Record<string, ReactionInfo>),
constant({ processor: undefined, supported: false }),
);
const failPromise = Promise.withResolvers<ElementCallError>();
callVM.configError$.subscribe((error) => {
if (error) {
failPromise.resolve(error);
}
});
const error = await failPromise.promise;
expect(error).toBeInstanceOf(MatrixRTCTransportMissingError);
});
test("participants are retained during a focus switch", () => {
withTestScheduler(({ behavior, expectObservable }) => {
// Participants disappear on frame 2 and come back on frame 3
@@ -346,7 +267,7 @@ describe("CallViewModel", () => {
});
});
it.skip("screen sharing activates spotlight layout", () => {
test("screen sharing activates spotlight layout", () => {
withTestScheduler(({ behavior, schedule, expectObservable }) => {
// Start with no screen shares, then have Alice and Bob share their screens,
// then return to no screen shares, then have just Alice share for a bit
@@ -581,6 +502,48 @@ describe("CallViewModel", () => {
});
});
test("layout reacts to window size", () => {
withTestScheduler(({ behavior, schedule, expectObservable }) => {
const windowSizeInputMarbles = "abc";
const expectedLayoutMarbles = " abc";
withCallViewModel(
{
remoteParticipants$: constant([aliceParticipant]),
rtcMembers$: constant([localRtcMember, aliceRtcMember]),
windowSize$: behavior(windowSizeInputMarbles, {
a: { width: 300, height: 600 }, // Start very narrow, like a phone
b: { width: 1000, height: 800 }, // Go to normal desktop window size
c: { width: 200, height: 180 }, // Go to PiP size
}),
},
(vm) => {
expectObservable(summarizeLayout$(vm.layout$)).toBe(
expectedLayoutMarbles,
{
a: {
// This is the expected one-on-one layout for a narrow window
type: "spotlight-expanded",
spotlight: [`${aliceId}:0`],
pip: `${localId}:0`,
},
b: {
// In a larger window, expect the normal one-on-one layout
type: "one-on-one",
local: `${localId}:0`,
remote: `${aliceId}:0`,
},
c: {
// In a PiP-sized window, we of course expect a PiP layout
type: "pip",
spotlight: [`${aliceId}:0`],
},
},
);
},
);
});
});
test("spotlight speakers swap places", () => {
withTestScheduler(({ behavior, schedule, expectObservable }) => {
// Go immediately into spotlight mode for the test

View File

@@ -1,4 +1,5 @@
/*
Copyright 2025 Element Creations Ltd.
Copyright 2023, 2024, 2025 New Vector Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
@@ -14,9 +15,9 @@ import {
} from "livekit-client";
import { type Room as MatrixRoom } from "matrix-js-sdk";
import {
catchError,
combineLatest,
distinctUntilChanged,
EMPTY,
filter,
fromEvent,
map,
@@ -27,7 +28,6 @@ import {
pairwise,
race,
scan,
skip,
skipWhile,
startWith,
Subject,
@@ -40,7 +40,10 @@ import {
timer,
} from "rxjs";
import { logger as rootLogger } from "matrix-js-sdk/lib/logger";
import { type MatrixRTCSession } from "matrix-js-sdk/lib/matrixrtc";
import {
type LivekitTransport,
type MatrixRTCSession,
} from "matrix-js-sdk/lib/matrixrtc";
import { type IWidgetApiRequest } from "matrix-widget-api";
import {
@@ -91,12 +94,14 @@ import {
type SpotlightLandscapeLayoutMedia,
type SpotlightPortraitLayoutMedia,
} from "../layout-types.ts";
import { type ElementCallError } from "../../utils/errors.ts";
import { ElementCallError } from "../../utils/errors.ts";
import { type ObservableScope } from "../ObservableScope.ts";
import { createHomeserverConnected$ } from "./localMember/HomeserverConnected.ts";
import {
createLocalMembership$,
type LocalMemberConnectionState,
} from "./localMember/LocalMembership.ts";
enterRTCSession,
TransportState,
} from "./localMember/LocalMember.ts";
import { createLocalTransport$ } from "./localMember/LocalTransport.ts";
import {
createMemberships$,
@@ -119,6 +124,9 @@ import {
createMatrixMemberMetadata$,
createRoomMembers$,
} from "./remoteMembers/MatrixMemberMetadata.ts";
import { Publisher } from "./localMember/Publisher.ts";
import { type Connection } from "./remoteMembers/Connection.ts";
import { createLayoutModeSwitch } from "./LayoutSwitch.ts";
const logger = rootLogger.getChild("[CallViewModel]");
//TODO
@@ -140,6 +148,8 @@ export interface CallViewModelOptions {
livekitRoomFactory?: (options?: RoomOptions) => LivekitRoom;
/** Optional behavior overriding the local connection state, mainly for testing purposes. */
connectionState$?: Behavior<ConnectionState>;
/** Optional behavior overriding the computed window size, mainly for testing purposes. */
windowSize$?: Behavior<{ width: number; height: number }>;
}
// Do not play any sounds if the participant count has exceeded this
@@ -193,7 +203,7 @@ export interface CallViewModel {
hangup: () => void;
// joining
join: () => LocalMemberConnectionState;
join: () => void;
// screen sharing
/**
@@ -229,7 +239,7 @@ export interface CallViewModel {
* This is a fatal error that prevents the call from being created/joined.
* Should render a blocking error screen.
*/
configError$: Behavior<ElementCallError | null>;
fatalError$: Behavior<ElementCallError | null>;
// participants and counts
/**
@@ -336,6 +346,7 @@ export interface CallViewModel {
// DISCUSSION own membership manager ALSO this probably can be simplifis
reconnecting$: Behavior<boolean>;
}
/**
* A view model providing all the application logic needed to show the in-call
* UI (may eventually be expanded to cover the lobby and feedback screens in the
@@ -356,9 +367,9 @@ export function createCallViewModel$(
reactionsSubject$: Observable<Record<string, ReactionInfo>>,
trackProcessorState$: Behavior<ProcessorState>,
): CallViewModel {
const userId = matrixRoom.client.getUserId()!;
const deviceId = matrixRoom.client.getDeviceId()!;
const client = matrixRoom.client;
const userId = client.getUserId()!;
const deviceId = client.getDeviceId()!;
const livekitKeyProvider = getE2eeKeyProvider(
options.encryptionSystem,
matrixRTCSession,
@@ -370,8 +381,8 @@ export function createCallViewModel$(
// For mocking purposes it is recommended to only mock the functions creating those outputs.
// All other fields are just temp computations for the mentioned output.
// The class does not need anything except the values underneath the bar.
// The creation of the values under the bar are all tested independently and testing the callViewModel Should
// not test their cretation. Call view model only needs:
// The creations of the values under the bar are all tested independently and testing the callViewModel Should
// not test their creation. Call view model only needs:
// - memberships$ via createMemberships$
// - localMembership via createLocalMembership$
// - callLifecycle via createCallNotificationLifecycle$
@@ -392,7 +403,7 @@ export function createCallViewModel$(
const localTransport$ = createLocalTransport$({
scope: scope,
memberships$: memberships$,
client: matrixRoom.client,
client,
roomId: matrixRoom.roomId,
useOldestMember$: scope.behavior(
matrixRTCMode.value$.pipe(map((v) => v === MatrixRTCMode.Legacy)),
@@ -400,12 +411,14 @@ export function createCallViewModel$(
});
const connectionFactory = new ECConnectionFactory(
matrixRoom.client,
client,
mediaDevices,
trackProcessorState$,
livekitKeyProvider,
getUrlParams().controlledAudioDevices,
options.livekitRoomFactory,
getUrlParams().echoCancellation,
getUrlParams().noiseSuppression,
);
const connectionManager = createConnectionManager$({
@@ -413,7 +426,18 @@ export function createCallViewModel$(
connectionFactory: connectionFactory,
inputTransports$: scope.behavior(
combineLatest(
[localTransport$, membershipsAndTransports.transports$],
[
localTransport$.pipe(
catchError((e: unknown) => {
logger.info(
"dont pass local transport to createConnectionManager$. localTransport$ threw an error",
e,
);
return of(null);
}),
),
membershipsAndTransports.transports$,
],
(localTransport, transports) => {
const localTransportAsArray = localTransport ? [localTransport] : [];
return transports.mapInner((transports) => [
@@ -437,7 +461,7 @@ export function createCallViewModel$(
matrixRTCMode.value$.pipe(
map((mode) => ({
encryptMedia: livekitKeyProvider !== undefined,
// TODO. This might need to get called again on each cahnge of matrixRTCMode...
// TODO. This might need to get called again on each change of matrixRTCMode...
matrixRTCMode: mode,
})),
),
@@ -445,15 +469,34 @@ export function createCallViewModel$(
const localMembership = createLocalMembership$({
scope: scope,
homeserverConnected: createHomeserverConnected$(
scope,
client,
matrixRTCSession,
),
muteStates: muteStates,
mediaDevices: mediaDevices,
joinMatrixRTC: (transport: LivekitTransport) => {
return enterRTCSession(
matrixRTCSession,
transport,
connectOptions$.value,
);
},
createPublisherFactory: (connection: Connection) => {
return new Publisher(
scope,
connection,
mediaDevices,
muteStates,
trackProcessorState$,
logger.getChild(
"[Publisher" + connection.transport.livekit_service_url + "]",
),
);
},
connectionManager: connectionManager,
matrixRTCSession: matrixRTCSession,
matrixRoom: matrixRoom,
localTransport$: localTransport$,
trackProcessorState$: trackProcessorState$,
widget,
options: connectOptions$,
logger: logger.getChild(`[${Date.now()}]`),
});
@@ -547,27 +590,6 @@ export function createCallViewModel$(
),
);
// CODESMELL?
// This is functionally the same Observable as leave$, except here it's
// hoisted to the top of the class. This enables the cyclic dependency between
// leave$ -> autoLeave$ -> callPickupState$ -> livekitConnectionState$ ->
// localConnection$ -> transports$ -> joined$ -> leave$.
const leaveHoisted$ = new Subject<
"user" | "timeout" | "decline" | "allOthersLeft"
>();
/**
* Whether various media/event sources should pretend to be disconnected from
* all network input, even if their connection still technically works.
*/
// We do this when the app is in the 'reconnecting' state, because it might be
// that the LiveKit connection is still functional while the homeserver is
// down, for example, and we want to avoid making people worry that the app is
// in a split-brained state.
// DISCUSSION own membership manager ALSO this probably can be simplifis
const reconnecting$ = localMembership.reconnecting$;
const pretendToBeDisconnected$ = reconnecting$;
const audioParticipants$ = scope.behavior(
matrixLivekitMembers$.pipe(
switchMap((membersWithEpoch) => {
@@ -615,7 +637,7 @@ export function createCallViewModel$(
);
const handsRaised$ = scope.behavior(
handsRaisedSubject$.pipe(pauseWhen(pretendToBeDisconnected$)),
handsRaisedSubject$.pipe(pauseWhen(localMembership.reconnecting$)),
);
const reactions$ = scope.behavior(
@@ -628,14 +650,13 @@ export function createCallViewModel$(
]),
),
),
pauseWhen(pretendToBeDisconnected$),
pauseWhen(localMembership.reconnecting$),
),
);
/**
* List of user media (camera feeds) that we want tiles for.
*/
// TODO this also needs the local participant to be added.
const userMedia$ = scope.behavior<UserMedia[]>(
combineLatest([
localMatrixLivekitMember$,
@@ -650,7 +671,7 @@ export function createCallViewModel$(
{ value: matrixLivekitMembers },
duplicateTiles,
]) {
let localParticipantId = undefined;
let localParticipantId: string | undefined = undefined;
// add local member if available
if (localMatrixLivekitMember) {
const { userId, participant$, connection$, membership$ } =
@@ -720,7 +741,7 @@ export function createCallViewModel$(
livekitRoom$,
focusUrl$,
mediaDevices,
pretendToBeDisconnected$,
localMembership.reconnecting$,
displayName$,
matrixMemberMetadataStore.createAvatarUrlBehavior$(userId),
handsRaised$.pipe(map((v) => v[participantId]?.time ?? null)),
@@ -816,10 +837,7 @@ export function createCallViewModel$(
merge(
autoLeave$,
merge(userHangup$, widgetHangup$).pipe(map(() => "user" as const)),
).pipe(
scope.share,
tap((reason) => leaveHoisted$.next(reason)),
);
).pipe(scope.share);
const spotlightSpeaker$ = scope.behavior<UserMediaViewModel | null>(
userMedia$.pipe(
@@ -937,11 +955,19 @@ export function createCallViewModel$(
const pipEnabled$ = scope.behavior(setPipEnabled$, false);
const windowSize$ =
options.windowSize$ ??
scope.behavior<{ width: number; height: number }>(
fromEvent(window, "resize").pipe(
startWith(null),
map(() => ({ width: window.innerWidth, height: window.innerHeight })),
),
);
// A guess at what the window's mode should be based on its size and shape.
const naturalWindowMode$ = scope.behavior<WindowMode>(
fromEvent(window, "resize").pipe(
map(() => {
const height = window.innerHeight;
const width = window.innerWidth;
windowSize$.pipe(
map(({ width, height }) => {
if (height <= 400 && width <= 340) return "pip";
// Our layouts for flat windows are better at adapting to a small width
// than our layouts for narrow windows are at adapting to a small height,
@@ -951,7 +977,6 @@ export function createCallViewModel$(
return "normal";
}),
),
"normal",
);
/**
@@ -968,36 +993,11 @@ export function createCallViewModel$(
spotlightExpandedToggle$.pipe(accumulate(false, (expanded) => !expanded)),
);
const gridModeUserSelection$ = new Subject<GridMode>();
/**
* The layout mode of the media tile grid.
*/
const gridMode$ =
// If the user hasn't selected spotlight and somebody starts screen sharing,
// automatically switch to spotlight mode and reset when screen sharing ends
scope.behavior<GridMode>(
gridModeUserSelection$.pipe(
switchMap((userSelection) =>
(userSelection === "spotlight"
? EMPTY
: combineLatest([hasRemoteScreenShares$, windowMode$]).pipe(
skip(userSelection === null ? 0 : 1),
map(
([hasScreenShares, windowMode]): GridMode =>
hasScreenShares || windowMode === "flat"
? "spotlight"
: "grid",
),
)
).pipe(startWith(userSelection ?? "grid")),
),
),
"grid",
);
const setGridMode = (value: GridMode): void => {
gridModeUserSelection$.next(value);
};
const { setGridMode, gridMode$ } = createLayoutModeSwitch(
scope,
windowMode$,
hasRemoteScreenShares$,
);
const gridLayoutMedia$: Observable<GridLayoutMedia> = combineLatest(
[grid$, spotlight$],
@@ -1424,16 +1424,44 @@ export function createCallViewModel$(
// reassigned here to make it publicly accessible
const toggleScreenSharing = localMembership.toggleScreenSharing;
const join = localMembership.requestConnect;
// TODO-MULTI-SFU: Use this view model for the lobby as well, and only call this once 'join' is clicked?
join();
const errors$ = scope.behavior<{
transportError?: ElementCallError;
matrixError?: ElementCallError;
connectionError?: ElementCallError;
publishError?: ElementCallError;
} | null>(
localMembership.localMemberState$.pipe(
map((value) => {
const returnObject: {
transportError?: ElementCallError;
matrixError?: ElementCallError;
connectionError?: ElementCallError;
publishError?: ElementCallError;
} = {};
if (value instanceof ElementCallError) return { transportError: value };
if (value === TransportState.Waiting) return null;
if (value.matrix instanceof ElementCallError)
returnObject.matrixError = value.matrix;
if (value.media instanceof ElementCallError)
returnObject.publishError = value.media;
else if (
typeof value.media === "object" &&
value.media.connection instanceof ElementCallError
)
returnObject.connectionError = value.media.connection;
return returnObject;
}),
),
null,
);
return {
autoLeave$: autoLeave$,
callPickupState$: callPickupState$,
ringOverlay$: ringOverlay$,
leave$: leave$,
hangup: (): void => userHangup$.next(),
join: join,
join: localMembership.requestJoinAndPublish,
toggleScreenSharing: toggleScreenSharing,
sharingScreen$: sharingScreen$,
@@ -1442,7 +1470,22 @@ export function createCallViewModel$(
hoverScreen: (): void => screenHover$.next(),
unhoverScreen: (): void => screenUnhover$.next(),
configError$: localMembership.configError$,
fatalError$: scope.behavior(
errors$.pipe(
map((errors) => {
logger.debug("errors$ to compute any fatal errors:", errors);
return (
errors?.transportError ??
errors?.matrixError ??
errors?.connectionError ??
null
);
}),
filter((error) => error !== null),
),
null,
),
participantCount$: participantCount$,
audioParticipants$: audioParticipants$,
@@ -1471,11 +1514,9 @@ export function createCallViewModel$(
showFooter$: showFooter$,
earpieceMode$: earpieceMode$,
audioOutputSwitcher$: audioOutputSwitcher$,
reconnecting$: reconnecting$,
reconnecting$: localMembership.reconnecting$,
};
}
// TODO-MULTI-SFU // Setup and update the keyProvider which was create by `createRoom` was a thing before. Now we never update if the E2EEsystem changes
// do we need this?
function getE2eeKeyProvider(
e2eeSystem: EncryptionSystem,

View File

@@ -75,6 +75,7 @@ export interface CallViewModelInputs {
speaking: Map<Participant, Observable<boolean>>;
mediaDevices: MediaDevices;
initialSyncState: SyncState;
windowSize$: Behavior<{ width: number; height: number }>;
}
const localParticipant = mockLocalParticipant({ identity: "" });
@@ -89,6 +90,7 @@ export function withCallViewModel(
speaking = new Map(),
mediaDevices = mockMediaDevices({}),
initialSyncState = SyncState.Syncing,
windowSize$ = constant({ width: 1000, height: 800 }),
}: Partial<CallViewModelInputs> = {},
continuation: (
vm: CallViewModel,
@@ -173,6 +175,7 @@ export function withCallViewModel(
setE2EEEnabled: async () => Promise.resolve(),
}),
connectionState$,
windowSize$,
},
raisedHands$,
reactions$,

View File

@@ -0,0 +1,202 @@
/*
Copyright 2025 Element Creations Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { afterEach, beforeEach, describe, expect, test } from "vitest";
import { firstValueFrom, of } from "rxjs";
import { createLayoutModeSwitch } from "./LayoutSwitch";
import { ObservableScope } from "../ObservableScope";
import { constant } from "../Behavior";
import { withTestScheduler } from "../../utils/test";
let scope: ObservableScope;
beforeEach(() => {
scope = new ObservableScope();
});
afterEach(() => {
scope.end();
});
describe("Default mode", () => {
test("Should be in grid layout by default", async () => {
const { gridMode$ } = createLayoutModeSwitch(
scope,
constant("normal"),
of(false),
);
const mode = await firstValueFrom(gridMode$);
expect(mode).toBe("grid");
});
test("Should switch to spotlight mode when window mode is flat", async () => {
const { gridMode$ } = createLayoutModeSwitch(
scope,
constant("flat"),
of(false),
);
const mode = await firstValueFrom(gridMode$);
expect(mode).toBe("spotlight");
});
});
test("Should allow switching modes manually", () => {
withTestScheduler(({ cold, behavior, expectObservable, schedule }): void => {
const { gridMode$, setGridMode } = createLayoutModeSwitch(
scope,
behavior("n", { n: "normal" }),
cold("f", { f: false, t: true }),
);
schedule("--sgs", {
s: () => setGridMode("spotlight"),
g: () => setGridMode("grid"),
});
expectObservable(gridMode$).toBe("g-sgs", {
g: "grid",
s: "spotlight",
});
});
});
test("Should switch to spotlight mode when there is a remote screen share", () => {
withTestScheduler(({ cold, behavior, expectObservable }): void => {
const shareMarble = "f--t";
const gridsMarble = "g--s";
const { gridMode$ } = createLayoutModeSwitch(
scope,
behavior("n", { n: "normal" }),
cold(shareMarble, { f: false, t: true }),
);
expectObservable(gridMode$).toBe(gridsMarble, {
g: "grid",
s: "spotlight",
});
});
});
test("Can manually force grid when there is a screenshare", () => {
withTestScheduler(({ cold, behavior, expectObservable, schedule }): void => {
const { gridMode$, setGridMode } = createLayoutModeSwitch(
scope,
behavior("n", { n: "normal" }),
cold("-ft", { f: false, t: true }),
);
schedule("---g", {
g: () => setGridMode("grid"),
});
expectObservable(gridMode$).toBe("ggsg", {
g: "grid",
s: "spotlight",
});
});
});
test("Should auto-switch after manually selected grid", () => {
withTestScheduler(({ cold, behavior, expectObservable, schedule }): void => {
const { gridMode$, setGridMode } = createLayoutModeSwitch(
scope,
behavior("n", { n: "normal" }),
// Two screenshares will happen in sequence
cold("-ft-ft", { f: false, t: true }),
);
// There was a screen-share that forced spotlight, then
// the user manually switch back to grid
schedule("---g", {
g: () => setGridMode("grid"),
});
// If we did want to respect manual selection, the expectation would be:
// const expectation = "ggsg";
const expectation = "ggsg-s";
expectObservable(gridMode$).toBe(expectation, {
g: "grid",
s: "spotlight",
});
});
});
test("Should switch back to grid mode when the remote screen share ends", () => {
withTestScheduler(({ cold, behavior, expectObservable }): void => {
const shareMarble = "f--t--f-";
const gridsMarble = "g--s--g-";
const { gridMode$ } = createLayoutModeSwitch(
scope,
behavior("n", { n: "normal" }),
cold(shareMarble, { f: false, t: true }),
);
expectObservable(gridMode$).toBe(gridsMarble, {
g: "grid",
s: "spotlight",
});
});
});
test("can auto-switch to spotlight again after first screen share ends", () => {
withTestScheduler(({ cold, behavior, expectObservable }): void => {
const shareMarble = "ftft";
const gridsMarble = "gsgs";
const { gridMode$ } = createLayoutModeSwitch(
scope,
behavior("n", { n: "normal" }),
cold(shareMarble, { f: false, t: true }),
);
expectObservable(gridMode$).toBe(gridsMarble, {
g: "grid",
s: "spotlight",
});
});
});
test("can switch manually to grid after screen share while manually in spotlight", () => {
withTestScheduler(({ cold, behavior, schedule, expectObservable }): void => {
// Initially, no one is sharing. Then the user manually switches to
// spotlight. After a screen share starts, the user manually switches to
// grid.
const shareMarbles = " f-t-";
const setModeMarbles = "-s-g";
const expectation = " gs-g";
const { gridMode$, setGridMode } = createLayoutModeSwitch(
scope,
behavior("n", { n: "normal" }),
cold(shareMarbles, { f: false, t: true }),
);
schedule(setModeMarbles, {
g: () => setGridMode("grid"),
s: () => setGridMode("spotlight"),
});
expectObservable(gridMode$).toBe(expectation, {
g: "grid",
s: "spotlight",
});
});
});
test("Should auto-switch to spotlight when in flat window mode", () => {
withTestScheduler(({ cold, behavior, expectObservable }): void => {
const { gridMode$ } = createLayoutModeSwitch(
scope,
behavior("naf", { n: "normal", a: "narrow", f: "flat" }),
cold("f", { f: false, t: true }),
);
expectObservable(gridMode$).toBe("g-s-", {
g: "grid",
s: "spotlight",
});
});
});

View File

@@ -0,0 +1,130 @@
/*
Copyright 2025 Element Creations Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import {
BehaviorSubject,
combineLatest,
map,
type Observable,
scan,
} from "rxjs";
import { logger } from "matrix-js-sdk/lib/logger";
import { type GridMode, type WindowMode } from "./CallViewModel.ts";
import { type Behavior } from "../Behavior.ts";
import { type ObservableScope } from "../ObservableScope.ts";
/**
* Creates a layout mode switch that allows switching between grid and spotlight modes.
* The actual layout mode can be overridden to spotlight mode if there is a remote screen share active
* or if the window mode is flat.
*
* @param scope - The observable scope to manage subscriptions.
* @param windowMode$ - The current window mode observable.
* @param hasRemoteScreenShares$ - An observable indicating if there are remote screen shares active.
*/
export function createLayoutModeSwitch(
scope: ObservableScope,
windowMode$: Behavior<WindowMode>,
hasRemoteScreenShares$: Observable<boolean>,
): {
gridMode$: Behavior<GridMode>;
setGridMode: (value: GridMode) => void;
} {
const gridModeUserSelection$ = new BehaviorSubject<GridMode>("grid");
// Callback to set the grid mode desired by the user.
// Notice that this is only a preference, the actual grid mode can be overridden
// if there is a remote screen share active.
const setGridMode = (value: GridMode): void => {
gridModeUserSelection$.next(value);
};
/**
* The layout mode of the media tile grid.
*/
const gridMode$ =
// If the user hasn't selected spotlight and somebody starts screen sharing,
// automatically switch to spotlight mode and reset when screen sharing ends
scope.behavior<GridMode>(
combineLatest([
gridModeUserSelection$,
hasRemoteScreenShares$,
windowMode$,
]).pipe(
// Scan to keep track if we have auto-switched already or not.
// To allow the user to override the auto-switch by selecting grid mode again.
scan<
[GridMode, boolean, WindowMode],
{
mode: GridMode;
/** Remember if the change was user driven or not */
hasAutoSwitched: boolean;
/** To know if it is new screen share or an already handled */
hasScreenShares: boolean;
}
>(
(prev, [userSelection, hasScreenShares, windowMode]) => {
const isFlatMode = windowMode === "flat";
// Always force spotlight in flat mode, grid layout is not supported
// in that mode.
// TODO: strange that we do that for flat mode but not for other modes?
// TODO: Why is this not handled in layoutMedia$ like other window modes?
if (isFlatMode) {
logger.debug(`Forcing spotlight mode, windowMode=${windowMode}`);
return {
mode: "spotlight",
hasAutoSwitched: prev.hasAutoSwitched,
hasScreenShares,
};
}
// User explicitly chose spotlight.
// Respect that choice.
if (userSelection === "spotlight") {
return {
mode: "spotlight",
hasAutoSwitched: prev.hasAutoSwitched,
hasScreenShares,
};
}
// User has chosen grid mode. If a screen share starts, we will
// auto-switch to spotlight mode for better experience.
// But we only do it once, if the user switches back to grid mode,
// we respect that choice until they explicitly change it again.
const isNewShare = hasScreenShares && !prev.hasScreenShares;
if (isNewShare && !prev.hasAutoSwitched) {
return {
mode: "spotlight",
hasAutoSwitched: true,
hasScreenShares: true,
};
}
// Respect user's grid choice
// XXX If we want to forbid switching automatically again after we can
// return hasAutoSwitched: acc.hasAutoSwitched here instead of setting to false.
return {
mode: "grid",
hasAutoSwitched: false,
hasScreenShares,
};
},
// initial value
{ mode: "grid", hasAutoSwitched: false, hasScreenShares: false },
),
map(({ mode }) => mode),
),
"grid",
);
return {
gridMode$,
setGridMode,
};
}

View File

@@ -0,0 +1,202 @@
/*
Copyright 2025 Element Creations Ltd.
Copyright 2024 New Vector Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { describe, it, expect, beforeEach, afterEach } from "vitest";
import { EventEmitter } from "events";
import { ClientEvent, SyncState } from "matrix-js-sdk";
import { MembershipManagerEvent, Status } from "matrix-js-sdk/lib/matrixrtc";
import { ObservableScope } from "../../ObservableScope";
import { createHomeserverConnected$ } from "./HomeserverConnected";
/**
* Minimal stub of a Matrix client sufficient for our tests:
```
createHomeserverConnected$(
scope: ObservableScope,
client: NodeStyleEventEmitter & Pick<MatrixClient, "getSyncState">,
matrixRTCSession: NodeStyleEventEmitter &
Pick<MatrixRTCSession, "membershipStatus" | "probablyLeft">,
)
```
*/
class MockMatrixClient extends EventEmitter {
private syncState: SyncState;
public constructor(initial: SyncState) {
super();
this.syncState = initial;
}
public setSyncState(state: SyncState): void {
this.syncState = state;
// Matrix's Sync event in createHomeserverConnected$ expects [SyncState]
this.emit(ClientEvent.Sync, [state]);
}
public getSyncState(): SyncState {
return this.syncState;
}
}
/**
* Minimal stub of MatrixRTCSession (membership manager):
```
createHomeserverConnected$(
scope: ObservableScope,
client: NodeStyleEventEmitter & Pick<MatrixClient, "getSyncState">,
matrixRTCSession: NodeStyleEventEmitter &
Pick<MatrixRTCSession, "membershipStatus" | "probablyLeft">,
)
```
*/
class MockMatrixRTCSession extends EventEmitter {
public membershipStatus: Status;
public probablyLeft: boolean;
public constructor(props: {
membershipStatus: Status;
probablyLeft: boolean;
}) {
super();
this.membershipStatus = props.membershipStatus;
this.probablyLeft = props.probablyLeft;
}
public setMembershipStatus(status: Status): void {
this.membershipStatus = status;
this.emit(MembershipManagerEvent.StatusChanged);
}
public setProbablyLeft(flag: boolean): void {
this.probablyLeft = flag;
this.emit(MembershipManagerEvent.ProbablyLeft);
}
}
describe("createHomeserverConnected$", () => {
let scope: ObservableScope;
let client: MockMatrixClient;
let session: MockMatrixRTCSession;
beforeEach(() => {
scope = new ObservableScope();
client = new MockMatrixClient(SyncState.Error); // start disconnected
session = new MockMatrixRTCSession({
membershipStatus: Status.Disconnected,
probablyLeft: false,
});
});
afterEach(() => {
scope.end();
});
// LLM generated test cases. They are a bit overkill but I improved the mocking so it is
// easy enough to read them so I think they can stay.
it("is false when sync state is not Syncing", () => {
const hsConnected = createHomeserverConnected$(scope, client, session);
expect(hsConnected.combined$.value).toBe(false);
});
it("remains false while membership status is not Connected even if sync is Syncing", () => {
const hsConnected = createHomeserverConnected$(scope, client, session);
client.setSyncState(SyncState.Syncing);
expect(hsConnected.combined$.value).toBe(false); // membership still disconnected
});
it("is false when membership status transitions to Connected but ProbablyLeft is true", () => {
const hsConnected = createHomeserverConnected$(scope, client, session);
// Make sync loop OK
client.setSyncState(SyncState.Syncing);
// Indicate probable leave before connection
session.setProbablyLeft(true);
session.setMembershipStatus(Status.Connected);
expect(hsConnected.combined$.value).toBe(false);
});
it("becomes true only when all three conditions are satisfied", () => {
const hsConnected = createHomeserverConnected$(scope, client, session);
// 1. Sync loop connected
client.setSyncState(SyncState.Syncing);
expect(hsConnected.combined$.value).toBe(false); // not yet membership connected
// 2. Membership connected
session.setMembershipStatus(Status.Connected);
expect(hsConnected.combined$.value).toBe(true); // probablyLeft is false
});
it("drops back to false when sync loop leaves Syncing", () => {
const hsConnected = createHomeserverConnected$(scope, client, session);
// Reach connected state
client.setSyncState(SyncState.Syncing);
session.setMembershipStatus(Status.Connected);
expect(hsConnected.combined$.value).toBe(true);
// Sync loop error => should flip false
client.setSyncState(SyncState.Error);
expect(hsConnected.combined$.value).toBe(false);
});
it("drops back to false when membership status becomes disconnected", () => {
const hsConnected = createHomeserverConnected$(scope, client, session);
client.setSyncState(SyncState.Syncing);
session.setMembershipStatus(Status.Connected);
expect(hsConnected.combined$.value).toBe(true);
session.setMembershipStatus(Status.Disconnected);
expect(hsConnected.combined$.value).toBe(false);
});
it("drops to false when ProbablyLeft is emitted after being true", () => {
const hsConnected = createHomeserverConnected$(scope, client, session);
client.setSyncState(SyncState.Syncing);
session.setMembershipStatus(Status.Connected);
expect(hsConnected.combined$.value).toBe(true);
session.setProbablyLeft(true);
expect(hsConnected.combined$.value).toBe(false);
});
it("recovers to true if ProbablyLeft becomes false again while other conditions remain true", () => {
const hsConnected = createHomeserverConnected$(scope, client, session);
client.setSyncState(SyncState.Syncing);
session.setMembershipStatus(Status.Connected);
expect(hsConnected.combined$.value).toBe(true);
session.setProbablyLeft(true);
expect(hsConnected.combined$.value).toBe(false);
// Simulate clearing the flag (in realistic scenario membership manager would update)
session.setProbablyLeft(false);
expect(hsConnected.combined$.value).toBe(true);
});
it("composite sequence reflects each individual failure reason", () => {
const hsConnected = createHomeserverConnected$(scope, client, session);
// Initially false (sync error + disconnected + not probably left)
expect(hsConnected.combined$.value).toBe(false);
// Fix sync only
client.setSyncState(SyncState.Syncing);
expect(hsConnected.combined$.value).toBe(false);
// Fix membership
session.setMembershipStatus(Status.Connected);
expect(hsConnected.combined$.value).toBe(true);
// Introduce probablyLeft -> false
session.setProbablyLeft(true);
expect(hsConnected.combined$.value).toBe(false);
// Restore notProbablyLeft -> true again
session.setProbablyLeft(false);
expect(hsConnected.combined$.value).toBe(true);
// Drop sync -> false
client.setSyncState(SyncState.Error);
expect(hsConnected.combined$.value).toBe(false);
});
});

View File

@@ -0,0 +1,91 @@
/*
Copyright 2025 Element Creations Ltd.
Copyright 2024 New Vector Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import {
MembershipManagerEvent,
Status,
type MatrixRTCSession,
} from "matrix-js-sdk/lib/matrixrtc";
import { ClientEvent, type MatrixClient, SyncState } from "matrix-js-sdk";
import { fromEvent, startWith, map, tap, type Observable } from "rxjs";
import { logger as rootLogger } from "matrix-js-sdk/lib/logger";
import { type ObservableScope } from "../../ObservableScope";
import { type Behavior } from "../../Behavior";
import { and$ } from "../../../utils/observable";
import { type NodeStyleEventEmitter } from "../../../utils/test";
/**
* Logger instance (scoped child) for homeserver connection updates.
*/
const logger = rootLogger.getChild("[HomeserverConnected]");
export interface HomeserverConnected {
combined$: Behavior<boolean>;
rtsSession$: Behavior<Status>;
}
/**
* Behavior representing whether we consider ourselves connected to the Matrix homeserver
* for the purposes of a MatrixRTC session.
*
* Becomes FALSE if ANY sub-condition is fulfilled:
* 1. Sync loop is not in SyncState.Syncing
* 2. membershipStatus !== Status.Connected
* 3. probablyLeft === true
*/
export function createHomeserverConnected$(
scope: ObservableScope,
client: NodeStyleEventEmitter & Pick<MatrixClient, "getSyncState">,
matrixRTCSession: NodeStyleEventEmitter &
Pick<MatrixRTCSession, "membershipStatus" | "probablyLeft">,
): HomeserverConnected {
const syncing$ = (
fromEvent(client, ClientEvent.Sync) as Observable<[SyncState]>
).pipe(
startWith([client.getSyncState()]),
map(([state]) => state === SyncState.Syncing),
);
const rtsSession$ = scope.behavior<Status>(
fromEvent(matrixRTCSession, MembershipManagerEvent.StatusChanged).pipe(
map(() => matrixRTCSession.membershipStatus ?? Status.Unknown),
),
Status.Unknown,
);
const membershipConnected$ = rtsSession$.pipe(
map((status) => status === Status.Connected),
);
// This is basically notProbablyLeft$
//
// probablyLeft is computed by a local timer that mimics the server delayed event.
// If we locally predict our server event timed out. We consider ourselves as probablyLeft
// even though we might not yet have received the delayed event leave.
//
// If that is not the case we certainly still have a valid membership on the matrix network
// independet if the sync currently works.
const certainlyConnected$ = fromEvent(
matrixRTCSession,
MembershipManagerEvent.ProbablyLeft,
).pipe(
startWith(null),
map(() => matrixRTCSession.probablyLeft !== true),
);
const combined$ = scope.behavior(
and$(syncing$, membershipConnected$, certainlyConnected$).pipe(
tap((connected) => {
logger.info(`Homeserver connected update: ${connected}`);
}),
),
);
return { combined$, rtsSession$ };
}

View File

@@ -0,0 +1,527 @@
/*
Copyright 2025 Element Creations Ltd.
Copyright 2024 New Vector Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import {
Status as RTCMemberStatus,
type LivekitTransport,
type MatrixRTCSession,
} from "matrix-js-sdk/lib/matrixrtc";
import { describe, expect, it, vi } from "vitest";
import { AutoDiscovery } from "matrix-js-sdk/lib/autodiscovery";
import { BehaviorSubject, map, of } from "rxjs";
import { logger } from "matrix-js-sdk/lib/logger";
import { type LocalParticipant, type LocalTrack } from "livekit-client";
import { MatrixRTCMode } from "../../../settings/settings";
import {
flushPromises,
mockConfig,
mockLivekitRoom,
mockMuteStates,
withTestScheduler,
} from "../../../utils/test";
import {
TransportState,
createLocalMembership$,
enterRTCSession,
PublishState,
TrackState,
} from "./LocalMember";
import { MatrixRTCTransportMissingError } from "../../../utils/errors";
import { Epoch, ObservableScope } from "../../ObservableScope";
import { constant } from "../../Behavior";
import { ConnectionManagerData } from "../remoteMembers/ConnectionManager";
import { ConnectionState, type Connection } from "../remoteMembers/Connection";
import { type Publisher } from "./Publisher";
const MATRIX_RTC_MODE = MatrixRTCMode.Legacy;
const getUrlParams = vi.hoisted(() => vi.fn(() => ({})));
vi.mock("../../../UrlParams", () => ({ getUrlParams }));
vi.mock("@livekit/components-core", () => ({
observeParticipantEvents: vi
.fn()
.mockReturnValue(of({ isScreenShareEnabled: false })),
}));
describe("LocalMembership", () => {
describe("enterRTCSession", () => {
it("It joins the correct Session", () => {
const focusFromOlderMembership = {
type: "livekit",
livekit_service_url: "http://my-oldest-member-service-url.com",
livekit_alias: "my-oldest-member-service-alias",
};
const focusConfigFromWellKnown = {
type: "livekit",
livekit_service_url: "http://my-well-known-service-url.com",
};
const focusConfigFromWellKnown2 = {
type: "livekit",
livekit_service_url: "http://my-well-known-service-url2.com",
};
const clientWellKnown = {
"org.matrix.msc4143.rtc_foci": [
focusConfigFromWellKnown,
focusConfigFromWellKnown2,
],
};
mockConfig({
livekit: { livekit_service_url: "http://my-default-service-url.com" },
});
vi.spyOn(AutoDiscovery, "getRawClientConfig").mockImplementation(
async (domain) => {
if (domain === "example.org") {
return Promise.resolve(clientWellKnown);
}
return Promise.resolve({});
},
);
const mockedSession = vi.mocked({
room: {
roomId: "roomId",
client: {
getDomain: vi.fn().mockReturnValue("example.org"),
getOpenIdToken: vi.fn().mockResolvedValue({
access_token: "ACCCESS_TOKEN",
token_type: "Bearer",
matrix_server_name: "localhost",
expires_in: 10000,
}),
},
},
memberships: [],
getFocusInUse: vi.fn().mockReturnValue(focusFromOlderMembership),
getOldestMembership: vi.fn().mockReturnValue({
getPreferredFoci: vi.fn().mockReturnValue([focusFromOlderMembership]),
}),
joinRoomSession: vi.fn(),
}) as unknown as MatrixRTCSession;
enterRTCSession(
mockedSession,
{
livekit_alias: "roomId",
livekit_service_url: "http://my-well-known-service-url.com",
type: "livekit",
},
{
encryptMedia: true,
matrixRTCMode: MATRIX_RTC_MODE,
},
);
expect(mockedSession.joinRoomSession).toHaveBeenLastCalledWith(
[
{
livekit_alias: "roomId",
livekit_service_url: "http://my-well-known-service-url.com",
type: "livekit",
},
],
undefined,
expect.objectContaining({
manageMediaKeys: true,
useLegacyMemberEvents: false,
}),
);
});
it("It should not fail with configuration error if homeserver config has livekit url but not fallback", () => {
mockConfig({});
vi.spyOn(AutoDiscovery, "getRawClientConfig").mockResolvedValue({
"org.matrix.msc4143.rtc_foci": [
{
type: "livekit",
livekit_service_url: "http://my-well-known-service-url.com",
},
],
});
const mockedSession = vi.mocked({
room: {
roomId: "roomId",
client: {
getDomain: vi.fn().mockReturnValue("example.org"),
getOpenIdToken: vi.fn().mockResolvedValue({
access_token: "ACCCESS_TOKEN",
token_type: "Bearer",
matrix_server_name: "localhost",
expires_in: 10000,
}),
},
},
memberships: [],
getFocusInUse: vi.fn(),
joinRoomSession: vi.fn(),
}) as unknown as MatrixRTCSession;
enterRTCSession(
mockedSession,
{
livekit_alias: "roomId",
livekit_service_url: "http://my-well-known-service-url.com",
type: "livekit",
},
{
encryptMedia: true,
matrixRTCMode: MATRIX_RTC_MODE,
},
);
});
});
const defaultCreateLocalMemberValues = {
options: constant({
encryptMedia: false,
matrixRTCMode: MatrixRTCMode.Matrix_2_0,
}),
matrixRTCSession: {
updateCallIntent: () => {},
leaveRoomSession: () => {},
} as unknown as MatrixRTCSession,
muteStates: mockMuteStates(),
trackProcessorState$: constant({
supported: false,
processor: undefined,
}),
logger: logger,
createPublisherFactory: vi.fn(),
joinMatrixRTC: async (): Promise<void> => {},
homeserverConnected: {
combined$: constant(true),
rtsSession$: constant(RTCMemberStatus.Connected),
},
};
it("throws error on missing RTC config error", () => {
withTestScheduler(({ scope, hot, expectObservable }) => {
const localTransport$ = scope.behavior<null | LivekitTransport>(
hot("1ms #", {}, new MatrixRTCTransportMissingError("domain.com")),
null,
);
// we do not need any connection data since we want to fail before reaching that.
const mockConnectionManager = {
transports$: scope.behavior(
localTransport$.pipe(map((t) => new Epoch([t]))),
),
connectionManagerData$: constant(
new Epoch(new ConnectionManagerData()),
),
};
const localMembership = createLocalMembership$({
scope,
...defaultCreateLocalMemberValues,
connectionManager: mockConnectionManager,
localTransport$,
});
localMembership.requestJoinAndPublish();
expectObservable(localMembership.localMemberState$).toBe("ne", {
n: TransportState.Waiting,
e: expect.toSatisfy((e) => e instanceof MatrixRTCTransportMissingError),
});
});
});
const aTransport = {
livekit_service_url: "a",
} as LivekitTransport;
const bTransport = {
livekit_service_url: "b",
} as LivekitTransport;
const connectionTransportAConnected = {
livekitRoom: mockLivekitRoom({
localParticipant: {
isScreenShareEnabled: false,
trackPublications: [],
} as unknown as LocalParticipant,
}),
state$: constant(ConnectionState.LivekitConnected),
transport: aTransport,
} as unknown as Connection;
const connectionTransportAConnecting = {
...connectionTransportAConnected,
state$: constant(ConnectionState.LivekitConnecting),
livekitRoom: mockLivekitRoom({}),
} as unknown as Connection;
const connectionTransportBConnected = {
state$: constant(ConnectionState.LivekitConnected),
transport: bTransport,
livekitRoom: mockLivekitRoom({}),
} as unknown as Connection;
it("recreates publisher if new connection is used and ENDS always unpublish and end tracks", async () => {
const scope = new ObservableScope();
const localTransport$ = new BehaviorSubject(aTransport);
const publishers: Publisher[] = [];
let seed = 0;
defaultCreateLocalMemberValues.createPublisherFactory.mockImplementation(
() => {
const a = seed;
seed += 1;
logger.info(`creating [${a}]`);
const p = {
stopPublishing: vi.fn().mockImplementation(() => {
logger.info(`stopPublishing [${a}]`);
}),
stopTracks: vi.fn(),
};
publishers.push(p as unknown as Publisher);
return p;
},
);
const publisherFactory =
defaultCreateLocalMemberValues.createPublisherFactory as ReturnType<
typeof vi.fn
>;
const connectionManagerData = new ConnectionManagerData();
connectionManagerData.add(connectionTransportAConnected, []);
connectionManagerData.add(connectionTransportBConnected, []);
createLocalMembership$({
scope,
...defaultCreateLocalMemberValues,
connectionManager: {
connectionManagerData$: constant(new Epoch(connectionManagerData)),
},
localTransport$,
});
await flushPromises();
localTransport$.next(bTransport);
await flushPromises();
expect(publisherFactory).toHaveBeenCalledTimes(2);
expect(publishers.length).toBe(2);
// stop the first Publisher and let the second one life.
expect(publishers[0].stopTracks).toHaveBeenCalled();
expect(publishers[1].stopTracks).not.toHaveBeenCalled();
expect(publishers[0].stopPublishing).toHaveBeenCalled();
expect(publishers[1].stopPublishing).not.toHaveBeenCalled();
expect(publisherFactory.mock.calls[0][0].transport).toBe(aTransport);
expect(publisherFactory.mock.calls[1][0].transport).toBe(bTransport);
scope.end();
await flushPromises();
// stop all tracks after ending scopes
expect(publishers[1].stopPublishing).toHaveBeenCalled();
// expect(publishers[1].stopTracks).toHaveBeenCalled();
defaultCreateLocalMemberValues.createPublisherFactory.mockReset();
});
it("only start tracks if requested", async () => {
const scope = new ObservableScope();
const localTransport$ = new BehaviorSubject(aTransport);
const publishers: Publisher[] = [];
const tracks$ = new BehaviorSubject<LocalTrack[]>([]);
const publishing$ = new BehaviorSubject<boolean>(false);
defaultCreateLocalMemberValues.createPublisherFactory.mockImplementation(
() => {
const p = {
stopPublishing: vi.fn(),
stopTracks: vi.fn(),
createAndSetupTracks: vi.fn().mockImplementation(async () => {
tracks$.next([{}, {}] as LocalTrack[]);
return Promise.resolve();
}),
tracks$,
publishing$,
};
publishers.push(p as unknown as Publisher);
return p;
},
);
const publisherFactory =
defaultCreateLocalMemberValues.createPublisherFactory as ReturnType<
typeof vi.fn
>;
const connectionManagerData = new ConnectionManagerData();
connectionManagerData.add(connectionTransportAConnected, []);
// connectionManagerData.add(connectionTransportB, []);
const localMembership = createLocalMembership$({
scope,
...defaultCreateLocalMemberValues,
connectionManager: {
connectionManagerData$: constant(new Epoch(connectionManagerData)),
},
localTransport$,
});
await flushPromises();
expect(publisherFactory).toHaveBeenCalledOnce();
// expect(localMembership.tracks$.value.length).toBe(0);
expect(publishers[0].createAndSetupTracks).not.toHaveBeenCalled();
localMembership.startTracks();
await flushPromises();
expect(publishers[0].createAndSetupTracks).toHaveBeenCalled();
// expect(localMembership.tracks$.value.length).toBe(2);
scope.end();
await flushPromises();
// stop all tracks after ending scopes
expect(publishers[0].stopPublishing).toHaveBeenCalled();
// expect(publishers[0].stopTracks).toHaveBeenCalled();
publisherFactory.mockClear();
});
// TODO add an integration test combining publisher and localMembership
//
it("tracks livekit state correctly", async () => {
const scope = new ObservableScope();
const connectionManagerData = new ConnectionManagerData();
const localTransport$ = new BehaviorSubject<null | LivekitTransport>(null);
const connectionManagerData$ = new BehaviorSubject(
new Epoch(connectionManagerData),
);
const publishers: Publisher[] = [];
const tracks$ = new BehaviorSubject<LocalTrack[]>([]);
const publishing$ = new BehaviorSubject<boolean>(false);
const createTrackResolver = Promise.withResolvers<void>();
const publishResolver = Promise.withResolvers<void>();
defaultCreateLocalMemberValues.createPublisherFactory.mockImplementation(
() => {
const p = {
stopPublishing: vi.fn(),
stopTracks: vi.fn().mockImplementation(() => {
logger.info("stopTracks");
tracks$.next([]);
}),
createAndSetupTracks: vi.fn().mockImplementation(async () => {
await createTrackResolver.promise;
tracks$.next([{}, {}] as LocalTrack[]);
}),
startPublishing: vi.fn().mockImplementation(async () => {
await publishResolver.promise;
publishing$.next(true);
}),
tracks$,
publishing$,
};
publishers.push(p as unknown as Publisher);
return p;
},
);
const publisherFactory =
defaultCreateLocalMemberValues.createPublisherFactory as ReturnType<
typeof vi.fn
>;
const localMembership = createLocalMembership$({
scope,
...defaultCreateLocalMemberValues,
connectionManager: {
connectionManagerData$,
},
localTransport$,
});
await flushPromises();
expect(localMembership.localMemberState$.value).toStrictEqual(
TransportState.Waiting,
);
localTransport$.next(aTransport);
await flushPromises();
expect(localMembership.localMemberState$.value).toStrictEqual({
matrix: RTCMemberStatus.Connected,
media: { connection: null, tracks: TrackState.WaitingForUser },
});
const connectionManagerData2 = new ConnectionManagerData();
connectionManagerData2.add(
// clone because we will mutate this later.
{ ...connectionTransportAConnecting } as unknown as Connection,
[],
);
connectionManagerData$.next(new Epoch(connectionManagerData2));
await flushPromises();
expect(localMembership.localMemberState$.value).toStrictEqual({
matrix: RTCMemberStatus.Connected,
media: {
connection: ConnectionState.LivekitConnecting,
tracks: TrackState.WaitingForUser,
},
});
(
connectionManagerData2.getConnectionForTransport(aTransport)!
.state$ as BehaviorSubject<ConnectionState>
).next(ConnectionState.LivekitConnected);
expect(localMembership.localMemberState$.value).toStrictEqual({
matrix: RTCMemberStatus.Connected,
media: {
connection: ConnectionState.LivekitConnected,
tracks: TrackState.WaitingForUser,
},
});
expect(publisherFactory).toHaveBeenCalledOnce();
// expect(localMembership.tracks$.value.length).toBe(0);
// -------
localMembership.startTracks();
// -------
await flushPromises();
// expect(localMembership.localMemberState$.value).toStrictEqual({
// matrix: RTCMemberStatus.Connected,
// media: {
// tracks: TrackState.Creating,
// connection: ConnectionState.LivekitConnected,
// },
// });
createTrackResolver.resolve();
await flushPromises();
expect(
// eslint-disable-next-line @typescript-eslint/no-explicit-any
(localMembership.localMemberState$.value as any).media,
).toStrictEqual(PublishState.WaitingForUser);
// -------
localMembership.requestJoinAndPublish();
// -------
expect(
// eslint-disable-next-line @typescript-eslint/no-explicit-any
(localMembership.localMemberState$.value as any).media,
).toStrictEqual(PublishState.Publishing);
publishResolver.resolve();
await flushPromises();
expect(
// eslint-disable-next-line @typescript-eslint/no-explicit-any
(localMembership.localMemberState$.value as any).media,
).toStrictEqual(PublishState.Publishing);
expect(publishers[0].stopPublishing).not.toHaveBeenCalled();
expect(localMembership.localMemberState$.isStopped).toBe(false);
scope.end();
await flushPromises();
// stays in connected state because it is stopped before the update to tracks update the state.
expect(
// eslint-disable-next-line @typescript-eslint/no-explicit-any
(localMembership.localMemberState$.value as any).media,
).toStrictEqual(PublishState.Publishing);
// stop all tracks after ending scopes
expect(publishers[0].stopPublishing).toHaveBeenCalled();
// expect(publishers[0].stopTracks).toHaveBeenCalled();
});
// TODO add tests for matrix local matrix participation.
});

View File

@@ -0,0 +1,723 @@
/*
Copyright 2025 Element Creations Ltd.
SPDX-License-IdFentifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import {
type Participant,
ParticipantEvent,
type LocalParticipant,
type ScreenShareCaptureOptions,
RoomEvent,
MediaDeviceFailure,
} from "livekit-client";
import { observeParticipantEvents } from "@livekit/components-core";
import {
Status as RTCSessionStatus,
type LivekitTransport,
type MatrixRTCSession,
} from "matrix-js-sdk/lib/matrixrtc";
import {
BehaviorSubject,
catchError,
combineLatest,
distinctUntilChanged,
from,
fromEvent,
map,
type Observable,
of,
pairwise,
startWith,
switchMap,
tap,
} from "rxjs";
import { type Logger } from "matrix-js-sdk/lib/logger";
import { deepCompare } from "matrix-js-sdk/lib/utils";
import { type Behavior } from "../../Behavior.ts";
import { type IConnectionManager } from "../remoteMembers/ConnectionManager.ts";
import { type ObservableScope } from "../../ObservableScope.ts";
import { type Publisher } from "./Publisher.ts";
import { type MuteStates } from "../../MuteStates.ts";
import {
ElementCallError,
FailToStartLivekitConnection,
MembershipManagerError,
UnknownCallError,
} from "../../../utils/errors.ts";
import { ElementWidgetActions, widget } from "../../../widget.ts";
import { getUrlParams } from "../../../UrlParams.ts";
import { PosthogAnalytics } from "../../../analytics/PosthogAnalytics.ts";
import { MatrixRTCMode } from "../../../settings/settings.ts";
import { Config } from "../../../config/Config.ts";
import {
ConnectionState,
type Connection,
type FailedToStartError,
} from "../remoteMembers/Connection.ts";
import { type HomeserverConnected } from "./HomeserverConnected.ts";
import { and$ } from "../../../utils/observable.ts";
export enum TransportState {
/** Not even a transport is available to the LocalMembership */
Waiting = "transport_waiting",
}
export enum PublishState {
WaitingForUser = "publish_waiting_for_user",
// XXX: This state is removed for now since we do not have full control over
// track publication anymore with the publisher abstraction, might come back in the future?
// /** Implies lk connection is connected */
// Starting = "publish_start_publishing",
/** Implies lk connection is connected */
Publishing = "publish_publishing",
}
// TODO not sure how to map that correctly with the
// new publisher that does not manage tracks itself anymore
export enum TrackState {
/** The track is waiting for user input to create tracks (waiting to call `startTracks()`) */
WaitingForUser = "tracks_waiting_for_user",
// XXX: This state is removed for now since we do not have full control over
// track creation anymore with the publisher abstraction, might come back in the future?
// /** Implies lk connection is connected */
// Creating = "tracks_creating",
/** Implies lk connection is connected */
Ready = "tracks_ready",
}
export type LocalMemberMediaState =
| {
tracks: TrackState;
connection: ConnectionState | FailedToStartError;
}
| PublishState
| ElementCallError;
export type LocalMemberState =
| ElementCallError
| TransportState.Waiting
| {
media: LocalMemberMediaState;
matrix: ElementCallError | RTCSessionStatus;
};
/*
* - get well known
* - get oldest membership
* - get transport to use
* - get openId + jwt token
* - wait for createTrack() call
* - create tracks
* - wait for join() call
* - Publisher.publishTracks()
* - send join state/sticky event
*/
interface Props {
// TODO add a comment into some code style readme or file header callviewmodel
// that the inputs for those createSomething$() functions should NOT contain any js-sdk objectes
scope: ObservableScope;
muteStates: MuteStates;
connectionManager: IConnectionManager;
createPublisherFactory: (connection: Connection) => Publisher;
joinMatrixRTC: (transport: LivekitTransport) => void;
homeserverConnected: HomeserverConnected;
localTransport$: Behavior<LivekitTransport | null>;
matrixRTCSession: Pick<
MatrixRTCSession,
"updateCallIntent" | "leaveRoomSession"
>;
logger: Logger;
}
/**
* This class is responsible for managing the own membership in a room.
* We want
* - a publisher
* -
* @param param0
* @returns
* - publisher: The handle to create tracks and publish them to the room.
* - connected$: the current connection state. Including matrix server and livekit server connection. (only considering the livekit server we are using for our own media publication)
* - transport$: the transport object the ownMembership$ ended up using.
* - connectionState: the current connection state. Including matrix server and livekit server connection.
* - sharingScreen$: Whether we are sharing our screen. `undefined` if we cannot share the screen.
*/
export const createLocalMembership$ = ({
scope,
connectionManager,
localTransport$: localTransportCanThrow$,
homeserverConnected,
createPublisherFactory,
joinMatrixRTC,
logger: parentLogger,
muteStates,
matrixRTCSession,
}: Props): {
/**
* This request to start audio and video tracks.
* Can be called early to pre-emptively get media permissions and start devices.
*/
startTracks: () => void;
/**
* This sets a inner state (shouldPublish) to true and instructs the js-sdk and livekit to keep the user
* connected to matrix and livekit.
*/
requestJoinAndPublish: () => void;
requestDisconnect: () => void;
localMemberState$: Behavior<LocalMemberState>;
sharingScreen$: Behavior<boolean>;
/**
* Callback to toggle screen sharing. If null, screen sharing is not possible.
*/
toggleScreenSharing: (() => void) | null;
// tracks$: Behavior<LocalTrack[]>;
participant$: Behavior<LocalParticipant | null>;
connection$: Behavior<Connection | null>;
/** Shorthand for homeserverConnected.rtcSession === Status.Reconnecting
* Direct translation to the js-sdk membership manager connection `Status`.
*/
reconnecting$: Behavior<boolean>;
/** Shorthand for homeserverConnected.rtcSession === Status.Disconnected
* Direct translation to the js-sdk membership manager connection `Status`.
*/
disconnected$: Behavior<boolean>;
} => {
const logger = parentLogger.getChild("[LocalMembership]");
logger.debug(`Creating local membership..`);
// Unwrap the local transport and set the state of the LocalMembership to error in case the transport is an error.
const localTransport$ = scope.behavior(
localTransportCanThrow$.pipe(
catchError((e: unknown) => {
let error: ElementCallError;
if (e instanceof ElementCallError) {
error = e;
} else {
error = new UnknownCallError(
e instanceof Error
? e
: new Error("Unknown error from localTransport"),
);
}
setTransportError(error);
return of(null);
}),
),
);
// Drop Epoch data here since we will not combine this anymore
const localConnection$ = scope.behavior(
combineLatest([
connectionManager.connectionManagerData$,
localTransport$,
]).pipe(
map(([{ value: connectionData }, localTransport]) => {
if (localTransport === null) {
return null;
}
return connectionData.getConnectionForTransport(localTransport);
}),
tap((connection) => {
logger.info(
`Local connection updated: ${connection?.transport?.livekit_service_url}`,
);
}),
),
);
// Tracks error that happen when creating the local tracks.
const mediaErrors$ = localConnection$.pipe(
switchMap((connection) => {
if (!connection) {
return of(null);
} else {
return fromEvent(
connection.livekitRoom,
RoomEvent.MediaDevicesError,
(error: Error) => {
return MediaDeviceFailure.getFailure(error) ?? null;
},
);
}
}),
);
mediaErrors$.pipe(scope.bind()).subscribe((error) => {
if (error) {
logger.error(`Failed to create local tracks:`, error);
setMatrixError(
// TODO is it fatal? Do we need to create a new Specialized Error?
new UnknownCallError(new Error(`Media device error: ${error}`)),
);
}
});
// MATRIX RELATED
// This should be used in a combineLatest with publisher$ to connect.
// to make it possible to call startTracks before the preferredTransport$ has resolved.
const trackStartRequested = Promise.withResolvers<void>();
// This should be used in a combineLatest with publisher$ to connect.
// to make it possible to call startTracks before the preferredTransport$ has resolved.
const joinAndPublishRequested$ = new BehaviorSubject(false);
/**
* The publisher is stored in here an abstracts creating and publishing tracks.
*/
const publisher$ = new BehaviorSubject<Publisher | null>(null);
const startTracks = (): void => {
trackStartRequested.resolve();
// This used to return the tracks, but now they are only accessible via the publisher.
};
const requestJoinAndPublish = (): void => {
trackStartRequested.resolve();
joinAndPublishRequested$.next(true);
};
const requestDisconnect = (): void => {
joinAndPublishRequested$.next(false);
};
// Take care of the publisher$
// create a new one as soon as a local Connection is available
//
// Recreate a new one once the local connection changes
// - stop publishing
// - destruct all current streams
// - overwrite current publisher
scope.reconcile(localConnection$, async (connection) => {
if (connection !== null) {
const publisher = createPublisherFactory(connection);
publisher$.next(publisher);
// Clean-up callback
return Promise.resolve(async (): Promise<void> => {
await publisher.stopPublishing();
await publisher.stopTracks();
});
}
});
// Use reconcile here to not run concurrent createAndSetupTracks calls
// `tracks$` will update once they are ready.
scope.reconcile(
scope.behavior(
combineLatest([
publisher$ /*, tracks$*/,
from(trackStartRequested.promise),
]),
null,
),
async (valueIfReady) => {
if (!valueIfReady) return;
const [publisher] = valueIfReady;
if (publisher) {
await publisher.createAndSetupTracks().catch((e) => logger.error(e));
}
},
);
// Based on `connectRequested$` we start publishing tracks. (once they are there!)
scope.reconcile(
scope.behavior(combineLatest([publisher$, joinAndPublishRequested$])),
async ([publisher, shouldJoinAndPublish]) => {
// Get the current publishing state to avoid redundant calls.
const isPublishing = publisher?.shouldPublish === true;
if (shouldJoinAndPublish && !isPublishing) {
try {
await publisher?.startPublishing();
} catch (error) {
const message =
error instanceof Error ? error.message : String(error);
setPublishError(new FailToStartLivekitConnection(message));
}
} else if (isPublishing) {
try {
await publisher?.stopPublishing();
} catch (error) {
setPublishError(new UnknownCallError(error as Error));
}
}
},
);
// STATE COMPUTATION
// These are non fatal since we can join a room and concume media even though publishing failed.
const publishError$ = new BehaviorSubject<ElementCallError | null>(null);
const setPublishError = (e: ElementCallError): void => {
if (publishError$.value !== null) {
logger.error("Multiple Media Errors:", e);
} else {
publishError$.next(e);
}
};
const fatalTransportError$ = new BehaviorSubject<ElementCallError | null>(
null,
);
const setTransportError = (e: ElementCallError): void => {
if (fatalTransportError$.value !== null) {
logger.error("Multiple Transport Errors:", e);
} else {
fatalTransportError$.next(e);
}
};
const localConnectionState$ = localConnection$.pipe(
switchMap((connection) => (connection ? connection.state$ : of(null))),
);
const mediaState$: Behavior<LocalMemberMediaState> = scope.behavior(
combineLatest([
localConnectionState$,
localTransport$,
joinAndPublishRequested$,
from(trackStartRequested.promise).pipe(
map(() => true),
startWith(false),
),
]).pipe(
map(
([
localConnectionState,
localTransport,
shouldPublish,
shouldStartTracks,
]) => {
if (!localTransport) return null;
const trackState: TrackState = shouldStartTracks
? TrackState.Ready
: TrackState.WaitingForUser;
if (
localConnectionState !== ConnectionState.LivekitConnected ||
trackState !== TrackState.Ready
)
return {
connection: localConnectionState,
tracks: trackState,
};
if (!shouldPublish) return PublishState.WaitingForUser;
// if (!publishing) return PublishState.Starting;
return PublishState.Publishing;
},
),
distinctUntilChanged(deepCompare),
),
);
const fatalMatrixError$ = new BehaviorSubject<ElementCallError | null>(null);
const setMatrixError = (e: ElementCallError): void => {
if (fatalMatrixError$.value !== null) {
logger.error("Multiple Matrix Errors:", e);
} else {
fatalMatrixError$.next(e);
}
};
const localMemberState$ = scope.behavior<LocalMemberState>(
combineLatest([
mediaState$,
homeserverConnected.rtsSession$,
fatalMatrixError$,
fatalTransportError$,
publishError$,
]).pipe(
map(
([
mediaState,
rtcSessionStatus,
fatalMatrixError,
fatalTransportError,
publishError,
]) => {
if (fatalTransportError !== null) return fatalTransportError;
// `mediaState` will be 'null' until the transport/connection appears.
if (mediaState && rtcSessionStatus)
return {
matrix: fatalMatrixError ?? rtcSessionStatus,
media: publishError ?? mediaState,
};
return TransportState.Waiting;
},
),
),
);
/**
* Whether we are "fully" connected to the call. Accounts for both the
* connection to the MatrixRTC session and the LiveKit publish connection.
*/
const matrixAndLivekitConnected$ = scope.behavior(
and$(
homeserverConnected.combined$,
localConnectionState$.pipe(
map((state) => state === ConnectionState.LivekitConnected),
),
).pipe(
tap((v) => logger.debug("livekit+matrix: Connected state changed", v)),
),
);
/**
* Whether we should tell the user that we're reconnecting to the call.
*/
const reconnecting$ = scope.behavior(
matrixAndLivekitConnected$.pipe(
pairwise(),
map(([prev, current]) => prev === true && current === false),
),
false,
);
// inform the widget about the connect and disconnect intent from the user.
scope
.behavior(joinAndPublishRequested$.pipe(pairwise(), scope.bind()), [
undefined,
joinAndPublishRequested$.value,
])
.subscribe(([prev, current]) => {
if (!widget) return;
// JOIN prev=false (was left) => current-true (now joiend)
if (!prev && current) {
widget.api.transport
.send(ElementWidgetActions.JoinCall, {})
.catch((e) => {
logger.error("Failed to send join action", e);
});
}
// LEAVE prev=false (was joined) => current-true (now left)
if (prev && !current) {
widget.api.transport
.send(ElementWidgetActions.HangupCall, {})
.catch((e) => {
logger.error("Failed to send hangup action", e);
});
}
});
combineLatest([muteStates.video.enabled$, homeserverConnected.combined$])
.pipe(scope.bind())
.subscribe(([videoEnabled, connected]) => {
if (!connected) return;
void matrixRTCSession.updateCallIntent(videoEnabled ? "video" : "audio");
});
// Keep matrix rtc session in sync with localTransport$, connectRequested$
scope.reconcile(
scope.behavior(combineLatest([localTransport$, joinAndPublishRequested$])),
async ([transport, shouldConnect]) => {
if (!transport) return;
// if shouldConnect=false we will do the disconnect as the cleanup from the previous reconcile iteration.
if (!shouldConnect) return;
try {
joinMatrixRTC(transport);
} catch (error) {
logger.error("Error entering RTC session", error);
if (error instanceof Error)
setMatrixError(new MembershipManagerError(error));
}
return Promise.resolve(async (): Promise<void> => {
try {
// TODO Update matrixRTCSession to allow udpating the transport without leaving the session!
await matrixRTCSession.leaveRoomSession(1000);
} catch (e) {
logger.error("Error leaving RTC session", e);
}
});
},
);
const participant$ = scope.behavior(
localConnection$.pipe(map((c) => c?.livekitRoom?.localParticipant ?? null)),
);
// Pause upstream of all local media tracks when we're disconnected from
// MatrixRTC, because it can be an unpleasant surprise for the app to say
// 'reconnecting' and yet still be transmitting your media to others.
// We use matrixConnected$ rather than reconnecting$ because we want to
// pause tracks during the initial joining sequence too until we're sure
// that our own media is displayed on screen.
// TODO refactor this based no livekitState$
combineLatest([participant$, homeserverConnected.combined$])
.pipe(scope.bind())
.subscribe(([participant, connected]) => {
if (!participant) return;
const publications = participant.trackPublications.values();
if (connected) {
for (const p of publications) {
if (p.track?.isUpstreamPaused === true) {
const kind = p.track.kind;
logger.info(
`Resuming ${kind} track (MatrixRTC connection present)`,
);
p.track
.resumeUpstream()
.catch((e) =>
logger.error(
`Failed to resume ${kind} track after MatrixRTC reconnection`,
e,
),
);
}
}
} else {
for (const p of publications) {
if (p.track?.isUpstreamPaused === false) {
const kind = p.track.kind;
logger.info(
`Pausing ${kind} track (uncertain MatrixRTC connection)`,
);
p.track
.pauseUpstream()
.catch((e) =>
logger.error(
`Failed to pause ${kind} track after entering uncertain MatrixRTC connection`,
e,
),
);
}
}
}
});
/**
* Whether the user is currently sharing their screen.
*/
const sharingScreen$ = scope.behavior(
participant$.pipe(
switchMap((p) => (p !== null ? observeSharingScreen$(p) : of(false))),
),
);
let toggleScreenSharing: (() => void) | null = null;
if (
"getDisplayMedia" in (navigator.mediaDevices ?? {}) &&
!getUrlParams().hideScreensharing
) {
toggleScreenSharing = (): void => {
const screenshareSettings: ScreenShareCaptureOptions = {
audio: true,
selfBrowserSurface: "include",
surfaceSwitching: "include",
systemAudio: "include",
};
const targetScreenshareState = !sharingScreen$.value;
logger.info(
`toggleScreenSharing called. Switching ${
targetScreenshareState ? "On" : "Off"
}`,
);
// If a connection is ready, toggle screen sharing.
// We deliberately do nothing in the case of a null connection because
// it looks nice for the call control buttons to all become available
// at once upon joining the call, rather than introducing a disabled
// state. The user can just click again.
// We also allow screen sharing to be toggled even if the connection
// is still initializing or publishing tracks, because there's no
// technical reason to disallow this. LiveKit will publish if it can.
participant$.value
?.setScreenShareEnabled(targetScreenshareState, screenshareSettings)
.catch(logger.error);
};
}
return {
startTracks,
requestJoinAndPublish,
requestDisconnect,
localMemberState$,
participant$,
reconnecting$,
disconnected$: scope.behavior(
homeserverConnected.rtsSession$.pipe(
map((state) => state === RTCSessionStatus.Disconnected),
),
),
sharingScreen$,
toggleScreenSharing,
connection$: localConnection$,
};
};
export function observeSharingScreen$(p: Participant): Observable<boolean> {
return observeParticipantEvents(
p,
ParticipantEvent.TrackPublished,
ParticipantEvent.TrackUnpublished,
ParticipantEvent.LocalTrackPublished,
ParticipantEvent.LocalTrackUnpublished,
).pipe(map((p) => p.isScreenShareEnabled));
}
interface EnterRTCSessionOptions {
encryptMedia: boolean;
matrixRTCMode: MatrixRTCMode;
}
/**
* Does the necessary steps to enter the RTC session on the matrix side:
* - Preparing the membership info (FOCUS to use, options)
* - Sends the matrix event to join the call, and starts the membership manager:
* - Delay events management
* - Handles retries (fails only after several attempts)
*
* @param rtcSession
* @param transport
* @param options
* @throws If the widget could not send ElementWidgetActions.JoinCall action.
*/
// Exported for unit testing
export function enterRTCSession(
rtcSession: MatrixRTCSession,
transport: LivekitTransport,
{ encryptMedia, matrixRTCMode }: EnterRTCSessionOptions,
): void {
PosthogAnalytics.instance.eventCallEnded.cacheStartCall(new Date());
PosthogAnalytics.instance.eventCallStarted.track(rtcSession.room.roomId);
// This must be called before we start trying to join the call, as we need to
// have started tracking by the time calls start getting created.
// groupCallOTelMembership?.onJoinCall();
const { features, matrix_rtc_session: matrixRtcSessionConfig } = Config.get();
const useDeviceSessionMemberEvents =
features?.feature_use_device_session_member_events;
const { sendNotificationType: notificationType, callIntent } = getUrlParams();
const multiSFU = matrixRTCMode !== MatrixRTCMode.Legacy;
// Multi-sfu does not need a preferred foci list. just the focus that is actually used.
// TODO where/how do we track errors originating from the ongoing rtcSession?
rtcSession.joinRoomSession(
multiSFU ? [] : [transport],
multiSFU ? transport : undefined,
{
notificationType,
callIntent,
manageMediaKeys: encryptMedia,
...(useDeviceSessionMemberEvents !== undefined && {
useLegacyMemberEvents: !useDeviceSessionMemberEvents,
}),
delayedLeaveEventRestartMs:
matrixRtcSessionConfig?.delayed_leave_event_restart_ms,
delayedLeaveEventDelayMs:
matrixRtcSessionConfig?.delayed_leave_event_delay_ms,
delayedLeaveEventRestartLocalTimeoutMs:
matrixRtcSessionConfig?.delayed_leave_event_restart_local_timeout_ms,
networkErrorRetryMs: matrixRtcSessionConfig?.network_error_retry_ms,
makeKeyDelay: matrixRtcSessionConfig?.wait_for_key_rotation_ms,
membershipEventExpiryMs:
matrixRtcSessionConfig?.membership_event_expiry_ms,
useExperimentalToDeviceTransport: true,
unstableSendStickyEvents: matrixRTCMode === MatrixRTCMode.Matrix_2_0,
},
);
}

View File

@@ -1,158 +0,0 @@
/*
Copyright 2024 New Vector Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { type MatrixRTCSession } from "matrix-js-sdk/lib/matrixrtc";
import { expect, test, vi } from "vitest";
import { AutoDiscovery } from "matrix-js-sdk/lib/autodiscovery";
import EventEmitter from "events";
import { MatrixRTCMode } from "../../../settings/settings";
import { mockConfig } from "../../../utils/test";
import { enterRTCSession } from "./LocalMembership";
const MATRIX_RTC_MODE = MatrixRTCMode.Legacy;
const getUrlParams = vi.hoisted(() => vi.fn(() => ({})));
vi.mock("../../../UrlParams", () => ({ getUrlParams }));
vi.mock("../../../widget", async (importOriginal) => ({
...(await importOriginal()),
widget: {
api: {
setAlwaysOnScreen: (): void => {},
transport: { send: vi.fn(), reply: vi.fn(), stop: vi.fn() },
},
lazyActions: new EventEmitter(),
},
}));
test("It joins the correct Session", async () => {
const focusFromOlderMembership = {
type: "livekit",
livekit_service_url: "http://my-oldest-member-service-url.com",
livekit_alias: "my-oldest-member-service-alias",
};
const focusConfigFromWellKnown = {
type: "livekit",
livekit_service_url: "http://my-well-known-service-url.com",
};
const focusConfigFromWellKnown2 = {
type: "livekit",
livekit_service_url: "http://my-well-known-service-url2.com",
};
const clientWellKnown = {
"org.matrix.msc4143.rtc_foci": [
focusConfigFromWellKnown,
focusConfigFromWellKnown2,
],
};
mockConfig({
livekit: { livekit_service_url: "http://my-default-service-url.com" },
});
vi.spyOn(AutoDiscovery, "getRawClientConfig").mockImplementation(
async (domain) => {
if (domain === "example.org") {
return Promise.resolve(clientWellKnown);
}
return Promise.resolve({});
},
);
const mockedSession = vi.mocked({
room: {
roomId: "roomId",
client: {
getDomain: vi.fn().mockReturnValue("example.org"),
getOpenIdToken: vi.fn().mockResolvedValue({
access_token: "ACCCESS_TOKEN",
token_type: "Bearer",
matrix_server_name: "localhost",
expires_in: 10000,
}),
},
},
memberships: [],
getFocusInUse: vi.fn().mockReturnValue(focusFromOlderMembership),
getOldestMembership: vi.fn().mockReturnValue({
getPreferredFoci: vi.fn().mockReturnValue([focusFromOlderMembership]),
}),
joinRoomSession: vi.fn(),
}) as unknown as MatrixRTCSession;
await enterRTCSession(
mockedSession,
{
livekit_alias: "roomId",
livekit_service_url: "http://my-well-known-service-url.com",
type: "livekit",
},
{
encryptMedia: true,
matrixRTCMode: MATRIX_RTC_MODE,
},
);
expect(mockedSession.joinRoomSession).toHaveBeenLastCalledWith(
[
{
livekit_alias: "roomId",
livekit_service_url: "http://my-well-known-service-url.com",
type: "livekit",
},
],
undefined,
expect.objectContaining({
manageMediaKeys: true,
useLegacyMemberEvents: false,
}),
);
});
test("It should not fail with configuration error if homeserver config has livekit url but not fallback", async () => {
mockConfig({});
vi.spyOn(AutoDiscovery, "getRawClientConfig").mockResolvedValue({
"org.matrix.msc4143.rtc_foci": [
{
type: "livekit",
livekit_service_url: "http://my-well-known-service-url.com",
},
],
});
const mockedSession = vi.mocked({
room: {
roomId: "roomId",
client: {
getDomain: vi.fn().mockReturnValue("example.org"),
getOpenIdToken: vi.fn().mockResolvedValue({
access_token: "ACCCESS_TOKEN",
token_type: "Bearer",
matrix_server_name: "localhost",
expires_in: 10000,
}),
},
},
memberships: [],
getFocusInUse: vi.fn(),
joinRoomSession: vi.fn(),
}) as unknown as MatrixRTCSession;
await enterRTCSession(
mockedSession,
{
livekit_alias: "roomId",
livekit_service_url: "http://my-well-known-service-url.com",
type: "livekit",
},
{
encryptMedia: true,
matrixRTCMode: MATRIX_RTC_MODE,
},
);
});

View File

@@ -1,628 +0,0 @@
/*
Copyright 2025 New Vector Ltd.
SPDX-License-IdFentifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import {
type LocalTrack,
type Participant,
ParticipantEvent,
type LocalParticipant,
} from "livekit-client";
import { observeParticipantEvents } from "@livekit/components-core";
import {
type LivekitTransport,
type MatrixRTCSession,
MembershipManagerEvent,
Status,
} from "matrix-js-sdk/lib/matrixrtc";
import { ClientEvent, SyncState, type Room as MatrixRoom } from "matrix-js-sdk";
import {
BehaviorSubject,
combineLatest,
distinctUntilChanged,
fromEvent,
map,
type Observable,
of,
scan,
startWith,
switchMap,
tap,
} from "rxjs";
import { type Logger } from "matrix-js-sdk/lib/logger";
import { type Behavior } from "../../Behavior";
import { type IConnectionManager } from "../remoteMembers/ConnectionManager";
import { ObservableScope } from "../../ObservableScope";
import { Publisher } from "./Publisher";
import { type MuteStates } from "../../MuteStates";
import { type ProcessorState } from "../../../livekit/TrackProcessorContext";
import { type MediaDevices } from "../../MediaDevices";
import { and$ } from "../../../utils/observable";
import { ElementCallError, UnknownCallError } from "../../../utils/errors";
import {
ElementWidgetActions,
widget,
type WidgetHelpers,
} from "../../../widget";
import { areLivekitTransportsEqual } from "../remoteMembers/MatrixLivekitMembers";
import { getUrlParams } from "../../../UrlParams.ts";
import { PosthogAnalytics } from "../../../analytics/PosthogAnalytics.ts";
import { MatrixRTCMode } from "../../../settings/settings.ts";
import { Config } from "../../../config/Config.ts";
import {
type Connection,
type ConnectionState,
} from "../remoteMembers/Connection.ts";
export enum LivekitState {
Uninitialized = "uninitialized",
Connecting = "connecting",
Connected = "connected",
Error = "error",
Disconnected = "disconnected",
Disconnecting = "disconnecting",
}
type LocalMemberLivekitState =
| { state: LivekitState.Error; error: string }
| { state: LivekitState.Connected }
| { state: LivekitState.Connecting }
| { state: LivekitState.Uninitialized }
| { state: LivekitState.Disconnected }
| { state: LivekitState.Disconnecting };
export enum MatrixState {
Connected = "connected",
Disconnected = "disconnected",
Connecting = "connecting",
}
type LocalMemberMatrixState =
| { state: MatrixState.Connected }
| { state: MatrixState.Connecting }
| { state: MatrixState.Disconnected };
export interface LocalMemberConnectionState {
livekit$: Behavior<LocalMemberLivekitState>;
matrix$: Behavior<LocalMemberMatrixState>;
}
/*
* - get well known
* - get oldest membership
* - get transport to use
* - get openId + jwt token
* - wait for createTrack() call
* - create tracks
* - wait for join() call
* - Publisher.publishTracks()
* - send join state/sticky event
*/
interface Props {
options: Behavior<EnterRTCSessionOptions>;
scope: ObservableScope;
mediaDevices: MediaDevices;
muteStates: MuteStates;
connectionManager: IConnectionManager;
matrixRTCSession: MatrixRTCSession;
matrixRoom: MatrixRoom;
localTransport$: Behavior<LivekitTransport | null>;
trackProcessorState$: Behavior<ProcessorState>;
widget: WidgetHelpers | null;
logger: Logger;
}
/**
* This class is responsible for managing the own membership in a room.
* We want
* - a publisher
* -
* @param param0
* @returns
* - publisher: The handle to create tracks and publish them to the room.
* - connected$: the current connection state. Including matrix server and livekit server connection. (only considering the livekit server we are using for our own media publication)
* - transport$: the transport object the ownMembership$ ended up using.
* - connectionState: the current connection state. Including matrix server and livekit server connection.
* - sharingScreen$: Whether we are sharing our screen. `undefined` if we cannot share the screen.
*/
export const createLocalMembership$ = ({
scope,
options,
muteStates,
mediaDevices,
connectionManager,
matrixRTCSession,
localTransport$,
matrixRoom,
trackProcessorState$,
widget,
logger: parentLogger,
}: Props): {
// publisher: Publisher
requestConnect: () => LocalMemberConnectionState;
startTracks: () => Behavior<LocalTrack[]>;
requestDisconnect: () => Observable<LocalMemberLivekitState> | null;
connectionState: LocalMemberConnectionState;
sharingScreen$: Behavior<boolean>;
/**
* Callback to toggle screen sharing. If null, screen sharing is not possible.
*/
toggleScreenSharing: (() => void) | null;
participant$: Behavior<LocalParticipant | null>;
connection$: Behavior<Connection | null>;
// deprecated fields
/** @deprecated use state instead*/
homeserverConnected$: Behavior<boolean>;
/** @deprecated use state instead*/
connected$: Behavior<boolean>;
// this needs to be discussed
/** @deprecated use state instead*/
reconnecting$: Behavior<boolean>;
// also needs to be disccues
/** @deprecated use state instead*/
configError$: Behavior<ElementCallError | null>;
} => {
const logger = parentLogger.getChild("[LocalMembership]");
logger.debug(`Creating local membership..`);
const state = {
livekit$: new BehaviorSubject<LocalMemberLivekitState>({
state: LivekitState.Uninitialized,
}),
matrix$: new BehaviorSubject<LocalMemberMatrixState>({
state: MatrixState.Disconnected,
}),
};
// This should be used in a combineLatest with publisher$ to connect.
// to make it possible to call startTracks before the preferredTransport$ has resolved.
const trackStartRequested$ = new BehaviorSubject(false);
// This should be used in a combineLatest with publisher$ to connect.
// to make it possible to call startTracks before the preferredTransport$ has resolved.
const connectRequested$ = new BehaviorSubject(false);
// This should be used in a combineLatest with publisher$ to connect.
const tracks$ = new BehaviorSubject<LocalTrack[]>([]);
// Drop Epoch data here since we will not combine this anymore
const localConnection$ = scope.behavior(
combineLatest([connectionManager.connections$, localTransport$]).pipe(
map(([connections, localTransport]) => {
if (localTransport === null) {
return null;
}
return (
connections.value.find((connection) =>
areLivekitTransportsEqual(connection.transport, localTransport),
) ?? null
);
}),
tap((connection) => {
logger.info(
`Local connection updated: ${connection?.transport?.livekit_service_url}`,
);
}),
),
);
/**
* Whether we are connected to the MatrixRTC session.
*/
const homeserverConnected$ = scope.behavior(
// To consider ourselves connected to MatrixRTC, we check the following:
and$(
// The client is connected to the sync loop
(
fromEvent(matrixRoom.client, ClientEvent.Sync) as Observable<
[SyncState]
>
).pipe(
startWith([matrixRoom.client.getSyncState()]),
map(([state]) => state === SyncState.Syncing),
),
// Room state observed by session says we're connected
fromEvent(matrixRTCSession, MembershipManagerEvent.StatusChanged).pipe(
startWith(null),
map(() => matrixRTCSession.membershipStatus === Status.Connected),
),
// Also watch out for warnings that we've likely hit a timeout and our
// delayed leave event is being sent (this condition is here because it
// provides an earlier warning than the sync loop timeout, and we wouldn't
// see the actual leave event until we reconnect to the sync loop)
fromEvent(matrixRTCSession, MembershipManagerEvent.ProbablyLeft).pipe(
startWith(null),
map(() => matrixRTCSession.probablyLeft !== true),
),
).pipe(
tap((connected) => {
logger.info(`Homeserver connected update: ${connected}`);
}),
),
);
// /**
// * Whether we are "fully" connected to the call. Accounts for both the
// * connection to the MatrixRTC session and the LiveKit publish connection.
// */
// // TODO use this in combination with the MemberState.
const connected$ = scope.behavior(
and$(
homeserverConnected$,
localConnection$.pipe(
switchMap((c) =>
c
? c.state$.pipe(map((state) => state.state === "ConnectedToLkRoom"))
: of(false),
),
),
),
);
const publisher$ = new BehaviorSubject<Publisher | null>(null);
localConnection$.subscribe((connection) => {
if (connection !== null && publisher$.value === null) {
// TODO looks strange to not change publisher if connection changes.
publisher$.next(
new Publisher(
scope,
connection,
mediaDevices,
muteStates,
trackProcessorState$,
),
);
}
});
combineLatest([publisher$, trackStartRequested$]).subscribe(
([publisher, shouldStartTracks]) => {
if (publisher && shouldStartTracks) {
publisher
.createAndSetupTracks()
.then((tracks) => {
tracks$.next(tracks);
})
.catch((error) => {
logger.error("Error creating tracks:", error);
});
}
},
);
// MATRIX RELATED
// /**
// * Whether we should tell the user that we're reconnecting to the call.
// */
// DISCUSSION is there a better way to do this?
// sth that is more deriectly implied from the membership manager of the js sdk. (fromEvent(matrixRTCSession, Reconnecting)) ??? or similar
const reconnecting$ = scope.behavior(
connected$.pipe(
// We are reconnecting if we previously had some successful initial
// connection but are now disconnected
scan(
({ connectedPreviously }, connectedNow) => ({
connectedPreviously: connectedPreviously || connectedNow,
reconnecting: connectedPreviously && !connectedNow,
}),
{ connectedPreviously: false, reconnecting: false },
),
map(({ reconnecting }) => reconnecting),
),
);
const startTracks = (): Behavior<LocalTrack[]> => {
trackStartRequested$.next(true);
return tracks$;
};
combineLatest([publisher$, tracks$]).subscribe(([publisher, tracks]) => {
if (
tracks.length === 0 ||
// change this to !== Publishing
state.livekit$.value.state !== LivekitState.Uninitialized
) {
return;
}
state.livekit$.next({ state: LivekitState.Connecting });
publisher
?.startPublishing()
.then(() => {
state.livekit$.next({ state: LivekitState.Connected });
})
.catch((error) => {
state.livekit$.next({ state: LivekitState.Error, error });
});
});
combineLatest([localTransport$, connectRequested$]).subscribe(
// TODO reconnect on options change.
([transport, connectRequested]) => {
if (
transport === null ||
!connectRequested ||
state.matrix$.value.state !== MatrixState.Disconnected
) {
logger.info(
"Not yet connecting because: ",
"transport === null:",
transport === null,
"!connectRequested:",
!connectRequested,
"state.matrix$.value.state !== MatrixState.Disconnected:",
state.matrix$.value.state !== MatrixState.Disconnected,
);
return;
}
state.matrix$.next({ state: MatrixState.Connecting });
logger.info("Matrix State connecting");
enterRTCSession(matrixRTCSession, transport, options.value).catch(
(error) => {
logger.error(error);
},
);
},
);
const requestConnect = (): LocalMemberConnectionState => {
trackStartRequested$.next(true);
connectRequested$.next(true);
return state;
};
const requestDisconnect = (): Behavior<LocalMemberLivekitState> | null => {
if (state.livekit$.value.state !== LivekitState.Connected) return null;
state.livekit$.next({ state: LivekitState.Disconnecting });
combineLatest([publisher$, tracks$], (publisher, tracks) => {
publisher
?.stopPublishing()
.then(() => {
tracks.forEach((track) => track.stop());
state.livekit$.next({ state: LivekitState.Disconnected });
})
.catch((error) => {
state.livekit$.next({ state: LivekitState.Error, error });
});
});
return state.livekit$;
};
// Pause upstream of all local media tracks when we're disconnected from
// MatrixRTC, because it can be an unpleasant surprise for the app to say
// 'reconnecting' and yet still be transmitting your media to others.
// We use matrixConnected$ rather than reconnecting$ because we want to
// pause tracks during the initial joining sequence too until we're sure
// that our own media is displayed on screen.
combineLatest([localConnection$, homeserverConnected$])
.pipe(scope.bind())
.subscribe(([connection, connected]) => {
if (connection?.state$.value.state !== "ConnectedToLkRoom") return;
const publications =
connection.livekitRoom.localParticipant.trackPublications.values();
if (connected) {
for (const p of publications) {
if (p.track?.isUpstreamPaused === true) {
const kind = p.track.kind;
logger.info(
`Resuming ${kind} track (MatrixRTC connection present)`,
);
p.track
.resumeUpstream()
.catch((e) =>
logger.error(
`Failed to resume ${kind} track after MatrixRTC reconnection`,
e,
),
);
}
}
} else {
for (const p of publications) {
if (p.track?.isUpstreamPaused === false) {
const kind = p.track.kind;
logger.info(
`Pausing ${kind} track (uncertain MatrixRTC connection)`,
);
p.track
.pauseUpstream()
.catch((e) =>
logger.error(
`Failed to pause ${kind} track after entering uncertain MatrixRTC connection`,
e,
),
);
}
}
}
});
const configError$ = new BehaviorSubject<ElementCallError | null>(null);
// TODO I do not fully understand what this does.
// Is it needed?
// Is this at the right place?
// Can this be simplified?
// Start and stop session membership as needed
scope.reconcile(localTransport$, async (advertised) => {
if (advertised !== null && advertised !== undefined) {
try {
await enterRTCSession(matrixRTCSession, advertised, options.value);
configError$.next(null);
} catch (e) {
logger.error("Error entering RTC session", e);
}
// Update our member event when our mute state changes.
const intentScope = new ObservableScope();
intentScope.reconcile(muteStates.video.enabled$, async (videoEnabled) =>
matrixRTCSession.updateCallIntent(videoEnabled ? "video" : "audio"),
);
return async (): Promise<void> => {
intentScope.end();
// Only sends Matrix leave event. The LiveKit session will disconnect
// as soon as either the stopConnection$ handler above gets to it or
// the view model is destroyed.
try {
await matrixRTCSession.leaveRoomSession();
} catch (e) {
logger.error("Error leaving RTC session", e);
}
try {
await widget?.api.transport.send(ElementWidgetActions.HangupCall, {});
} catch (e) {
logger.error("Failed to send hangup action", e);
}
};
}
});
localConnection$
.pipe(
distinctUntilChanged(),
switchMap((c) =>
c === null ? of({ state: "Initialized" } as ConnectionState) : c.state$,
),
map((s) => {
logger.trace(`Local connection state update: ${s.state}`);
if (s.state == "FailedToStart") {
return s.error instanceof ElementCallError
? s.error
: new UnknownCallError(s.error);
} else {
return null;
}
}),
scope.bind(),
)
.subscribe((fatalError) => {
configError$.next(fatalError);
});
/**
* Whether the user is currently sharing their screen.
*/
const sharingScreen$ = scope.behavior(
localConnection$.pipe(
switchMap((c) =>
c === null
? of(false)
: observeSharingScreen$(c.livekitRoom.localParticipant),
),
),
);
const toggleScreenSharing =
"getDisplayMedia" in (navigator.mediaDevices ?? {}) &&
!getUrlParams().hideScreensharing
? (): void =>
// If a connection is ready, toggle screen sharing.
// We deliberately do nothing in the case of a null connection because
// it looks nice for the call control buttons to all become available
// at once upon joining the call, rather than introducing a disabled
// state. The user can just click again.
// We also allow screen sharing to be toggled even if the connection
// is still initializing or publishing tracks, because there's no
// technical reason to disallow this. LiveKit will publish if it can.
void localConnection$.value?.livekitRoom.localParticipant
.setScreenShareEnabled(!sharingScreen$.value, {
audio: true,
selfBrowserSurface: "include",
surfaceSwitching: "include",
systemAudio: "include",
})
.catch(logger.error)
: null;
const participant$ = scope.behavior(
localConnection$.pipe(map((c) => c?.livekitRoom.localParticipant ?? null)),
);
return {
startTracks,
requestConnect,
requestDisconnect,
connectionState: state,
homeserverConnected$,
connected$,
reconnecting$,
configError$,
sharingScreen$,
toggleScreenSharing,
participant$,
connection$: localConnection$,
};
};
export function observeSharingScreen$(p: Participant): Observable<boolean> {
return observeParticipantEvents(
p,
ParticipantEvent.TrackPublished,
ParticipantEvent.TrackUnpublished,
ParticipantEvent.LocalTrackPublished,
ParticipantEvent.LocalTrackUnpublished,
).pipe(map((p) => p.isScreenShareEnabled));
}
interface EnterRTCSessionOptions {
encryptMedia: boolean;
matrixRTCMode: MatrixRTCMode;
}
/**
* TODO! document this function properly
* @param rtcSession
* @param transport
* @param options
* @throws If the widget could not send ElementWidgetActions.JoinCall action.
*/
// Exported for unit testing
export async function enterRTCSession(
rtcSession: MatrixRTCSession,
transport: LivekitTransport,
{ encryptMedia, matrixRTCMode }: EnterRTCSessionOptions,
): Promise<void> {
PosthogAnalytics.instance.eventCallEnded.cacheStartCall(new Date());
PosthogAnalytics.instance.eventCallStarted.track(rtcSession.room.roomId);
// This must be called before we start trying to join the call, as we need to
// have started tracking by the time calls start getting created.
// groupCallOTelMembership?.onJoinCall();
const { features, matrix_rtc_session: matrixRtcSessionConfig } = Config.get();
const useDeviceSessionMemberEvents =
features?.feature_use_device_session_member_events;
const { sendNotificationType: notificationType, callIntent } = getUrlParams();
const multiSFU = matrixRTCMode !== MatrixRTCMode.Legacy;
// Multi-sfu does not need a preferred foci list. just the focus that is actually used.
rtcSession.joinRoomSession(
multiSFU ? [] : [transport],
multiSFU ? transport : undefined,
{
notificationType,
callIntent,
manageMediaKeys: encryptMedia,
...(useDeviceSessionMemberEvents !== undefined && {
useLegacyMemberEvents: !useDeviceSessionMemberEvents,
}),
delayedLeaveEventRestartMs:
matrixRtcSessionConfig?.delayed_leave_event_restart_ms,
delayedLeaveEventDelayMs:
matrixRtcSessionConfig?.delayed_leave_event_delay_ms,
delayedLeaveEventRestartLocalTimeoutMs:
matrixRtcSessionConfig?.delayed_leave_event_restart_local_timeout_ms,
networkErrorRetryMs: matrixRtcSessionConfig?.network_error_retry_ms,
makeKeyDelay: matrixRtcSessionConfig?.wait_for_key_rotation_ms,
membershipEventExpiryMs:
matrixRtcSessionConfig?.membership_event_expiry_ms,
useExperimentalToDeviceTransport: true,
unstableSendStickyEvents: matrixRTCMode === MatrixRTCMode.Matrix_2_0,
},
);
if (widget) {
await widget.api.transport.send(ElementWidgetActions.JoinCall, {});
}
}

View File

@@ -0,0 +1,156 @@
/*
Copyright 2025 Element Creations Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
import { type CallMembership } from "matrix-js-sdk/lib/matrixrtc";
import { BehaviorSubject } from "rxjs";
import { mockConfig, flushPromises } from "../../../utils/test";
import { createLocalTransport$ } from "./LocalTransport";
import { constant } from "../../Behavior";
import { Epoch, ObservableScope } from "../../ObservableScope";
import {
MatrixRTCTransportMissingError,
FailToGetOpenIdToken,
} from "../../../utils/errors";
import * as openIDSFU from "../../../livekit/openIDSFU";
describe("LocalTransport", () => {
let scope: ObservableScope;
beforeEach(() => (scope = new ObservableScope()));
afterEach(() => scope.end());
it("throws if config is missing", async () => {
const localTransport$ = createLocalTransport$({
scope,
roomId: "!room:example.org",
useOldestMember$: constant(false),
memberships$: constant(new Epoch<CallMembership[]>([])),
client: {
getDomain: () => "",
// These won't be called in this error path but satisfy the type
getOpenIdToken: vi.fn(),
getDeviceId: vi.fn(),
},
});
await flushPromises();
expect(() => localTransport$.value).toThrow(
new MatrixRTCTransportMissingError(""),
);
});
it("throws FailToGetOpenIdToken when OpenID fetch fails", async () => {
// Provide a valid config so makeTransportInternal resolves a transport
const scope = new ObservableScope();
mockConfig({
livekit: { livekit_service_url: "https://lk.example.org" },
});
const resolver = Promise.withResolvers<void>();
vi.spyOn(openIDSFU, "getSFUConfigWithOpenID").mockImplementation(
async () => {
await resolver.promise;
throw new FailToGetOpenIdToken(new Error("no openid"));
},
);
const observations: unknown[] = [];
const errors: Error[] = [];
const localTransport$ = createLocalTransport$({
scope,
roomId: "!room:example.org",
useOldestMember$: constant(false),
memberships$: constant(new Epoch<CallMembership[]>([])),
client: {
// Use empty domain to skip .well-known and use config directly
getDomain: () => "",
getOpenIdToken: vi.fn(),
getDeviceId: vi.fn(),
},
});
localTransport$.subscribe(
(o) => observations.push(o),
(e) => errors.push(e),
);
resolver.resolve();
await flushPromises();
const expectedError = new FailToGetOpenIdToken(new Error("no openid"));
expect(observations).toStrictEqual([null]);
expect(errors).toStrictEqual([expectedError]);
expect(() => localTransport$.value).toThrow(expectedError);
});
it("emits preferred transport after OpenID resolves", async () => {
// Use config so transport discovery succeeds, but delay OpenID JWT fetch
mockConfig({
livekit: { livekit_service_url: "https://lk.example.org" },
});
const openIdResolver = Promise.withResolvers<openIDSFU.SFUConfig>();
vi.spyOn(openIDSFU, "getSFUConfigWithOpenID").mockReturnValue(
openIdResolver.promise,
);
const localTransport$ = createLocalTransport$({
scope,
roomId: "!room:example.org",
useOldestMember$: constant(false),
memberships$: constant(new Epoch<CallMembership[]>([])),
client: {
getDomain: () => "",
getOpenIdToken: vi.fn(),
getDeviceId: vi.fn(),
},
});
openIdResolver.resolve?.({ url: "https://lk.example.org", jwt: "jwt" });
expect(localTransport$.value).toBe(null);
await flushPromises();
// final
expect(localTransport$.value).toStrictEqual({
livekit_alias: "!room:example.org",
livekit_service_url: "https://lk.example.org",
type: "livekit",
});
});
it("updates local transport when oldest member changes", async () => {
// Use config so transport discovery succeeds, but delay OpenID JWT fetch
mockConfig({
livekit: { livekit_service_url: "https://lk.example.org" },
});
const memberships$ = new BehaviorSubject(new Epoch([]));
const openIdResolver = Promise.withResolvers<openIDSFU.SFUConfig>();
vi.spyOn(openIDSFU, "getSFUConfigWithOpenID").mockReturnValue(
openIdResolver.promise,
);
const localTransport$ = createLocalTransport$({
scope,
roomId: "!room:example.org",
useOldestMember$: constant(true),
memberships$,
client: {
getDomain: () => "",
getOpenIdToken: vi.fn(),
getDeviceId: vi.fn(),
},
});
openIdResolver.resolve?.({ url: "https://lk.example.org", jwt: "jwt" });
expect(localTransport$.value).toBe(null);
await flushPromises();
// final
expect(localTransport$.value).toStrictEqual({
livekit_alias: "!room:example.org",
livekit_service_url: "https://lk.example.org",
type: "livekit",
});
});
});

View File

@@ -1,5 +1,5 @@
/*
Copyright 2025 New Vector Ltd.
Copyright 2025 Element Creations Ltd.
SPDX-License-IdFentifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
@@ -13,32 +13,39 @@ import {
isLivekitTransportConfig,
} from "matrix-js-sdk/lib/matrixrtc";
import { type MatrixClient } from "matrix-js-sdk";
import { combineLatest, distinctUntilChanged, first, from, map } from "rxjs";
import { logger } from "matrix-js-sdk/lib/logger";
import {
combineLatest,
distinctUntilChanged,
first,
from,
map,
switchMap,
} from "rxjs";
import { logger as rootLogger } from "matrix-js-sdk/lib/logger";
import { AutoDiscovery } from "matrix-js-sdk/lib/autodiscovery";
import { type Behavior } from "../../Behavior.ts";
import { type Epoch, type ObservableScope } from "../../ObservableScope.ts";
import { Config } from "../../../config/Config.ts";
import { MatrixRTCTransportMissingError } from "../../../utils/errors.ts";
import { getSFUConfigWithOpenID } from "../../../livekit/openIDSFU.ts";
import {
getSFUConfigWithOpenID,
type OpenIDClientParts,
} from "../../../livekit/openIDSFU.ts";
import { areLivekitTransportsEqual } from "../remoteMembers/MatrixLivekitMembers.ts";
import { customLivekitUrl } from "../../../settings/settings.ts";
const logger = rootLogger.getChild("[LocalTransport]");
/*
* - get well known
* - get oldest membership
* - get transport to use
* - get openId + jwt token
* - wait for createTrack() call
* - create tracks
* - wait for join() call
* - Publisher.publishTracks()
* - send join state/sticky event
* It figures out “which LiveKit focus URL/alias the local user should use,”
* optionally aligning with the oldest member, and ensures the SFU path is primed
* before advertising that choice.
*/
interface Props {
scope: ObservableScope;
memberships$: Behavior<Epoch<CallMembership[]>>;
client: MatrixClient;
client: Pick<MatrixClient, "getDomain"> & OpenIDClientParts;
roomId: string;
useOldestMember$: Behavior<boolean>;
}
@@ -49,6 +56,8 @@ interface Props {
*
* @prop useOldestMember Whether to use the same transport as the oldest member.
* This will only update once the first oldest member appears. Will not recompute if the oldest member leaves.
*
* @throws MatrixRTCTransportMissingError | FailToGetOpenIdToken
*/
export const createLocalTransport$ = ({
scope,
@@ -75,16 +84,20 @@ export const createLocalTransport$ = ({
/**
* The transport that we would personally prefer to publish on (if not for the
* transport preferences of others, perhaps).
*
* @throws MatrixRTCTransportMissingError | FailToGetOpenIdToken
*/
const preferredTransport$: Behavior<LivekitTransport | null> = scope.behavior(
from(makeTransport(client, roomId)),
customLivekitUrl.value$.pipe(
switchMap((customUrl) => from(makeTransport(client, roomId, customUrl))),
),
null,
);
/**
* The transport we should advertise in our MatrixRTC membership.
* The chosen transport we should advertise in our MatrixRTC membership.
*/
const advertisedTransport$ = scope.behavior(
return scope.behavior(
combineLatest([
useOldestMember$,
oldestMemberTransport$,
@@ -98,82 +111,78 @@ export const createLocalTransport$ = ({
distinctUntilChanged(areLivekitTransportsEqual),
),
);
return advertisedTransport$;
};
const FOCI_WK_KEY = "org.matrix.msc4143.rtc_foci";
async function makeTransportInternal(
client: MatrixClient,
/**
*
* @param client
* @param roomId
* @returns
* @throws MatrixRTCTransportMissingError | FailToGetOpenIdToken
*/
async function makeTransport(
client: Pick<MatrixClient, "getDomain"> & OpenIDClientParts,
roomId: string,
urlFromDevSettings: string | null,
): Promise<LivekitTransport> {
logger.log("Searching for a preferred transport");
let transport: LivekitTransport | undefined;
logger.trace("Searching for a preferred transport");
//TODO refactor this to use the jwt service returned alias.
const livekitAlias = roomId;
// TODO-MULTI-SFU: Either remove this dev tool or make it more official
const urlFromStorage =
localStorage.getItem("robin-matrixrtc-auth") ??
localStorage.getItem("timo-focus-url");
if (urlFromStorage !== null) {
// DEVTOOL: Highest priority: Load from devtool setting
if (urlFromDevSettings !== null) {
const transportFromStorage: LivekitTransport = {
type: "livekit",
livekit_service_url: urlFromStorage,
livekit_service_url: urlFromDevSettings,
livekit_alias: livekitAlias,
};
logger.log(
"Using LiveKit transport from local storage: ",
logger.info(
"Using LiveKit transport from dev tools: ",
transportFromStorage,
);
return transportFromStorage;
transport = transportFromStorage;
}
// Prioritize the .well-known/matrix/client, if available, over the configured SFU
// WELL_KNOWN: Prioritize the .well-known/matrix/client, if available, over the configured SFU
const domain = client.getDomain();
if (domain) {
if (domain && transport === undefined) {
// we use AutoDiscovery instead of relying on the MatrixClient having already
// been fully configured and started
const wellKnownFoci = (await AutoDiscovery.getRawClientConfig(domain))?.[
FOCI_WK_KEY
];
if (Array.isArray(wellKnownFoci)) {
const transport: LivekitTransportConfig | undefined = wellKnownFoci.find(
(f) => f && isLivekitTransportConfig(f),
);
if (transport !== undefined) {
logger.log("Using LiveKit transport from .well-known: ", transport);
return { ...transport, livekit_alias: livekitAlias };
const wellKnownTransport: LivekitTransportConfig | undefined =
wellKnownFoci.find((f) => f && isLivekitTransportConfig(f));
if (wellKnownTransport !== undefined) {
logger.info("Using LiveKit transport from .well-known: ", transport);
transport = { ...wellKnownTransport, livekit_alias: livekitAlias };
}
}
}
// CONFIG: Least prioritized; Load from config file
const urlFromConf = Config.get().livekit?.livekit_service_url;
if (urlFromConf) {
if (urlFromConf && transport === undefined) {
const transportFromConf: LivekitTransport = {
type: "livekit",
livekit_service_url: urlFromConf,
livekit_alias: livekitAlias,
};
logger.log("Using LiveKit transport from config: ", transportFromConf);
return transportFromConf;
logger.info("Using LiveKit transport from config: ", transportFromConf);
transport = transportFromConf;
}
throw new MatrixRTCTransportMissingError(domain ?? "");
}
if (!transport) throw new MatrixRTCTransportMissingError(domain ?? ""); // this will call the jwt/sfu/get endpoint to pre create the livekit room.
await getSFUConfigWithOpenID(
client,
transport.livekit_service_url,
transport.livekit_alias,
);
async function makeTransport(
client: MatrixClient,
roomId: string,
): Promise<LivekitTransport> {
const transport = await makeTransportInternal(client, roomId);
// this will call the jwt/sfu/get endpoint to pre create the livekit room.
try {
await getSFUConfigWithOpenID(
client,
transport.livekit_service_url,
transport.livekit_alias,
);
} catch (e) {
logger.warn(`Failed to get SFU config for transport: ${e}`);
}
return transport;
}

View File

@@ -0,0 +1,360 @@
/*
Copyright 2025 Element Creations Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { afterEach, beforeEach, describe, expect, it, test, vi } from "vitest";
import {
ConnectionState as LivekitConnectionState,
LocalParticipant,
type LocalTrack,
type LocalTrackPublication,
ParticipantEvent,
Track,
} from "livekit-client";
import { BehaviorSubject } from "rxjs";
import { logger } from "matrix-js-sdk/lib/logger";
import { ObservableScope } from "../../ObservableScope";
import { constant } from "../../Behavior";
import {
flushPromises,
mockLivekitRoom,
mockMediaDevices,
} from "../../../utils/test";
import { Publisher } from "./Publisher";
import { type Connection } from "../remoteMembers/Connection";
import { type MuteStates } from "../../MuteStates";
let scope: ObservableScope;
beforeEach(() => {
scope = new ObservableScope();
});
afterEach(() => scope.end());
function createMockLocalTrack(source: Track.Source): LocalTrack {
const track = {
source,
isMuted: false,
isUpstreamPaused: false,
} as Partial<LocalTrack> as LocalTrack;
vi.mocked(track).mute = vi.fn().mockImplementation(() => {
track.isMuted = true;
});
vi.mocked(track).unmute = vi.fn().mockImplementation(() => {
track.isMuted = false;
});
vi.mocked(track).pauseUpstream = vi.fn().mockImplementation(() => {
// @ts-expect-error - for that test we want to set isUpstreamPaused directly
track.isUpstreamPaused = true;
});
vi.mocked(track).resumeUpstream = vi.fn().mockImplementation(() => {
// @ts-expect-error - for that test we want to set isUpstreamPaused directly
track.isUpstreamPaused = false;
});
return track;
}
function createMockMuteState(enabled$: BehaviorSubject<boolean>): {
enabled$: BehaviorSubject<boolean>;
setHandler: (h: (enabled: boolean) => void) => void;
unsetHandler: () => void;
} {
let currentHandler = (enabled: boolean): void => {};
const ms = {
enabled$,
setHandler: vi.fn().mockImplementation((h: (enabled: boolean) => void) => {
currentHandler = h;
}),
unsetHandler: vi.fn().mockImplementation(() => {
currentHandler = (enabled: boolean): void => {};
}),
};
// forward enabled$ emissions to the current handler
enabled$.subscribe((enabled) => {
logger.info(`MockMuteState: enabled changed to ${enabled}`);
currentHandler(enabled);
});
return ms;
}
let connection: Connection;
let muteStates: MuteStates;
let localParticipant: LocalParticipant;
let audioEnabled$: BehaviorSubject<boolean>;
let videoEnabled$: BehaviorSubject<boolean>;
let trackPublications: LocalTrackPublication[];
// use it to control when track creation resolves, default to resolved
let createTrackLock: Promise<void>;
beforeEach(() => {
trackPublications = [];
audioEnabled$ = new BehaviorSubject(false);
videoEnabled$ = new BehaviorSubject(false);
createTrackLock = Promise.resolve();
muteStates = {
audio: createMockMuteState(audioEnabled$),
video: createMockMuteState(videoEnabled$),
} as unknown as MuteStates;
const mockSendDataPacket = vi.fn();
const mockEngine = {
client: {
sendUpdateLocalMetadata: vi.fn(),
},
on: vi.fn().mockReturnThis(),
sendDataPacket: mockSendDataPacket,
};
localParticipant = new LocalParticipant(
"local-sid",
"local-identity",
// @ts-expect-error - for that test we want a real LocalParticipant to have the pending publications logic
mockEngine,
{
adaptiveStream: true,
dynacase: false,
audioCaptureDefaults: {},
videoCaptureDefaults: {},
stopLocalTrackOnUnpublish: true,
reconnectPolicy: "always",
disconnectOnPageLeave: true,
},
new Map(),
{},
);
vi.mocked(localParticipant).createTracks = vi
.fn()
.mockImplementation(async (opts) => {
const tracks: LocalTrack[] = [];
if (opts.audio) {
tracks.push(createMockLocalTrack(Track.Source.Microphone));
}
if (opts.video) {
tracks.push(createMockLocalTrack(Track.Source.Camera));
}
await createTrackLock;
return tracks;
});
vi.mocked(localParticipant).publishTrack = vi
.fn()
.mockImplementation(async (track: LocalTrack) => {
const pub = {
track,
source: track.source,
mute: track.mute,
unmute: track.unmute,
} as Partial<LocalTrackPublication> as LocalTrackPublication;
trackPublications.push(pub);
localParticipant.emit(ParticipantEvent.LocalTrackPublished, pub);
return Promise.resolve(pub);
});
vi.mocked(localParticipant).getTrackPublication = vi
.fn()
.mockImplementation((source: Track.Source) => {
return trackPublications.find((pub) => pub.track?.source === source);
});
connection = {
state$: constant({
state: "ConnectedToLkRoom",
livekitConnectionState$: constant(LivekitConnectionState.Connected),
}),
livekitRoom: mockLivekitRoom({
localParticipant: localParticipant,
}),
} as unknown as Connection;
});
describe("Publisher", () => {
let publisher: Publisher;
beforeEach(() => {
publisher = new Publisher(
scope,
connection,
mockMediaDevices({}),
muteStates,
constant({ supported: false, processor: undefined }),
logger,
);
});
afterEach(() => {});
it("Should not create tracks if started muted to avoid unneeded permission requests", async () => {
const createTracksSpy = vi.spyOn(
connection.livekitRoom.localParticipant,
"createTracks",
);
audioEnabled$.next(false);
videoEnabled$.next(false);
await publisher.createAndSetupTracks();
expect(createTracksSpy).not.toHaveBeenCalled();
});
it("Should minimize permission request by querying create at once", async () => {
const enableCameraAndMicrophoneSpy = vi.spyOn(
localParticipant,
"enableCameraAndMicrophone",
);
const createTracksSpy = vi.spyOn(localParticipant, "createTracks");
audioEnabled$.next(true);
videoEnabled$.next(true);
await publisher.createAndSetupTracks();
await flushPromises();
expect(enableCameraAndMicrophoneSpy).toHaveBeenCalled();
// It should create both at once
expect(createTracksSpy).toHaveBeenCalledWith({
audio: true,
video: true,
});
});
it("Ensure no data is streamed until publish has been called", async () => {
audioEnabled$.next(true);
await publisher.createAndSetupTracks();
// The track should be created and paused
expect(localParticipant.createTracks).toHaveBeenCalledWith({
audio: true,
video: undefined,
});
await flushPromises();
expect(localParticipant.publishTrack).toHaveBeenCalled();
await flushPromises();
const track = localParticipant.getTrackPublication(
Track.Source.Microphone,
)?.track;
expect(track).toBeDefined();
expect(track!.pauseUpstream).toHaveBeenCalled();
expect(track!.isUpstreamPaused).toBe(true);
});
it("Ensure resume upstream when published is called", async () => {
videoEnabled$.next(true);
await publisher.createAndSetupTracks();
// await flushPromises();
await publisher.startPublishing();
const track = localParticipant.getTrackPublication(
Track.Source.Camera,
)?.track;
expect(track).toBeDefined();
// expect(track.pauseUpstream).toHaveBeenCalled();
expect(track!.isUpstreamPaused).toBe(false);
});
describe("Mute states", () => {
let publisher: Publisher;
beforeEach(() => {
publisher = new Publisher(
scope,
connection,
mockMediaDevices({}),
muteStates,
constant({ supported: false, processor: undefined }),
logger,
);
});
test.each([
{ mutes: { audioEnabled: true, videoEnabled: false } },
{ mutes: { audioEnabled: true, videoEnabled: false } },
])("only create the tracks that are unmuted $mutes", async ({ mutes }) => {
// Ensure all muted
audioEnabled$.next(mutes.audioEnabled);
videoEnabled$.next(mutes.videoEnabled);
vi.mocked(connection.livekitRoom.localParticipant).createTracks = vi
.fn()
.mockResolvedValue([]);
await publisher.createAndSetupTracks();
expect(
connection.livekitRoom.localParticipant.createTracks,
).toHaveBeenCalledOnce();
expect(
connection.livekitRoom.localParticipant.createTracks,
).toHaveBeenCalledWith({
audio: mutes.audioEnabled ? true : undefined,
video: mutes.videoEnabled ? true : undefined,
});
});
});
it("does mute unmute audio", async () => {});
});
describe("Bug fix", () => {
// There is a race condition when creating and publishing tracks while the mute state changes.
// This race condition could cause tracks to be published even though they are muted at the
// beginning of a call coming from lobby.
// This is caused by our stack using manually the low level API to create and publish tracks,
// but also using the higher level setMicrophoneEnabled and setCameraEnabled functions that also create
// and publish tracks, and managing pending publications.
// Race is as follow, on creation of the Publisher we create the tracks then publish them.
// If in the middle of that process the mute state changes:
// - the `setMicrophoneEnabled` will be no-op because it is not aware of our created track and can't see any pending publication
// - If start publication is requested it will publish the track even though there was a mute request.
it("wrongly publish tracks while muted", async () => {
// setLogLevel(`debug`);
const publisher = new Publisher(
scope,
connection,
mockMediaDevices({}),
muteStates,
constant({ supported: false, processor: undefined }),
logger,
);
audioEnabled$.next(true);
const resolvers = Promise.withResolvers<void>();
createTrackLock = resolvers.promise;
// Initially the audio is unmuted, so creating tracks should publish the audio track
const createTracks = publisher.createAndSetupTracks();
void publisher.startPublishing();
void createTracks.then(() => {
void publisher.startPublishing();
});
// now mute the audio before allowing track creation to complete
audioEnabled$.next(false);
resolvers.resolve(undefined);
await createTracks;
await flushPromises();
const track = localParticipant.getTrackPublication(
Track.Source.Microphone,
)?.track;
expect(track).toBeDefined();
try {
expect(localParticipant.publishTrack).not.toHaveBeenCalled();
} catch {
expect(track!.mute).toHaveBeenCalled();
expect(track!.isMuted).toBe(true);
}
});
});

View File

@@ -1,16 +1,17 @@
/*
Copyright 2025 Element Creations Ltd.
Copyright 2025 New Vector Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import {
ConnectionState as LivekitConnectionState,
type LocalTrackPublication,
LocalVideoTrack,
ParticipantEvent,
type Room as LivekitRoom,
Track,
type LocalTrack,
type LocalTrackPublication,
ConnectionState as LivekitConnectionState,
} from "livekit-client";
import {
map,
@@ -39,31 +40,36 @@ import { type ObservableScope } from "../../ObservableScope.ts";
* The Publisher is also responsible for creating the media tracks.
*/
export class Publisher {
public tracks: LocalTrack<Track.Kind>[] = [];
/**
* By default, livekit will start publishing tracks as soon as they are created.
* In the matrix RTC world, we want to control when tracks are published based
* on whether the user is part of the RTC session or not.
*/
public shouldPublish = false;
/**
* Creates a new Publisher.
* @param scope - The observable scope to use for managing the publisher.
* @param connection - The connection to use for publishing.
* @param devices - The media devices to use for audio and video input.
* @param muteStates - The mute states for audio and video.
* @param e2eeLivekitOptions - The E2EE options to use for the LiveKit room. Use to share the same key provider across connections!.
* @param trackerProcessorState$ - The processor state for the video track processor (e.g. background blur).
* @param logger - The logger to use for logging :D.
*/
public constructor(
private scope: ObservableScope,
private connection: Connection,
private connection: Pick<Connection, "livekitRoom" | "state$">, //setE2EEEnabled,
devices: MediaDevices,
private readonly muteStates: MuteStates,
trackerProcessorState$: Behavior<ProcessorState>,
private logger?: Logger,
private logger: Logger,
) {
this.logger?.info("[PublishConnection] Create LiveKit room");
const { controlledAudioDevices } = getUrlParams();
const room = connection.livekitRoom;
room.setE2EEEnabled(room.options.e2ee !== undefined)?.catch((e: Error) => {
this.logger?.error("Failed to set E2EE enabled on room", e);
this.logger.error("Failed to set E2EE enabled on room", e);
});
// Setup track processor syncing (blur)
@@ -73,117 +79,187 @@ export class Publisher {
this.workaroundRestartAudioInputTrackChrome(devices, scope);
this.scope.onEnd(() => {
this.logger?.info(
"[PublishConnection] Scope ended -> stop publishing all tracks",
);
this.logger.info("Scope ended -> stop publishing all tracks");
void this.stopPublishing();
muteStates.audio.unsetHandler();
muteStates.video.unsetHandler();
});
this.connection.livekitRoom.localParticipant.on(
ParticipantEvent.LocalTrackPublished,
this.onLocalTrackPublished.bind(this),
);
}
// LiveKit will publish the tracks as soon as they are created
// but we want to control when tracks are published.
// We cannot just mute the tracks, even if this will effectively stop the publishing,
// it would also prevent the user from seeing their own video/audio preview.
// So for that we use pauseUpStream(): Stops sending media to the server by replacing
// the sender track with null, but keeps the local MediaStreamTrack active.
// The user can still see/hear themselves locally, but remote participants see nothing.
private onLocalTrackPublished(
localTrackPublication: LocalTrackPublication,
): void {
this.logger.info("Local track published", localTrackPublication);
const lkRoom = this.connection.livekitRoom;
if (!this.shouldPublish) {
this.pauseUpstreams(lkRoom, [localTrackPublication.source]).catch((e) => {
this.logger.error(`Failed to pause upstreams`, e);
});
}
// also check the mute state and apply it
if (localTrackPublication.source === Track.Source.Microphone) {
const enabled = this.muteStates.audio.enabled$.value;
lkRoom.localParticipant.setMicrophoneEnabled(enabled).catch((e) => {
this.logger.error(
`Failed to enable microphone track, enabled:${enabled}`,
e,
);
});
} else if (localTrackPublication.source === Track.Source.Camera) {
const enabled = this.muteStates.video.enabled$.value;
lkRoom.localParticipant.setCameraEnabled(enabled).catch((e) => {
this.logger.error(
`Failed to enable camera track, enabled:${enabled}`,
e,
);
});
}
}
/**
* Start the connection to LiveKit and publish local tracks.
* Create and setup local audio and video tracks based on the current mute states.
* It creates the tracks only if audio and/or video is enabled, to avoid unnecessary
* permission prompts.
*
* This will:
* wait for the connection to be ready.
// * 1. Request an OpenId token `request_token` (allows matrix users to verify their identity with a third-party service.)
// * 2. Use this token to request the SFU config to the MatrixRtc authentication service.
// * 3. Connect to the configured LiveKit room.
// * 4. Create local audio and video tracks based on the current mute states and publish them to the room.
* It also observes mute state changes to update LiveKit microphone/camera states accordingly.
* If a track is not created initially because disabled, it will be created when unmuting.
*
* This call is not blocking anymore, instead callers can listen to the
* `RoomEvent.MediaDevicesError` event in the LiveKit room to be notified of any errors.
*
* @throws {InsufficientCapacityError} if the LiveKit server indicates that it has insufficient capacity to accept the connection.
* @throws {SFURoomCreationRestrictedError} if the LiveKit server indicates that the room does not exist and cannot be created.
*/
public async createAndSetupTracks(): Promise<LocalTrack[]> {
public async createAndSetupTracks(): Promise<void> {
this.logger.debug("createAndSetupTracks called");
const lkRoom = this.connection.livekitRoom;
// Observe mute state changes and update LiveKit microphone/camera states accordingly
this.observeMuteStates(this.scope);
// TODO: This should be an autostarted connection no need to start here. just check the connection state.
// TODO: This will fetch the JWT token. Perhaps we could keep it preloaded
// instead? This optimization would only be safe for a publish connection,
// because we don't want to leak the user's intent to perhaps join a call to
// remote servers before they actually commit to it.
// const { promise, resolve, reject } = Promise.withResolvers<void>();
// const sub = this.connection.state$.subscribe((s) => {
// if (s.state === "FailedToStart") {
// reject(new Error("Disconnected from LiveKit server"));
// } else if (s.state === "ConnectedToLkRoom") {
// resolve();
// }
// });
// try {
// await promise;
// } catch (e) {
// throw e;
// } finally {
// sub.unsubscribe();
// }
// TODO-MULTI-SFU: Prepublish a microphone track
// Check if audio and/or video is enabled. We only create tracks if enabled,
// because it could prompt for permission, and we don't want to do that unnecessarily.
const audio = this.muteStates.audio.enabled$.value;
const video = this.muteStates.video.enabled$.value;
// createTracks throws if called with audio=false and video=false
if (audio || video) {
// TODO this can still throw errors? It will also prompt for permissions if not already granted
this.tracks =
(await lkRoom.localParticipant
.createTracks({
audio,
video,
})
.catch((error) => {
this.logger?.error("Failed to create tracks", error);
})) ?? [];
// We don't await the creation, because livekit could block until the tracks
// are fully published, and not only that they are created.
// We don't have control on that, localParticipant creates and publishes the tracks
// asap.
// We are using the `ParticipantEvent.LocalTrackPublished` to be notified
// when tracks are actually published, and at that point
// we can pause upstream if needed (depending on if startPublishing has been called).
if (audio && video) {
// Enable both at once in order to have a single permission prompt!
void lkRoom.localParticipant.enableCameraAndMicrophone();
} else if (audio) {
void lkRoom.localParticipant.setMicrophoneEnabled(true);
} else if (video) {
void lkRoom.localParticipant.setCameraEnabled(true);
}
return this.tracks;
return Promise.resolve();
}
public async startPublishing(): Promise<LocalTrack[]> {
const lkRoom = this.connection.livekitRoom;
const { promise, resolve, reject } = Promise.withResolvers<void>();
const sub = this.connection.state$.subscribe((s) => {
switch (s.state) {
case "ConnectedToLkRoom":
resolve();
break;
case "FailedToStart":
reject(new Error("Failed to connect to LiveKit server"));
break;
default:
this.logger?.info("waiting for connection: ", s.state);
private async pauseUpstreams(
lkRoom: LivekitRoom,
sources: Track.Source[],
): Promise<void> {
for (const source of sources) {
const track = lkRoom.localParticipant.getTrackPublication(source)?.track;
if (track) {
await track.pauseUpstream();
} else {
this.logger.warn(
`No track found for source ${source} to pause upstream`,
);
}
});
try {
await promise;
} catch (e) {
throw e;
} finally {
sub.unsubscribe();
}
for (const track of this.tracks) {
// TODO: handle errors? Needs the signaling connection to be up, but it has some retries internally
// with a timeout.
await lkRoom.localParticipant.publishTrack(track).catch((error) => {
this.logger?.error("Failed to publish track", error);
});
}
// TODO: check if the connection is still active? and break the loop if not?
private async resumeUpstreams(
lkRoom: LivekitRoom,
sources: Track.Source[],
): Promise<void> {
for (const source of sources) {
const track = lkRoom.localParticipant.getTrackPublication(source)?.track;
if (track) {
await track.resumeUpstream();
} else {
this.logger.warn(
`No track found for source ${source} to resume upstream`,
);
}
}
}
/**
*
* Request to publish local tracks to the LiveKit room.
* This will wait for the connection to be ready before publishing.
* Livekit also have some local retry logic for publishing tracks.
* Can be called multiple times, localparticipant manages the state of published tracks (or pending publications).
*
* @returns
*/
public async startPublishing(): Promise<void> {
if (this.shouldPublish) {
this.logger.debug(`Already publishing, ignoring startPublishing call`);
return;
}
this.shouldPublish = true;
this.logger.debug("startPublishing called");
const lkRoom = this.connection.livekitRoom;
// Resume upstream for both audio and video tracks
// We need to call it explicitly because call setTrackEnabled does not always
// resume upstream. It will only if you switch the track from disabled to enabled,
// but if the track is already enabled but upstream is paused, it won't resume it.
// TODO what about screen share?
try {
await this.resumeUpstreams(lkRoom, [
Track.Source.Microphone,
Track.Source.Camera,
]);
} catch (e) {
this.logger.error(`Failed to resume upstreams`, e);
}
return this.tracks;
}
public async stopPublishing(): Promise<void> {
// TODO-MULTI-SFU: Move these calls back to ObservableScope.onEnd once scope
// actually has the right lifetime
this.muteStates.audio.unsetHandler();
this.muteStates.video.unsetHandler();
this.logger.debug("stopPublishing called");
this.shouldPublish = false;
// Pause upstream will stop sending media to the server, while keeping
// the local MediaStreamTrack active, so the user can still see themselves.
await this.pauseUpstreams(this.connection.livekitRoom, [
Track.Source.Microphone,
Track.Source.Camera,
Track.Source.ScreenShare,
]);
}
const localParticipant = this.connection.livekitRoom.localParticipant;
const tracks: LocalTrack[] = [];
const addToTracksIfDefined = (p: LocalTrackPublication): void => {
if (p.track !== undefined) tracks.push(p.track);
};
localParticipant.trackPublications.forEach(addToTracksIfDefined);
await localParticipant.unpublishTracks(tracks);
public async stopTracks(): Promise<void> {
const lkRoom = this.connection.livekitRoom;
for (const source of [
Track.Source.Microphone,
Track.Source.Camera,
Track.Source.ScreenShare,
]) {
const localPub = lkRoom.localParticipant.getTrackPublication(source);
if (localPub?.track) {
// stops and unpublishes the track
await lkRoom.localParticipant.unpublishTrack(localPub!.track, true);
}
}
}
/// Private methods
@@ -220,6 +296,9 @@ export class Publisher {
// the process of being restarted.
activeMicTrack.mediaStreamTrack.readyState !== "ended"
) {
this.logger?.info(
"Restarting audio device track due to active media device changed (workaroundRestartAudioInputTrackChrome)",
);
// Restart the track, which will cause Livekit to do another
// getUserMedia() call with deviceId: default to get the *new* default device.
// Note that room.switchActiveDevice() won't work: Livekit will ignore it because
@@ -228,7 +307,7 @@ export class Publisher {
.getTrackPublication(Track.Source.Microphone)
?.audioTrack?.restartTrack()
.catch((e) => {
this.logger?.error(`Failed to restart audio device track`, e);
this.logger.error(`Failed to restart audio device track`, e);
});
}
});
@@ -248,7 +327,7 @@ export class Publisher {
selected$.pipe(scope.bind()).subscribe((device) => {
if (lkRoom.state != LivekitConnectionState.Connected) return;
// if (this.connectionState$.value !== ConnectionState.Connected) return;
this.logger?.info(
this.logger.info(
"[LivekitRoom] syncDevice room.getActiveDevice(kind) !== d.id :",
lkRoom.getActiveDevice(kind),
" !== ",
@@ -261,7 +340,7 @@ export class Publisher {
lkRoom
.switchActiveDevice(kind, device.id)
.catch((e: Error) =>
this.logger?.error(
this.logger.error(
`Failed to sync ${kind} device with LiveKit`,
e,
),
@@ -282,25 +361,33 @@ export class Publisher {
*/
private observeMuteStates(scope: ObservableScope): void {
const lkRoom = this.connection.livekitRoom;
this.muteStates.audio.setHandler(async (desired) => {
this.muteStates.audio.setHandler(async (enable) => {
try {
await lkRoom.localParticipant.setMicrophoneEnabled(desired);
} catch (e) {
this.logger?.error(
"Failed to update LiveKit audio input mute state",
e,
this.logger.debug(
`handler: Setting LiveKit microphone enabled: ${enable}`,
);
await lkRoom.localParticipant.setMicrophoneEnabled(enable);
// Unmute will restart the track if it was paused upstream,
// but until explicitly requested, we want to keep it paused.
if (!this.shouldPublish && enable) {
await this.pauseUpstreams(lkRoom, [Track.Source.Microphone]);
}
} catch (e) {
this.logger.error("Failed to update LiveKit audio input mute state", e);
}
return lkRoom.localParticipant.isMicrophoneEnabled;
});
this.muteStates.video.setHandler(async (desired) => {
this.muteStates.video.setHandler(async (enable) => {
try {
await lkRoom.localParticipant.setCameraEnabled(desired);
this.logger.debug(`handler: Setting LiveKit camera enabled: ${enable}`);
await lkRoom.localParticipant.setCameraEnabled(enable);
// Unmute will restart the track if it was paused upstream,
// but until explicitly requested, we want to keep it paused.
if (!this.shouldPublish && enable) {
await this.pauseUpstreams(lkRoom, [Track.Source.Camera]);
}
} catch (e) {
this.logger?.error(
"Failed to update LiveKit video input mute state",
e,
);
this.logger.error("Failed to update LiveKit video input mute state", e);
}
return lkRoom.localParticipant.isCameraEnabled;
});

View File

@@ -1,4 +1,5 @@
/*
Copyright 2025 Element Creations Ltd.
Copyright 2025 New Vector Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
@@ -29,13 +30,16 @@ import { logger } from "matrix-js-sdk/lib/logger";
import type { LivekitTransport } from "matrix-js-sdk/lib/matrixrtc";
import {
Connection,
ConnectionState,
type ConnectionOpts,
type ConnectionState,
type PublishingParticipant,
} from "./Connection.ts";
import { ObservableScope } from "../../ObservableScope.ts";
import { type OpenIDClientParts } from "../../../livekit/openIDSFU.ts";
import { FailToGetOpenIdToken } from "../../../utils/errors.ts";
import {
ElementCallError,
FailToGetOpenIdToken,
} from "../../../utils/errors.ts";
let testScope: ObservableScope;
@@ -46,11 +50,6 @@ let fakeLivekitRoom: MockedObject<LivekitRoom>;
let localParticipantEventEmiter: EventEmitter;
let fakeLocalParticipant: MockedObject<LocalParticipant>;
let fakeRoomEventEmiter: EventEmitter;
// let fakeMembershipsFocusMap$: BehaviorSubject<
// { membership: CallMembership; transport: LivekitTransport }[]
// >;
const livekitFocus: LivekitTransport = {
livekit_alias: "!roomID:example.org",
livekit_service_url: "https://matrix-rtc.example.org/livekit/jwt",
@@ -87,22 +86,25 @@ function setupTest(): void {
localParticipantEventEmiter,
),
} as unknown as LocalParticipant);
fakeRoomEventEmiter = new EventEmitter();
const fakeRoomEventEmitter = new EventEmitter();
fakeLivekitRoom = vi.mocked<LivekitRoom>({
connect: vi.fn(),
disconnect: vi.fn(),
remoteParticipants: new Map(),
localParticipant: fakeLocalParticipant,
state: LivekitConnectionState.Disconnected,
on: fakeRoomEventEmiter.on.bind(fakeRoomEventEmiter),
off: fakeRoomEventEmiter.off.bind(fakeRoomEventEmiter),
addListener: fakeRoomEventEmiter.addListener.bind(fakeRoomEventEmiter),
on: fakeRoomEventEmitter.on.bind(fakeRoomEventEmitter),
off: fakeRoomEventEmitter.off.bind(fakeRoomEventEmitter),
addListener: fakeRoomEventEmitter.addListener.bind(fakeRoomEventEmitter),
removeListener:
fakeRoomEventEmiter.removeListener.bind(fakeRoomEventEmiter),
fakeRoomEventEmitter.removeListener.bind(fakeRoomEventEmitter),
removeAllListeners:
fakeRoomEventEmiter.removeAllListeners.bind(fakeRoomEventEmiter),
fakeRoomEventEmitter.removeAllListeners.bind(fakeRoomEventEmitter),
setE2EEEnabled: vi.fn().mockResolvedValue(undefined),
emit: (eventName: string | symbol, ...args: unknown[]) => {
fakeRoomEventEmitter.emit(eventName, ...args);
},
} as unknown as LivekitRoom);
}
@@ -124,7 +126,16 @@ function setupRemoteConnection(): Connection {
};
});
fakeLivekitRoom.connect.mockResolvedValue(undefined);
fakeLivekitRoom.connect.mockImplementation(async (): Promise<void> => {
const changeEv = RoomEvent.ConnectionStateChanged;
fakeLivekitRoom.state = LivekitConnectionState.Connecting;
fakeLivekitRoom.emit(changeEv, fakeLivekitRoom.state);
fakeLivekitRoom.state = LivekitConnectionState.Connected;
fakeLivekitRoom.emit(changeEv, fakeLivekitRoom.state);
return Promise.resolve();
});
return new Connection(opts, logger);
}
@@ -147,7 +158,7 @@ describe("Start connection states", () => {
};
const connection = new Connection(opts, logger);
expect(connection.state$.getValue().state).toEqual("Initialized");
expect(connection.state$.getValue()).toEqual("Initialized");
});
it("fail to getOpenId token then error state", async () => {
@@ -163,7 +174,7 @@ describe("Start connection states", () => {
const connection = new Connection(opts, logger);
const capturedStates: ConnectionState[] = [];
const capturedStates: (ConnectionState | Error)[] = [];
const s = connection.state$.subscribe((value) => {
capturedStates.push(value);
});
@@ -183,22 +194,20 @@ describe("Start connection states", () => {
let capturedState = capturedStates.pop();
expect(capturedState).toBeDefined();
expect(capturedState!.state).toEqual("FetchingConfig");
expect(capturedState!).toEqual("FetchingConfig");
deferred.reject(new FailToGetOpenIdToken(new Error("Failed to get token")));
await vi.runAllTimersAsync();
capturedState = capturedStates.pop();
if (capturedState!.state === "FailedToStart") {
expect(capturedState!.error.message).toEqual("Something went wrong");
expect(capturedState!.transport.livekit_alias).toEqual(
if (capturedState instanceof Error) {
expect(capturedState.message).toEqual("Something went wrong");
expect(connection.transport.livekit_alias).toEqual(
livekitFocus.livekit_alias,
);
} else {
expect.fail(
"Expected FailedToStart state but got " + capturedState?.state,
);
expect.fail("Expected FailedToStart state but got " + capturedState);
}
});
@@ -215,7 +224,7 @@ describe("Start connection states", () => {
const connection = new Connection(opts, logger);
const capturedStates: ConnectionState[] = [];
const capturedStates: (ConnectionState | Error)[] = [];
const s = connection.state$.subscribe((value) => {
capturedStates.push(value);
});
@@ -237,24 +246,25 @@ describe("Start connection states", () => {
let capturedState = capturedStates.pop();
expect(capturedState).toBeDefined();
expect(capturedState?.state).toEqual("FetchingConfig");
expect(capturedState).toEqual(ConnectionState.FetchingConfig);
deferredSFU.resolve();
await vi.runAllTimersAsync();
capturedState = capturedStates.pop();
if (capturedState?.state === "FailedToStart") {
expect(capturedState?.error.message).toContain(
if (
capturedState instanceof ElementCallError &&
capturedState.cause instanceof Error
) {
expect(capturedState.cause.message).toContain(
"SFU Config fetch failed with exception Error",
);
expect(capturedState?.transport.livekit_alias).toEqual(
expect(connection.transport.livekit_alias).toEqual(
livekitFocus.livekit_alias,
);
} else {
expect.fail(
"Expected FailedToStart state but got " + capturedState?.state,
);
expect.fail("Expected FailedToStart state but got " + capturedState);
}
});
@@ -271,7 +281,7 @@ describe("Start connection states", () => {
const connection = new Connection(opts, logger);
const capturedStates: ConnectionState[] = [];
const capturedStates: (ConnectionState | Error)[] = [];
const s = connection.state$.subscribe((value) => {
capturedStates.push(value);
});
@@ -301,18 +311,21 @@ describe("Start connection states", () => {
let capturedState = capturedStates.pop();
expect(capturedState).toBeDefined();
expect(capturedState?.state).toEqual("FetchingConfig");
expect(capturedState).toEqual(ConnectionState.FetchingConfig);
deferredSFU.resolve();
await vi.runAllTimersAsync();
capturedState = capturedStates.pop();
if (capturedState && capturedState?.state === "FailedToStart") {
expect(capturedState.error.message).toContain(
if (
capturedState instanceof ElementCallError &&
capturedState.cause instanceof Error
) {
expect(capturedState.cause.message).toContain(
"Failed to connect to livekit",
);
expect(capturedState.transport.livekit_alias).toEqual(
expect(connection.transport.livekit_alias).toEqual(
livekitFocus.livekit_alias,
);
} else {
@@ -328,7 +341,7 @@ describe("Start connection states", () => {
const connection = setupRemoteConnection();
const capturedStates: ConnectionState[] = [];
const capturedStates: (ConnectionState | Error)[] = [];
const s = connection.state$.subscribe((value) => {
capturedStates.push(value);
});
@@ -338,13 +351,15 @@ describe("Start connection states", () => {
await vi.runAllTimersAsync();
const initialState = capturedStates.shift();
expect(initialState?.state).toEqual("Initialized");
expect(initialState).toEqual(ConnectionState.Initialized);
const fetchingState = capturedStates.shift();
expect(fetchingState?.state).toEqual("FetchingConfig");
expect(fetchingState).toEqual(ConnectionState.FetchingConfig);
const disconnectedState = capturedStates.shift();
expect(disconnectedState).toEqual(ConnectionState.LivekitDisconnected);
const connectingState = capturedStates.shift();
expect(connectingState?.state).toEqual("ConnectingToLkRoom");
expect(connectingState).toEqual(ConnectionState.LivekitConnecting);
const connectedState = capturedStates.shift();
expect(connectedState?.state).toEqual("ConnectedToLkRoom");
expect(connectedState).toEqual(ConnectionState.LivekitConnected);
});
it("shutting down the scope should stop the connection", async () => {
@@ -410,7 +425,7 @@ describe("Publishing participants observations", () => {
);
participants.forEach((p) =>
fakeRoomEventEmiter.emit(RoomEvent.ParticipantConnected, p),
fakeLivekitRoom.emit(RoomEvent.ParticipantConnected, p),
);
// At this point there should be no publishers
@@ -423,7 +438,7 @@ describe("Publishing participants observations", () => {
fakeRemoteLivekitParticipant("@dan:example.org:DEV333", 2),
];
participants.forEach((p) =>
fakeRoomEventEmiter.emit(RoomEvent.ParticipantConnected, p),
fakeLivekitRoom.emit(RoomEvent.ParticipantConnected, p),
);
// At this point there should be no publishers
@@ -453,7 +468,7 @@ describe("Publishing participants observations", () => {
);
for (const participant of participants) {
fakeRoomEventEmiter.emit(RoomEvent.ParticipantConnected, participant);
fakeLivekitRoom.emit(RoomEvent.ParticipantConnected, participant);
}
// At this point there should be no publishers
@@ -462,7 +477,7 @@ describe("Publishing participants observations", () => {
participants = [fakeRemoteLivekitParticipant("@bob:example.org:DEV111", 1)];
for (const participant of participants) {
fakeRoomEventEmiter.emit(RoomEvent.ParticipantConnected, participant);
fakeLivekitRoom.emit(RoomEvent.ParticipantConnected, participant);
}
// We should have bob has a publisher now
@@ -479,7 +494,7 @@ describe("Publishing participants observations", () => {
(p) => p.identity !== "@bob:example.org:DEV111",
);
fakeRoomEventEmiter.emit(
fakeLivekitRoom.emit(
RoomEvent.ParticipantDisconnected,
fakeRemoteLivekitParticipant("@bob:example.org:DEV111"),
);

View File

@@ -1,4 +1,5 @@
/*
Copyright 2025 Element Creations Ltd.
Copyright 2025 New Vector Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
@@ -11,14 +12,13 @@ import {
} from "@livekit/components-core";
import {
ConnectionError,
type ConnectionState as LivekitConenctionState,
type Room as LivekitRoom,
type LocalParticipant,
type RemoteParticipant,
RoomEvent,
} from "livekit-client";
import { type LivekitTransport } from "matrix-js-sdk/lib/matrixrtc";
import { BehaviorSubject, map, type Observable } from "rxjs";
import { BehaviorSubject, map } from "rxjs";
import { type Logger } from "matrix-js-sdk/lib/logger";
import {
@@ -29,8 +29,10 @@ import {
import { type Behavior } from "../../Behavior.ts";
import { type ObservableScope } from "../../ObservableScope.ts";
import {
ElementCallError,
InsufficientCapacityError,
SFURoomCreationRestrictedError,
UnknownCallError,
} from "../../../utils/errors.ts";
export type PublishingParticipant = LocalParticipant | RemoteParticipant;
@@ -46,19 +48,30 @@ export interface ConnectionOpts {
/** Optional factory to create the LiveKit room, mainly for testing purposes. */
livekitRoomFactory: () => LivekitRoom;
}
export class FailedToStartError extends Error {
public constructor(message: string) {
super(message);
this.name = "FailedToStartError";
}
}
export type ConnectionState =
| { state: "Initialized" }
| { state: "FetchingConfig"; transport: LivekitTransport }
| { state: "ConnectingToLkRoom"; transport: LivekitTransport }
| { state: "PublishingTracks"; transport: LivekitTransport }
| { state: "FailedToStart"; error: Error; transport: LivekitTransport }
| {
state: "ConnectedToLkRoom";
livekitConnectionState$: Observable<LivekitConenctionState>;
transport: LivekitTransport;
}
| { state: "Stopped"; transport: LivekitTransport };
export enum ConnectionState {
/** The start state of a connection. It has been created but nothing has loaded yet. */
Initialized = "Initialized",
/** `start` has been called on the connection. It aquires the jwt info to conenct to the LK Room */
FetchingConfig = "FetchingConfig",
Stopped = "Stopped",
/** The same as ConnectionState.Disconnected from `livekit-client` */
LivekitDisconnected = "disconnected",
/** The same as ConnectionState.Connecting from `livekit-client` */
LivekitConnecting = "connecting",
/** The same as ConnectionState.Connected from `livekit-client` */
LivekitConnected = "connected",
/** The same as ConnectionState.Reconnecting from `livekit-client` */
LivekitReconnecting = "reconnecting",
/** The same as ConnectionState.SignalReconnecting from `livekit-client` */
LivekitSignalReconnecting = "signalReconnecting",
}
/**
* A connection to a Matrix RTC LiveKit backend.
@@ -67,14 +80,32 @@ export type ConnectionState =
*/
export class Connection {
// Private Behavior
private readonly _state$ = new BehaviorSubject<ConnectionState>({
state: "Initialized",
});
private readonly _state$ = new BehaviorSubject<
ConnectionState | ElementCallError
>(ConnectionState.Initialized);
/**
* The current state of the connection to the media transport.
*/
public readonly state$: Behavior<ConnectionState> = this._state$;
public readonly state$: Behavior<ConnectionState | Error> = this._state$;
/**
* The media transport to connect to.
*/
public readonly transport: LivekitTransport;
public readonly livekitRoom: LivekitRoom;
private scope: ObservableScope;
/**
* An observable of the participants that are publishing on this connection. (Excluding our local participant)
* This is derived from `participantsIncludingSubscribers$` and `remoteTransports$`.
* It filters the participants to only those that are associated with a membership that claims to publish on this connection.
*/
public readonly remoteParticipantsWithTracks$: Behavior<
PublishingParticipant[]
>;
/**
* Whether the connection has been stopped.
@@ -95,24 +126,29 @@ export class Connection {
* @throws {InsufficientCapacityError} if the LiveKit server indicates that it has insufficient capacity to accept the connection.
* @throws {SFURoomCreationRestrictedError} if the LiveKit server indicates that the room does not exist and cannot be created.
*/
// TODO dont make this throw and instead store a connection error state in this class?
// TODO consider an autostart pattern...
public async start(): Promise<void> {
this.logger.debug("Starting Connection");
this.stopped = false;
try {
this._state$.next({
state: "FetchingConfig",
transport: this.transport,
});
this._state$.next(ConnectionState.FetchingConfig);
// We should already have this information after creating the localTransport.
// It would probably be better to forward this here.
const { url, jwt } = await this.getSFUConfigWithOpenID();
// If we were stopped while fetching the config, don't proceed to connect
if (this.stopped) return;
this._state$.next({
state: "ConnectingToLkRoom",
transport: this.transport,
});
// Setup observer once we are done with getSFUConfigWithOpenID
connectionStateObserver(this.livekitRoom)
.pipe(
this.scope.bind(),
map((s) => s as unknown as ConnectionState),
)
.subscribe((lkState) => {
// It is save to cast lkState to ConnectionState as they are fully overlapping.
this._state$.next(lkState);
});
try {
await this.livekitRoom.connect(url, jwt);
} catch (e) {
@@ -127,7 +163,8 @@ export class Connection {
throw new InsufficientCapacityError();
}
if (e.status === 404) {
// error msg is "Could not establish signal connection: requested room does not exist"
// error msg is "Failed to create call"
// error description is "Call creation might be restricted to authorized users only. Try again later, or contact your server admin if the problem persists."
// The room does not exist. There are two different modes of operation for the SFU:
// - the room is created on the fly when connecting (livekit `auto_create` option)
// - Only authorized users can create rooms, so the room must exist before connecting (done by the auth jwt service)
@@ -139,19 +176,16 @@ export class Connection {
}
// If we were stopped while connecting, don't proceed to update state.
if (this.stopped) return;
this._state$.next({
state: "ConnectedToLkRoom",
transport: this.transport,
livekitConnectionState$: connectionStateObserver(this.livekitRoom),
});
} catch (error) {
this.logger.debug(`Failed to connect to LiveKit room: ${error}`);
this._state$.next({
state: "FailedToStart",
error: error instanceof Error ? error : new Error(`${error}`),
transport: this.transport,
});
this._state$.next(
error instanceof ElementCallError
? error
: error instanceof Error
? new UnknownCallError(error)
: new UnknownCallError(new Error(`${error}`)),
);
// Its okay to ignore the throw. The error is part of the state.
throw error;
}
}
@@ -176,30 +210,11 @@ export class Connection {
);
if (this.stopped) return;
await this.livekitRoom.disconnect();
this._state$.next({
state: "Stopped",
transport: this.transport,
});
this._state$.next(ConnectionState.Stopped);
this.stopped = true;
}
/**
* An observable of the participants that are publishing on this connection. (Excluding our local participant)
* This is derived from `participantsIncludingSubscribers$` and `remoteTransports$`.
* It filters the participants to only those that are associated with a membership that claims to publish on this connection.
*/
public readonly remoteParticipantsWithTracks$: Behavior<
PublishingParticipant[]
>;
/**
* The media transport to connect to.
*/
public readonly transport: LivekitTransport;
private readonly client: OpenIDClientParts;
public readonly livekitRoom: LivekitRoom;
private readonly logger: Logger;
/**
@@ -216,6 +231,7 @@ export class Connection {
);
const { transport, client, scope } = opts;
this.scope = scope;
this.livekitRoom = opts.livekitRoomFactory();
this.transport = transport;
this.client = client;

View File

@@ -7,10 +7,11 @@ Please see LICENSE in the repository root for full details.
import { type LivekitTransport } from "matrix-js-sdk/lib/matrixrtc";
import {
type E2EEOptions,
Room as LivekitRoom,
type RoomOptions,
type BaseKeyProvider,
type E2EEManagerOptions,
type BaseE2EEManager,
} from "livekit-client";
import { type Logger } from "matrix-js-sdk/lib/logger";
import E2EEWorker from "livekit-client/e2ee-worker?worker";
@@ -23,6 +24,7 @@ import type { Behavior } from "../../Behavior.ts";
import type { ProcessorState } from "../../../livekit/TrackProcessorContext.tsx";
import { defaultLiveKitOptions } from "../../../livekit/options.ts";
// TODO evaluate if this should be done like the Publisher Factory
export interface ConnectionFactory {
createConnection(
transport: LivekitTransport,
@@ -40,8 +42,10 @@ export class ECConnectionFactory implements ConnectionFactory {
* @param client - The OpenID client parts for authentication, needed to get openID and JWT tokens.
* @param devices - Used for video/audio out/in capture options.
* @param processorState$ - Effects like background blur (only for publishing connection?)
* @param e2eeLivekitOptions - The E2EE options to use for the LiveKit Room.
* @param livekitKeyProvider
* @param controlledAudioDevices - Option to indicate whether audio output device is controlled externally (native mobile app).
* @param echoCancellation - Whether to enable echo cancellation for audio capture.
* @param noiseSuppression - Whether to enable noise suppression for audio capture.
* @param livekitRoomFactory - Optional factory function (for testing) to create LivekitRoom instances. If not provided, a default factory is used.
*/
public constructor(
@@ -51,20 +55,24 @@ export class ECConnectionFactory implements ConnectionFactory {
livekitKeyProvider: BaseKeyProvider | undefined,
private controlledAudioDevices: boolean,
livekitRoomFactory?: () => LivekitRoom,
echoCancellation: boolean = true,
noiseSuppression: boolean = true,
) {
const defaultFactory = (): LivekitRoom =>
new LivekitRoom(
generateRoomOption(
this.devices,
this.processorState$.value,
livekitKeyProvider && {
generateRoomOption({
devices: this.devices,
processorState: this.processorState$.value,
e2eeLivekitOptions: livekitKeyProvider && {
keyProvider: livekitKeyProvider,
// It's important that every room use a separate E2EE worker.
// They get confused if given streams from multiple rooms.
worker: new E2EEWorker(),
},
this.controlledAudioDevices,
),
controlledAudioDevices: this.controlledAudioDevices,
echoCancellation,
noiseSuppression,
}),
);
this.livekitRoomFactory = livekitRoomFactory ?? defaultFactory;
}
@@ -89,12 +97,24 @@ export class ECConnectionFactory implements ConnectionFactory {
/**
* Generate the initial LiveKit RoomOptions based on the current media devices and processor state.
*/
function generateRoomOption(
devices: MediaDevices,
processorState: ProcessorState,
e2eeLivekitOptions: E2EEOptions | undefined,
controlledAudioDevices: boolean,
): RoomOptions {
function generateRoomOption({
devices,
processorState,
e2eeLivekitOptions,
controlledAudioDevices,
echoCancellation,
noiseSuppression,
}: {
devices: MediaDevices;
processorState: ProcessorState;
e2eeLivekitOptions:
| E2EEManagerOptions
| { e2eeManager: BaseE2EEManager }
| undefined;
controlledAudioDevices: boolean;
echoCancellation: boolean;
noiseSuppression: boolean;
}): RoomOptions {
return {
...defaultLiveKitOptions,
videoCaptureDefaults: {
@@ -105,6 +125,8 @@ function generateRoomOption(
audioCaptureDefaults: {
...defaultLiveKitOptions.audioCaptureDefaults,
deviceId: devices.audioInput.selected$.value?.id,
echoCancellation,
noiseSuppression,
},
audioOutput: {
// When using controlled audio devices, we don't want to set the

View File

@@ -11,7 +11,7 @@ import { type LivekitTransport } from "matrix-js-sdk/lib/matrixrtc";
import { type Participant as LivekitParticipant } from "livekit-client";
import { logger } from "matrix-js-sdk/lib/logger";
import { Epoch, ObservableScope } from "../../ObservableScope.ts";
import { Epoch, mapEpoch, ObservableScope } from "../../ObservableScope.ts";
import {
createConnectionManager$,
type ConnectionManagerData,
@@ -73,7 +73,7 @@ afterEach(() => {
describe("connections$ stream", () => {
test("Should create and start new connections for each transports", () => {
withTestScheduler(({ behavior, expectObservable }) => {
const { connections$ } = createConnectionManager$({
const { connectionManagerData$ } = createConnectionManager$({
scope: testScope,
connectionFactory: fakeConnectionFactory,
inputTransports$: behavior("a", {
@@ -82,7 +82,9 @@ describe("connections$ stream", () => {
logger: logger,
});
expectObservable(connections$).toBe("a", {
expectObservable(
connectionManagerData$.pipe(mapEpoch((d) => d.getConnections())),
).toBe("a", {
a: expect.toSatisfy((e: Epoch<Connection[]>) => {
const connections = e.value;
expect(connections.length).toBe(2);
@@ -110,7 +112,7 @@ describe("connections$ stream", () => {
test("Should start connection only once", () => {
withTestScheduler(({ behavior, expectObservable }) => {
const { connections$ } = createConnectionManager$({
const { connectionManagerData$ } = createConnectionManager$({
scope: testScope,
connectionFactory: fakeConnectionFactory,
inputTransports$: behavior("abcdef", {
@@ -124,7 +126,9 @@ describe("connections$ stream", () => {
logger: logger,
});
expectObservable(connections$).toBe("xxxxxa", {
expectObservable(
connectionManagerData$.pipe(mapEpoch((d) => d.getConnections())),
).toBe("xxxxxa", {
x: expect.anything(),
a: expect.toSatisfy((e: Epoch<Connection[]>) => {
const connections = e.value;
@@ -153,7 +157,7 @@ describe("connections$ stream", () => {
test("Should cleanup connections when not needed anymore", () => {
withTestScheduler(({ behavior, expectObservable }) => {
const { connections$ } = createConnectionManager$({
const { connectionManagerData$ } = createConnectionManager$({
scope: testScope,
connectionFactory: fakeConnectionFactory,
inputTransports$: behavior("abc", {
@@ -164,7 +168,9 @@ describe("connections$ stream", () => {
logger: logger,
});
expectObservable(connections$).toBe("xab", {
expectObservable(
connectionManagerData$.pipe(mapEpoch((d) => d.getConnections())),
).toBe("xab", {
x: expect.anything(),
a: expect.toSatisfy((e: Epoch<Connection[]>) => {
const connections = e.value;

View File

@@ -1,6 +1,3 @@
// TODOs:
// - make ConnectionManager its own actual class
/*
Copyright 2025 Element Creations Ltd.
Copyright 2025 New Vector Ltd.
@@ -13,7 +10,7 @@ import {
type LivekitTransport,
type ParticipantId,
} from "matrix-js-sdk/lib/matrixrtc";
import { BehaviorSubject, combineLatest, map, of, switchMap, tap } from "rxjs";
import { combineLatest, map, of, switchMap, tap } from "rxjs";
import { type Logger } from "matrix-js-sdk/lib/logger";
import { type LocalParticipant, type RemoteParticipant } from "livekit-client";
@@ -63,11 +60,7 @@ export class ConnectionManagerData {
transport: LivekitTransport,
): (LocalParticipant | RemoteParticipant)[] {
const key = transport.livekit_service_url + "|" + transport.livekit_alias;
const existing = this.store.get(key);
if (existing) {
return existing[1];
}
return [];
return this.store.get(key)?.[1] ?? [];
}
/**
* Get all connections where the given participant is publishing.
@@ -95,9 +88,7 @@ interface Props {
}
// TODO - write test for scopes (do we really need to bind scope)
export interface IConnectionManager {
transports$: Behavior<Epoch<LivekitTransport[]>>;
connectionManagerData$: Behavior<Epoch<ConnectionManagerData>>;
connections$: Behavior<Epoch<Connection[]>>;
}
/**
* Crete a `ConnectionManager`
@@ -120,9 +111,6 @@ export function createConnectionManager$({
logger: parentLogger,
}: Props): IConnectionManager {
const logger = parentLogger.getChild("[ConnectionManager]");
const running$ = new BehaviorSubject(true);
scope.onEnd(() => running$.next(false));
// TODO logger: only construct one logger from the client and make it compatible via a EC specific sing
/**
@@ -134,10 +122,7 @@ export function createConnectionManager$({
* externally this is modified via `registerTransports()`.
*/
const transports$ = scope.behavior(
combineLatest([running$, inputTransports$]).pipe(
map(([running, transports]) =>
transports.mapInner((transport) => (running ? transport : [])),
),
inputTransports$.pipe(
map((transports) => transports.mapInner(removeDuplicateTransports)),
tap(({ value: transports }) => {
logger.trace(
@@ -220,7 +205,7 @@ export function createConnectionManager$({
new Epoch(new ConnectionManagerData()),
);
return { transports$, connectionManagerData$, connections$ };
return { connectionManagerData$ };
}
function removeDuplicateTransports(

View File

@@ -0,0 +1,133 @@
/*
Copyright 2025 Element Creations Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { afterEach, beforeEach, describe, expect, test, vi } from "vitest";
import { Room as LivekitRoom } from "livekit-client";
import { BehaviorSubject } from "rxjs";
import fetchMock from "fetch-mock";
import { logger } from "matrix-js-sdk/lib/logger";
import EventEmitter from "events";
import { ObservableScope } from "../../ObservableScope.ts";
import { ECConnectionFactory } from "./ConnectionFactory.ts";
import type { OpenIDClientParts } from "../../../livekit/openIDSFU.ts";
import { exampleTransport, mockMediaDevices } from "../../../utils/test.ts";
import type { ProcessorState } from "../../../livekit/TrackProcessorContext.tsx";
import { constant } from "../../Behavior";
// At the top of your test file, after imports
vi.mock("livekit-client", async (importOriginal) => {
return {
...(await importOriginal()),
Room: vi.fn().mockImplementation(function (this: LivekitRoom, options) {
const emitter = new EventEmitter();
return {
on: emitter.on.bind(emitter),
off: emitter.off.bind(emitter),
emit: emitter.emit.bind(emitter),
disconnect: vi.fn(),
remoteParticipants: new Map(),
} as unknown as LivekitRoom;
}),
};
});
let testScope: ObservableScope;
let mockClient: OpenIDClientParts;
beforeEach(() => {
testScope = new ObservableScope();
mockClient = {
getOpenIdToken: vi.fn().mockReturnValue(""),
getDeviceId: vi.fn().mockReturnValue("DEV000"),
};
});
describe("ECConnectionFactory - Audio inputs options", () => {
test.each([
{ echo: true, noise: true },
{ echo: true, noise: false },
{ echo: false, noise: true },
{ echo: false, noise: false },
])(
"it sets echoCancellation=$echo and noiseSuppression=$noise based on constructor parameters",
({ echo, noise }) => {
// test("it sets echoCancellation and noiseSuppression based on constructor parameters", () => {
const RoomConstructor = vi.mocked(LivekitRoom);
const ecConnectionFactory = new ECConnectionFactory(
mockClient,
mockMediaDevices({}),
new BehaviorSubject<ProcessorState>({
supported: true,
processor: undefined,
}),
undefined,
false,
undefined,
echo,
noise,
);
ecConnectionFactory.createConnection(exampleTransport, testScope, logger);
// Check if Room was constructed with expected options
expect(RoomConstructor).toHaveBeenCalledWith(
expect.objectContaining({
audioCaptureDefaults: expect.objectContaining({
echoCancellation: echo,
noiseSuppression: noise,
}),
}),
);
},
);
});
describe("ECConnectionFactory - ControlledAudioDevice", () => {
test.each([{ controlled: true }, { controlled: false }])(
"it sets controlledAudioDevice=$controlled then uses deviceId accordingly",
({ controlled }) => {
// test("it sets echoCancellation and noiseSuppression based on constructor parameters", () => {
const RoomConstructor = vi.mocked(LivekitRoom);
const ecConnectionFactory = new ECConnectionFactory(
mockClient,
mockMediaDevices({
audioOutput: {
available$: constant(new Map<never, never>()),
selected$: constant({ id: "DEV00", virtualEarpiece: false }),
select: () => {},
},
}),
new BehaviorSubject<ProcessorState>({
supported: true,
processor: undefined,
}),
undefined,
controlled,
undefined,
false,
false,
);
ecConnectionFactory.createConnection(exampleTransport, testScope, logger);
// Check if Room was constructed with expected options
expect(RoomConstructor).toHaveBeenCalledWith(
expect.objectContaining({
audioOutput: expect.objectContaining({
deviceId: controlled ? undefined : "DEV00",
}),
}),
);
},
);
});
afterEach(() => {
testScope.end();
fetchMock.reset();
});

View File

@@ -108,7 +108,7 @@ export function createMatrixLivekitMembers$({
// Each update where the key of the generator array do not change will result in updates to the `data$` observable in the factory.
(scope, data$, participantId, userId) => {
logger.debug(
`Updating data$ for participantId: ${participantId}, userId: ${userId}`,
`Generating member for participantId: ${participantId}, userId: ${userId}`,
);
// will only get called once per `participantId, userId` pair.
// updates to data$ and as a result to displayName$ and mxcAvatarUrl$ are more frequent.

View File

@@ -104,7 +104,6 @@ describe("MatrixMemberMetadata", () => {
}
}
// TODO this is a regression, now there the own user is not always in the map. Ask Timo if fine
it("should show our own user if present in rtc session and room", () => {
withTestScheduler(({ behavior, expectObservable }) => {
fakeMemberWith({
@@ -517,7 +516,6 @@ describe("MatrixMemberMetadata", () => {
}
}
// TODO this is a regression, now there the own user is not always in the map. Ask Timo if fine
it("should use avatar url from room members", () => {
withTestScheduler(({ behavior, expectObservable }) => {
fakeMemberWith({

View File

@@ -1,5 +1,5 @@
/*
Copyright 2025 New Vector Ltd.
Copyright 2025 Element Creations Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.

View File

@@ -0,0 +1,212 @@
/*
Copyright 2025 Element Creations Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { afterEach, beforeEach, describe, expect, test, vi } from "vitest";
import { BehaviorSubject } from "rxjs";
import { logger } from "matrix-js-sdk/lib/logger";
import { MuteStates, MuteState } from "./MuteStates";
import {
type AudioOutputDeviceLabel,
type DeviceLabel,
type MediaDevice,
type SelectedAudioOutputDevice,
type SelectedDevice,
} from "./MediaDevices";
import { constant } from "./Behavior";
import { ObservableScope } from "./ObservableScope";
import { flushPromises, mockMediaDevices } from "../utils/test";
const getUrlParams = vi.hoisted(() => vi.fn(() => ({})));
vi.mock("../UrlParams", () => ({ getUrlParams }));
let testScope: ObservableScope;
beforeEach(() => {
testScope = new ObservableScope();
});
afterEach(() => {
testScope.end();
});
describe("MuteState", () => {
test("should automatically mute if force mute is set", async () => {
const forceMute$ = new BehaviorSubject<boolean>(false);
const deviceStub = {
available$: constant(
new Map<string, DeviceLabel>([
["fbac11", { type: "name", name: "HD Camera" }],
]),
),
selected$: constant({ id: "fbac11" }),
select(): void {},
} as unknown as MediaDevice<DeviceLabel, SelectedDevice>;
const muteState = new MuteState(
testScope,
deviceStub,
constant(true),
true,
forceMute$,
);
let lastEnabled: boolean = false;
muteState.enabled$.subscribe((enabled) => {
lastEnabled = enabled;
});
let setEnabled: ((enabled: boolean) => void) | null = null;
muteState.setEnabled$.subscribe((setter) => {
setEnabled = setter;
});
await flushPromises();
setEnabled!(true);
await flushPromises();
expect(lastEnabled).toBe(true);
// Now force mute
forceMute$.next(true);
await flushPromises();
// Should automatically mute
expect(lastEnabled).toBe(false);
// Try to unmute can not work
expect(setEnabled).toBeNull();
// Disable force mute
forceMute$.next(false);
await flushPromises();
// TODO I'd expect it to go back to previous state (enabled)
// but actually it goes back to the initial state from construction (disabled)
// Should go back to previous state (enabled)
// Skip for now
// expect(lastEnabled).toBe(true);
// But yet it can be unmuted now
expect(setEnabled).not.toBeNull();
setEnabled!(true);
await flushPromises();
expect(lastEnabled).toBe(true);
});
});
describe("MuteStates", () => {
function aAudioOutputDevices(): MediaDevice<
AudioOutputDeviceLabel,
SelectedAudioOutputDevice
> {
const selected$ = new BehaviorSubject<
SelectedAudioOutputDevice | undefined
>({
id: "default",
virtualEarpiece: false,
});
return {
available$: constant(
new Map<string, AudioOutputDeviceLabel>([
["default", { type: "speaker" }],
["0000", { type: "speaker" }],
["1111", { type: "earpiece" }],
["222", { type: "name", name: "Bluetooth Speaker" }],
]),
),
selected$,
select(id: string): void {
if (!this.available$.getValue().has(id)) {
logger.warn(`Attempted to select unknown device id: ${id}`);
return;
}
selected$.next({
id,
/** For test purposes we ignore this */
virtualEarpiece: false,
});
},
};
}
function aVideoInput(): MediaDevice<DeviceLabel, SelectedDevice> {
const selected$ = new BehaviorSubject<SelectedDevice | undefined>(
undefined,
);
return {
available$: constant(
new Map<string, DeviceLabel>([
["0000", { type: "name", name: "HD Camera" }],
["1111", { type: "name", name: "WebCam Pro" }],
]),
),
selected$,
select(id: string): void {
if (!this.available$.getValue().has(id)) {
logger.warn(`Attempted to select unknown device id: ${id}`);
return;
}
selected$.next({ id });
},
};
}
test("should mute camera when in earpiece mode", async () => {
const audioOutputDevice = aAudioOutputDevices();
const mediaDevices = mockMediaDevices({
audioOutput: audioOutputDevice,
videoInput: aVideoInput(),
// other devices are not relevant for this test
});
const muteStates = new MuteStates(
testScope,
mediaDevices,
// consider joined
constant(true),
);
let latestSyncedState: boolean | null = null;
muteStates.video.setHandler(async (enabled: boolean): Promise<boolean> => {
logger.info(`Video mute state set to: ${enabled}`);
latestSyncedState = enabled;
return Promise.resolve(enabled);
});
let lastVideoEnabled: boolean = false;
muteStates.video.enabled$.subscribe((enabled) => {
lastVideoEnabled = enabled;
});
expect(muteStates.video.setEnabled$.value).toBeDefined();
muteStates.video.setEnabled$.value?.(true);
await flushPromises();
expect(lastVideoEnabled).toBe(true);
// Select earpiece audio output
audioOutputDevice.select("1111");
await flushPromises();
// Video should be automatically muted
expect(lastVideoEnabled).toBe(false);
expect(latestSyncedState).toBe(false);
// Try to switch to speaker
audioOutputDevice.select("0000");
await flushPromises();
// TODO I'd expect it to go back to previous state (enabled)??
// But maybe not? If you move the phone away from your ear you may not want it
// to automatically enable video?
expect(lastVideoEnabled).toBe(false);
// But yet it can be unmuted now
expect(muteStates.video.setEnabled$.value).toBeDefined();
muteStates.video.setEnabled$.value?.(true);
await flushPromises();
expect(lastVideoEnabled).toBe(true);
});
});

View File

@@ -27,7 +27,7 @@ import { ElementWidgetActions, widget } from "../widget";
import { Config } from "../config/Config";
import { getUrlParams } from "../UrlParams";
import { type ObservableScope } from "./ObservableScope";
import { type Behavior } from "./Behavior";
import { type Behavior, constant } from "./Behavior";
interface MuteStateData {
enabled$: Observable<boolean>;
@@ -38,31 +38,58 @@ interface MuteStateData {
export type Handler = (desired: boolean) => Promise<boolean>;
const defaultHandler: Handler = async (desired) => Promise.resolve(desired);
class MuteState<Label, Selected> {
/**
* Internal class - exported only for testing purposes.
* Do not use directly outside of tests.
*/
export class MuteState<Label, Selected> {
// TODO: rewrite this to explain behavior, it is not understandable, and cannot add logging
private readonly enabledByDefault$ =
this.enabledByConfig && !getUrlParams().skipLobby
? this.joined$.pipe(map((isJoined) => !isJoined))
: of(false);
private readonly handler$ = new BehaviorSubject(defaultHandler);
public setHandler(handler: Handler): void {
if (this.handler$.value !== defaultHandler)
throw new Error("Multiple mute state handlers are not supported");
this.handler$.next(handler);
}
public unsetHandler(): void {
this.handler$.next(defaultHandler);
}
private readonly canControlDevices$ = combineLatest([
this.device.available$,
this.forceMute$,
]).pipe(
map(([available, forceMute]) => {
return !forceMute && available.size > 0;
}),
);
private readonly data$ = this.scope.behavior<MuteStateData>(
this.device.available$.pipe(
map((available) => available.size > 0),
this.canControlDevices$.pipe(
distinctUntilChanged(),
withLatestFrom(
this.enabledByDefault$,
(devicesConnected, enabledByDefault) => {
if (!devicesConnected)
(canControlDevices, enabledByDefault) => {
logger.info(
`MuteState: canControlDevices: ${canControlDevices}, enabled by default: ${enabledByDefault}`,
);
if (!canControlDevices) {
logger.info(
`MuteState: devices connected: ${canControlDevices}, disabling`,
);
// We need to sync the mute state with the handler
// to ensure nothing is beeing published.
this.handler$.value(false).catch((err) => {
logger.error("MuteState-disable: handler error", err);
});
return { enabled$: of(false), set: null, toggle: null };
}
// Assume the default value only once devices are actually connected
let enabled = enabledByDefault;
@@ -135,21 +162,45 @@ class MuteState<Label, Selected> {
private readonly device: MediaDevice<Label, Selected>,
private readonly joined$: Observable<boolean>,
private readonly enabledByConfig: boolean,
/**
* An optional observable which, when it emits `true`, will force the mute.
* Used for video to stop camera when earpiece mode is on.
* @private
*/
private readonly forceMute$: Observable<boolean>,
) {}
}
export class MuteStates {
/**
* True if the selected audio output device is an earpiece.
* Used to force-disable video when on earpiece.
*/
private readonly isEarpiece$ = combineLatest(
this.mediaDevices.audioOutput.available$,
this.mediaDevices.audioOutput.selected$,
).pipe(
map(([available, selected]) => {
if (!selected?.id) return false;
const device = available.get(selected.id);
logger.info(`MuteStates: selected audio output device:`, device);
return device?.type === "earpiece";
}),
);
public readonly audio = new MuteState(
this.scope,
this.mediaDevices.audioInput,
this.joined$,
Config.get().media_devices.enable_audio,
constant(false),
);
public readonly video = new MuteState(
this.scope,
this.mediaDevices.videoInput,
this.joined$,
Config.get().media_devices.enable_video,
this.isEarpiece$,
);
public constructor(

View File

@@ -1,13 +1,14 @@
/*
Copyright 2025 New Vector Ltd.
Copyright 2025 Element Creations Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { describe, expect, it } from "vitest";
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
import { BehaviorSubject, combineLatest, Subject } from "rxjs";
import { logger } from "matrix-js-sdk/lib/logger";
import { sleep } from "matrix-js-sdk/lib/utils";
import {
Epoch,
@@ -102,3 +103,137 @@ describe("Epoch", () => {
s$.complete();
});
});
describe("Reconcile", () => {
beforeEach(() => {
vi.useFakeTimers();
});
afterEach(() => {
vi.useRealTimers();
});
it("should wait clean up before processing next", async () => {
vi.useFakeTimers();
const scope = new ObservableScope();
const behavior$ = new BehaviorSubject<number>(0);
const setup = vi.fn().mockImplementation(async () => await sleep(100));
const cleanup = vi
.fn()
.mockImplementation(async (n: number) => await sleep(100));
scope.reconcile(behavior$, async (value) => {
await setup();
return async (): Promise<void> => {
await cleanup(value);
};
});
// Let the initial setup process
await vi.advanceTimersByTimeAsync(120);
expect(setup).toHaveBeenCalledTimes(1);
expect(cleanup).toHaveBeenCalledTimes(0);
// Send next value
behavior$.next(1);
await vi.advanceTimersByTimeAsync(50);
// Should not have started setup for 1 yet
expect(setup).toHaveBeenCalledTimes(1);
expect(cleanup).toHaveBeenCalledTimes(1);
expect(cleanup).toHaveBeenCalledWith(0);
// Let cleanup finish
await vi.advanceTimersByTimeAsync(50);
// Now setup for 1 should have started
expect(setup).toHaveBeenCalledTimes(2);
});
it("should skip intermediates values that are not setup", async () => {
vi.useFakeTimers();
const scope = new ObservableScope();
const behavior$ = new BehaviorSubject<number>(0);
const setup = vi
.fn()
.mockImplementation(async (n: number) => await sleep(100));
const cleanupLock = Promise.withResolvers();
const cleanup = vi
.fn()
.mockImplementation(async (n: number) => await cleanupLock.promise);
scope.reconcile(behavior$, async (value) => {
await setup(value);
return async (): Promise<void> => {
await cleanup(value);
};
});
// Let the initial setup process (0)
await vi.advanceTimersByTimeAsync(120);
// Send 4 next values quickly
behavior$.next(1);
behavior$.next(2);
behavior$.next(3);
behavior$.next(4);
await vi.advanceTimersByTimeAsync(3000);
// should have only called cleanup for 0
expect(cleanup).toHaveBeenCalledTimes(1);
expect(cleanup).toHaveBeenCalledWith(0);
// Let cleanup finish
cleanupLock.resolve(undefined);
await vi.advanceTimersByTimeAsync(120);
// Now setup for 4 should have started, skipping 1,2,3
expect(setup).toHaveBeenCalledTimes(2);
expect(setup).toHaveBeenCalledWith(4);
expect(setup).not.toHaveBeenCalledWith(1);
expect(setup).not.toHaveBeenCalledWith(2);
expect(setup).not.toHaveBeenCalledWith(3);
});
it("should wait for setup to complete before starting cleanup", async () => {
vi.useFakeTimers();
const scope = new ObservableScope();
const behavior$ = new BehaviorSubject<number>(0);
const setup = vi
.fn()
.mockImplementation(async (n: number) => await sleep(3000));
const cleanupLock = Promise.withResolvers();
const cleanup = vi
.fn()
.mockImplementation(async (n: number) => await cleanupLock.promise);
scope.reconcile(behavior$, async (value) => {
await setup(value);
return async (): Promise<void> => {
await cleanup(value);
};
});
await vi.advanceTimersByTimeAsync(500);
// Setup for 0 should be in progress
expect(setup).toHaveBeenCalledTimes(1);
behavior$.next(1);
await vi.advanceTimersByTimeAsync(500);
// Should not have started setup for 1 yet
expect(setup).not.toHaveBeenCalledWith(1);
// Should not have called cleanup yet, because the setup for 0 is not done
expect(cleanup).toHaveBeenCalledTimes(0);
// Let setup for 0 finish
await vi.advanceTimersByTimeAsync(2500 + 100);
// Now cleanup for 0 should have started
expect(cleanup).toHaveBeenCalledTimes(1);
expect(cleanup).toHaveBeenCalledWith(0);
cleanupLock.resolve(undefined);
await vi.advanceTimersByTimeAsync(100);
// Now setup for 1 should have started
expect(setup).toHaveBeenCalledWith(1);
});
});

View File

@@ -123,8 +123,22 @@ export class ObservableScope {
callback: (value: T) => Promise<(() => Promise<void>) | void>,
): void {
let latestValue: T | typeof nothing = nothing;
let reconciledValue: T | typeof nothing = nothing;
let reconcilePromise: Promise<void> | undefined = undefined;
let cleanUp: (() => Promise<void>) | void = undefined;
let prevVal: T | typeof nothing = nothing;
// While this loop runs it will process the latest from `value$` until it caught up with the updates.
// It might skip updates from `value$` and only process the newest value after callback has resolved.
const reconcileLoop = async (): Promise<void> => {
while (latestValue !== prevVal) {
await cleanUp?.(); // Call the previous value's clean-up handler
prevVal = latestValue;
if (latestValue !== nothing) cleanUp = await callback(latestValue); // Sync current value...
// `latestValue` might have gotten updated during the `await callback`. That is why we loop here
}
};
value$
.pipe(
catchError(() => EMPTY), // Ignore errors
@@ -132,23 +146,15 @@ export class ObservableScope {
endWith(nothing), // Clean up when the scope ends
)
.subscribe((value) => {
void (async (): Promise<void> => {
if (latestValue === nothing) {
latestValue = value;
while (latestValue !== reconciledValue) {
await cleanUp?.(); // Call the previous value's clean-up handler
reconciledValue = latestValue;
if (latestValue !== nothing)
cleanUp = await callback(latestValue); // Sync current value
}
// Reset to signal that reconciliation is done for now
latestValue = nothing;
} else {
// There's already an instance of the above 'while' loop running
// concurrently. Just update the latest value and let it be handled.
latestValue = value;
}
})();
// Always track the latest value! The `reconcileLoop` will run until it "processed" the "last" `latestValue`.
latestValue = value;
// There's already an instance of the below 'reconcileLoop' loop running
// concurrently. So lets let the loop handle it. NEVER instanciate two `reconcileLoop`s.
if (reconcilePromise) return;
reconcilePromise = reconcileLoop().finally(() => {
reconcilePromise = undefined;
});
});
}

View File

@@ -1,5 +1,5 @@
/*
Copyright 2025 New Vector Ltd.
Copyright 2025 Element Creations Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.

View File

@@ -13,6 +13,8 @@ export enum ErrorCode {
*/
MISSING_MATRIX_RTC_TRANSPORT = "MISSING_MATRIX_RTC_TRANSPORT",
CONNECTION_LOST_ERROR = "CONNECTION_LOST_ERROR",
INTERNAL_MEMBERSHIP_MANAGER = "INTERNAL_MEMBERSHIP_MANAGER",
FAILED_TO_START_LIVEKIT = "FAILED_TO_START_LIVEKIT",
/** LiveKit indicates that the server has hit its track limits */
INSUFFICIENT_CAPACITY_ERROR = "INSUFFICIENT_CAPACITY_ERROR",
E2EE_NOT_SUPPORTED = "E2EE_NOT_SUPPORTED",
@@ -27,6 +29,7 @@ export enum ErrorCategory {
NETWORK_CONNECTIVITY = "NETWORK_CONNECTIVITY",
CLIENT_CONFIGURATION = "CLIENT_CONFIGURATION",
UNKNOWN = "UNKNOWN",
SYSTEM_FAILURE = "SYSTEM_FAILURE",
// SYSTEM_FAILURE / FEDERATION_FAILURE ..
}
@@ -83,6 +86,18 @@ export class ConnectionLostError extends ElementCallError {
}
}
export class MembershipManagerError extends ElementCallError {
public constructor(error: Error) {
super(
t("error.membership_manager"),
ErrorCode.INTERNAL_MEMBERSHIP_MANAGER,
ErrorCategory.SYSTEM_FAILURE,
t("error.membership_manager_description"),
error,
);
}
}
export class E2EENotSupportedError extends ElementCallError {
public constructor() {
super(
@@ -120,6 +135,17 @@ export class FailToGetOpenIdToken extends ElementCallError {
}
}
export class FailToStartLivekitConnection extends ElementCallError {
public constructor(e?: string) {
super(
t("error.failed_to_start_livekit"),
ErrorCode.FAILED_TO_START_LIVEKIT,
ErrorCategory.NETWORK_CONNECTIVITY,
e,
);
}
}
export class InsufficientCapacityError extends ElementCallError {
public constructor() {
super(

View File

@@ -44,12 +44,12 @@ import {
Track,
} from "livekit-client";
import { randomUUID } from "crypto";
import {
type RoomAndToDeviceEvents,
type RoomAndToDeviceEventsHandlerMap,
} from "matrix-js-sdk/lib/matrixrtc/RoomAndToDeviceKeyTransport";
import { type TrackReference } from "@livekit/components-core";
import EventEmitter from "events";
import {
type KeyTransportEvents,
type KeyTransportEventsHandlerMap,
} from "matrix-js-sdk/lib/matrixrtc/IKeyTransport";
import {
LocalUserMediaViewModel,
@@ -79,6 +79,13 @@ export async function flushPromises(): Promise<void> {
await new Promise<void>((resolve) => window.setTimeout(resolve));
}
export type NodeEventHandler = (...args: unknown[]) => void;
export interface NodeStyleEventEmitter {
addListener(eventName: string | symbol, handler: NodeEventHandler): this;
removeListener(eventName: string | symbol, handler: NodeEventHandler): this;
}
export interface OurRunHelpers extends RunHelpers {
/**
* Schedules a sequence of actions to happen, as described by a marble
@@ -277,6 +284,8 @@ export function mockLivekitRoom(
): LivekitRoom {
const livekitRoom = {
options: {},
setE2EEEnabled: vi.fn(),
...mockEmitter(),
...room,
} as Partial<LivekitRoom> as LivekitRoom;
@@ -299,7 +308,11 @@ export function mockLocalParticipant(
return {
isLocal: true,
trackPublications: new Map(),
unpublishTracks: async () => Promise.resolve(),
publishTrack: vi.fn(),
unpublishTracks: vi.fn().mockResolvedValue([]),
createTracks: vi.fn(),
setMicrophoneEnabled: vi.fn(),
setCameraEnabled: vi.fn(),
getTrackPublication: () =>
({}) as Partial<LocalTrackPublication> as LocalTrackPublication,
...mockEmitter(),
@@ -387,9 +400,9 @@ export function mockConfig(
}
export class MockRTCSession extends TypedEventEmitter<
MatrixRTCSessionEvent | RoomAndToDeviceEvents | MembershipManagerEvent,
MatrixRTCSessionEventHandlerMap &
RoomAndToDeviceEventsHandlerMap &
MatrixRTCSessionEvent | MembershipManagerEvent | KeyTransportEvents,
KeyTransportEventsHandlerMap &
MatrixRTCSessionEventHandlerMap &
MembershipManagerEventHandlerMap
> {
public asMockedSession(): MockedObject<MatrixRTCSession> {

120
yarn.lock
View File

@@ -2731,37 +2731,39 @@ __metadata:
languageName: node
linkType: hard
"@livekit/components-core@npm:0.12.10, @livekit/components-core@npm:^0.12.0":
version: 0.12.10
resolution: "@livekit/components-core@npm:0.12.10"
"@livekit/components-core@npm:0.12.11, @livekit/components-core@npm:^0.12.0":
version: 0.12.11
resolution: "@livekit/components-core@npm:0.12.11"
dependencies:
"@floating-ui/dom": "npm:1.6.13"
loglevel: "npm:1.9.1"
rxjs: "npm:7.8.2"
peerDependencies:
livekit-client: ^2.13.3
livekit-client: ^2.15.14
tslib: ^2.6.2
checksum: 10c0/bfd84fb950f72dd037bd5329658c1e750a8ac6b8f2953ea673e3e944b8ea9d412ef9c98eb8b690052323e03c675964b162aacb00e60530cdc5187f77d21979bd
checksum: 10c0/9c2ac3d30bb8cc9067ae0b2049784f81e90e57df9eabf7edbaf3c8ceb65a63f644a4e6abeb6cc38d3ebe52663d8dbb88535e01a965011f365d5ae1f3daf86052
languageName: node
linkType: hard
"@livekit/components-react@npm:^2.0.0":
version: 2.9.15
resolution: "@livekit/components-react@npm:2.9.15"
version: 2.9.16
resolution: "@livekit/components-react@npm:2.9.16"
dependencies:
"@livekit/components-core": "npm:0.12.10"
"@livekit/components-core": "npm:0.12.11"
clsx: "npm:2.1.1"
events: "npm:^3.3.0"
jose: "npm:^6.0.12"
usehooks-ts: "npm:3.1.1"
peerDependencies:
"@livekit/krisp-noise-filter": ^0.2.12 || ^0.3.0
livekit-client: ^2.13.3
livekit-client: ^2.15.14
react: ">=18"
react-dom: ">=18"
tslib: ^2.6.2
peerDependenciesMeta:
"@livekit/krisp-noise-filter":
optional: true
checksum: 10c0/58a93d85c3b8267d0afd00eceb4f34992ce66124f93450e828a78dd825ecc20e254c3123ed22ec33061a3728f50f4f020ff896769fb3a0ff79656ed8cf452a2b
checksum: 10c0/4ba4ff473c5a29d3107412733a6676a3b708d70684ed463e9b34cda26abb3d2f317c2828a52e730837b756de9df3fc248260d6f390aedebfb6ec96ef63c7b151
languageName: node
linkType: hard
@@ -3371,14 +3373,14 @@ __metadata:
languageName: node
linkType: hard
"@playwright/test@npm:^1.56.1":
version: 1.56.1
resolution: "@playwright/test@npm:1.56.1"
"@playwright/test@npm:^1.57.0":
version: 1.57.0
resolution: "@playwright/test@npm:1.57.0"
dependencies:
playwright: "npm:1.56.1"
playwright: "npm:1.57.0"
bin:
playwright: cli.js
checksum: 10c0/2b5b0e1f2e6a18f6e5ce6897c7440ca78f64e0b004834e9808e93ad2b78b96366b562ae4366602669cf8ad793a43d85481b58541e74be71e905e732d833dd691
checksum: 10c0/35ba4b28be72bf0a53e33dbb11c6cff848fb9a37f49e893ce63a90675b5291ec29a1ba82c8a3b043abaead129400f0589623e9ace2e6a1c8eaa409721ecc3774
languageName: node
linkType: hard
@@ -5247,12 +5249,12 @@ __metadata:
languageName: node
linkType: hard
"@types/node@npm:^22.0.0":
version: 22.17.0
resolution: "@types/node@npm:22.17.0"
"@types/node@npm:^24.0.0":
version: 24.10.0
resolution: "@types/node@npm:24.10.0"
dependencies:
undici-types: "npm:~6.21.0"
checksum: 10c0/e1c603b660d3de3243dfc02ded5d40623ff3f36315ffbdd8cdc81bc2c5a8da172035879d437b72e9fa61ca01827f28e9c2b0c32898f411a8e9ba0a5efac0b4ca
undici-types: "npm:~7.16.0"
checksum: 10c0/f82ed7194e16f5590ef7afdc20c6d09068c76d50278b485ede8f0c5749683536e3064ffa8def8db76915196afb3724b854aa5723c64d6571b890b14492943b46
languageName: node
linkType: hard
@@ -7490,7 +7492,7 @@ __metadata:
"@opentelemetry/sdk-trace-base": "npm:^2.0.0"
"@opentelemetry/sdk-trace-web": "npm:^2.0.0"
"@opentelemetry/semantic-conventions": "npm:^1.25.1"
"@playwright/test": "npm:^1.56.1"
"@playwright/test": "npm:^1.57.0"
"@radix-ui/react-dialog": "npm:^1.0.4"
"@radix-ui/react-slider": "npm:^1.1.2"
"@radix-ui/react-visually-hidden": "npm:^1.0.3"
@@ -7507,7 +7509,7 @@ __metadata:
"@types/grecaptcha": "npm:^3.0.9"
"@types/jsdom": "npm:^21.1.7"
"@types/lodash-es": "npm:^4.17.12"
"@types/node": "npm:^22.0.0"
"@types/node": "npm:^24.0.0"
"@types/pako": "npm:^2.0.3"
"@types/qrcode": "npm:^1.5.5"
"@types/react": "npm:^19.0.0"
@@ -7545,8 +7547,8 @@ __metadata:
livekit-client: "npm:^2.13.0"
lodash-es: "npm:^4.17.21"
loglevel: "npm:^1.9.1"
matrix-js-sdk: "github:matrix-org/matrix-js-sdk#head=toger5/sticky-events&commit=e7f5bec51b6f70501a025b79fe5021c933385b21"
matrix-widget-api: "npm:^1.13.0"
matrix-js-sdk: "npm:^39.2.0"
matrix-widget-api: "npm:^1.14.0"
normalize.css: "npm:^8.0.1"
observable-hooks: "npm:^4.2.3"
pako: "npm:^2.0.4"
@@ -9828,6 +9830,13 @@ __metadata:
languageName: node
linkType: hard
"jose@npm:^6.0.12":
version: 6.1.2
resolution: "jose@npm:6.1.2"
checksum: 10c0/55f79426f43e652ed6d5de938d50f66bb0a10dcae078db81a23f8d3303e889ce226f000e815f3211f9956bb84badce10da892d130d40fe2eca658045a6f1778e
languageName: node
linkType: hard
"jose@npm:^6.1.0":
version: 6.1.0
resolution: "jose@npm:6.1.0"
@@ -10110,8 +10119,8 @@ __metadata:
linkType: hard
"livekit-client@npm:^2.13.0":
version: 2.15.13
resolution: "livekit-client@npm:2.15.13"
version: 2.16.0
resolution: "livekit-client@npm:2.16.0"
dependencies:
"@livekit/mutex": "npm:1.1.1"
"@livekit/protocol": "npm:1.42.2"
@@ -10125,7 +10134,7 @@ __metadata:
webrtc-adapter: "npm:^9.0.1"
peerDependencies:
"@types/dom-mediacapture-record": ^1
checksum: 10c0/5a061df9000461a6d40ef8aa1e72e8aedc640181cc57fe6f2c48c5c7f90ce96a735b125aede377fc43f4692a685e098f17eeae0f42c5b2fed473305867bf2789
checksum: 10c0/5d03adc5d09efde343ab894db397529dff26117598e773b23a5df90a4fb166bde12c6bb1f2cfd1d28dbaf93fe9f275026d7abb75f2ffd2ba816393a2d58e6c7e
languageName: node
linkType: hard
@@ -10343,9 +10352,9 @@ __metadata:
languageName: node
linkType: hard
"matrix-js-sdk@github:matrix-org/matrix-js-sdk#head=toger5/sticky-events&commit=e7f5bec51b6f70501a025b79fe5021c933385b21":
version: 38.4.0
resolution: "matrix-js-sdk@https://github.com/matrix-org/matrix-js-sdk.git#commit=e7f5bec51b6f70501a025b79fe5021c933385b21"
"matrix-js-sdk@npm:^39.2.0":
version: 39.3.0
resolution: "matrix-js-sdk@npm:39.3.0"
dependencies:
"@babel/runtime": "npm:^7.12.5"
"@matrix-org/matrix-sdk-crypto-wasm": "npm:^15.3.0"
@@ -10355,23 +10364,23 @@ __metadata:
jwt-decode: "npm:^4.0.0"
loglevel: "npm:^1.9.2"
matrix-events-sdk: "npm:0.0.1"
matrix-widget-api: "npm:^1.10.0"
matrix-widget-api: "npm:^1.14.0"
oidc-client-ts: "npm:^3.0.1"
p-retry: "npm:7"
sdp-transform: "npm:^2.14.1"
sdp-transform: "npm:^3.0.0"
unhomoglyph: "npm:^1.0.6"
uuid: "npm:13"
checksum: 10c0/7adffdc183affd2d3ee1e8497cad6ca7904a37f98328ff7bc15aa6c1829dc9f9a92f8e1bd6260432a33626ff2a839644de938270163e73438b7294675cd954e4
checksum: 10c0/031c9ec042e00c32dc531f82fc59c64cc25fb665abfc642b1f0765c530d60684f8bd63daf0cdd0dbe96b4f87ea3f4148f9d3f024a59d57eceaec1ce5d0164755
languageName: node
linkType: hard
"matrix-widget-api@npm:^1.10.0, matrix-widget-api@npm:^1.13.0":
version: 1.13.1
resolution: "matrix-widget-api@npm:1.13.1"
"matrix-widget-api@npm:^1.14.0":
version: 1.15.0
resolution: "matrix-widget-api@npm:1.15.0"
dependencies:
"@types/events": "npm:^3.0.0"
events: "npm:^3.2.0"
checksum: 10c0/25ded744922755b3eb65f4e171cf6cff1a2e0fe43fc3fecbb13e565e41d8af066daa817dd2c3c7d921b996af399eec3b23df70ab7b682cf422d9cee7ca202512
checksum: 10c0/1c08b5284cd98aed312d95594335e1391d937dfad70ef862a1f90fdbaaa27709e1c44dcda37f8045e4814779d8d5816d240aee396d52cfd9b37fbf243a6baf6a
languageName: node
linkType: hard
@@ -11168,27 +11177,27 @@ __metadata:
languageName: node
linkType: hard
"playwright-core@npm:1.56.1":
version: 1.56.1
resolution: "playwright-core@npm:1.56.1"
"playwright-core@npm:1.57.0":
version: 1.57.0
resolution: "playwright-core@npm:1.57.0"
bin:
playwright-core: cli.js
checksum: 10c0/ffd40142b99c68678b387445d5b42f1fee4ab0b65d983058c37f342e5629f9cdbdac0506ea80a0dfd41a8f9f13345bad54e9a8c35826ef66dc765f4eb3db8da7
checksum: 10c0/798e35d83bf48419a8c73de20bb94d68be5dde68de23f95d80a0ebe401e3b83e29e3e84aea7894d67fa6c79d2d3d40cc5bcde3e166f657ce50987aaa2421b6a9
languageName: node
linkType: hard
"playwright@npm:1.56.1":
version: 1.56.1
resolution: "playwright@npm:1.56.1"
"playwright@npm:1.57.0":
version: 1.57.0
resolution: "playwright@npm:1.57.0"
dependencies:
fsevents: "npm:2.3.2"
playwright-core: "npm:1.56.1"
playwright-core: "npm:1.57.0"
dependenciesMeta:
fsevents:
optional: true
bin:
playwright: cli.js
checksum: 10c0/8e9965aede86df0f4722063385748498977b219630a40a10d1b82b8bd8d4d4e9b6b65ecbfa024331a30800163161aca292fb6dd7446c531a1ad25f4155625ab4
checksum: 10c0/ab03c99a67b835bdea9059f516ad3b6e42c21025f9adaa161a4ef6bc7ca716dcba476d287140bb240d06126eb23f889a8933b8f5f1f1a56b80659d92d1358899
languageName: node
linkType: hard
@@ -12544,7 +12553,7 @@ __metadata:
languageName: node
linkType: hard
"sdp-transform@npm:^2.14.1, sdp-transform@npm:^2.15.0":
"sdp-transform@npm:^2.15.0":
version: 2.15.0
resolution: "sdp-transform@npm:2.15.0"
bin:
@@ -12553,6 +12562,15 @@ __metadata:
languageName: node
linkType: hard
"sdp-transform@npm:^3.0.0":
version: 3.0.0
resolution: "sdp-transform@npm:3.0.0"
bin:
sdp-verify: checker.js
checksum: 10c0/828a4595041ba64c86b29075aa4007ab384519b1fa29882db59ccb83b54b2b2a33b60848293f8da537fe151c52f5844fc17c8325396cac309fb19e2e81ec5bf4
languageName: node
linkType: hard
"sdp@npm:^3.2.0":
version: 3.2.0
resolution: "sdp@npm:3.2.0"
@@ -13603,10 +13621,10 @@ __metadata:
languageName: node
linkType: hard
"undici-types@npm:~6.21.0":
version: 6.21.0
resolution: "undici-types@npm:6.21.0"
checksum: 10c0/c01ed51829b10aa72fc3ce64b747f8e74ae9b60eafa19a7b46ef624403508a54c526ffab06a14a26b3120d055e1104d7abe7c9017e83ced038ea5cf52f8d5e04
"undici-types@npm:~7.16.0":
version: 7.16.0
resolution: "undici-types@npm:7.16.0"
checksum: 10c0/3033e2f2b5c9f1504bdc5934646cb54e37ecaca0f9249c983f7b1fc2e87c6d18399ebb05dc7fd5419e02b2e915f734d872a65da2e3eeed1813951c427d33cc9a
languageName: node
linkType: hard