Compare commits

...

156 Commits

Author SHA1 Message Date
Timo K
3e0e1ccde5 add todo comment
Signed-off-by: Timo K <toger5@hotmail.de>
2025-10-22 12:51:10 +02:00
Robin
bd94e415e7 Fix test type errors 2025-10-21 13:19:38 -04:00
Robin
e313cf04a6 Always consider multi-SFU mode enabled when using sticky events
CallViewModel would pass the wrong transport to enterRtcSession when the user enabled sticky events but didn't manually enable multi-SFU mode as well. This likely would've added some confusion to our attempts to test these modes.
2025-10-21 13:01:38 -04:00
Timo K
49d5a54d7e lint
Signed-off-by: Timo K <toger5@hotmail.de>
2025-10-21 14:42:46 +02:00
Timo K
e0c0923993 review
Signed-off-by: Timo K <toger5@hotmail.de>
2025-10-21 14:16:24 +02:00
Timo K
e523776d06 always use multi sfu if we are using sticky events.
Signed-off-by: Timo K <toger5@hotmail.de>
2025-10-20 17:07:07 +02:00
Timo K
63be9939b1 fix demo
Signed-off-by: Timo K <toger5@hotmail.de>
2025-10-16 01:02:25 +02:00
Timo K
b34bee05c9 bump js-sdk fix sticky events type
Signed-off-by: Timo K <toger5@hotmail.de>
2025-10-15 17:47:48 +02:00
Will Hunt
a9586fe64b update checksum 2025-10-15 15:38:40 +01:00
Will Hunt
3337768098 Merge remote-tracking branch 'origin/voip-team/rebased-multiSFU' into toger5/sticky-events-version 2025-10-15 15:27:34 +01:00
Robin
2c66e11a0a Fix connection tests
Removing one of these tests because it was doing a lot of work to test something that was made trivial in b0eb566a4f.
2025-10-14 16:04:52 -04:00
Robin
102e581c41 Serialize updates to the call intent
So that the value advertised in your membership can't desync from the actual value if you toggle video too fast.
2025-10-14 15:20:07 -04:00
Robin
b0eb566a4f Simplify local transport connection state tracking 2025-10-14 14:43:13 -04:00
Robin
2dc6134606 Stick to the term 'transport' rather than 'focus' 2025-10-14 14:38:37 -04:00
Robin
ea17ed7253 Rename 'localTransport' to 'transport', since it's often remote 2025-10-14 14:34:51 -04:00
Robin
9f4e99310b Elaborate on a TODO for preloading JWT tokens 2025-10-14 14:32:56 -04:00
Robin
625cfa15ff Move sorting bin calculation into UserMedia 2025-10-14 14:31:35 -04:00
Robin
95069fd3fa Fix joining call with audio and video muted 2025-10-14 12:19:23 -04:00
Robin
13636b78d9 Replace deprecated CallMembership.sender with userId 2025-10-14 12:07:51 -04:00
Robin
bcbf7a90f0 Initialize preferredTransport$ in a less unusual way 2025-10-14 12:00:00 -04:00
Robin
2de7a20d8a Remove outdated comment 2025-10-14 11:54:36 -04:00
Valere
a6c4fb4148 review: count as publishing even if not yet connected to LK 2025-10-14 17:47:38 +02:00
Robin
b030d304df Fix crash during focus switching 2025-10-14 09:29:48 -04:00
Robin
392c51fbf6 Merge branch 'voip-team/rebased-multiSFU' of github.com:element-hq/element-call into voip-team/rebased-multiSFU 2025-10-14 09:22:08 -04:00
Robin
a18700cbcd Avoid updating membership during focus switch 2025-10-14 09:16:46 -04:00
Valere
93d763f58f devtool: quick display of focus URL in stats tile 2025-10-14 14:06:54 +02:00
Valere
58d60b35fd fix CI failing with Invalid value "iife" for option "worker.format"
UMD and IIFE output formats are not supported for code-splitting builds. see https://github.com/vitejs/vite/issues/18585
2025-10-14 12:25:31 +02:00
Valere
60332dc2db fix js-sdk dependency format 2025-10-14 12:16:24 +02:00
Valere
a9db9c8b59 ErrorHandling: publish connection error handling 2025-10-14 10:46:57 +02:00
Valere
a5aba928dd dependency: depends on js-sdk develop 2025-10-13 16:39:14 +02:00
Valere
0e1b61a5e8 refactor: Split out all exports of CallViewModel to their own file 2025-10-13 16:24:55 +02:00
Valere
8823be67c5 refactor extract inner classes to their own files 2025-10-13 15:43:12 +02:00
Valere
8e6eb70e5b refactor: use EnterRTCSessionOptions instead of unnamed bools 2025-10-13 13:52:01 +02:00
Half-Shot
b90bf7da2d More tidy 2025-10-13 12:05:01 +01:00
Half-Shot
6580555bda update docker image 2025-10-13 11:58:33 +01:00
Half-Shot
abcf30083a lint 2025-10-13 11:58:13 +01:00
Half-Shot
1546c04a5b Fixup call detection logic to allow sticky events 2025-10-13 11:58:13 +01:00
Half-Shot
3ffaf33701 Add prefer sticky setting] 2025-10-13 11:58:13 +01:00
Half-Shot
f3abcb61cf Remove unused useNewMembershipmanager setting 2025-10-13 11:57:32 +01:00
Timo K
2dcb823899 enable sticky events in the joinSessionConfig
Signed-off-by: Timo K <toger5@hotmail.de>
2025-10-13 11:56:16 +01:00
Timo K
4bfe5c5a7c add sticky event support
- use new js-sdk
 - use custom synapse
 - don't filter rooms by existing call state events

Signed-off-by: Timo K <toger5@hotmail.de>
2025-10-13 11:56:04 +01:00
Valere Fedronic
c846ea6447 Merge pull request #3521 from element-hq/valere/multi-sfu/connection_states
Connection State: Error Handling + Cleaning / refactoring /
2025-10-13 10:15:55 +02:00
Valere
fc2384e330 post merge fixes (js-sdk changes) 2025-10-10 19:23:24 +02:00
Valere
2fc7f11bea prettier fix 2025-10-10 16:51:58 +02:00
Valere
cca46bd8dd Merge branch 'voip-team/rebased-multiSFU' into valere/multi-sfu/connection_states 2025-10-10 16:51:09 +02:00
Valere
1e75f9adcc test: fix additional test with proper mutestate fix 2025-10-10 15:13:55 +02:00
Valere
4a8f5bccac post merge lint fixes 2025-10-10 15:04:12 +02:00
Valere
4608d68cd9 Merge branch 'voip-team/rebased-multiSFU' into valere/multi-sfu/connection_states 2025-10-10 15:01:49 +02:00
Timo K
0fd414358c Merge branch 'livekit' into voip-team/rebased-multiSFU
Signed-off-by: Timo K <toger5@hotmail.de>
2025-10-10 12:26:47 +02:00
Valere
1ab081d636 test: MISSING_MATRIX_RTC_FOCUS renamed as MISSING_MATRIX_RTC_TRANSPORT 2025-10-10 11:41:26 +02:00
Valere
6710f4c72a Test: Fix mocking to fix failing tests 2025-10-10 11:09:41 +02:00
Robin
de5f5198dd Merge branch 'livekit' into voip-team/rebased-multiSFU
Note that this effectively *reverts* 3ac2aa8526 because this branch now has what is a better UX (at least I think so): the in-call view is presented instantly when pressing the join button. Errors that occur during initial connection procedures will be surfaced more uniformly.
2025-10-09 16:44:12 -04:00
Robin
f9f2896a6a Merge branch 'voip-team/rebased-multiSFU' of github.com:element-hq/element-call into voip-team/rebased-multiSFU 2025-10-09 16:37:48 -04:00
Valere
a500915c43 test: Fix mute test, behavior change from setMuted to setAudioEnabled
useCallViewKeyboardShortcuts() changed a param from `setMicrophoneMuted` to `setAudioEnabled`, the boolean arg of the callback is inverse tht it used to be
2025-10-09 19:24:44 +02:00
Valere
7cbb1ec1e8 Simplify AudioRenderer and add more tests 2025-10-09 15:33:25 +02:00
Timo K
39f8cb95ec make compatible with related_event (async CallMembership) branch
Signed-off-by: Timo K <toger5@hotmail.de>
2025-10-09 15:29:47 +02:00
Valere
4c6b960da3 fix: use correct TestEachFunction 2025-10-09 11:00:45 +02:00
Robin
85ffe68d98 Remove outdated comment 2025-10-08 19:20:21 -04:00
Robin
2c576a7477 Clean up subscriptions in Connection tests 2025-10-08 19:09:27 -04:00
Robin
64c2e5911c Update outdated comment 2025-10-08 19:09:08 -04:00
Robin
5be3b91509 Fix focus connection state typo, simplify its initialization 2025-10-08 19:09:03 -04:00
Robin
2d7e4247f3 Note a potential resource leak 2025-10-08 18:51:30 -04:00
Robin
00daf834b6 Remove local participant case (now enforced by types) from audio tests 2025-10-08 17:53:11 -04:00
Robin
dcc3ab641f Remove MockedObject from mockMediaDevices type signature 2025-10-08 17:40:03 -04:00
Robin
dee06a4b70 Remove unused useIsEarpiece hook 2025-10-08 17:37:54 -04:00
Robin
3691e7120d Restore a hidden 'null' state for the local transport/connection 2025-10-08 17:35:53 -04:00
Robin
8778be8351 Fix doc comment typo 2025-10-08 17:34:33 -04:00
Robin
e88474452f Correct / document some missing bits in tests 2025-10-08 17:33:17 -04:00
Robin
b1d143720a Add comments to Async 2025-10-08 17:08:51 -04:00
Robin
5da780ed30 Remove dead MuteStates file
It's been replaced by a refactored RxJS version living in src/state.
2025-10-08 16:43:43 -04:00
Robin
c96e81bfd3 Simplify type of audio participants exposed from CallViewModel 2025-10-08 16:42:36 -04:00
Robin
e346c8c148 Re-enable React strict mode 2025-10-08 16:39:27 -04:00
Robin
1a4b38cf93 Document ObservableScope.reconcile 2025-10-08 12:54:24 -04:00
Robin
5835cd423e Merge pull request #3517 from element-hq/renovate/github-actions
Update GitHub Actions
2025-10-08 10:19:22 -04:00
Valere
427a8dd644 test: Fix Audio render tests and added more 2025-10-08 14:48:40 +02:00
Valere
afe004c6e7 Remove un-necessary transport field, already accessible from connection 2025-10-08 14:30:52 +02:00
Valere
f5ea734a5c esLint fix 2025-10-08 14:29:59 +02:00
Valere
13fb46644c test: Fix mediaView test, ,member is not optional anymore
Updated the test because now name will be the userId instead of default display name
2025-10-08 08:50:35 +02:00
Robin
669bc76dd5 Replace calls to deprecated resolveActiveFocus 2025-10-08 01:04:58 -04:00
Valere
05e7b5a7ff fixup MediaView tests 2025-10-07 17:35:25 +02:00
Valere
18ba02c9c2 knip: remove dead code 2025-10-07 16:29:11 +02:00
Valere
529cb8a7ec prettier ! 2025-10-07 16:24:02 +02:00
Valere
7437961195 lint: fix import order 2025-10-07 16:12:23 +02:00
Valere
c820ba3983 build: update lock file 2025-10-07 16:07:46 +02:00
Valere
c3c0516f0d Lint: fix all the lint errors 2025-10-07 16:00:59 +02:00
Valere
597e6782a8 Merge branch 'voip-team/rebased-multiSFU' into valere/multi-sfu/connection_states 2025-10-07 10:33:31 +02:00
renovate[bot]
f08805d3ed Update GitHub Actions 2025-10-07 05:31:36 +00:00
Will Hunt
3ac2aa8526 Disable call button while the call is connecting. (#3531)
* Disable call button while the call is connecting.

* cleanup

* fix

* Update src/room/LobbyView.tsx

Co-authored-by: Robin <robin@robin.town>

* fixup

* appease linter

---------

Co-authored-by: Robin <robin@robin.town>
2025-10-06 18:52:58 +00:00
Valere
91a366fa2a tests: Publish connection states 2025-10-06 10:50:10 +02:00
Robin
1fff71ace1 Actually leave the MatrixRTC session again 2025-10-03 21:00:45 -04:00
Robin
1820cac3f6 Create media items for session members not joined to LiveKit 2025-10-03 19:14:48 -04:00
Robin
86fb026be8 Turn multi-SFU media transport into a developer option 2025-10-03 14:43:22 -04:00
Valere
00401ca38a refactor: PublishConnection extract from giant constructor 2025-10-02 15:15:23 +02:00
Valere
84f95be48d test: Ensure scope for publishers observer 2025-10-02 13:08:00 +02:00
Valere
0502f66e21 tests: Add publisher observable tests 2025-10-02 12:53:59 +02:00
Timo K
68aae4a8e3 fix another rename + another js-sdk bump
Signed-off-by: Timo K <toger5@hotmail.de>
2025-10-02 11:23:11 +02:00
Valere
dfaa6a33f4 fix lint errors 2025-10-01 17:24:19 +02:00
Valere
e8bf817f88 tests: end scope tests 2025-10-01 16:39:21 +02:00
Valere
6a1f7dd057 ConnectionState: test livekit connection states 2025-10-01 15:23:24 +02:00
Valere
22900161d6 extract common test setup 2025-10-01 14:47:45 +02:00
Valere
47c876f3df lint fixes 2025-10-01 14:37:03 +02:00
Valere
3d8639df03 Connection states tests 2025-10-01 14:21:37 +02:00
Valere
879a1d4af1 Connection: add Connection state and handle error on start 2025-10-01 11:23:16 +02:00
Robin
71127a4e94 Merge pull request #3516 from element-hq/toger5/voip-team/rebased-multiSFU-bump-to-new-js-sdk-version
Use updated multi sfu js-sdk
2025-09-30 11:24:12 -04:00
Valere
b00f7d5409 refactor: Remote / Publish Connection and constructor 2025-09-30 17:02:48 +02:00
Timo K
2819c79590 use updated multi sfu js-sdk
Signed-off-by: Timo K <toger5@hotmail.de>
2025-09-30 16:47:45 +02:00
Valere
edf68d16b7 refactoring: prep work extract to file + documentation 2025-09-30 11:33:45 +02:00
Robin
a4a0a58a72 Remove the option to show non-member ("ghost") participants
As we'd like to get the multi-SFU feature branch shipped, this is not the most important debugging tool to expend effort on at the moment.
2025-09-26 13:27:18 -04:00
Robin
dbdf853d55 Stop connections on view model destroy 2025-09-26 13:20:55 -04:00
Robin
0759f9b27d Don't render audio from participants that aren't meant to be publishing 2025-09-25 21:29:02 -04:00
Robin
4980d8a622 Merge branch 'livekit' into voip-team/rebased-multiSFU 2025-09-25 18:40:59 -04:00
Will Hunt
d24da1859e Add media hints for notification events. (#3493)
* Add media hints for notification events.

* Prevent showing calling view when disconnected from Livekit. (#3491)

* Refactor disconnection handling

* Use "unknown"

* Update signature

* Add tests

* Expose livekitConnectionState directly

* fix whoopsie

* Update dependency livekit-client to v2.15.7 (#3496)

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

* Fix the interactivity of buttons while reconnecting or in earpiece mode (#3486)

* Fix the interactivity of buttons while reconnecting or in earpiece mode

When we're in one of these modes, we need to ensure that everything above the overlay (the header and footer buttons) is interactive, while everything obscured by the overlay (the media tiles) is non-interactive and removed from the accessibility tree. It's not a very easy task to trap focus *outside* an element, so the best solution I could come up with is to set tabindex="-1" manually on all interactive elements belonging to the media tiles.

* Write a Playwright test for reconnecting

* fix lints

Signed-off-by: Timo K <toger5@hotmail.de>

* fix test

Signed-off-by: Timo K <toger5@hotmail.de>

* enable http2 for matrx-rtc host to allow the jwt service to talk to the SFU

* remove rate limit for delayed events

* more time to connect to livekit SFU

* Due to a Firefox issue we set the start anchor for the tab test to the Mute microphone button

* adapt to most recent Element Web version

* Use the "End call" button as proofe for a started call

* Currrenty disabled due to recent Element Web
- not indicating the number of participants
- bypassing Lobby

* linting

* disable 'can only interact with header and footer while reconnecting' for firefox

---------

Signed-off-by: Timo K <toger5@hotmail.de>
Co-authored-by: Timo <16718859+toger5@users.noreply.github.com>
Co-authored-by: Timo K <toger5@hotmail.de>
Co-authored-by: fkwp <github-fkwp@w4ve.de>

* Log when a track is unpublished or runs into an error (#3495)

* default mute states (unmuted!) in widget mode (embedded + intent) (#3494)

* default mute states (unmuted!) in widget mode (embedded + intent)

Signed-off-by: Timo K <toger5@hotmail.de>

* review

Signed-off-by: Timo K <toger5@hotmail.de>

* introduce a cache for the url params.

Signed-off-by: Timo K <toger5@hotmail.de>

* Add an option to skip the cache.

Signed-off-by: Timo K <toger5@hotmail.de>

---------

Signed-off-by: Timo K <toger5@hotmail.de>

* Apply new hint code

* missed a bit

* fix intent

* Automatically update intent on mute change

* update packages

* lint

* Fix tests

* fix merge fails

---------

Signed-off-by: Timo K <toger5@hotmail.de>
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: Robin <robin@robin.town>
Co-authored-by: Timo <16718859+toger5@users.noreply.github.com>
Co-authored-by: Timo K <toger5@hotmail.de>
Co-authored-by: fkwp <github-fkwp@w4ve.de>
2025-09-25 13:02:43 +01:00
Robin
530fbaf90a Clear up the room membership confusion around reading session members 2025-09-24 21:39:36 -04:00
Robin
6cf020763e Make UI react instantly to hanging up but also wait for leave sound
This ensures that we don't see a mistaken 'reconnecting' toast while we're hanging up (and also that the leave sound gets a chance to play in widgets once again).
2025-09-24 21:26:16 -04:00
Robin
edd3eb8747 Implement screen sharing 2025-09-24 13:54:54 -04:00
Robin
f99a256c86 Reset matrix-js-sdk to multi SFU branch 2025-09-24 13:53:39 -04:00
Robin
8e21ea6d51 Merge branch 'livekit' into voip-team/rebased-multiSFU 2025-09-24 10:12:26 -04:00
Timo K
6b44f3b008 a tiny bit of tests lint fixes.
Signed-off-by: Timo K <toger5@hotmail.de>
2025-09-23 12:25:31 +02:00
Timo K
96e96a5e43 fix leaving
Signed-off-by: Timo K <toger5@hotmail.de>
2025-09-23 12:25:05 +02:00
Timo K
7777179935 cleanup (delete files useLivekit) now covered by Connection.ts
Signed-off-by: Timo K <toger5@hotmail.de>
2025-09-23 11:40:29 +02:00
Timo K
78e9521f22 Make track processor work
Signed-off-by: Timo K <toger5@hotmail.de>
2025-09-23 11:38:34 +02:00
fkwp
342dd2e7d7 DevX: Add another Matrix site to docker-compose (#3511)
* remove redis, since we dont use it

* update localhost TLS certificat to add *.othersite.m.localhost wildcard

* allow for federation

* Add services and config files for Matrix site othersite.m.localhost

* add element web instance app.othersite.m.localhost

* update README

* exclude synapse database for othersite.m.localhost

* linting
2025-09-22 21:14:12 +02:00
Timo K
8bf24895ce TODO: settings modal with multiple connections
Signed-off-by: Timo K <toger5@hotmail.de>
2025-09-22 14:18:23 +02:00
Timo K
dddda7057f add todo comments and who works on them
Signed-off-by: Timo K <toger5@hotmail.de>
2025-09-22 14:17:52 +02:00
Timo K
02f23e25fd remove todo from matrix audio renderer
Signed-off-by: Timo K <toger5@hotmail.de>
2025-09-22 14:16:24 +02:00
Timo
b2c594d36c Never skip lobby for huddles (#3508)
* Never skip lobby for huddles

Signed-off-by: Timo K <toger5@hotmail.de>

* tests

Signed-off-by: Timo K <toger5@hotmail.de>

---------

Signed-off-by: Timo K <toger5@hotmail.de>
2025-09-22 11:35:20 +02:00
Timo K
d9fe31039f start fixing CallViewModel tests.
Signed-off-by: Timo K <toger5@hotmail.de>
2025-09-19 18:01:45 +02:00
Timo K
41e152f420 dont throw disconnected error at start of the call
Signed-off-by: Timo K <toger5@hotmail.de>
2025-09-19 17:52:12 +02:00
Timo K
ccfd32c9b2 move leave logic into view model
Signed-off-by: Timo K <toger5@hotmail.de>
2025-09-19 17:52:12 +02:00
Timo K
38d78ddce4 make audio work
Signed-off-by: Timo K <toger5@hotmail.de>
2025-09-19 17:50:43 +02:00
Timo K
cc870c3cc2 enable encryption in per sender case
Signed-off-by: Timo K <toger5@hotmail.de>
2025-09-19 17:50:43 +02:00
Timo K
c8098734cd one e2ee worker per session
Signed-off-by: Timo K <toger5@hotmail.de>
2025-09-19 17:50:43 +02:00
Robin
e08f16f889 All my Friday work. Demoable! 2025-09-19 17:50:43 +02:00
Timo K
386dc6c84d temp before holiday
Signed-off-by: Timo K <toger5@hotmail.de>
2025-09-19 17:50:43 +02:00
Robin
d46fe55a67 Import unfinished mute states refactor 2025-09-19 17:50:43 +02:00
Robin
02f4c73759 Add my own local storage SFU config stuff too 2025-09-19 17:50:43 +02:00
Timo K
598371b807 lots of work. noone knows if it works.
Signed-off-by: Timo K <toger5@hotmail.de>
2025-09-19 17:50:43 +02:00
Timo K
802ebf828d refactor connection
Signed-off-by: Timo K <toger5@hotmail.de>
2025-09-19 17:50:43 +02:00
Timo K
e4a54e3a19 refactor connnection class
Signed-off-by: Timo K <toger5@hotmail.de>
2025-09-19 17:50:43 +02:00
Timo K
a617a92e88 make it work
Signed-off-by: Timo K <toger5@hotmail.de>
2025-09-19 17:50:43 +02:00
Timo K
33bc78eec1 add logging
Signed-off-by: Timo K <toger5@hotmail.de>
2025-09-19 17:50:43 +02:00
Timo K
8ffb360114 add local storage + more readable + remoteParticipants + use publishingParticipants
Signed-off-by: Timo K <toger5@hotmail.de>
2025-09-19 17:50:43 +02:00
Timo K
55b46b3f33 introduce publishingParticipants$
Signed-off-by: Timo K <toger5@hotmail.de>
2025-09-19 17:50:43 +02:00
Timo K
6bdfd7fbd8 add comment
Signed-off-by: Timo K <toger5@hotmail.de>
2025-09-19 17:50:43 +02:00
Timo K
a10489d7df publish audio in remote rooms
Signed-off-by: Timo K <toger5@hotmail.de>
2025-09-19 17:50:43 +02:00
Timo K
7b88420f6a first video!
Signed-off-by: Timo K <toger5@hotmail.de>
2025-09-19 17:50:43 +02:00
Robin
cb91f1ad4f Make it actually join the session 2025-09-19 17:50:43 +02:00
Robin
c6e8c94fd6 Fix makeFocus 2025-09-19 17:50:43 +02:00
Timo K
376a4b4e4a initial compiling version
Signed-off-by: Timo K <toger5@hotmail.de>
2025-09-19 17:50:43 +02:00
Robin
d9b6302bf7 Fix crash? 2025-09-19 17:50:43 +02:00
Robin
35319dd6b5 Fix some errors in CallViewModel 2025-09-19 17:50:43 +02:00
Timo K
9011ae4e1f temp
Signed-off-by: Timo K <toger5@hotmail.de>
2025-09-19 17:50:43 +02:00
Timo
9b9c08ed61 Refactor ring$ observable (#3504)
* Refactor ring$ observable

Signed-off-by: Timo K <toger5@hotmail.de>

* fix ci

Signed-off-by: Timo K <toger5@hotmail.de>

* fix regression test

Signed-off-by: Timo K <toger5@hotmail.de>

---------

Signed-off-by: Timo K <toger5@hotmail.de>
2025-09-19 17:43:31 +02:00
Timo
317b2dc796 Fix: never stop ring feedback on the sender side. (#3502)
* make ring$ a behavior and add code comments to justify/explain the change.

Signed-off-by: Timo K <toger5@hotmail.de>

* Add test: reproduce "ring does not stop" race.

Signed-off-by: Timo K <toger5@hotmail.de>

---------

Signed-off-by: Timo K <toger5@hotmail.de>
2025-09-19 16:42:47 +02:00
Jorge Martin Espinosa
0774d18f92 Android: allow RCs to be published as non-SNAPSHOT versions (#3501)
* Android: allow RCs to be published as non-SNAPSHOT versions

* Fix broken string in build script
2025-09-18 19:22:24 +02:00
Timo
6e9dc34008 Add ring notification to UserIntent.StartNewCallDM (#3499)
* Add ring notification to UserIntent.StartNewCallDM

Signed-off-by: Timo K <toger5@hotmail.de>

* Add more tests (refactor to compute + get)

Signed-off-by: Timo K <toger5@hotmail.de>

---------

Signed-off-by: Timo K <toger5@hotmail.de>
2025-09-18 17:45:49 +02:00
99 changed files with 4789 additions and 2478 deletions

View File

@@ -10,7 +10,7 @@ jobs:
pull-requests: read
steps:
- name: Add notice
uses: actions/github-script@60a0d83039c74a4aee543508d2ffcb1c3799cdea # v7
uses: actions/github-script@f28e40c7f34bde8b3046d885e986cb6290c5673b # v7
if: contains(github.event.pull_request.labels.*.name, 'X-Blocked')
with:
script: |

View File

@@ -23,7 +23,7 @@ jobs:
packages: write
steps:
- name: Check it out
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4
uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4
- name: 📥 Download artifact
uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4
@@ -34,7 +34,7 @@ jobs:
path: dist
- name: Log in to container registry
uses: docker/login-action@74a5d142397b4f367a81961eba4e8cd7edddf772 # v3.4.0
uses: docker/login-action@5e57cd118135c172c3672efd75eb46360885c0ef # v3.6.0
with:
registry: ${{ env.REGISTRY }}
username: ${{ github.actor }}
@@ -42,7 +42,7 @@ jobs:
- name: Extract metadata (tags, labels) for Docker
id: meta
uses: docker/metadata-action@902fa8ec7d6ecbf8d84d538b9b233a880e428804 # v5.7.0
uses: docker/metadata-action@c1e51972afc2121e065aed6d45c65596fe445f3f # v5.8.0
with:
images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
tags: ${{ inputs.docker_tags}}
@@ -50,10 +50,10 @@ jobs:
org.opencontainers.image.licenses=AGPL-3.0-only OR LicenseRef-Element-Commercial
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@b5ca514318bd6ebac0fb2aedd5d36ec1b5c232a2 # v3.10.0
uses: docker/setup-buildx-action@e468171a9de216ec08956ac3ada2f0791b6bd435 # v3.11.1
- name: Build and push Docker image
uses: docker/build-push-action@14487ce63c7a62a4a324b0bfb37086795e31c6c1 # v6.16.0
uses: docker/build-push-action@263435318d21b8e681c14492fe198d362a7d2c83 # v6.18.0
with:
context: .
platforms: linux/amd64,linux/arm64

View File

@@ -32,7 +32,7 @@ jobs:
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4
uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4
- name: Enable Corepack
run: corepack enable
- name: Yarn cache

View File

@@ -7,7 +7,7 @@ jobs:
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4
uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4
- name: Enable Corepack
run: corepack enable
- name: Yarn cache

View File

@@ -44,6 +44,8 @@ jobs:
run: |
if [[ "${VERSION}" =~ ^v[0-9]+\.[0-9]+\.[0-9]+$ ]]; then
echo "TAG=latest" >> "$GITHUB_OUTPUT"
elif [[ "${VERSION}" =~ ^v[0-9]+\.[0-9]+\.[0-9]+\-rc\.[0-9]+$ ]]; then
echo "TAG=rc" >> "$GITHUB_OUTPUT"
else
echo "TAG=other" >> "$GITHUB_OUTPUT"
fi
@@ -83,7 +85,7 @@ jobs:
run: find ${{ env.FILENAME_PREFIX }} -type f -print0 | sort -z | xargs -0 sha256sum | tee ${{ env.FILENAME_PREFIX }}.sha256
- name: Upload
if: ${{ needs.versioning.outputs.DRY_RUN == 'false' }}
uses: softprops/action-gh-release@da05d552573ad5aba039eaac05058a918a7bf631 # v2
uses: softprops/action-gh-release@aec2ec56f94eb8180ceec724245f64ef008b89f5 # v2
with:
files: |
${{ env.FILENAME_PREFIX }}.tar.gz
@@ -101,7 +103,7 @@ jobs:
id-token: write # required for the provenance flag on npm publish
steps:
- name: Checkout
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4
uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4
- name: 📥 Download built element-call artifact
uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4
@@ -142,7 +144,7 @@ jobs:
contents: read
steps:
- name: Checkout
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4
uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4
- name: 📥 Download built element-call artifact
uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4
@@ -163,6 +165,8 @@ jobs:
run: |
if [[ "${{ needs.versioning.outputs.TAG }}" == "latest" ]]; then
echo "ARTIFACT_VERSION=${{ needs.versioning.outputs.UNPREFIXED_VERSION }}" >> "$GITHUB_ENV"
elif [[ "${{ needs.versioning.outputs.TAG }}" == "rc" ]]; then
echo "ARTIFACT_VERSION=${{ needs.versioning.outputs.UNPREFIXED_VERSION }}" >> "$GITHUB_ENV"
else
echo "ARTIFACT_VERSION=${{ needs.versioning.outputs.UNPREFIXED_VERSION }}-SNAPSHOT" >> "$GITHUB_ENV"
fi
@@ -195,7 +199,7 @@ jobs:
contents: read
steps:
- name: Checkout
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4
uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4
with:
path: element-call
@@ -208,7 +212,7 @@ jobs:
path: element-call/embedded/ios/Sources/dist
- name: Checkout element-call-swift
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4
uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4
with:
repository: element-hq/element-call-swift
path: element-call-swift
@@ -260,7 +264,7 @@ jobs:
echo "iOS: ${{ needs.publish_ios.outputs.ARTIFACT_VERSION }}"
- name: Add release notes
if: ${{ needs.versioning.outputs.DRY_RUN == 'false' }}
uses: softprops/action-gh-release@da05d552573ad5aba039eaac05058a918a7bf631 # v2
uses: softprops/action-gh-release@aec2ec56f94eb8180ceec724245f64ef008b89f5 # v2
with:
append_body: true
body: |

View File

@@ -42,7 +42,7 @@ jobs:
- name: Create Checksum
run: find ${{ env.FILENAME_PREFIX }} -type f -print0 | sort -z | xargs -0 sha256sum | tee ${{ env.FILENAME_PREFIX }}.sha256
- name: Upload
uses: softprops/action-gh-release@da05d552573ad5aba039eaac05058a918a7bf631 # v2
uses: softprops/action-gh-release@aec2ec56f94eb8180ceec724245f64ef008b89f5 # v2
with:
files: |
${{ env.FILENAME_PREFIX }}.tar.gz
@@ -68,7 +68,7 @@ jobs:
runs-on: ubuntu-latest
steps:
- name: Add release note
uses: softprops/action-gh-release@da05d552573ad5aba039eaac05058a918a7bf631 # v2
uses: softprops/action-gh-release@aec2ec56f94eb8180ceec724245f64ef008b89f5 # v2
with:
append_body: true
body: |

View File

@@ -9,7 +9,7 @@ jobs:
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4
uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4
- name: Enable Corepack
run: corepack enable
- name: Yarn cache
@@ -22,7 +22,7 @@ jobs:
- name: Vitest
run: "yarn run test:coverage"
- name: Upload to codecov
uses: codecov/codecov-action@18283e04ce6e62d37312384ff67231eb8fd56d24 # v5
uses: codecov/codecov-action@5a1091511ad55cbe89839c7260b706298ca349f7 # v5
env:
CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
with:
@@ -33,7 +33,7 @@ jobs:
timeout-minutes: 30
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4
- uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4
- name: Enable Corepack
run: corepack enable
- uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4

View File

@@ -13,7 +13,7 @@ jobs:
steps:
- name: Checkout the code
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4
uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4
- name: Enable Corepack
run: corepack enable

View File

@@ -14,7 +14,7 @@ jobs:
steps:
- name: Checkout the code
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4
uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4
- name: Upload
uses: localazy/upload@27e6b5c0fddf4551596b42226b1c24124335d24a # v1

3
.gitignore vendored
View File

@@ -8,6 +8,7 @@ dist-ssr
.idea/
public/config.json
backend/synapse_tmp/*
backend/synapse_tmp_othersite/*
/coverage
config.json
@@ -28,4 +29,4 @@ yarn-error.log
/test-results/
/playwright-report/
/blob-report/
/playwright/.cache/
/playwright/.cache/

View File

@@ -206,22 +206,22 @@ See also:
### Backend
A docker compose file `dev-backend-docker-compose.yml` is provided to start the
whole stack of components which is required for a local development environment:
whole stack of components which is required for a local development environment
including federation:
- Minimum Synapse Setup (servername: `synapse.m.localhost`)
- Minimum Synapse Setup (servernameis: `synapse.m.localhost`, `synapse.othersite.m.localhost`)
- MatrixRTC Authorization Service (Note requires Federation API and hence a TLS reverse proxy)
- Minimum LiveKit SFU Setup using dev defaults for config
- Redis db for completeness
- Minimum LiveKit SFU setup using dev defaults for config
- Minimum `localhost` Certificate Authority (CA) for Transport Layer Security (TLS)
- Hostnames: `m.localhost`, `*.m.localhost`
- Hostnames: `m.localhost`, `*.m.localhost`, `*.othersite.m.localhost`
- Add [./backend/dev_tls_local-ca.crt](./backend/dev_tls_local-ca.crt) to your web browsers trusted
certificates
- Minimum TLS reverse proxy for
- Synapse homeserver: `synapse.m.localhost`
- MatrixRTC backend: `matrix-rtc.m.localhost`
- Synapse homeserver: `synapse.m.localhost` and `synapse.othersite.m.localhost`
- MatrixRTC backend: `matrix-rtc.m.localhost` and `matrix-rtc.othersite.m.localhost`
- Local Element Call development `call.m.localhost` via `yarn dev --host `
- Element Web `app.m.localhost`
- Note certificates will expire on Thu, 03 May 2035 10:32:02 GMT
- Element Web `app.m.localhost` and `app.othersite.m.localhost`
- Note certificates will expire on Thr, 20 September 2035 14:27:35 CEST
These use a test 'secret' published in this repository, so this must be used
only for local development and **_never be exposed to the public Internet._**

View File

@@ -0,0 +1,64 @@
server_name: "synapse.othersite.m.localhost"
public_baseurl: https://synapse.othersite.m.localhost/
pid_file: /data/homeserver.pid
listeners:
- port: 18008
tls: false
type: http
x_forwarded: true
resources:
- names: [client, federation, openid]
compress: false
database:
name: sqlite3
args:
database: /data/homeserver.db
media_store_path: /data/media_store
signing_key_path: "/data/SERVERNAME.signing.key"
# Due to custom TLS certificate with domains
# - m.localhost, localhost
# - *.m.localhost
# - *.othersite.m.localhost
# we disable certificate verification to allow for federation
# WARNING: DO NOT USE IN PRODUCTION!!!
federation_verify_certificates: false
ip_range_blacklist: []
trusted_key_servers:
- server_name: "synapse.m.localhost"
accept_keys_insecurely: true
experimental_features:
# MSC3266: Room summary API. Used for knocking over federation
msc3266_enabled: true
# MSC4222 needed for syncv2 state_after. This allow clients to
# correctly track the state of the room.
msc4222_enabled: true
# The maximum allowed duration by which sent events can be delayed, as
# per MSC4140. Must be a positive value if set. Defaults to no
# duration (null), which disallows sending delayed events.
max_event_delay_duration: 24h
# Required for Element Call in Single Page Mode due to on-the-fly user registration
enable_registration: true
enable_registration_without_verification: true
report_stats: false
serve_server_wellknown: true
# Ratelimiting settings for client actions (registration, login, messaging).
#
# Each ratelimiting configuration is made of two parameters:
# - per_second: number of requests a client can send per second.
# - burst_count: number of requests a client can send before being throttled.
rc_message:
# This needs to match at least the heart-beat frequency plus a bit of headroom
# Currently the heart-beat is every 5 seconds which translates into a rate of 0.2s
per_second: 0.5
burst_count: 30

View File

@@ -19,8 +19,18 @@ database:
media_store_path: /data/media_store
signing_key_path: "/data/SERVERNAME.signing.key"
# Due to custom TLS certificate with domains
# - m.localhost, localhost
# - *.m.localhost
# - *.othersite.m.localhost
# we disable certificate verification to allow for federation.
# WARNING: DO NOT USE IN PRODUCTION!!!
federation_verify_certificates: false
ip_range_blacklist: []
trusted_key_servers:
- server_name: "matrix.org"
- server_name: "synapse.othersite.m.localhost"
accept_keys_insecurely: true
experimental_features:
# MSC3266: Room summary API. Used for knocking over federation
@@ -28,12 +38,21 @@ experimental_features:
# MSC4222 needed for syncv2 state_after. This allow clients to
# correctly track the state of the room.
msc4222_enabled: true
# sticky events for matrixRTC user state
msc4354_enabled: true
# The maximum allowed duration by which sent events can be delayed, as
# per MSC4140. Must be a positive value if set. Defaults to no
# duration (null), which disallows sending delayed events.
max_event_delay_duration: 24h
# Required for Element Call in Single Page Mode due to on-the-fly user registration
enable_registration: true
enable_registration_without_verification: true
report_stats: false
serve_server_wellknown: true
# Ratelimiting settings for client actions (registration, login, messaging).
#
# Each ratelimiting configuration is made of two parameters:
@@ -45,10 +64,3 @@ rc_message:
# Currently the heart-beat is every 5 seconds which translates into a rate of 0.2s
per_second: 0.5
burst_count: 30
# Required for Element Call in Single Page Mode due to on-the-fly user registration
enable_registration: true
enable_registration_without_verification: true
report_stats: false
serve_server_wellknown: true

View File

@@ -0,0 +1,20 @@
port: 17880
bind_addresses:
- "0.0.0.0"
rtc:
tcp_port: 17881
port_range_start: 50300
port_range_end: 50400
use_external_ip: false
turn:
enabled: false
domain: localhost
cert_file: ""
key_file: ""
tls_port: 5349
udp_port: 443
external_tls: true
keys:
devkey: secret
room:
auto_create: false

View File

@@ -6,11 +6,6 @@ rtc:
port_range_start: 50100
port_range_end: 50200
use_external_ip: false
#redis:
# address: redis:6379
# username: ""
# password: ""
# db: 0
turn:
enabled: false
domain: localhost

View File

@@ -1,4 +1,5 @@
# Synapse reverse proxy including .well-known/matrix/client
# domain synapse.m.localhost
server {
listen 80;
listen [::]:80;
@@ -26,14 +27,53 @@ server {
# This is also required for development environment.
# Reason: the lk-jwt-service uses the federation API for the openid token
# verification, which requires TLS
location / {
proxy_pass "http://homeserver:8008";
location ~ ^(/_matrix|/_synapse/client) {
proxy_pass "http://homeserver:8008";
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
}
error_page 500 502 503 504 /50x.html;
}
# Synapse reverse proxy including .well-known/matrix/client
# domain synapse.othersite.m.localhost
server {
listen 80;
listen [::]:80;
listen 443 ssl;
listen 8448 ssl;
listen [::]:443 ssl;
listen [::]:8448 ssl;
server_name synapse.othersite.m.localhost;
ssl_certificate /root/ssl/cert.pem;
ssl_certificate_key /root/ssl/key.pem;
# well-known config adding rtc_foci backend
# Note well-known is currently not effective due to:
# https://spec.matrix.org/v1.12/client-server-api/#well-known-uri the spec
# says it must be at https://$server_name/... (implied port 443) Hence, we
# currently rely for local development environment on deprecated config.json
# setting for livekit_service_url
location /.well-known/matrix/client {
add_header Access-Control-Allow-Origin *;
return 200 '{"m.homeserver": {"base_url": "https://synapse.othersite.m.localhost"}, "org.matrix.msc4143.rtc_foci": [{"type": "livekit", "livekit_service_url": "https://matrix-rtc.othersite.m.localhost/livekit/jwt"}]}';
default_type application/json;
}
# Reverse proxy for Matrix Synapse Homeserver
# This is also required for development environment.
# Reason: the lk-jwt-service uses the federation API for the openid token
# verification, which requires TLS
location ~ ^(/_matrix|/_synapse/client) {
proxy_pass "http://homeserver-1:18008";
proxy_http_version 1.1;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
}
error_page 500 502 503 504 /50x.html;
@@ -41,6 +81,7 @@ server {
}
# MatrixRTC reverse proxy
# domain matrix-rtc.m.localhost
# - MatrixRTC Authorization Service
# - LiveKit SFU websocket signaling connection
upstream jwt-auth-services {
@@ -49,12 +90,8 @@ upstream jwt-auth-services {
}
server {
listen 80;
listen [::]:80;
listen 443 ssl;
listen [::]:443 ssl;
listen 8448 ssl;
listen [::]:8448 ssl;
server_name matrix-rtc.m.localhost;
ssl_certificate /root/ssl/cert.pem;
ssl_certificate_key /root/ssl/key.pem;
@@ -96,6 +133,54 @@ server {
}
# MatrixRTC reverse proxy
# domain matrix-rtc.othersite.m.localhost
# - MatrixRTC Authorization Service
# - LiveKit SFU websocket signaling connection
server {
listen 443 ssl;
listen [::]:443 ssl;
server_name matrix-rtc.othersite.m.localhost;
ssl_certificate /root/ssl/cert.pem;
ssl_certificate_key /root/ssl/key.pem;
http2 on;
location ^~ /livekit/jwt/ {
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
# JWT Service running at port 16080
proxy_pass http://auth-service-1:16080/;
}
location ^~ /livekit/sfu/ {
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_send_timeout 120;
proxy_read_timeout 120;
proxy_buffering off;
proxy_set_header Accept-Encoding gzip;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
# LiveKit SFU websocket connection running at port 17880
proxy_pass http://livekit-sfu-1:17880/;
}
error_page 500 502 503 504 /50x.html;
}
# Convenience reverse proxy for the call.m.localhost domain to yarn dev --host
server {
listen 80;
@@ -161,3 +246,36 @@ server {
error_page 500 502 503 504 /50x.html;
}
# Convenience reverse proxy app.othersite.m.localhost for element web
server {
listen 80;
listen [::]:80;
server_name app.othersite.m.localhost;
return 301 https://$host$request_uri;
}
server {
listen 443 ssl;
listen [::]:443 ssl;
server_name app.othersite.m.localhost;
ssl_certificate /root/ssl/cert.pem;
ssl_certificate_key /root/ssl/key.pem;
location ^~ / {
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_pass http://element-web-1:18081;
proxy_ssl_verify off;
}
error_page 500 502 503 504 /50x.html;
}

View File

@@ -1,19 +1,19 @@
-----BEGIN CERTIFICATE-----
MIIDGjCCAgKgAwIBAgIUGdiFHhH4KL2pqBjMQHQ+PVIkSV8wDQYJKoZIhvcNAQEL
BQAwHjEcMBoGA1UEAwwTRWxlbWVudCBDYWxsIERldiBDQTAeFw0yNTA1MDUxMDMy
MDJaFw0zNTA1MDMxMDMyMDJaMB4xHDAaBgNVBAMME0VsZW1lbnQgQ2FsbCBEZXYg
Q0EwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDA2y0hjmNn1vRsVSdy
8IOfo8N1q9UgkhQWpGKXzPh+D5d1fnuJEmHIVwtDEtS/PwQ43LTmegChPtKH9jdT
tG0IihW9Ja5YNG+9xAwaoA/sB3CGCBYsz+2/XjVUpXoBJXIPoFBWsn+K0oeFw9fw
eRO1z9abM4cl+LjKzMNM8CCyu9uI1MaGjYez2YIWvG854VucLxX7HSlMJxZNWnie
Ui7fMakuJhB2+aiIQjdKxy4E5RHNhzYG/LXhvP+wBYBDPNRsP3rtzEaE9HAveL9K
FGqd3R4cBia6r1WIXmpAzyu5RGP5Eou0TZlGkal96/bF0I7q/pKlL23Jt1BLPiQU
KGKrAgMBAAGjUDBOMB0GA1UdDgQWBBQJqBjMu61c1p24txw/y+kv3D+V6DAfBgNV
HSMEGDAWgBQJqBjMu61c1p24txw/y+kv3D+V6DAMBgNVHRMEBTADAQH/MA0GCSqG
SIb3DQEBCwUAA4IBAQB8m2YfFGLugNt5vAAOvNxVqDA8c72yCVYr3CBCpmTIEY5Z
d3qVGhG9//ux6+J8ntkSwd9nV5GJyYXHukCG1VavnAWolWdNF/WAllf0jhLuz7kD
/cJnuI1By4tBsBmSz851i6HJ4t5k99Be+6GQVzi0e7zzfxTHZE4xP2J6Ox8QbPsP
n0m76nIp/WbWaJqzvIIjJhmUUPPv+4wN+eOArgjiGLzptM2qTtGZtd0c9nS5gvep
+mEbSUN9zkhAroZf80wf+hEvy+fJ94VbZ9QjTzTg7odZLrsXGIe8DaG63EYRQ25b
W5iYBAreln5fGSt7qHsGfqwZibTEk/Lx3dydO1Kg
MIIDGjCCAgKgAwIBAgIUbSbx+1UGptOTGefqEn7Zh3yoChIwDQYJKoZIhvcNAQEL
BQAwHjEcMBoGA1UEAwwTRWxlbWVudCBDYWxsIERldiBDQTAeFw0yNTA5MjIxMjI3
MzVaFw0zNTA5MjAxMjI3MzVaMB4xHDAaBgNVBAMME0VsZW1lbnQgQ2FsbCBEZXYg
Q0EwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDHODfkrFsOkqCnXnTb
QWz3LkUtNCoVvM7wFouirRnITJYG+lFwF+zNl89Eaq+uUN4bwd8ml1ZuR9p+1azC
SlklD5adhCR/ErknfUWamQEf6amSs3p0NnqnhXbnDEEbQOwNaPU/aGc6aw0+I9O6
NQ/H830GlVuKd24Bfv0mx6Imo0Hi9jxKYhqFh80nmltk2uyXefaJxuo1jXBhwLyC
DW8RVj55QvkZyBUzid8yslxrlo0LHKCCjZflwJJv5f+jaubkH5c0qxVaoR4+Liyt
X/4viIwt3Mhj04ppudTvt973mTbjRG5haCz9y7OkT1mMWhc0xrdMFX+gjPERYS2H
Ru/RAgMBAAGjUDBOMB0GA1UdDgQWBBTXNfLAKVayGQda/JZLPszrpz6LVzAfBgNV
HSMEGDAWgBTXNfLAKVayGQda/JZLPszrpz6LVzAMBgNVHRMEBTADAQH/MA0GCSqG
SIb3DQEBCwUAA4IBAQCvGfyopHHgZB+horGH6i/Xg41V+r4d0o092F1Lfr4vh86e
XMakRw92vsyk/iWOnLPNPcpVWzPcvINaCs/bahgnGSOAnrA4jjcXqymyGIy/6xc5
1EeZAxehiL9E5q4LQ841HDX0gps4ZzUO1BRYQcjG9Rdts83JO2ekkfHkZdNj2eQr
KOrr92Na1/w+EQdo/T9Rs2ES623xKEOxPqb8d/rx5Z4DdeuGx1u+3AfS76Lpo4ni
EJ0g1ImqdSUtiOLzeCQh6pqqb+vuFbxAyeyYSAJ49847EtFBvZCmWmPL2JICg9uq
7rKW/qDfEK9GUs0GWCs3+mJkNvOOxBwtMuQrL7ZF
-----END CERTIFICATE-----

View File

@@ -1,28 +1,28 @@
-----BEGIN PRIVATE KEY-----
MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQDA2y0hjmNn1vRs
VSdy8IOfo8N1q9UgkhQWpGKXzPh+D5d1fnuJEmHIVwtDEtS/PwQ43LTmegChPtKH
9jdTtG0IihW9Ja5YNG+9xAwaoA/sB3CGCBYsz+2/XjVUpXoBJXIPoFBWsn+K0oeF
w9fweRO1z9abM4cl+LjKzMNM8CCyu9uI1MaGjYez2YIWvG854VucLxX7HSlMJxZN
WnieUi7fMakuJhB2+aiIQjdKxy4E5RHNhzYG/LXhvP+wBYBDPNRsP3rtzEaE9HAv
eL9KFGqd3R4cBia6r1WIXmpAzyu5RGP5Eou0TZlGkal96/bF0I7q/pKlL23Jt1BL
PiQUKGKrAgMBAAECggEAAPX2kxi5AQ7ul82SzT1KgpSXyDHLdYaUyAoYnaX9RO+B
8ylmpyeqygs4+KQS4EMJm9jpo85Oy37bIKdG3kljU6wQcKlL5Y+ZUOo1nzpV6fid
hGVs6ts8VXw8KshKQ9AyccZ8L/pirUfgOffgTwfjY7/90zceAL/s98GuZWc62nkX
55joQv/OikqYfAGP/U6Bp2Zyf23DwJB09Z3B6NnZj/ZyAbDrDEHuA15LhCOcCczp
IU/mFEywBPHT9Tg4w4Beq78PeAETvku2UalYRLhP3RLlXr2oEbwUtINRVt2QjZ85
Esps4uCqL/mgQluIebtudD9HL/YMlNPXue1mDXFxJQKBgQDgZZY4yJBcf488T1V6
HNm06b/LvVGj253pKgw14hpY1xQu3Ymgzv1GEqzhSYdzxhpmj0tMUNHxAp+YdGQu
SZ0wcPKhw0aYVkIjDRYDC3Wn5GJhyIEYHGYMo/n4l49UzHRBPOTDzp49DkHTKBgh
XgIIazYT3CkjTIMRrkUv+qfIPQKBgQDcBGu/mqbjxs4sN3zqPS4aB21o6t6W0sXs
ZP9w6RlTPQi5U2oRbftjZtYc0bbEgkMUImB1HwYPQT5pJ+MyC414xDvSc2exBr5d
To6yyPIy78Tf5PHM12fpKV92nSvoz/pSjYcGxxDtKfPqu+t8mOJfjCV1lLLA+xuB
DDaE4p8dBwKBgQCdAne6A5v/HMH8UQZeCxHJpESvKiiVnnU/UEx651nID7XvlNNX
0X0mKqsMd4ZvW43ddSYan/JF0LAa3FW8jYWO/3jF9vzOWoysOdvNBZetgf/Uq5ao
aDZ/YbzmVCXWD7jIbPMkjs3pqrAkL0mzDzQc7+dGviWKrV6IYIfIqnn7gQKBgDCz
vdIk/qpO+JZrFfiX4Fucp0hhLTJ/p5ZDaRPqVVPKn+K+Jy2ChfIj8mNgvK9VEloj
nexvGJ1J2PHYBX+vdPp1nbRhHWPfVUY8PHQw7QP/dToGaMvqJrNDGEGeWvjnCMc7
UtdaO1H0Rm0AegkTopB56lTTvJnhO95eALd7nrMDAoGAEPdzJtWoKafp49svhSj0
hiXQv2SPBwVUN4LZ4SOWiXUcmYYm80aNpYKLkBxYjrfqFWhE7NUHLGp8YorQWKY2
acD9AReHk/xku0ABy6jeYmSCmCxASxst5liKD+l12sk0gB0rk5MBxB4Uu1MIbQZ2
aCASX3AVD2/XyC2MKkzc8Eg=
MIIEvwIBADANBgkqhkiG9w0BAQEFAASCBKkwggSlAgEAAoIBAQDHODfkrFsOkqCn
XnTbQWz3LkUtNCoVvM7wFouirRnITJYG+lFwF+zNl89Eaq+uUN4bwd8ml1ZuR9p+
1azCSlklD5adhCR/ErknfUWamQEf6amSs3p0NnqnhXbnDEEbQOwNaPU/aGc6aw0+
I9O6NQ/H830GlVuKd24Bfv0mx6Imo0Hi9jxKYhqFh80nmltk2uyXefaJxuo1jXBh
wLyCDW8RVj55QvkZyBUzid8yslxrlo0LHKCCjZflwJJv5f+jaubkH5c0qxVaoR4+
LiytX/4viIwt3Mhj04ppudTvt973mTbjRG5haCz9y7OkT1mMWhc0xrdMFX+gjPER
YS2HRu/RAgMBAAECggEABhB9CxYAE5p9D3s9nWsJcSDUdELRQSYlOoPFLmeMkF9c
dcvq7LmduMh1Q8TnoivOBxRIwbj7pZHEYfYJM0TmH82wrQzXu5KLVltm4gTkVt9b
DR8vjBgYdb8HVpM17Cl2xhW62XpJIiseFRUsHc/9sf2Egc3MIpPuIleGR0budbSW
ybBkqEokTYTSiAztcu3G+VN0U9MsJgLMa8HApya7M48ojdrhzngVHZRUOXul9o7u
zYJWSxPHIIYp5C4pYQBAx8OttThwKK1A9lwbQ2EJx0KnTbBC6O5Gna/jENpGd1h2
rzK/9MONtsjln7IejP+4mDlNupS6SF3zzHPBHjqKAQKBgQDtXUIKPiVTFS45yWtK
XD62s3j8jfIi+22b/C30fCPtppn0cm/0zY+vovgWVUBnQXkExafRthZCuxnE8ry7
E29S40+4z9yivAC9dz7vHZUbyIFP6VG9WyhUYo+/WqOIePyh+iBISQ9TA1DneIYz
+VZ8iU5GvdybUPl2C5WN8seaoQKBgQDW3EwVN2EEkChLRJbQYN2qpjn+0vYESMJ8
K0sgMRtgh4+/T2Xb9b8O/dd87Fi/4oaUqWZ2E2sdsXq8P/IEo0cv6SRfHMy7GyxL
RM7ztwUfMC4LVWi0ZIXMrm4gRDGN2XjGvhkX6fU2lSf6azWL1K3wI3amNV2b7P7d
ItpvdkH3MQKBgQCXf29YJEQkXB9t6J3fDzND3xb4cwy5wSo7ZeBa7CTuWOhoeeX1
JIJyAp0/e9goT0SThChRlFtu6gZPivJkoMnr6IOInLrg7we15fc4HPR/kCDgxTVT
m2wJOAMxigNYZogwRfn2yRLL1BD+PBHD+H936xcX1bSJOUyPSGOC/xLhIQKBgQCb
kCDd85ygyycBaAWxlZCor3WqFF/fNjbp5Aaepi9mMoBXSUs8eK7+UbelURHozEAY
fpYaw3B4rTlp9vppdTZjb+/PlXB9v+zQCl+0gTyKGj4cIpiOk4F0co51eipOw7f4
XUaZ0+CgxlmNq/W26iONjH+pU1YVQQA+Z6+zp/GW4QKBgQCrzYgeugxxqgJzyIRu
0njJkIg+T5gHvsQrtpzq7LVob+HBiBiT7eDOeGDXTK8F//sk969QVrDMQsTMvGW9
sG1oTqxciALTMqkJTf8+hT9Uogir0/iTbJUzTt5vPYpQOEQwQHIXMUTjZ9C6NDKT
QlmeMCxeWyPYqoMfwKmdtDP/Iw==
-----END PRIVATE KEY-----

View File

@@ -1,21 +1,21 @@
-----BEGIN CERTIFICATE-----
MIIDZzCCAk+gAwIBAgIUXizLjwkdqepX0bh0K3abeJxj68IwDQYJKoZIhvcNAQEL
BQAwHjEcMBoGA1UEAwwTRWxlbWVudCBDYWxsIERldiBDQTAeFw0yNTA1MDUxMzU5
MTFaFw0zNTA1MDMxMzU5MTFaMBgxFjAUBgNVBAMMDSoubS5sb2NhbGhvc3QwggEi
MA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCrzGSScSgaQuZdELGFYiLiYRwr
LKyUdNr0rsPcOo0bvbeZ3zQMeUMRNlA69zGFdarumiDRXUoAmZI39WmH95aX3d+A
U7EFnWev7xpWSVhSYj8T0d4rke8HjGk3LpaffJ93tbJuagBIH1ouuN6AOdzWs8hp
RYIomWleEeeuVnnfaMwaXOdc+ihJJ6wzm2hwQSfdpjZPWBDd/DFft1ZXxIZOCjDs
rEIiI7uU8iZPLB3QEM/tgxSSAOxrcKvQvxZokk+FD7aMJFP71IfieLCEzMTP1VXa
tP7UTAKAqB2NyDJ8m3IHbOINiqcdFvFR3R1D9bXOYE4oRynNvYZrQUGnL2RtAgMB
AAGjgaIwgZ8wHwYDVR0jBBgwFoAUCagYzLutXNaduLccP8vpL9w/legwCQYDVR0T
BAIwADALBgNVHQ8EBAMCBaAwEwYDVR0lBAwwCgYIKwYBBQUHAwEwMAYDVR0RBCkw
J4IJbG9jYWxob3N0ggttLmxvY2FsaG9zdIINKi5tLmxvY2FsaG9zdDAdBgNVHQ4E
FgQUfdh1p52ZgWyZcBgBXGwKi4EnUE0wDQYJKoZIhvcNAQELBQADggEBAKrHEuB6
33j8+EwSHw3zrvt/DRXK2BDHI1Ir9JcztSunaKAjZXVvf/dvZp0Xs1dEdJIdnv6G
iZYhBbOqDqpQZbf2h/h0kuu5yZSBUdnQXnYNxlhp2UaC/UEgw5iZT/p1rm7RjVie
y4Dp2WytV5iZOLmLj6xDvd3DXazgJPWIRX8p8qJZbKTkwCjTr7nDIj8jjG1sVFf7
1RJBO5/6WSnImrpDmlLUrvjiKvbxcdseDJyBOhTwdRdSk4S2M+s5tR5j2I1gXLOq
J5ioN76+SCrTY0K0WKRy9oOXWO1/X3+VYcekp+0F3SGkd5w17jylCv1XIGHAdEsQ
v2z2/aMI/7sAD2Q=
MIIDgDCCAmigAwIBAgIUT9NYpZbrAKokSPSTE3zzsAMowvEwDQYJKoZIhvcNAQEL
BQAwHjEcMBoGA1UEAwwTRWxlbWVudCBDYWxsIERldiBDQTAeFw0yNTA5MjIxMjI3
MzVaFw0zNTA5MjAxMjI3MzVaMBgxFjAUBgNVBAMMDSoubS5sb2NhbGhvc3QwggEi
MA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDK1DNwTQWmyK71Ar56NvmSMQ8s
qUY3jGqqPVORjfDUtDCrPPdCxT+ZlnsAgdonElWoWqczMrSyBRgfJlZMd4lEvt6V
EEiZGUvA/lG1XIVgrx1kMSHKBoJj7lCBN6r3IWmYe6CxgfZurgp+7Z22i6cGMOnQ
0XduX5Asup6zk5V7AE6i9eKrJsUjYmRBXtk099IitkER4TMqh6WxJmFF+eV9P/ax
fxkon+bQWITwP1PLC1UOTK7lR0EcVan5aY6WMs/6RfO4Gw/dvuiVG1jCrVcaKNGT
PYqmQqs+MOvyIqJ9kYELRZu+6bhPWSXk2ESpSIUIPH9twfnmWrncneIJR24/AgMB
AAGjgbswgbgwHwYDVR0jBBgwFoAU1zXywClWshkHWvyWSz7M66c+i1cwCQYDVR0T
BAIwADALBgNVHQ8EBAMCBaAwEwYDVR0lBAwwCgYIKwYBBQUHAwEwSQYDVR0RBEIw
QIIJbG9jYWxob3N0ggttLmxvY2FsaG9zdIINKi5tLmxvY2FsaG9zdIIXKi5vdGhl
cnNpdGUubS5sb2NhbGhvc3QwHQYDVR0OBBYEFIkGX+cEJ1ISKIwuT1zzp7uHJ90e
MA0GCSqGSIb3DQEBCwUAA4IBAQBnnnfB7KmyYo16ZYUCmoqGhbM4p8npeYTh5ySb
K01YwGCnMU1qGfJnKHaRwQ2+KtVGZnpBdjmsHcOUetA3V2BirPaYowMCMtaI36LD
LnxvboSZLX0mgEYuN7HmxW4a7fSelDecTYa7xti1sNhE/w8xW7Lky046/DousyUy
d9x3wJ183GGj1W2p6bR1E4sqTr/VbmoULQxnqA3GUNOxW3lRL5e8lQ6jJVRmMF4k
92BtMPrI/m7jwHj0f/WBLI8mdJ/O/W/NxQOG475FZePDfrg+MkeXPChPggf42/ou
AMm56FNB7e1l0b1Fots730RfpCPuXpiAxL4pisS0X1dMVeeM
-----END CERTIFICATE-----

View File

@@ -1,28 +1,28 @@
-----BEGIN PRIVATE KEY-----
MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQCrzGSScSgaQuZd
ELGFYiLiYRwrLKyUdNr0rsPcOo0bvbeZ3zQMeUMRNlA69zGFdarumiDRXUoAmZI3
9WmH95aX3d+AU7EFnWev7xpWSVhSYj8T0d4rke8HjGk3LpaffJ93tbJuagBIH1ou
uN6AOdzWs8hpRYIomWleEeeuVnnfaMwaXOdc+ihJJ6wzm2hwQSfdpjZPWBDd/DFf
t1ZXxIZOCjDsrEIiI7uU8iZPLB3QEM/tgxSSAOxrcKvQvxZokk+FD7aMJFP71Ifi
eLCEzMTP1VXatP7UTAKAqB2NyDJ8m3IHbOINiqcdFvFR3R1D9bXOYE4oRynNvYZr
QUGnL2RtAgMBAAECggEAJaFQii8U/KOYt9vXNoMnZvSkaeSQLLhn2V6Kciu1CtWE
aMTWLsFE6nk+G5xXkYcTmM3T0GghtH3u5CjyI6EcsEkeEorCZJt0wbmayDmqiekR
LfMzOdHuTHX5+edPgMGYYG1BFyRKyYFsjH1b5zRFZhXdGQnrl5760GsVlz9D1KZQ
iHcT+q1S2tmZeoUukQnADENKXUMCyTGM5FCddgNtsWnGDsTDayh7hUdvDkB+mW4G
lSp+BZuc3PCwpbD6qkXvfugWs6CUAAtXoV3ceWgxQ+TEnNlwxaG1AyugfgNUBolk
8xgeZt4r5QId03jsHDf7hpBAofcaCd5EMIIQYFvWoQKBgQDlbAvAzEFPTZZn2nRV
Xagw4xjqVc1LLEKLCWq0N5rEkwn0h90Dz5N7/3NuonP/sIDsDHCbyiOYBI1Ck6Xi
0WuB+OyKDh+xeF2mekN9G9ywPahdK5lT/TVsxXFyZlwtVv1x/6KBO4yv5URizxqU
gyAPDDxfD/KcNjkOBaodWEwQGQKBgQC/s2gPDBtQkjLwkHXchBomLww5eLlVrac1
WK4UX6uSdOgrjJ375OOgMTxe8NVZdOuAKytGXRWDwgH3nVWvuZhe7dGlX3JMuSer
e9VwDpBESrvqcR4ruL6wm8wej6BXyjH0wD3FHb0S5HfuBDxTn+4bDwrbRzOUMNgy
lSppuflxdQKBgQDiZcIfazFT8evn5nMAvuC4BZNTxIJHmZC9JfjPiUPIkpWzYtOe
7BvNtKOT3Op9uw8uYYRKqKqBXJSNy6ha8XCXHS9HeXKbLn20SFkLQBCDNwVLlDfF
40zyXtF6JDr4XyzSb4NM5pgKCER5AYloXxGm59s3sEQpFXUuOjbKqJS/GQKBgAoI
c7vF4HAZFr1sch62cz/oWnVvkhOf4Q5zs7ixQSOLJtOQqnwSgK9TpFs7s47ZBbJR
kBRAru2Ua9Hv1Bo8VnMxczV6h1roneDlvEf/GyHX33nnrbKQGrrXjJlU3wl5NaAf
p5v3cHvapUQ5yIZ/6lBUOzc6xMJOxCHxmKSr7Rg5AoGAbEE4lt6Xh2dnBPJ81eNI
IDrw/3ITY53qAY4Bx88CByIFuu8CEUdUZprh98jSl6ic1tMinZfUhRMwABLrUD51
DGst8iGLPD9u83iMcUHI/L+p7AbxrKLvWXZrF5UZm440c9mSWqfhPaTBosPtNDsG
LfETwH1flKXMTXd2xA9RTE4=
MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQDK1DNwTQWmyK71
Ar56NvmSMQ8sqUY3jGqqPVORjfDUtDCrPPdCxT+ZlnsAgdonElWoWqczMrSyBRgf
JlZMd4lEvt6VEEiZGUvA/lG1XIVgrx1kMSHKBoJj7lCBN6r3IWmYe6CxgfZurgp+
7Z22i6cGMOnQ0XduX5Asup6zk5V7AE6i9eKrJsUjYmRBXtk099IitkER4TMqh6Wx
JmFF+eV9P/axfxkon+bQWITwP1PLC1UOTK7lR0EcVan5aY6WMs/6RfO4Gw/dvuiV
G1jCrVcaKNGTPYqmQqs+MOvyIqJ9kYELRZu+6bhPWSXk2ESpSIUIPH9twfnmWrnc
neIJR24/AgMBAAECggEANRp6vzfDN4gKWoaV5TyYegCTNv+4rRl74cd9pjmx6Jam
uWaUXCx1etpNqPPWcG1Z9OKLLRnk+kjgKGOqq4mObGvGreNeBot7bHOJZADtwMMI
YG+Gp7StlclS1YoEHoDmezA/AcqDgTXa9KF0rdMBb1sGFJCLAuBNSJCxtVV6CQIz
X26uT0m+Wx8MQyQWA7Sqy6DQNJo++IZkvr7a3cidqBOUPs+QvnIV5JsUb2gp5tGn
zk+ObeRjoFFWYAN/NK7bneRenkP40m3MSL8ZfaEuuonui7CrxM1SiQyq2N1u/Aoy
OE1JtNaVPbLBo6kG5al7Sj4Z0zhRt+iv93S2lZMkBQKBgQD2+FpLTqyLO1NDOFkE
kxU+LdLOx0OV9wASC0ApPOu1dHMG6ksByr7TWeiu6GJDgajusPB7NVPOt2cm4iWU
xPxXPO5l87uiSvu80h5uG4Qdj8KEijHkdap2wbVkU/mm8lBKC36jyBQIlJKySyXY
zSEMfLK9jQPKz5cKKT3dVj/fAwKBgQDSPq9oks6K96MAB66o6cm214otQlnTQkPM
xgjtjddX+Lp9tgihGvtSfPbyy89oUDHCfKvW/AHG52e5dec5YUi6mVdHEWbk33Kt
BoQuxeK3XseIDlD/JD9Dd7KfUyO5w2jtYLfNdqez41O4qj2N52m1KwJYTwMsc8Kq
izVgkC5hFQKBgBFAc/5CtqbbNAvECePZ6mf3h3xOSxhUsrqP8qFu0gBQ7CAVibvM
T9wvsaNWNFcG3age0A2rQfl0sk3zCjEEOaRWa0jP59GEb2VXQCzs2yO9gRcFGEsf
NRMqoOMrQos47gbeGrCSL2QSDNVLjo9AdQiMRWgcS6GFMsXQ77NgbQHFAoGBAI4a
YGTGFWRITJvQlXUFz5kNxg8hMaVgvILDt3UY0dxb+XDOgLajjgsK+77Pkrhmu7tA
mMUOQAU4kxr/XfGil43H5v3Z/Tnk7ZWVOfKDPeHC5gpH4ucQkNIBLXISt6rvMRSA
srrk4CTuGcBPEJvBNemF0Gfvv61j8MdkoAdMbIyhAoGAfGR6yZLBmRMsW5PKmcpT
nq2oSeUpmtGZra6pWz/3XU7AgrCLcx1DmqEjm4w7y5NQJmxyMZqqdTJILCjr3Srt
+2F0NqQL6Li+xQGibAvDj0Jxyol38RvFC0J/w2vQmuF0hTuH95yknSd7FPXK+DPG
qYgXLjun9dht6kx9vGJ69wI=
-----END PRIVATE KEY-----

View File

@@ -34,5 +34,6 @@ subjectAltName = @alt_names
DNS.1 = localhost
DNS.2 = m.localhost
DNS.3 = *.m.localhost
DNS.4 = *.othersite.m.localhost
EOF
)

View File

@@ -0,0 +1,53 @@
{
"default_server_config": {
"m.homeserver": {
"base_url": "https://synapse.othersite.m.localhost",
"server_name": "synapse.othersite.m.localhost"
}
},
"disable_custom_urls": false,
"disable_guests": false,
"disable_login_language_selector": false,
"disable_3pid_login": false,
"force_verification": false,
"brand": "Element",
"integrations_ui_url": "https://scalar.vector.im/",
"integrations_rest_url": "https://scalar.vector.im/api",
"integrations_widgets_urls": [
"https://scalar.vector.im/_matrix/integrations/v1",
"https://scalar.vector.im/api",
"https://scalar-staging.vector.im/_matrix/integrations/v1",
"https://scalar-staging.vector.im/api",
"https://scalar-staging.riot.im/scalar/api"
],
"default_widget_container_height": 280,
"default_country_code": "GB",
"show_labs_settings": false,
"features": {
"feature_element_call_video_rooms": true,
"feature_video_rooms": true,
"feature_group_calls": true,
"feature_release_announcement": false
},
"default_federate": true,
"default_theme": "light",
"room_directory": {
"servers": ["matrix.org"]
},
"enable_presence_by_hs_url": {
"https://matrix.org": false,
"https://matrix-client.matrix.org": false
},
"setting_defaults": {
"breadcrumbs": true,
"feature_group_calls": true
},
"jitsi": {
"preferred_domain": "meet.element.io"
},
"element_call": {
"participant_limit": 8,
"brand": "Element Call"
},
"map_style_url": "https://api.maptiler.com/maps/streets/style.json?key=fU3vlMsMn4Jb6dnEIFsx"
}

View File

@@ -0,0 +1,81 @@
server_name: "synapse.othersite.m.localhost"
public_baseurl: https://synapse.othersite.m.localhost/
pid_file: /data/homeserver.pid
listeners:
- port: 18008
tls: false
type: http
x_forwarded: true
resources:
- names: [client, federation, openid]
compress: false
database:
name: sqlite3
args:
database: /data/homeserver.db
media_store_path: /data/media_store
signing_key_path: "/data/SERVERNAME.signing.key"
# Due to custom TLS certificate with domains
# - m.localhost, localhost
# - *.m.localhost
# - *.othersite.m.localhost
# we disable certificate verification to allow for federation.
# WARNING: DO NOT USE IN PRODUCTION!!!
federation_verify_certificates: false
ip_range_blacklist: []
trusted_key_servers:
- server_name: "synapse.m.localhost"
accept_keys_insecurely: true
experimental_features:
# MSC3266: Room summary API. Used for knocking over federation
msc3266_enabled: true
# MSC4222 needed for syncv2 state_after. This allow clients to
# correctly track the state of the room.
msc4222_enabled: true
# The maximum allowed duration by which sent events can be delayed, as
# per MSC4140. Must be a positive value if set. Defaults to no
# duration (null), which disallows sending delayed events.
max_event_delay_duration: 24h
# Required for Element Call in Single Page Mode due to on-the-fly user registration
enable_registration: true
enable_registration_without_verification: true
report_stats: false
serve_server_wellknown: true
# Ratelimiting settings for client actions (registration, login, messaging).
#
# Each ratelimiting configuration is made of two parameters:
# - per_second: number of requests a client can send per second.
# - burst_count: number of requests a client can send before being throttled.
rc_message:
per_second: 10000
burst_count: 10000
rc_delayed_event_mgmt:
per_second: 10000
burst_count: 10000
rc_login:
address:
per_second: 10000
burst_count: 10000
account:
per_second: 10000
burst_count: 10000
failed_attempts:
per_second: 10000
burst_count: 10000
rc_registration:
per_second: 10000
burst_count: 10000

View File

@@ -19,8 +19,18 @@ database:
media_store_path: /data/media_store
signing_key_path: "/data/SERVERNAME.signing.key"
# Due to custom TLS certificate with domains
# - m.localhost, localhost
# - *.m.localhost
# - *.othersite.m.localhost
# we disable certificate verification to allow for federation.
# WARNING: DO NOT USE IN PRODUCTION!!!
federation_verify_certificates: false
ip_range_blacklist: []
trusted_key_servers:
- server_name: "matrix.org"
- server_name: "synapse.othersite.m.localhost"
accept_keys_insecurely: true
experimental_features:
# MSC3266: Room summary API. Used for knocking over federation
@@ -34,6 +44,13 @@ experimental_features:
# duration (null), which disallows sending delayed events.
max_event_delay_duration: 24h
# Required for Element Call in Single Page Mode due to on-the-fly user registration
enable_registration: true
enable_registration_without_verification: true
report_stats: false
serve_server_wellknown: true
# Ratelimiting settings for client actions (registration, login, messaging).
#
# Each ratelimiting configuration is made of two parameters:
@@ -62,10 +79,3 @@ rc_login:
rc_registration:
per_second: 10000
burst_count: 10000
# Required for Element Call in Single Page Mode due to on-the-fly user registration
enable_registration: true
enable_registration_without_verification: true
report_stats: false
serve_server_wellknown: true

View File

@@ -1,5 +0,0 @@
bind 0.0.0.0
protected-mode yes
port 6379
timeout 0
tcp-keepalive 300

View File

@@ -24,6 +24,28 @@ services:
networks:
- ecbackend
auth-service-1:
image: ghcr.io/element-hq/lk-jwt-service:latest-ci
pull_policy: always
hostname: auth-server-1
environment:
- LIVEKIT_JWT_PORT=16080
- LIVEKIT_URL=wss://matrix-rtc.othersite.m.localhost/livekit/sfu
- LIVEKIT_KEY=devkey
- LIVEKIT_SECRET=secret
# If the configured homeserver runs on localhost, it'll probably be using
# a self-signed certificate
- LIVEKIT_INSECURE_SKIP_VERIFY_TLS=YES_I_KNOW_WHAT_I_AM_DOING
- LIVEKIT_FULL_ACCESS_HOMESERVERS=*
deploy:
restart_policy:
condition: on-failure
ports:
# HOST_PORT:CONTAINER_PORT
- 16080:16080
networks:
- ecbackend
livekit:
image: livekit/livekit-server:latest
pull_policy: always
@@ -44,21 +66,29 @@ services:
networks:
- ecbackend
redis:
image: redis:6-alpine
livekit-1:
image: livekit/livekit-server:latest
pull_policy: always
command: redis-server /etc/redis.conf
hostname: livekit-sfu-1
command: --dev --config /etc/livekit.yaml
restart: unless-stopped
# The SFU seems to work far more reliably when we let it share the host
# network rather than opening specific ports (but why?? we're not missing
# any…)
ports:
# HOST_PORT:CONTAINER_PORT
- 6379:6379
- 17880:17880/tcp
- 17881:17881/tcp
- 17882:17882/tcp
- 50300-50400:50300-50400/udp
volumes:
- ./backend/redis.conf:/etc/redis.conf:Z
- ./backend/dev_livekit-othersite.yaml:/etc/livekit.yaml:Z
networks:
- ecbackend
synapse:
hostname: homeserver
image: docker.io/matrixdotorg/synapse:latest
image: ghcr.io/element-hq/synapse:msc4354-5
pull_policy: always
environment:
- SYNAPSE_CONFIG_PATH=/data/cfg/homeserver.yaml
@@ -74,6 +104,24 @@ services:
networks:
- ecbackend
synapse-1:
hostname: homeserver-1
image: docker.io/matrixdotorg/synapse:latest
pull_policy: always
environment:
- SYNAPSE_CONFIG_PATH=/data/cfg/homeserver.yaml
# Needed for rootless podman-compose such that the uid/gid mapping does
# fit local user uid. If the container runs as root (uid 0) it is fine as
# it actually maps to your non-root user on the host (e.g. 1000).
# Otherwise uid mapping will not match your non-root user.
- UID=0
- GID=0
volumes:
- ./backend/synapse_tmp_othersite:/data:Z
- ./backend/dev_homeserver-othersite.yaml:/data/cfg/homeserver.yaml:Z
networks:
- ecbackend
element-web:
image: ghcr.io/element-hq/element-web:develop
pull_policy: always
@@ -86,6 +134,19 @@ services:
networks:
- ecbackend
element-web-1:
image: ghcr.io/element-hq/element-web:develop
pull_policy: always
volumes:
- ./backend/ew.test.othersite.config.json:/app/config.json:Z
environment:
ELEMENT_WEB_PORT: 18081
ports:
# HOST_PORT:CONTAINER_PORT
- "18081:18081"
networks:
- ecbackend
nginx:
# see backend/dev_tls_setup for how to generate the tls certs
hostname: synapse.m.localhost
@@ -108,4 +169,7 @@ services:
networks:
ecbackend:
aliases:
- synapse.m.localhost
- synapse.othersite.m.localhost
- matrix-rtc.m.localhost
- matrix-rtc.othersite.m.localhost

View File

@@ -72,11 +72,14 @@
"livekit_server_info": "LiveKit Server Info",
"livekit_sfu": "LiveKit SFU: {{url}}",
"matrix_id": "Matrix ID: {{id}}",
"multi_sfu": "Multi-SFU media transport",
"mute_all_audio": "Mute all audio (participants, reactions, join sounds)",
"prefer_sticky_events": {
"description": "Improves reliability of calls (requires homeserver support)",
"label": "Prefer sticky events"
},
"show_connection_stats": "Show connection statistics",
"show_non_member_tiles": "Show tiles for non-member media",
"url_params": "URL parameters",
"use_new_membership_manager": "Use the new implementation of the call MembershipManager",
"use_to_device_key_transport": "Use to device key transport. This will fallback to room key transport when another call member sent a room key"
},
"disconnected_banner": "Connectivity to the server has been lost.",
@@ -92,7 +95,7 @@
"generic_description": "Submitting debug logs will help us track down the problem.",
"insufficient_capacity": "Insufficient capacity",
"insufficient_capacity_description": "The server has reached its maximum capacity and you cannot join the call at this time. Try again later, or contact your server admin if the problem persists.",
"matrix_rtc_focus_missing": "The server is not configured to work with {{brand}}. Please contact your server admin (Domain: {{domain}}, Error Code: {{ errorCode }}).",
"matrix_rtc_transport_missing": "The server is not configured to work with {{brand}}. Please contact your server admin (Domain: {{domain}}, Error Code: {{ errorCode }}).",
"open_elsewhere": "Opened in another tab",
"open_elsewhere_description": "{{brand}} has been opened in another tab. If that doesn't sound right, try reloading the page.",
"room_creation_restricted": "Failed to create call",

View File

@@ -99,6 +99,7 @@
"eslint-plugin-react-hooks": "^5.0.0",
"eslint-plugin-rxjs": "^5.0.3",
"eslint-plugin-unicorn": "^56.0.0",
"fetch-mock": "11.1.5",
"global-jsdom": "^26.0.0",
"i18next": "^24.0.0",
"i18next-browser-languagedetector": "^8.0.0",
@@ -108,7 +109,7 @@
"livekit-client": "^2.13.0",
"lodash-es": "^4.17.21",
"loglevel": "^1.9.1",
"matrix-js-sdk": "github:matrix-org/matrix-js-sdk#head=develop",
"matrix-js-sdk": "github:matrix-org/matrix-js-sdk#head=toger5/sticky-events&commit=e7f5bec51b6f70501a025b79fe5021c933385b21",
"matrix-widget-api": "^1.13.0",
"normalize.css": "^8.0.1",
"observable-hooks": "^4.2.3",

View File

@@ -2,3 +2,6 @@ services:
synapse:
volumes:
- ./backend/playwright_homeserver.yaml:/data/cfg/homeserver.yaml:Z
synapse-1:
volumes:
- ./backend/playwright_homeserver-othersite.yaml:/data/cfg/homeserver.yaml:Z

View File

@@ -23,14 +23,6 @@ export function useMediaDevices(): MediaDevices {
return mediaDevices;
}
export const useIsEarpiece = (): boolean => {
const devices = useMediaDevices();
const audioOutput = useObservableEagerState(devices.audioOutput.selected$);
const available = useObservableEagerState(devices.audioOutput.available$);
if (!audioOutput?.id) return false;
return available.get(audioOutput.id)?.type === "earpiece";
};
/**
* A convenience hook to get the audio node configuration for the earpiece.
* It will check the `useAsEarpiece` of the `audioOutput` device and return

View File

@@ -19,10 +19,26 @@ import mediaViewStyles from "../src/tile/MediaView.module.css";
interface Props {
audio?: RTCInboundRtpStreamStats | RTCOutboundRtpStreamStats;
video?: RTCInboundRtpStreamStats | RTCOutboundRtpStreamStats;
focusUrl?: string;
}
const extractDomain = (url: string): string => {
try {
const parsedUrl = new URL(url);
return parsedUrl.hostname; // Returns "kdk.cpm"
} catch (error) {
console.error("Invalid URL:", error);
return url;
}
};
// This is only used in developer mode for debugging purposes, so we don't need full localization
export const RTCConnectionStats: FC<Props> = ({ audio, video, ...rest }) => {
export const RTCConnectionStats: FC<Props> = ({
audio,
video,
focusUrl,
...rest
}) => {
const [showModal, setShowModal] = useState(false);
const [modalContents, setModalContents] = useState<
"video" | "audio" | "none"
@@ -55,6 +71,13 @@ export const RTCConnectionStats: FC<Props> = ({ audio, video, ...rest }) => {
</pre>
</div>
</Modal>
{focusUrl && (
<div>
<Text as="span" size="xs" title="focusURL">
&nbsp;{extractDomain(focusUrl)}
</Text>
</div>
)}
{audio && (
<div>
<Button

View File

@@ -5,12 +5,15 @@ SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { describe, expect, it } from "vitest";
import { describe, expect, it, onTestFinished, vi } from "vitest";
import { logger } from "matrix-js-sdk/lib/logger";
import * as PlatformMod from "../src/Platform";
import {
getRoomIdentifierFromUrl,
getUrlParams,
computeUrlParams,
HeaderStyle,
getUrlParams,
} from "../src/UrlParams";
const ROOM_NAME = "roomNameHere";
@@ -103,16 +106,16 @@ describe("UrlParams", () => {
describe("preload", () => {
it("defaults to false", () => {
expect(getUrlParams().preload).toBe(false);
expect(computeUrlParams().preload).toBe(false);
});
it("ignored in SPA mode", () => {
expect(getUrlParams("?preload=true").preload).toBe(false);
expect(computeUrlParams("?preload=true").preload).toBe(false);
});
it("respected in widget mode", () => {
expect(
getUrlParams(
computeUrlParams(
"?preload=true&widgetId=12345&parentUrl=https%3A%2F%2Flocalhost%2Ffoo",
).preload,
).toBe(true);
@@ -121,19 +124,20 @@ describe("UrlParams", () => {
describe("returnToLobby", () => {
it("is false in SPA mode", () => {
expect(getUrlParams("?returnToLobby=true").returnToLobby).toBe(false);
expect(computeUrlParams("?returnToLobby=true").returnToLobby).toBe(false);
});
it("defaults to false in widget mode", () => {
expect(
getUrlParams("?widgetId=12345&parentUrl=https%3A%2F%2Flocalhost%2Ffoo")
.returnToLobby,
computeUrlParams(
"?widgetId=12345&parentUrl=https%3A%2F%2Flocalhost%2Ffoo",
).returnToLobby,
).toBe(false);
});
it("respected in widget mode", () => {
expect(
getUrlParams(
computeUrlParams(
"?returnToLobby=true&widgetId=12345&parentUrl=https%3A%2F%2Flocalhost%2Ffoo",
).returnToLobby,
).toBe(true);
@@ -142,12 +146,12 @@ describe("UrlParams", () => {
describe("userId", () => {
it("is ignored in SPA mode", () => {
expect(getUrlParams("?userId=asd").userId).toBe(null);
expect(computeUrlParams("?userId=asd").userId).toBe(null);
});
it("is parsed in widget mode", () => {
expect(
getUrlParams(
computeUrlParams(
"?userId=asd&widgetId=12345&parentUrl=https%3A%2F%2Flocalhost%2Ffoo",
).userId,
).toBe("asd");
@@ -156,12 +160,12 @@ describe("UrlParams", () => {
describe("deviceId", () => {
it("is ignored in SPA mode", () => {
expect(getUrlParams("?deviceId=asd").deviceId).toBe(null);
expect(computeUrlParams("?deviceId=asd").deviceId).toBe(null);
});
it("is parsed in widget mode", () => {
expect(
getUrlParams(
computeUrlParams(
"?deviceId=asd&widgetId=12345&parentUrl=https%3A%2F%2Flocalhost%2Ffoo",
).deviceId,
).toBe("asd");
@@ -170,12 +174,12 @@ describe("UrlParams", () => {
describe("baseUrl", () => {
it("is ignored in SPA mode", () => {
expect(getUrlParams("?baseUrl=asd").baseUrl).toBe(null);
expect(computeUrlParams("?baseUrl=asd").baseUrl).toBe(null);
});
it("is parsed in widget mode", () => {
expect(
getUrlParams(
computeUrlParams(
"?baseUrl=asd&widgetId=12345&parentUrl=https%3A%2F%2Flocalhost%2Ffoo",
).baseUrl,
).toBe("asd");
@@ -185,28 +189,28 @@ describe("UrlParams", () => {
describe("viaServers", () => {
it("is ignored in widget mode", () => {
expect(
getUrlParams(
computeUrlParams(
"?viaServers=asd&widgetId=12345&parentUrl=https%3A%2F%2Flocalhost%2Ffoo",
).viaServers,
).toBe(null);
});
it("is parsed in SPA mode", () => {
expect(getUrlParams("?viaServers=asd").viaServers).toBe("asd");
expect(computeUrlParams("?viaServers=asd").viaServers).toBe("asd");
});
});
describe("homeserver", () => {
it("is ignored in widget mode", () => {
expect(
getUrlParams(
computeUrlParams(
"?homeserver=asd&widgetId=12345&parentUrl=https%3A%2F%2Flocalhost%2Ffoo",
).homeserver,
).toBe(null);
});
it("is parsed in SPA mode", () => {
expect(getUrlParams("?homeserver=asd").homeserver).toBe("asd");
expect(computeUrlParams("?homeserver=asd").homeserver).toBe("asd");
});
});
@@ -237,7 +241,7 @@ describe("UrlParams", () => {
controlledAudioDevices: platform === "desktop" ? false : true,
skipLobby: true,
returnToLobby: false,
sendNotificationType: "notification",
sendNotificationType: platform === "desktop" ? "notification" : "ring",
});
const joinExistingCallDefaults = (platform: string): object => ({
confineToRoom: true,
@@ -252,24 +256,55 @@ describe("UrlParams", () => {
skipLobby: false,
returnToLobby: false,
sendNotificationType: "notification",
defaultAudioEnabled: true,
defaultVideoEnabled: true,
});
it("use no-intent-defaults with unknown intent", () => {
expect(getUrlParams()).toMatchObject(noIntentDefaults);
expect(computeUrlParams()).toMatchObject(noIntentDefaults);
});
it("ignores intent if it is not a valid value", () => {
expect(getUrlParams("?intent=foo")).toMatchObject(noIntentDefaults);
expect(computeUrlParams("?intent=foo")).toMatchObject(noIntentDefaults);
});
it("accepts start_call", () => {
expect(
getUrlParams("?intent=start_call&widgetId=1234&parentUrl=parent.org"),
).toMatchObject(startNewCallDefaults("desktop"));
computeUrlParams(
"?intent=start_call&widgetId=1234&parentUrl=parent.org",
),
).toMatchObject({ ...startNewCallDefaults("desktop"), skipLobby: false });
});
it("accepts start_call_dm mobile", () => {
vi.spyOn(PlatformMod, "platform", "get").mockReturnValue("android");
onTestFinished(() => {
vi.spyOn(PlatformMod, "platform", "get").mockReturnValue("desktop");
});
expect(
computeUrlParams(
"?intent=start_call_dm&widgetId=1234&parentUrl=parent.org",
),
).toMatchObject(startNewCallDefaults("android"));
});
it("accepts start_call_dm mobile and prioritizes overwritten params", () => {
vi.spyOn(PlatformMod, "platform", "get").mockReturnValue("android");
onTestFinished(() => {
vi.spyOn(PlatformMod, "platform", "get").mockReturnValue("desktop");
});
expect(
computeUrlParams(
"?intent=start_call_dm&widgetId=1234&parentUrl=parent.org&sendNotificationType=notification",
),
).toMatchObject({
...startNewCallDefaults("android"),
sendNotificationType: "notification",
});
});
it("accepts join_existing", () => {
expect(
getUrlParams(
computeUrlParams(
"?intent=join_existing&widgetId=1234&parentUrl=parent.org",
),
).toMatchObject(joinExistingCallDefaults("desktop"));
@@ -278,31 +313,55 @@ describe("UrlParams", () => {
describe("skipLobby", () => {
it("defaults to false", () => {
expect(getUrlParams().skipLobby).toBe(false);
expect(computeUrlParams().skipLobby).toBe(false);
});
it("defaults to false if intent is start_call in SPA mode", () => {
expect(getUrlParams("?intent=start_call").skipLobby).toBe(false);
expect(computeUrlParams("?intent=start_call").skipLobby).toBe(false);
});
it("defaults to true if intent is start_call in widget mode", () => {
it("defaults to false if intent is start_call in widget mode", () => {
expect(
getUrlParams(
computeUrlParams(
"?intent=start_call&widgetId=12345&parentUrl=https%3A%2F%2Flocalhost%2Ffoo",
).skipLobby,
).toBe(true);
).toBe(false);
});
it("default to false if intent is join_existing", () => {
expect(getUrlParams("?intent=join_existing").skipLobby).toBe(false);
expect(computeUrlParams("?intent=join_existing").skipLobby).toBe(false);
});
});
describe("header", () => {
it("uses header if provided", () => {
expect(getUrlParams("?header=app_bar&hideHeader=true").header).toBe(
expect(computeUrlParams("?header=app_bar&hideHeader=true").header).toBe(
"app_bar",
);
expect(getUrlParams("?header=none&hideHeader=false").header).toBe("none");
expect(computeUrlParams("?header=none&hideHeader=false").header).toBe(
"none",
);
});
});
describe("getUrlParams", () => {
it("uses cached values", () => {
const spy = vi.spyOn(logger, "info");
// call get once
const params = getUrlParams("?header=app_bar&hideHeader=true", "");
// call get twice
expect(getUrlParams("?header=app_bar&hideHeader=true", "")).toBe(params);
// expect compute to only be called once
// it will only log when it is computing the values
expect(spy).toHaveBeenCalledExactlyOnceWith(
"UrlParams: final set of url params\n",
"intent:",
"unknown",
"\nproperties:",
expect.any(Object),
"configuration:",
expect.any(Object),
"intentAndPlatformDerivedConfiguration:",
{},
);
});
});
});

View File

@@ -8,7 +8,10 @@ Please see LICENSE in the repository root for full details.
import { useMemo } from "react";
import { useLocation } from "react-router-dom";
import { logger } from "matrix-js-sdk/lib/logger";
import { type RTCNotificationType } from "matrix-js-sdk/lib/matrixrtc";
import {
type RTCCallIntent,
type RTCNotificationType,
} from "matrix-js-sdk/lib/matrixrtc";
import { pickBy } from "lodash-es";
import { Config } from "./config/Config";
@@ -26,7 +29,9 @@ export enum UserIntent {
StartNewCall = "start_call",
JoinExistingCall = "join_existing",
StartNewCallDM = "start_call_dm",
StartNewCallDMVoice = "start_call_dm_voice",
JoinExistingCallDM = "join_existing_dm",
JoinExistingCallDMVoice = "join_existing_dm_voice",
Unknown = "unknown",
}
@@ -227,6 +232,12 @@ export interface UrlConfiguration {
* - auto-dismiss the call widget once the notification lifetime expires on the receivers side.
*/
waitForCallPickup: boolean;
callIntent?: RTCCallIntent;
}
interface IntentAndPlatformDerivedConfiguration {
defaultAudioEnabled?: boolean;
defaultVideoEnabled?: boolean;
}
interface IntentAndPlatformDerivedConfiguration {
defaultAudioEnabled?: boolean;
@@ -322,8 +333,9 @@ let urlParamCache: {
hash?: string;
params?: UrlParams;
} = {};
/**
* Gets the app parameters for the current URL.
* Gets the url params and loads them from a cache if already computed.
* @param search The URL search string
* @param hash The URL hash
* @returns The app parameters encoded in the URL
@@ -331,18 +343,27 @@ let urlParamCache: {
export const getUrlParams = (
search = window.location.search,
hash = window.location.hash,
/** Skipping the cache might be needed in tests, to allow recomputing based on mocked platform changes. */
skipCache = false,
): UrlParams => {
// Only run the param configuration if we do not yet have it cached for this url.
if (
urlParamCache.search === search &&
urlParamCache.hash === hash &&
urlParamCache.params &&
!skipCache
urlParamCache.params
) {
return urlParamCache.params;
}
const params = computeUrlParams(search, hash);
urlParamCache = { search, hash, params };
return params;
};
/**
* Gets the app parameters for the current URL.
* @param search The URL search string
* @param hash The URL hash
* @returns The app parameters encoded in the URL
*/
export const computeUrlParams = (search = "", hash = ""): UrlParams => {
const parser = new ParamParser(search, hash);
const fontScale = parseFloat(parser.getParam("fontScale") ?? "");
@@ -378,28 +399,38 @@ export const getUrlParams = (
controlledAudioDevices: platform === "desktop" ? false : true,
skipLobby: true,
returnToLobby: false,
sendNotificationType: "notification" as RTCNotificationType,
sendNotificationType: "notification",
autoLeaveWhenOthersLeft: false,
waitForCallPickup: false,
};
switch (intent) {
case UserIntent.StartNewCall:
intentPreset.skipLobby = true;
intentPreset.skipLobby = false;
intentPreset.callIntent = "video";
break;
case UserIntent.JoinExistingCall:
// On desktop this will be overridden based on which button was used to join the call
intentPreset.skipLobby = false;
intentPreset.callIntent = "video";
break;
case UserIntent.StartNewCallDMVoice:
intentPreset.callIntent = "audio";
// Fall through
case UserIntent.StartNewCallDM:
intentPreset.skipLobby = true;
intentPreset.sendNotificationType = "ring";
intentPreset.autoLeaveWhenOthersLeft = true;
intentPreset.waitForCallPickup = true;
intentPreset.callIntent = intentPreset.callIntent ?? "video";
break;
case UserIntent.JoinExistingCallDMVoice:
intentPreset.callIntent = "audio";
// Fall through
case UserIntent.JoinExistingCallDM:
// On desktop this will be overridden based on which button was used to join the call
intentPreset.skipLobby = true;
intentPreset.autoLeaveWhenOthersLeft = true;
intentPreset.callIntent = intentPreset.callIntent ?? "video";
break;
// Non widget usecase defaults
default:
@@ -436,6 +467,11 @@ export const getUrlParams = (
intentAndPlatformDerivedConfiguration.defaultAudioEnabled = true;
intentAndPlatformDerivedConfiguration.defaultVideoEnabled = true;
break;
case UserIntent.StartNewCallDMVoice:
case UserIntent.JoinExistingCallDMVoice:
intentAndPlatformDerivedConfiguration.defaultAudioEnabled = true;
intentAndPlatformDerivedConfiguration.defaultVideoEnabled = false;
break;
}
}
@@ -505,15 +541,12 @@ export const getUrlParams = (
intentAndPlatformDerivedConfiguration,
);
const params = {
return {
...properties,
...intentPreset,
...pickBy(configuration, (v?: unknown) => v !== undefined),
...intentAndPlatformDerivedConfiguration,
};
urlParamCache = { search, hash, params };
return params;
};
/**

View File

@@ -10,7 +10,6 @@ import { expect, test } from "vitest";
import { TooltipProvider } from "@vector-im/compound-web";
import { userEvent } from "@testing-library/user-event";
import { type ReactNode } from "react";
import { type MatrixRTCSession } from "matrix-js-sdk/lib/matrixrtc";
import { ReactionToggleButton } from "./ReactionToggleButton";
import { ElementCallReactionEventType } from "../reactions";
@@ -20,7 +19,7 @@ import { alice, local, localRtcMember } from "../utils/test-fixtures";
import { type MockRTCSession } from "../utils/test";
import { ReactionsSenderProvider } from "../reactions/useReactionsSender";
const localIdent = `${localRtcMember.sender}:${localRtcMember.deviceId}`;
const localIdent = `${localRtcMember.userId}:${localRtcMember.deviceId}`;
function TestComponent({
rtcSession,
@@ -33,7 +32,7 @@ function TestComponent({
<TooltipProvider>
<ReactionsSenderProvider
vm={vm}
rtcSession={rtcSession as unknown as MatrixRTCSession}
rtcSession={rtcSession.asMockedSession()}
>
<ReactionToggleButton vm={vm} identifier={localIdent} />
</ReactionsSenderProvider>

View File

@@ -14,7 +14,7 @@ import {
import { distinctUntilChanged } from "rxjs";
import { useObservableEagerState } from "observable-hooks";
import { type GridLayout as GridLayoutModel } from "../state/CallViewModel";
import { type GridLayout as GridLayoutModel } from "../state/layout-types.ts";
import styles from "./GridLayout.module.css";
import { useInitial } from "../useInitial";
import { type CallLayout, arrangeTiles } from "./CallLayout";

View File

@@ -9,7 +9,7 @@ import { type ReactNode, useCallback, useMemo } from "react";
import { useObservableEagerState } from "observable-hooks";
import classNames from "classnames";
import { type OneOnOneLayout as OneOnOneLayoutModel } from "../state/CallViewModel";
import { type OneOnOneLayout as OneOnOneLayoutModel } from "../state/layout-types.ts";
import { type CallLayout, arrangeTiles } from "./CallLayout";
import styles from "./OneOnOneLayout.module.css";
import { type DragCallback, useUpdateLayout } from "./Grid";

View File

@@ -7,7 +7,7 @@ Please see LICENSE in the repository root for full details.
import { type ReactNode, useCallback } from "react";
import { type SpotlightExpandedLayout as SpotlightExpandedLayoutModel } from "../state/CallViewModel";
import { type SpotlightExpandedLayout as SpotlightExpandedLayoutModel } from "../state/layout-types.ts";
import { type CallLayout } from "./CallLayout";
import { type DragCallback, useUpdateLayout } from "./Grid";
import styles from "./SpotlightExpandedLayout.module.css";

View File

@@ -10,7 +10,7 @@ import { useObservableEagerState } from "observable-hooks";
import classNames from "classnames";
import { type CallLayout } from "./CallLayout";
import { type SpotlightLandscapeLayout as SpotlightLandscapeLayoutModel } from "../state/CallViewModel";
import { type SpotlightLandscapeLayout as SpotlightLandscapeLayoutModel } from "../state/layout-types.ts";
import styles from "./SpotlightLandscapeLayout.module.css";
import { useUpdateLayout, useVisibleTiles } from "./Grid";

View File

@@ -10,7 +10,7 @@ import { useObservableEagerState } from "observable-hooks";
import classNames from "classnames";
import { type CallLayout, arrangeTiles } from "./CallLayout";
import { type SpotlightPortraitLayout as SpotlightPortraitLayoutModel } from "../state/CallViewModel";
import { type SpotlightPortraitLayout as SpotlightPortraitLayoutModel } from "../state/layout-types.ts";
import styles from "./SpotlightPortraitLayout.module.css";
import { useUpdateLayout, useVisibleTiles } from "./Grid";
import { useBehavior } from "../useBehavior";

View File

@@ -113,19 +113,49 @@ const roomIsJoinable = (room: Room): boolean => {
}
};
/**
* Determines if a given room has call events in it, and therefore
* is likely to be a call room.
* @param room The Matrix room instance.
* @returns `true` if the room has call events.
*/
const roomHasCallMembershipEvents = (room: Room): boolean => {
switch (room.getMyMembership()) {
case KnownMembership.Join:
return !!room
.getLiveTimeline()
.getState(EventTimeline.FORWARDS)
?.events?.get(EventType.GroupCallMemberPrefix);
case KnownMembership.Knock:
// Assume that a room you've knocked on is able to hold calls
return true;
default:
return false;
// Check our room membership first, to rule out any rooms
// we can't have a call in.
const myMembership = room.getMyMembership();
if (myMembership === KnownMembership.Knock) {
// Assume that a room you've knocked on is able to hold calls
return true;
} else if (myMembership !== KnownMembership.Join) {
// Otherwise, non-joined rooms should never show up.
return false;
}
// Legacy member state checks (cheaper to check.)
const timeline = room.getLiveTimeline();
if (
timeline
.getState(EventTimeline.FORWARDS)
?.events?.has(EventType.GroupCallMemberPrefix)
) {
return true;
}
// Check for *active* calls using sticky events.
for (const sticky of room._unstable_getStickyEvents()) {
if (sticky.getType() === EventType.RTCMembership) {
return true;
}
}
// Otherwise, check recent event history to see if anyone had
// sent a call membership in here.
return timeline.getEvents().some(
(e) =>
// Membership events only count if both of these are true
e.unstableStickyInfo && e.getType() === EventType.GroupCallMemberPrefix,
);
// Otherwise, it's *unlikely* this room was ever a call.
};
export function useGroupCallRooms(client: MatrixClient): GroupCallRoom[] {

View File

@@ -6,20 +6,28 @@ Please see LICENSE in the repository root for full details.
*/
import { afterEach, beforeEach, expect, it, vi } from "vitest";
import { render } from "@testing-library/react";
import { type CallMembership } from "matrix-js-sdk/lib/matrixrtc";
import { render, type RenderResult } from "@testing-library/react";
import {
getTrackReferenceId,
type TrackReference,
} from "@livekit/components-core";
import { type RemoteAudioTrack } from "livekit-client";
import {
type Participant,
type RemoteAudioTrack,
type Room,
Track,
} from "livekit-client";
import { type ReactNode } from "react";
import { useTracks } from "@livekit/components-react";
import { testAudioContext } from "../useAudioContext.test";
import * as MediaDevicesContext from "../MediaDevicesContext";
import { MatrixAudioRenderer } from "./MatrixAudioRenderer";
import { mockMediaDevices, mockTrack } from "../utils/test";
import { LivekitRoomAudioRenderer } from "./MatrixAudioRenderer";
import {
mockMediaDevices,
mockRemoteParticipant,
mockTrack,
} from "../utils/test";
export const TestAudioContextConstructor = vi.fn(() => testAudioContext);
@@ -48,42 +56,203 @@ vi.mock("@livekit/components-react", async (importOriginal) => {
};
});
const tracks = [mockTrack("test:123")];
vi.mocked(useTracks).mockReturnValue(tracks);
let tracks: TrackReference[] = [];
it("should render for member", () => {
const { container, queryAllByTestId } = render(
/**
* Render the test component with given rtc members and livekit participant identities.
*
* It is possible to have rtc members that are not in livekit (e.g. not yet joined) and vice versa.
*
* @param rtcMembers - Array of active rtc members with userId and deviceId.
* @param livekitParticipantIdentities - Array of livekit participant (that are publishing).
* @param explicitTracks - Array of tracks available in livekit, if not provided, one audio track per livekitParticipantIdentities will be created.
* */
function renderTestComponent(
rtcMembers: { userId: string; deviceId: string }[],
livekitParticipantIdentities: string[],
explicitTracks?: {
participantId: string;
kind: Track.Kind;
source: Track.Source;
}[],
): RenderResult {
const liveKitParticipants = livekitParticipantIdentities.map((identity) =>
mockRemoteParticipant({ identity }),
);
const participants = rtcMembers.flatMap(({ userId, deviceId }) => {
const p = liveKitParticipants.find(
(p) => p.identity === `${userId}:${deviceId}`,
);
return p === undefined ? [] : [p];
});
const livekitRoom = {
remoteParticipants: new Map<string, Participant>(
liveKitParticipants.map((p) => [p.identity, p]),
),
} as unknown as Room;
if (explicitTracks?.length ?? 0 > 0) {
tracks = explicitTracks!.map(({ participantId, source, kind }) => {
const participant =
liveKitParticipants.find((p) => p.identity === participantId) ??
mockRemoteParticipant({ identity: participantId });
return mockTrack(participant, kind, source);
});
} else {
tracks = participants.map((p) => mockTrack(p));
}
vi.mocked(useTracks).mockReturnValue(tracks);
return render(
<MediaDevicesProvider value={mockMediaDevices({})}>
<MatrixAudioRenderer
members={[{ sender: "test", deviceId: "123" }] as CallMembership[]}
<LivekitRoomAudioRenderer
validIdentities={participants.map((p) => p.identity)}
livekitRoom={livekitRoom}
url={""}
/>
</MediaDevicesProvider>,
);
}
it("should render for member", () => {
const { container, queryAllByTestId } = renderTestComponent(
[{ userId: "@alice", deviceId: "DEV0" }],
["@alice:DEV0"],
);
expect(container).toBeTruthy();
expect(queryAllByTestId("audio")).toHaveLength(1);
});
it("should not render without member", () => {
const memberships = [
{ sender: "othermember", deviceId: "123" },
] as CallMembership[];
const { container, queryAllByTestId } = render(
<MediaDevicesProvider value={mockMediaDevices({})}>
<MatrixAudioRenderer members={memberships} />
</MediaDevicesProvider>,
const { container, queryAllByTestId } = renderTestComponent(
[{ userId: "@bob", deviceId: "DEV0" }],
["@alice:DEV0"],
);
expect(container).toBeTruthy();
expect(queryAllByTestId("audio")).toHaveLength(0);
});
const TEST_CASES: {
name: string;
rtcUsers: { userId: string; deviceId: string }[];
livekitParticipantIdentities: string[];
explicitTracks?: {
participantId: string;
kind: Track.Kind;
source: Track.Source;
}[];
expectedAudioTracks: number;
}[] = [
{
name: "single user single device",
rtcUsers: [
{ userId: "@alice", deviceId: "DEV0" },
{ userId: "@alice", deviceId: "DEV1" },
{ userId: "@bob", deviceId: "DEV0" },
],
livekitParticipantIdentities: ["@alice:DEV0", "@bob:DEV0", "@alice:DEV1"],
expectedAudioTracks: 3,
},
// Charlie is a rtc member but not in livekit
{
name: "Charlie is rtc member but not in livekit",
rtcUsers: [
{ userId: "@alice", deviceId: "DEV0" },
{ userId: "@bob", deviceId: "DEV0" },
{ userId: "@charlie", deviceId: "DEV0" },
],
livekitParticipantIdentities: ["@alice:DEV0", "@bob:DEV0"],
expectedAudioTracks: 2,
},
// Charlie is in livekit but not rtc member
{
name: "Charlie is in livekit but not rtc member",
rtcUsers: [
{ userId: "@alice", deviceId: "DEV0" },
{ userId: "@bob", deviceId: "DEV0" },
],
livekitParticipantIdentities: ["@alice:DEV0", "@bob:DEV0", "@charlie:DEV0"],
expectedAudioTracks: 2,
},
{
name: "no audio track, only video track",
rtcUsers: [{ userId: "@alice", deviceId: "DEV0" }],
livekitParticipantIdentities: ["@alice:DEV0"],
explicitTracks: [
{
participantId: "@alice:DEV0",
kind: Track.Kind.Video,
source: Track.Source.Camera,
},
],
expectedAudioTracks: 0,
},
{
name: "Audio track from unknown source",
rtcUsers: [{ userId: "@alice", deviceId: "DEV0" }],
livekitParticipantIdentities: ["@alice:DEV0"],
explicitTracks: [
{
participantId: "@alice:DEV0",
kind: Track.Kind.Audio,
source: Track.Source.Unknown,
},
],
expectedAudioTracks: 1,
},
{
name: "Audio track from other device",
rtcUsers: [{ userId: "@alice", deviceId: "DEV0" }],
livekitParticipantIdentities: ["@alice:DEV0"],
explicitTracks: [
{
participantId: "@alice:DEV1",
kind: Track.Kind.Audio,
source: Track.Source.Microphone,
},
],
expectedAudioTracks: 0,
},
{
name: "two audio tracks, microphone and screenshare",
rtcUsers: [{ userId: "@alice", deviceId: "DEV0" }],
livekitParticipantIdentities: ["@alice:DEV0"],
explicitTracks: [
{
participantId: "@alice:DEV0",
kind: Track.Kind.Audio,
source: Track.Source.Microphone,
},
{
participantId: "@alice:DEV0",
kind: Track.Kind.Audio,
source: Track.Source.ScreenShareAudio,
},
],
expectedAudioTracks: 2,
},
];
it.each(TEST_CASES)(
`should render sound test cases $name`,
({
rtcUsers,
livekitParticipantIdentities,
explicitTracks,
expectedAudioTracks,
}) => {
const { queryAllByTestId } = renderTestComponent(
rtcUsers,
livekitParticipantIdentities,
explicitTracks,
);
expect(queryAllByTestId("audio")).toHaveLength(expectedAudioTracks);
},
);
it("should not setup audioContext gain and pan if there is no need to.", () => {
render(
<MediaDevicesProvider value={mockMediaDevices({})}>
<MatrixAudioRenderer
members={[{ sender: "test", deviceId: "123" }] as CallMembership[]}
/>
</MediaDevicesProvider>,
);
renderTestComponent([{ userId: "@bob", deviceId: "DEV0" }], ["@bob:DEV0"]);
const audioTrack = tracks[0].publication.track! as RemoteAudioTrack;
expect(audioTrack.setAudioContext).toHaveBeenCalledTimes(1);
@@ -100,13 +269,8 @@ it("should setup audioContext gain and pan", () => {
pan: 1,
volume: 0.1,
});
render(
<MediaDevicesProvider value={mockMediaDevices({})}>
<MatrixAudioRenderer
members={[{ sender: "test", deviceId: "123" }] as CallMembership[]}
/>
</MediaDevicesProvider>,
);
renderTestComponent([{ userId: "@bob", deviceId: "DEV0" }], ["@bob:DEV0"]);
const audioTrack = tracks[0].publication.track! as RemoteAudioTrack;
expect(audioTrack.setAudioContext).toHaveBeenCalled();

View File

@@ -6,15 +6,16 @@ Please see LICENSE in the repository root for full details.
*/
import { getTrackReferenceId } from "@livekit/components-core";
import { type Room as LivekitRoom } from "livekit-client";
import { type RemoteAudioTrack, Track } from "livekit-client";
import { useEffect, useMemo, useRef, useState, type ReactNode } from "react";
import { useEffect, useMemo, useState, type ReactNode } from "react";
import {
useTracks,
AudioTrack,
type AudioTrackProps,
} from "@livekit/components-react";
import { type CallMembership } from "matrix-js-sdk/lib/matrixrtc";
import { logger } from "matrix-js-sdk/lib/logger";
import { type ParticipantId } from "matrix-js-sdk/lib/matrixrtc";
import { useEarpieceAudioConfig } from "../MediaDevicesContext";
import { useReactiveState } from "../useReactiveState";
@@ -22,11 +23,16 @@ import * as controls from "../controls";
export interface MatrixAudioRendererProps {
/**
* The list of participants to render audio for.
* This list needs to be composed based on the matrixRTC members so that we do not play audio from users
* that are not expected to be in the rtc session.
* The service URL of the LiveKit room.
*/
members: CallMembership[];
url: string;
livekitRoom: LivekitRoom;
/**
* The list of participant identities to render audio for.
* This list needs to be composed based on the matrixRTC members so that we do not play audio from users
* that are not expected to be in the rtc session (local user is excluded).
*/
validIdentities: ParticipantId[];
/**
* If set to `true`, mutes all audio tracks rendered by the component.
* @remarks
@@ -35,9 +41,9 @@ export interface MatrixAudioRendererProps {
muted?: boolean;
}
const prefixedLogger = logger.getChild("[MatrixAudioRenderer]");
/**
* The `MatrixAudioRenderer` component is a drop-in solution for adding audio to your LiveKit app.
* It takes care of handling remote participants audio tracks and makes sure that microphones and screen share are audible.
* Takes care of handling remote participants audio tracks and makes sure that microphones and screen share are audible.
*
* It also takes care of the earpiece audio configuration for iOS devices.
* This is done by using the WebAudio API to create a stereo pan effect that mimics the earpiece audio.
@@ -49,35 +55,12 @@ export interface MatrixAudioRendererProps {
* ```
* @public
*/
export function MatrixAudioRenderer({
members,
export function LivekitRoomAudioRenderer({
url,
livekitRoom,
validIdentities,
muted,
}: MatrixAudioRendererProps): ReactNode {
const validIdentities = useMemo(
() =>
new Set(members?.map((member) => `${member.sender}:${member.deviceId}`)),
[members],
);
const loggedInvalidIdentities = useRef(new Set<string>());
/**
* Log an invalid livekit track identity.
* A invalid identity is one that does not match any of the matrix rtc members.
*
* @param identity The identity of the track that is invalid
* @param validIdentities The list of valid identities
*/
const logInvalid = (identity: string, validIdentities: Set<string>): void => {
if (loggedInvalidIdentities.current.has(identity)) return;
logger.warn(
`[MatrixAudioRenderer] Audio track ${identity} has no matching matrix call member`,
`current members: ${Array.from(validIdentities.values())}`,
`track will not get rendered`,
);
loggedInvalidIdentities.current.add(identity);
};
const tracks = useTracks(
[
Track.Source.Microphone,
@@ -87,25 +70,25 @@ export function MatrixAudioRenderer({
{
updateOnlyOn: [],
onlySubscribed: true,
room: livekitRoom,
},
).filter((ref) => {
const isValid = validIdentities?.has(ref.participant.identity);
if (!isValid && !ref.participant.isLocal)
logInvalid(ref.participant.identity, validIdentities);
return (
!ref.participant.isLocal &&
ref.publication.kind === Track.Kind.Audio &&
isValid
);
});
useEffect(() => {
if (!tracks.some((t) => !validIdentities.has(t.participant.identity))) {
logger.debug(
`[MatrixAudioRenderer] All audio tracks have a matching matrix call member identity.`,
);
loggedInvalidIdentities.current.clear();
}
}, [tracks, validIdentities]);
)
// Only keep audio tracks
.filter((ref) => ref.publication.kind === Track.Kind.Audio)
// Only keep tracks from participants that are in the validIdentities list
.filter((ref) => {
const isValid = validIdentities.includes(ref.participant.identity);
if (!isValid) {
// Log that there is an invalid identity, that means that someone is publishing audio that is not expected to be in the call.
prefixedLogger.warn(
`Audio track ${ref.participant.identity} from ${url} has no matching matrix call member`,
`current members: ${validIdentities.join()}`,
`track will not get rendered`,
);
return false;
}
return true;
});
// This component is also (in addition to the "only play audio for connected members" logic above)
// responsible for mimicking earpiece audio on iPhones.

View File

@@ -19,14 +19,21 @@ import {
useMemo,
} from "react";
import { type LocalVideoTrack } from "livekit-client";
import { combineLatest, map, type Observable } from "rxjs";
import { useObservable } from "observable-hooks";
import {
backgroundBlur as backgroundBlurSettings,
useSetting,
} from "../settings/settings";
import { BlurBackgroundTransformer } from "./BlurBackgroundTransformer";
import { type Behavior } from "../state/Behavior";
type ProcessorState = {
//TODO-MULTI-SFU: This is not yet fully there.
// it is a combination of exposing observable and react hooks.
// preferably we should not make this a context anymore and instead just a vm?
export type ProcessorState = {
supported: boolean | undefined;
processor: undefined | ProcessorWrapper<BackgroundOptions>;
};
@@ -42,6 +49,40 @@ export function useTrackProcessor(): ProcessorState {
return state;
}
export function useTrackProcessorObservable$(): Observable<ProcessorState> {
const state = use(ProcessorContext);
if (state === undefined)
throw new Error(
"useTrackProcessor must be used within a ProcessorProvider",
);
const state$ = useObservable(
(init$) => init$.pipe(map(([init]) => init)),
[state],
);
return state$;
}
export const trackProcessorSync = (
videoTrack$: Behavior<LocalVideoTrack | null>,
processor$: Behavior<ProcessorState>,
): void => {
// TODO-MULTI-SFU: Bind to an ObservableScope to avoid leaking resources.
combineLatest([videoTrack$, processor$]).subscribe(
([videoTrack, processorState]) => {
if (!processorState) return;
if (!videoTrack) return;
const { processor } = processorState;
if (processor && !videoTrack.getProcessor()) {
void videoTrack.setProcessor(processor);
}
if (!processor && videoTrack.getProcessor()) {
void videoTrack.stopProcessor();
}
},
);
};
export const useTrackProcessorSync = (
videoTrack: LocalVideoTrack | null,
): void => {

View File

@@ -7,12 +7,7 @@ Please see LICENSE in the repository root for full details.
import { type IOpenIDToken, type MatrixClient } from "matrix-js-sdk";
import { logger } from "matrix-js-sdk/lib/logger";
import { type MatrixRTCSession } from "matrix-js-sdk/lib/matrixrtc";
import { useEffect, useState } from "react";
import { type LivekitFocus } from "matrix-js-sdk/lib/matrixrtc";
import { useActiveLivekitFocus } from "../room/useActiveFocus";
import { useErrorBoundary } from "../useErrorBoundary";
import { FailToGetOpenIdToken } from "../utils/errors";
import { doNetworkOperationWithRetry } from "../utils/matrix";
@@ -34,38 +29,11 @@ export type OpenIDClientParts = Pick<
"getOpenIdToken" | "getDeviceId"
>;
export function useOpenIDSFU(
client: OpenIDClientParts,
rtcSession: MatrixRTCSession,
): SFUConfig | undefined {
const [sfuConfig, setSFUConfig] = useState<SFUConfig | undefined>(undefined);
const activeFocus = useActiveLivekitFocus(rtcSession);
const { showErrorBoundary } = useErrorBoundary();
useEffect(() => {
if (activeFocus) {
getSFUConfigWithOpenID(client, activeFocus).then(
(sfuConfig) => {
setSFUConfig(sfuConfig);
},
(e) => {
showErrorBoundary(new FailToGetOpenIdToken(e));
logger.error("Failed to get SFU config", e);
},
);
} else {
setSFUConfig(undefined);
}
}, [client, activeFocus, showErrorBoundary]);
return sfuConfig;
}
export async function getSFUConfigWithOpenID(
client: OpenIDClientParts,
activeFocus: LivekitFocus,
): Promise<SFUConfig | undefined> {
serviceUrl: string,
livekitAlias: string,
): Promise<SFUConfig> {
let openIdToken: IOpenIDToken;
try {
openIdToken = await doNetworkOperationWithRetry(async () =>
@@ -78,26 +46,16 @@ export async function getSFUConfigWithOpenID(
}
logger.debug("Got openID token", openIdToken);
try {
logger.info(
`Trying to get JWT from call's active focus URL of ${activeFocus.livekit_service_url}...`,
);
const sfuConfig = await getLiveKitJWT(
client,
activeFocus.livekit_service_url,
activeFocus.livekit_alias,
openIdToken,
);
logger.info(`Got JWT from call's active focus URL.`);
logger.info(`Trying to get JWT for focus ${serviceUrl}...`);
const sfuConfig = await getLiveKitJWT(
client,
serviceUrl,
livekitAlias,
openIdToken,
);
logger.info(`Got JWT from call's active focus URL.`);
return sfuConfig;
} catch (e) {
logger.warn(
`Failed to get JWT from RTC session's active focus URL of ${activeFocus.livekit_service_url}.`,
e,
);
return undefined;
}
return sfuConfig;
}
async function getLiveKitJWT(

View File

@@ -1,3 +1,4 @@
// TODO not used anymore - remove
/*
Copyright 2023, 2024 New Vector Ltd.

View File

@@ -1,431 +0,0 @@
/*
Copyright 2023, 2024 New Vector Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import {
ConnectionState,
type E2EEManagerOptions,
ExternalE2EEKeyProvider,
type LocalTrackPublication,
LocalVideoTrack,
Room,
type RoomOptions,
Track,
} from "livekit-client";
import { useEffect, useRef } from "react";
import E2EEWorker from "livekit-client/e2ee-worker?worker";
import { logger } from "matrix-js-sdk/lib/logger";
import { type MatrixRTCSession } from "matrix-js-sdk/lib/matrixrtc";
import { useObservable, useObservableEagerState } from "observable-hooks";
import {
map,
NEVER,
type Observable,
type Subscription,
switchMap,
} from "rxjs";
import { defaultLiveKitOptions } from "./options";
import { type SFUConfig } from "./openIDSFU";
import { type MuteStates } from "../room/MuteStates";
import { useMediaDevices } from "../MediaDevicesContext";
import {
type ECConnectionState,
useECConnectionState,
} from "./useECConnectionState";
import { MatrixKeyProvider } from "../e2ee/matrixKeyProvider";
import { E2eeType } from "../e2ee/e2eeType";
import { type EncryptionSystem } from "../e2ee/sharedKeyManagement";
import {
useTrackProcessor,
useTrackProcessorSync,
} from "./TrackProcessorContext";
import { observeTrackReference$ } from "../state/MediaViewModel";
import { useUrlParams } from "../UrlParams";
import { useInitial } from "../useInitial";
import { getValue } from "../utils/observable";
import { type SelectedDevice } from "../state/MediaDevices";
interface UseLivekitResult {
livekitRoom?: Room;
connState: ECConnectionState;
}
export function useLivekit(
rtcSession: MatrixRTCSession,
muteStates: MuteStates,
sfuConfig: SFUConfig | undefined,
e2eeSystem: EncryptionSystem,
): UseLivekitResult {
const { controlledAudioDevices } = useUrlParams();
const initialMuteStates = useInitial(() => muteStates);
const devices = useMediaDevices();
const initialAudioInputId = useInitial(
() => getValue(devices.audioInput.selected$)?.id,
);
// Store if audio/video are currently updating. If to prohibit unnecessary calls
// to setMicrophoneEnabled/setCameraEnabled
const audioMuteUpdating = useRef(false);
const videoMuteUpdating = useRef(false);
// Store the current button mute state that gets passed to this hook via props.
// We need to store it for awaited code that relies on the current value.
const buttonEnabled = useRef({
audio: initialMuteStates.audio.enabled,
video: initialMuteStates.video.enabled,
});
const { processor } = useTrackProcessor();
// Only ever create the room once via useInitial.
const room = useInitial(() => {
logger.info("[LivekitRoom] Create LiveKit room");
let e2ee: E2EEManagerOptions | undefined;
if (e2eeSystem.kind === E2eeType.PER_PARTICIPANT) {
logger.info("Created MatrixKeyProvider (per participant)");
e2ee = {
keyProvider: new MatrixKeyProvider(),
worker: new E2EEWorker(),
};
} else if (e2eeSystem.kind === E2eeType.SHARED_KEY && e2eeSystem.secret) {
logger.info("Created ExternalE2EEKeyProvider (shared key)");
e2ee = {
keyProvider: new ExternalE2EEKeyProvider(),
worker: new E2EEWorker(),
};
}
const roomOptions: RoomOptions = {
...defaultLiveKitOptions,
videoCaptureDefaults: {
...defaultLiveKitOptions.videoCaptureDefaults,
deviceId: getValue(devices.videoInput.selected$)?.id,
processor,
},
audioCaptureDefaults: {
...defaultLiveKitOptions.audioCaptureDefaults,
deviceId: initialAudioInputId,
},
audioOutput: {
// When using controlled audio devices, we don't want to set the
// deviceId here, because it will be set by the native app.
// (also the id does not need to match a browser device id)
deviceId: controlledAudioDevices
? undefined
: getValue(devices.audioOutput.selected$)?.id,
},
e2ee,
};
// We have to create the room manually here due to a bug inside
// @livekit/components-react. JSON.stringify() is used in deps of a
// useEffect() with an argument that references itself, if E2EE is enabled
const room = new Room(roomOptions);
room.setE2EEEnabled(e2eeSystem.kind !== E2eeType.NONE).catch((e) => {
logger.error("Failed to set E2EE enabled on room", e);
});
return room;
});
// Setup and update the keyProvider which was create by `createRoom`
useEffect(() => {
const e2eeOptions = room.options.e2ee;
if (
e2eeSystem.kind === E2eeType.NONE ||
!(e2eeOptions && "keyProvider" in e2eeOptions)
)
return;
if (e2eeSystem.kind === E2eeType.PER_PARTICIPANT) {
(e2eeOptions.keyProvider as MatrixKeyProvider).setRTCSession(rtcSession);
} else if (e2eeSystem.kind === E2eeType.SHARED_KEY && e2eeSystem.secret) {
(e2eeOptions.keyProvider as ExternalE2EEKeyProvider)
.setKey(e2eeSystem.secret)
.catch((e) => {
logger.error("Failed to set shared key for E2EE", e);
});
}
}, [room.options.e2ee, e2eeSystem, rtcSession]);
// Sync the requested track processors with LiveKit
useTrackProcessorSync(
useObservableEagerState(
useObservable(
(room$) =>
room$.pipe(
switchMap(([room]) =>
observeTrackReference$(
room.localParticipant,
Track.Source.Camera,
),
),
map((trackRef) => {
const track = trackRef?.publication?.track;
return track instanceof LocalVideoTrack ? track : null;
}),
),
[room],
),
),
);
const connectionState = useECConnectionState(
initialAudioInputId,
initialMuteStates.audio.enabled,
room,
sfuConfig,
);
// Log errors when local participant has issues publishing a track.
useEffect(() => {
const localTrackUnpublishedFn = (
publication: LocalTrackPublication,
): void => {
logger.info(
"Local track unpublished",
publication.trackName,
publication.trackInfo,
);
};
const mediaDevicesErrorFn = (error: Error): void => {
logger.warn("Media devices error when publishing a track", error);
};
room.localParticipant.on("localTrackUnpublished", localTrackUnpublishedFn);
room.localParticipant.on("mediaDevicesError", mediaDevicesErrorFn);
return (): void => {
room.localParticipant.off(
"localTrackUnpublished",
localTrackUnpublishedFn,
);
room.localParticipant.off("mediaDevicesError", mediaDevicesErrorFn);
};
}, [room.localParticipant]);
useEffect(() => {
// Sync the requested mute states with LiveKit's mute states. We do it this
// way around rather than using LiveKit as the source of truth, so that the
// states can be consistent throughout the lobby and loading screens.
// It's important that we only do this in the connected state, because
// LiveKit's internal mute states aren't consistent during connection setup,
// and setting tracks to be enabled during this time causes errors.
if (room !== undefined && connectionState === ConnectionState.Connected) {
const participant = room.localParticipant;
// Always update the muteButtonState Ref so that we can read the current
// state in awaited blocks.
buttonEnabled.current = {
audio: muteStates.audio.enabled,
video: muteStates.video.enabled,
};
enum MuteDevice {
Microphone,
Camera,
}
const syncMuteState = async (
iterCount: number,
type: MuteDevice,
): Promise<void> => {
// The approach for muting is to always bring the actual livekit state in sync with the button
// This allows for a very predictable and reactive behavior for the user.
// (the new state is the old state when pressing the button n times (where n is even))
// (the new state is different to the old state when pressing the button n times (where n is uneven))
// In case there are issues with the device there might be situations where setMicrophoneEnabled/setCameraEnabled
// return immediately. This should be caught with the Error("track with new mute state could not be published").
// For now we are still using an iterCount to limit the recursion loop to 10.
// This could happen if the device just really does not want to turn on (hardware based issue)
// but the mute button is in unmute state.
// For now our fail mode is to just stay in this state.
// TODO: decide for a UX on how that fail mode should be treated (disable button, hide button, sync button back to muted without user input)
if (iterCount > 10) {
logger.error(
"Stop trying to sync the input device with current mute state after 10 failed tries",
);
return;
}
let devEnabled;
let btnEnabled;
let updating;
switch (type) {
case MuteDevice.Microphone:
devEnabled = participant.isMicrophoneEnabled;
btnEnabled = buttonEnabled.current.audio;
updating = audioMuteUpdating.current;
break;
case MuteDevice.Camera:
devEnabled = participant.isCameraEnabled;
btnEnabled = buttonEnabled.current.video;
updating = videoMuteUpdating.current;
break;
}
if (devEnabled !== btnEnabled && !updating) {
try {
let trackPublication;
switch (type) {
case MuteDevice.Microphone:
audioMuteUpdating.current = true;
trackPublication = await participant.setMicrophoneEnabled(
buttonEnabled.current.audio,
room.options.audioCaptureDefaults,
);
audioMuteUpdating.current = false;
break;
case MuteDevice.Camera:
videoMuteUpdating.current = true;
trackPublication = await participant.setCameraEnabled(
buttonEnabled.current.video,
room.options.videoCaptureDefaults,
);
videoMuteUpdating.current = false;
break;
}
if (trackPublication) {
// await participant.setMicrophoneEnabled can return immediately in some instances,
// so that participant.isMicrophoneEnabled !== buttonEnabled.current.audio still holds true.
// This happens if the device is still in a pending state
// "sleeping" here makes sure we let react do its thing so that participant.isMicrophoneEnabled is updated,
// so we do not end up in a recursion loop.
await new Promise((r) => setTimeout(r, 100));
// track got successfully changed to mute/unmute
// Run the check again after the change is done. Because the user
// can update the state (presses mute button) while the device is enabling
// itself we need might need to update the mute state right away.
// This async recursion makes sure that setCamera/MicrophoneEnabled is
// called as little times as possible.
await syncMuteState(iterCount + 1, type);
} else {
throw new Error(
"track with new mute state could not be published",
);
}
} catch (e) {
if ((e as DOMException).name === "NotAllowedError") {
logger.error(
"Fatal error while syncing mute state: resetting",
e,
);
if (type === MuteDevice.Microphone) {
audioMuteUpdating.current = false;
muteStates.audio.setEnabled?.(false);
} else {
videoMuteUpdating.current = false;
muteStates.video.setEnabled?.(false);
}
} else {
logger.error(
"Failed to sync audio mute state with LiveKit (will retry to sync in 1s):",
e,
);
setTimeout(() => {
syncMuteState(iterCount + 1, type).catch((e) => {
logger.error(
`Failed to sync ${MuteDevice[type]} mute state with LiveKit iterCount=${iterCount + 1}`,
e,
);
});
}, 1000);
}
}
}
};
syncMuteState(0, MuteDevice.Microphone).catch((e) => {
logger.error("Failed to sync audio mute state with LiveKit", e);
});
syncMuteState(0, MuteDevice.Camera).catch((e) => {
logger.error("Failed to sync video mute state with LiveKit", e);
});
}
}, [room, muteStates, connectionState]);
useEffect(() => {
// Sync the requested devices with LiveKit's devices
if (room !== undefined && connectionState === ConnectionState.Connected) {
const syncDevice = (
kind: MediaDeviceKind,
selected$: Observable<SelectedDevice | undefined>,
): Subscription =>
selected$.subscribe((device) => {
logger.info(
"[LivekitRoom] syncDevice room.getActiveDevice(kind) !== d.id :",
room.getActiveDevice(kind),
" !== ",
device?.id,
);
if (
device !== undefined &&
room.getActiveDevice(kind) !== device.id
) {
room
.switchActiveDevice(kind, device.id)
.catch((e) =>
logger.error(`Failed to sync ${kind} device with LiveKit`, e),
);
}
});
const subscriptions = [
syncDevice("audioinput", devices.audioInput.selected$),
!controlledAudioDevices
? syncDevice("audiooutput", devices.audioOutput.selected$)
: undefined,
syncDevice("videoinput", devices.videoInput.selected$),
// Restart the audio input track whenever we detect that the active media
// device has changed to refer to a different hardware device. We do this
// for the sake of Chrome, which provides a "default" device that is meant
// to match the system's default audio input, whatever that may be.
// This is special-cased for only audio inputs because we need to dig around
// in the LocalParticipant object for the track object and there's not a nice
// way to do that generically. There is usually no OS-level default video capture
// device anyway, and audio outputs work differently.
devices.audioInput.selected$
.pipe(switchMap((device) => device?.hardwareDeviceChange$ ?? NEVER))
.subscribe(() => {
const activeMicTrack = Array.from(
room.localParticipant.audioTrackPublications.values(),
).find((d) => d.source === Track.Source.Microphone)?.track;
if (
activeMicTrack &&
// only restart if the stream is still running: LiveKit will detect
// when a track stops & restart appropriately, so this is not our job.
// Plus, we need to avoid restarting again if the track is already in
// the process of being restarted.
activeMicTrack.mediaStreamTrack.readyState !== "ended"
) {
// Restart the track, which will cause Livekit to do another
// getUserMedia() call with deviceId: default to get the *new* default device.
// Note that room.switchActiveDevice() won't work: Livekit will ignore it because
// the deviceId hasn't changed (was & still is default).
room.localParticipant
.getTrackPublication(Track.Source.Microphone)
?.audioTrack?.restartTrack()
.catch((e) => {
logger.error(`Failed to restart audio device track`, e);
});
}
}),
];
return (): void => {
for (const s of subscriptions) s?.unsubscribe();
};
}
}, [room, devices, connectionState, controlledAudioDevices]);
return {
connState: connectionState,
livekitRoom: room,
};
}

View File

@@ -61,7 +61,7 @@ Initializer.initBeforeReact()
.then(() => {
root.render(
<StrictMode>
<App vm={new AppViewModel()} />
<App vm={new AppViewModel()} />,
</StrictMode>,
);
})

View File

@@ -7,7 +7,6 @@ Please see LICENSE in the repository root for full details.
import { renderHook } from "@testing-library/react";
import { afterEach, test, vitest } from "vitest";
import { type MatrixRTCSession } from "matrix-js-sdk/lib/matrixrtc";
import {
RoomEvent as MatrixRoomEvent,
MatrixEvent,
@@ -38,7 +37,7 @@ test("handles a hand raised reaction", () => {
withTestScheduler(({ schedule, expectObservable }) => {
renderHook(() => {
const { raisedHands$ } = new ReactionsReader(
rtcSession as unknown as MatrixRTCSession,
rtcSession.asMockedSession(),
);
schedule("ab", {
a: () => {},
@@ -48,7 +47,7 @@ test("handles a hand raised reaction", () => {
new MatrixEvent({
room_id: rtcSession.room.roomId,
event_id: reactionEventId,
sender: localRtcMember.sender,
sender: localRtcMember.userId,
type: EventType.Reaction,
origin_server_ts: localTimestamp.getTime(),
content: {
@@ -68,7 +67,7 @@ test("handles a hand raised reaction", () => {
expectObservable(raisedHands$).toBe("ab", {
a: {},
b: {
[`${localRtcMember.sender}:${localRtcMember.deviceId}`]: {
[`${localRtcMember.userId}:${localRtcMember.deviceId}`]: {
reactionEventId,
membershipEventId: localRtcMember.eventId,
time: localTimestamp,
@@ -86,7 +85,7 @@ test("handles a redaction", () => {
withTestScheduler(({ schedule, expectObservable }) => {
renderHook(() => {
const { raisedHands$ } = new ReactionsReader(
rtcSession as unknown as MatrixRTCSession,
rtcSession.asMockedSession(),
);
schedule("abc", {
a: () => {},
@@ -96,7 +95,7 @@ test("handles a redaction", () => {
new MatrixEvent({
room_id: rtcSession.room.roomId,
event_id: reactionEventId,
sender: localRtcMember.sender,
sender: localRtcMember.userId,
type: EventType.Reaction,
origin_server_ts: localTimestamp.getTime(),
content: {
@@ -118,7 +117,7 @@ test("handles a redaction", () => {
new MatrixEvent({
room_id: rtcSession.room.roomId,
event_id: reactionEventId,
sender: localRtcMember.sender,
sender: localRtcMember.userId,
type: EventType.RoomRedaction,
redacts: reactionEventId,
}),
@@ -130,7 +129,7 @@ test("handles a redaction", () => {
expectObservable(raisedHands$).toBe("abc", {
a: {},
b: {
[`${localRtcMember.sender}:${localRtcMember.deviceId}`]: {
[`${localRtcMember.userId}:${localRtcMember.deviceId}`]: {
reactionEventId,
membershipEventId: localRtcMember.eventId,
time: localTimestamp,
@@ -149,7 +148,7 @@ test("handles waiting for event decryption", () => {
withTestScheduler(({ schedule, expectObservable }) => {
renderHook(() => {
const { raisedHands$ } = new ReactionsReader(
rtcSession as unknown as MatrixRTCSession,
rtcSession.asMockedSession(),
);
schedule("abc", {
a: () => {},
@@ -157,7 +156,7 @@ test("handles waiting for event decryption", () => {
const encryptedEvent = new MatrixEvent({
room_id: rtcSession.room.roomId,
event_id: reactionEventId,
sender: localRtcMember.sender,
sender: localRtcMember.userId,
type: EventType.Reaction,
origin_server_ts: localTimestamp.getTime(),
content: {
@@ -184,7 +183,7 @@ test("handles waiting for event decryption", () => {
new MatrixEvent({
room_id: rtcSession.room.roomId,
event_id: reactionEventId,
sender: localRtcMember.sender,
sender: localRtcMember.userId,
type: EventType.Reaction,
origin_server_ts: localTimestamp.getTime(),
content: {
@@ -200,7 +199,7 @@ test("handles waiting for event decryption", () => {
expectObservable(raisedHands$).toBe("a-c", {
a: {},
c: {
[`${localRtcMember.sender}:${localRtcMember.deviceId}`]: {
[`${localRtcMember.userId}:${localRtcMember.deviceId}`]: {
reactionEventId,
membershipEventId: localRtcMember.eventId,
time: localTimestamp,
@@ -218,7 +217,7 @@ test("hands rejecting events without a proper membership", () => {
withTestScheduler(({ schedule, expectObservable }) => {
renderHook(() => {
const { raisedHands$ } = new ReactionsReader(
rtcSession as unknown as MatrixRTCSession,
rtcSession.asMockedSession(),
);
schedule("ab", {
a: () => {},
@@ -228,7 +227,7 @@ test("hands rejecting events without a proper membership", () => {
new MatrixEvent({
room_id: rtcSession.room.roomId,
event_id: reactionEventId,
sender: localRtcMember.sender,
sender: localRtcMember.userId,
type: EventType.Reaction,
origin_server_ts: localTimestamp.getTime(),
content: {
@@ -262,9 +261,7 @@ test("handles a reaction", () => {
withTestScheduler(({ schedule, time, expectObservable }) => {
renderHook(() => {
const { reactions$ } = new ReactionsReader(
rtcSession as unknown as MatrixRTCSession,
);
const { reactions$ } = new ReactionsReader(rtcSession.asMockedSession());
schedule(`abc`, {
a: () => {},
b: () => {
@@ -273,7 +270,7 @@ test("handles a reaction", () => {
new MatrixEvent({
room_id: rtcSession.room.roomId,
event_id: reactionEventId,
sender: localRtcMember.sender,
sender: localRtcMember.userId,
type: ElementCallReactionEventType,
content: {
emoji: reaction.emoji,
@@ -298,7 +295,7 @@ test("handles a reaction", () => {
{
a: {},
b: {
[`${localRtcMember.sender}:${localRtcMember.deviceId}`]: {
[`${localRtcMember.userId}:${localRtcMember.deviceId}`]: {
reactionOption: reaction,
expireAfter: new Date(REACTION_ACTIVE_TIME_MS),
},
@@ -320,9 +317,7 @@ test("ignores bad reaction events", () => {
withTestScheduler(({ schedule, expectObservable }) => {
renderHook(() => {
const { reactions$ } = new ReactionsReader(
rtcSession as unknown as MatrixRTCSession,
);
const { reactions$ } = new ReactionsReader(rtcSession.asMockedSession());
schedule("ab", {
a: () => {},
b: () => {
@@ -332,7 +327,7 @@ test("ignores bad reaction events", () => {
new MatrixEvent({
room_id: rtcSession.room.roomId,
event_id: reactionEventId,
sender: localRtcMember.sender,
sender: localRtcMember.userId,
type: ElementCallReactionEventType,
content: {},
}),
@@ -347,7 +342,7 @@ test("ignores bad reaction events", () => {
new MatrixEvent({
room_id: rtcSession.room.roomId,
event_id: reactionEventId,
sender: localRtcMember.sender,
sender: localRtcMember.userId,
type: ElementCallReactionEventType,
content: {
emoji: reaction.emoji,
@@ -368,7 +363,7 @@ test("ignores bad reaction events", () => {
new MatrixEvent({
room_id: rtcSession.room.roomId,
event_id: reactionEventId,
sender: aliceRtcMember.sender,
sender: aliceRtcMember.userId,
type: ElementCallReactionEventType,
content: {
emoji: reaction.emoji,
@@ -389,7 +384,7 @@ test("ignores bad reaction events", () => {
new MatrixEvent({
room_id: rtcSession.room.roomId,
event_id: reactionEventId,
sender: localRtcMember.sender,
sender: localRtcMember.userId,
type: ElementCallReactionEventType,
content: {
name: reaction.name,
@@ -409,7 +404,7 @@ test("ignores bad reaction events", () => {
new MatrixEvent({
room_id: rtcSession.room.roomId,
event_id: reactionEventId,
sender: localRtcMember.sender,
sender: localRtcMember.userId,
type: ElementCallReactionEventType,
content: {
emoji: " ",
@@ -444,9 +439,7 @@ test("that reactions cannot be spammed", () => {
withTestScheduler(({ schedule, expectObservable }) => {
renderHook(() => {
const { reactions$ } = new ReactionsReader(
rtcSession as unknown as MatrixRTCSession,
);
const { reactions$ } = new ReactionsReader(rtcSession.asMockedSession());
schedule("abcd", {
a: () => {},
b: () => {
@@ -455,7 +448,7 @@ test("that reactions cannot be spammed", () => {
new MatrixEvent({
room_id: rtcSession.room.roomId,
event_id: reactionEventId,
sender: localRtcMember.sender,
sender: localRtcMember.userId,
type: ElementCallReactionEventType,
content: {
emoji: reactionA.emoji,
@@ -477,7 +470,7 @@ test("that reactions cannot be spammed", () => {
new MatrixEvent({
room_id: rtcSession.room.roomId,
event_id: reactionEventId,
sender: localRtcMember.sender,
sender: localRtcMember.userId,
type: ElementCallReactionEventType,
content: {
emoji: reactionB.emoji,
@@ -502,7 +495,7 @@ test("that reactions cannot be spammed", () => {
{
a: {},
b: {
[`${localRtcMember.sender}:${localRtcMember.deviceId}`]: {
[`${localRtcMember.userId}:${localRtcMember.deviceId}`]: {
reactionOption: reactionA,
expireAfter: new Date(REACTION_ACTIVE_TIME_MS),
},

View File

@@ -130,7 +130,7 @@ export class ReactionsReader {
private onMembershipsChanged = (oldMemberships: CallMembership[]): void => {
// Remove any raised hands for users no longer joined to the call.
for (const identifier of Object.keys(this.raisedHandsSubject$.value).filter(
(rhId) => oldMemberships.find((u) => u.sender == rhId),
(rhId) => oldMemberships.find((u) => u.userId == rhId),
)) {
this.removeRaisedHand(identifier);
}
@@ -138,10 +138,10 @@ export class ReactionsReader {
// For each member in the call, check to see if a reaction has
// been raised and adjust.
for (const m of this.rtcSession.memberships) {
if (!m.sender || !m.eventId) {
if (!m.userId || !m.eventId) {
continue;
}
const identifier = `${m.sender}:${m.deviceId}`;
const identifier = `${m.userId}:${m.deviceId}`;
if (
this.raisedHandsSubject$.value[identifier] &&
this.raisedHandsSubject$.value[identifier].membershipEventId !==
@@ -151,13 +151,13 @@ export class ReactionsReader {
// was raised, reset.
this.removeRaisedHand(identifier);
}
const reaction = this.getLastReactionEvent(m.eventId, m.sender);
const reaction = this.getLastReactionEvent(m.eventId, m.userId);
if (reaction) {
const eventId = reaction?.getId();
if (!eventId) {
continue;
}
this.addRaisedHand(`${m.sender}:${m.deviceId}`, {
this.addRaisedHand(`${m.userId}:${m.deviceId}`, {
membershipEventId: m.eventId,
reactionEventId: eventId,
time: new Date(reaction.localTimestamp),
@@ -219,7 +219,7 @@ export class ReactionsReader {
const membershipEventId = content?.["m.relates_to"]?.event_id;
const membershipEvent = this.rtcSession.memberships.find(
(e) => e.eventId === membershipEventId && e.sender === sender,
(e) => e.eventId === membershipEventId && e.userId === sender,
);
// Check to see if this reaction was made to a membership event (and the
// sender of the reaction matches the membership)
@@ -229,7 +229,7 @@ export class ReactionsReader {
);
return;
}
const identifier = `${membershipEvent.sender}:${membershipEvent.deviceId}`;
const identifier = `${membershipEvent.userId}:${membershipEvent.deviceId}`;
if (!content.emoji) {
logger.warn(`Reaction had no emoji from ${reactionEventId}`);
@@ -278,7 +278,7 @@ export class ReactionsReader {
// Check to see if this reaction was made to a membership event (and the
// sender of the reaction matches the membership)
const membershipEvent = this.rtcSession.memberships.find(
(e) => e.eventId === membershipEventId && e.sender === sender,
(e) => e.eventId === membershipEventId && e.userId === sender,
);
if (!membershipEvent) {
logger.warn(
@@ -289,7 +289,7 @@ export class ReactionsReader {
if (content?.["m.relates_to"].key === "🖐️") {
this.addRaisedHand(
`${membershipEvent.sender}:${membershipEvent.deviceId}`,
`${membershipEvent.userId}:${membershipEvent.deviceId}`,
{
reactionEventId,
membershipEventId,

View File

@@ -65,7 +65,7 @@ export const ReactionsSenderProvider = ({
const myMembershipEvent = useMemo(
() =>
memberships.find(
(m) => m.sender === myUserId && m.deviceId === myDeviceId,
(m) => m.userId === myUserId && m.deviceId === myDeviceId,
)?.eventId,
[memberships, myUserId, myDeviceId],
);

View File

@@ -155,7 +155,8 @@ test("plays one sound when a hand is raised", () => {
act(() => {
handRaisedSubject$.next({
[bobRtcMember.callId]: {
// TODO: What is this string supposed to be?
[`${bobRtcMember.userId}:${bobRtcMember.deviceId}`]: {
time: new Date(),
membershipEventId: "",
reactionEventId: "",

View File

@@ -26,7 +26,7 @@ import {
E2EENotSupportedError,
type ElementCallError,
InsufficientCapacityError,
MatrixRTCFocusMissingError,
MatrixRTCTransportMissingError,
UnknownCallError,
} from "../utils/errors.ts";
import { mockConfig } from "../utils/test.ts";
@@ -34,7 +34,7 @@ import { ElementWidgetActions, type WidgetHelpers } from "../widget.ts";
test.each([
{
error: new MatrixRTCFocusMissingError("example.com"),
error: new MatrixRTCTransportMissingError("example.com"),
expectedTitle: "Call is not supported",
},
{
@@ -85,7 +85,7 @@ test.each([
);
test("should render the error page with link back to home", async () => {
const error = new MatrixRTCFocusMissingError("example.com");
const error = new MatrixRTCTransportMissingError("example.com");
const TestComponent = (): ReactNode => {
throw error;
};
@@ -106,7 +106,7 @@ test("should render the error page with link back to home", async () => {
await screen.findByText("Call is not supported");
expect(screen.getByText(/Domain: example\.com/i)).toBeInTheDocument();
expect(
screen.getByText(/Error Code: MISSING_MATRIX_RTC_FOCUS/i),
screen.getByText(/Error Code: MISSING_MATRIX_RTC_TRANSPORT/i),
).toBeInTheDocument();
await screen.findByRole("button", { name: "Return to home screen" });
@@ -213,7 +213,7 @@ describe("Rageshake button", () => {
});
test("should have a close button in widget mode", async () => {
const error = new MatrixRTCFocusMissingError("example.com");
const error = new MatrixRTCTransportMissingError("example.com");
const TestComponent = (): ReactNode => {
throw error;
};

View File

@@ -26,7 +26,6 @@ import { type RelationsContainer } from "matrix-js-sdk/lib/models/relations-cont
import { useState } from "react";
import { TooltipProvider } from "@vector-im/compound-web";
import { type MuteStates } from "./MuteStates";
import { prefetchSounds } from "../soundUtils";
import { useAudioContext } from "../useAudioContext";
import { ActiveCall } from "./InCallView";
@@ -42,11 +41,12 @@ import {
import { GroupCallView } from "./GroupCallView";
import { type WidgetHelpers } from "../widget";
import { LazyEventEmitter } from "../LazyEventEmitter";
import { MatrixRTCFocusMissingError } from "../utils/errors";
import { MatrixRTCTransportMissingError } from "../utils/errors";
import { ProcessorProvider } from "../livekit/TrackProcessorContext";
import { MediaDevicesContext } from "../MediaDevicesContext";
import { HeaderStyle } from "../UrlParams";
import { constant } from "../state/Behavior";
import { type MuteStates } from "../state/MuteStates.ts";
vi.mock("../soundUtils");
vi.mock("../useAudioContext");
@@ -103,7 +103,7 @@ beforeEach(() => {
});
// A trivial implementation of Active call to ensure we are testing GroupCallView exclusively here.
(ActiveCall as MockedFunction<typeof ActiveCall>).mockImplementation(
({ onLeave }) => {
({ onLeft: onLeave }) => {
return (
<div>
<button onClick={() => onLeave("user")}>Leave</button>
@@ -117,12 +117,12 @@ function createGroupCallView(
widget: WidgetHelpers | null,
joined = true,
): {
rtcSession: MockRTCSession;
rtcSession: MatrixRTCSession;
getByText: ReturnType<typeof render>["getByText"];
} {
const client = {
getUser: () => null,
getUserId: () => localRtcMember.sender,
getUserId: () => localRtcMember.userId,
getDeviceId: () => localRtcMember.deviceId,
getRoom: (rId) => (rId === roomId ? room : null),
} as Partial<MatrixClient> as MatrixClient;
@@ -150,7 +150,8 @@ function createGroupCallView(
const muteState = {
audio: { enabled: false },
video: { enabled: false },
} as MuteStates;
// TODO-MULTI-SFU: This cast isn't valid, it's likely the cause of some current test failures
} as unknown as MuteStates;
const { getByText } = render(
<BrowserRouter>
<TooltipProvider>
@@ -163,10 +164,12 @@ function createGroupCallView(
preload={false}
skipLobby={false}
header={HeaderStyle.Standard}
rtcSession={rtcSession as unknown as MatrixRTCSession}
isJoined={joined}
rtcSession={rtcSession.asMockedSession()}
muteStates={muteState}
widget={widget}
// TODO-MULTI-SFU: Make joined and setJoined work
joined={true}
setJoined={function (value: boolean): void {}}
/>
</ProcessorProvider>
</MediaDevicesContext>
@@ -175,7 +178,7 @@ function createGroupCallView(
);
return {
getByText,
rtcSession,
rtcSession: rtcSession.asMockedSession(),
};
}
@@ -258,7 +261,7 @@ test("GroupCallView leaves the session when an error occurs", async () => {
test("GroupCallView shows errors that occur during joining", async () => {
const user = userEvent.setup();
enterRTCSession.mockRejectedValue(new MatrixRTCFocusMissingError(""));
enterRTCSession.mockRejectedValue(new MatrixRTCTransportMissingError(""));
onTestFinished(() => {
enterRTCSession.mockReset();
});

View File

@@ -38,10 +38,9 @@ import { PosthogAnalytics } from "../analytics/PosthogAnalytics";
import { useProfile } from "../profile/useProfile";
import { findDeviceByName } from "../utils/media";
import { ActiveCall } from "./InCallView";
import { MUTE_PARTICIPANT_COUNT, type MuteStates } from "./MuteStates";
import { type MuteStates } from "../state/MuteStates";
import { useMediaDevices } from "../MediaDevicesContext";
import { useMatrixRTCSessionMemberships } from "../useMatrixRTCSessionMemberships";
import { enterRTCSession, leaveRTCSession } from "../rtcSessionHelpers";
import {
saveKeyForRoom,
useRoomEncryptionSystem,
@@ -50,7 +49,12 @@ import { useRoomAvatar } from "./useRoomAvatar";
import { useRoomName } from "./useRoomName";
import { useJoinRule } from "./useJoinRule";
import { InviteModal } from "./InviteModal";
import { HeaderStyle, type UrlParams, useUrlParams } from "../UrlParams";
import {
getUrlParams,
HeaderStyle,
type UrlParams,
useUrlParams,
} from "../UrlParams";
import { E2eeType } from "../e2ee/e2eeType";
import { useAudioContext } from "../useAudioContext";
import {
@@ -66,16 +70,17 @@ import {
UnknownCallError,
} from "../utils/errors.ts";
import { GroupCallErrorBoundary } from "./GroupCallErrorBoundary.tsx";
import {
useNewMembershipManager as useNewMembershipManagerSetting,
useExperimentalToDeviceTransport as useExperimentalToDeviceTransportSetting,
useSetting,
} from "../settings/settings";
import { useTypedEventEmitter } from "../useEvents";
import { muteAllAudio$ } from "../state/MuteAllAudioModel.ts";
import { useAppBarTitle } from "../AppBar.tsx";
import { useBehavior } from "../useBehavior.ts";
/**
* If there already are this many participants in the call, we automatically mute
* the user.
*/
export const MUTE_PARTICIPANT_COUNT = 8;
declare global {
interface Window {
rtcSession?: MatrixRTCSession;
@@ -90,7 +95,8 @@ interface Props {
skipLobby: UrlParams["skipLobby"];
header: HeaderStyle;
rtcSession: MatrixRTCSession;
isJoined: boolean;
joined: boolean;
setJoined: (value: boolean) => void;
muteStates: MuteStates;
widget: WidgetHelpers | null;
}
@@ -103,7 +109,8 @@ export const GroupCallView: FC<Props> = ({
skipLobby,
header,
rtcSession,
isJoined,
joined,
setJoined,
muteStates,
widget,
}) => {
@@ -124,7 +131,7 @@ export const GroupCallView: FC<Props> = ({
// This should use `useEffectEvent` (only available in experimental versions)
useEffect(() => {
if (memberships.length >= MUTE_PARTICIPANT_COUNT)
muteStates.audio.setEnabled?.(false);
muteStates.audio.setEnabled$.value?.(false);
// eslint-disable-next-line react-hooks/exhaustive-deps
}, []);
@@ -175,10 +182,6 @@ export const GroupCallView: FC<Props> = ({
password: passwordFromUrl,
} = useUrlParams();
const e2eeSystem = useRoomEncryptionSystem(room.roomId);
const [useNewMembershipManager] = useSetting(useNewMembershipManagerSetting);
const [useExperimentalToDeviceTransport] = useSetting(
useExperimentalToDeviceTransportSetting,
);
// Save the password once we start the groupCallView
useEffect(() => {
@@ -203,7 +206,7 @@ export const GroupCallView: FC<Props> = ({
// Count each member only once, regardless of how many devices they use
const participantCount = useMemo(
() => new Set<string>(memberships.map((m) => m.sender!)).size,
() => new Set<string>(memberships.map((m) => m.userId!)).size,
[memberships],
);
@@ -213,12 +216,9 @@ export const GroupCallView: FC<Props> = ({
const enterRTCSessionOrError = useCallback(
async (rtcSession: MatrixRTCSession): Promise<void> => {
try {
await enterRTCSession(
rtcSession,
perParticipantE2EE,
useNewMembershipManager,
useExperimentalToDeviceTransport,
);
setJoined(true);
// TODO-MULTI-SFU what to do with error handling now that we don't use this function?
// @BillCarsonFr
} catch (e) {
if (e instanceof ElementCallError) {
setExternalError(e);
@@ -230,12 +230,9 @@ export const GroupCallView: FC<Props> = ({
setExternalError(error);
}
}
return Promise.resolve();
},
[
perParticipantE2EE,
useExperimentalToDeviceTransport,
useNewMembershipManager,
],
[setJoined],
);
useEffect(() => {
@@ -254,7 +251,7 @@ export const GroupCallView: FC<Props> = ({
if (!deviceId) {
logger.warn("Unknown audio input: " + audioInput);
// override the default mute state
latestMuteStates.current!.audio.setEnabled?.(false);
latestMuteStates.current!.audio.setEnabled$.value?.(false);
} else {
logger.debug(
`Found audio input ID ${deviceId} for name ${audioInput}`,
@@ -268,7 +265,7 @@ export const GroupCallView: FC<Props> = ({
if (!deviceId) {
logger.warn("Unknown video input: " + videoInput);
// override the default mute state
latestMuteStates.current!.video.setEnabled?.(false);
latestMuteStates.current!.video.setEnabled$.value?.(false);
} else {
logger.debug(
`Found video input ID ${deviceId} for name ${videoInput}`,
@@ -284,7 +281,7 @@ export const GroupCallView: FC<Props> = ({
const onJoin = (ev: CustomEvent<IWidgetApiRequest>): void => {
(async (): Promise<void> => {
await defaultDeviceSetup(ev.detail.data as unknown as JoinCallData);
await enterRTCSessionOrError(rtcSession);
setJoined(true);
widget.api.transport.reply(ev.detail, {});
})().catch((e) => {
logger.error("Error joining RTC session on preload", e);
@@ -296,11 +293,7 @@ export const GroupCallView: FC<Props> = ({
};
} else {
// No lobby and no preload: we enter the rtc session right away
(async (): Promise<void> => {
await enterRTCSessionOrError(rtcSession);
})().catch((e) => {
logger.error("Error joining RTC session immediately", e);
});
setJoined(true);
}
}
}, [
@@ -311,61 +304,85 @@ export const GroupCallView: FC<Props> = ({
perParticipantE2EE,
mediaDevices,
latestMuteStates,
enterRTCSessionOrError,
useNewMembershipManager,
setJoined,
]);
// TODO refactor this + "joined" to just one callState
const [left, setLeft] = useState(false);
const navigate = useNavigate();
const onLeave = useCallback(
(
cause: "user" | "error" = "user",
playSound: CallEventSounds = "left",
): void => {
const audioPromise = leaveSoundContext.current?.playSound(playSound);
// In embedded/widget mode the iFrame will be killed right after the call ended prohibiting the posthog event from getting sent,
// therefore we want the event to be sent instantly without getting queued/batched.
const sendInstantly = !!widget;
const onLeft = useCallback(
(reason: "timeout" | "user" | "allOthersLeft" | "decline"): void => {
let playSound: CallEventSounds = "left";
if (reason === "timeout" || reason === "decline") playSound = reason;
setJoined(false);
setLeft(true);
// we need to wait until the callEnded event is tracked on posthog.
// Otherwise the iFrame gets killed before the callEnded event got tracked.
const audioPromise = leaveSoundContext.current?.playSound(playSound);
// We need to wait until the callEnded event is tracked on PostHog,
// otherwise the iframe may get killed first.
const posthogRequest = new Promise((resolve) => {
// To increase the likelihood of the PostHog event being sent out in
// widget mode before the iframe is killed, we ask it to skip the
// usual queuing/batching of requests.
const sendInstantly = widget !== null;
PosthogAnalytics.instance.eventCallEnded.track(
room.roomId,
rtcSession.memberships.length,
sendInstantly,
rtcSession,
);
// Unfortunately the PostHog library provides no way to await the
// tracking of an event, but we don't really want it to hold up the
// closing of the widget that long anyway, so giving it 10 ms will do.
window.setTimeout(resolve, 10);
});
leaveRTCSession(
rtcSession,
cause,
// Wait for the sound in widget mode (it's not long)
Promise.all([audioPromise, posthogRequest]),
)
// Only sends matrix leave event. The Livekit session will disconnect once the ActiveCall-view unmounts.
void Promise.all([audioPromise, posthogRequest])
.catch((e) =>
logger.error(
"Failed to play leave audio and/or send PostHog leave event",
e,
),
)
.then(async () => {
if (
!isPasswordlessUser &&
!confineToRoom &&
!PosthogAnalytics.instance.isEnabled()
) {
await navigate("/");
)
void navigate("/");
if (widget) {
// After this point the iframe could die at any moment!
try {
await widget.api.setAlwaysOnScreen(false);
} catch (e) {
logger.error(
"Failed to set call widget `alwaysOnScreen` to false",
e,
);
}
// On a normal user hangup we can shut down and close the widget. But if an
// error occurs we should keep the widget open until the user reads it.
if (reason === "user" && !getUrlParams().returnToLobby) {
try {
await widget.api.transport.send(ElementWidgetActions.Close, {});
} catch (e) {
logger.error("Failed to send close action", e);
}
widget.api.transport.stop();
}
}
})
.catch((e) => {
logger.error("Error leaving RTC session", e);
});
},
[
setJoined,
leaveSoundContext,
widget,
rtcSession,
room.roomId,
rtcSession,
isPasswordlessUser,
confineToRoom,
navigate,
@@ -373,25 +390,12 @@ export const GroupCallView: FC<Props> = ({
);
useEffect(() => {
if (widget && isJoined) {
if (widget && joined)
// set widget to sticky once joined.
widget.api.setAlwaysOnScreen(true).catch((e) => {
logger.error("Error calling setAlwaysOnScreen(true)", e);
});
const onHangup = (ev: CustomEvent<IWidgetApiRequest>): void => {
widget.api.transport.reply(ev.detail, {});
// Only sends matrix leave event. The Livekit session will disconnect once the ActiveCall-view unmounts.
leaveRTCSession(rtcSession, "user").catch((e) => {
logger.error("Failed to leave RTC session", e);
});
};
widget.lazyActions.once(ElementWidgetActions.HangupCall, onHangup);
return (): void => {
widget.lazyActions.off(ElementWidgetActions.HangupCall, onHangup);
};
}
}, [widget, isJoined, rtcSession]);
}, [widget, joined, rtcSession]);
const joinRule = useJoinRule(room);
@@ -426,7 +430,7 @@ export const GroupCallView: FC<Props> = ({
client={client}
matrixInfo={matrixInfo}
muteStates={muteStates}
onEnter={() => void enterRTCSessionOrError(rtcSession)}
onEnter={() => setJoined(true)}
confineToRoom={confineToRoom}
hideHeader={header === HeaderStyle.None}
participantCount={participantCount}
@@ -444,7 +448,7 @@ export const GroupCallView: FC<Props> = ({
throw externalError;
};
body = <ErrorComponent />;
} else if (isJoined) {
} else if (joined) {
body = (
<>
{shareModal}
@@ -453,7 +457,7 @@ export const GroupCallView: FC<Props> = ({
matrixInfo={matrixInfo}
rtcSession={rtcSession as MatrixRTCSession}
matrixRoom={room}
onLeave={onLeave}
onLeft={onLeft}
header={header}
muteStates={muteStates}
e2eeSystem={e2eeSystem}
@@ -514,7 +518,8 @@ export const GroupCallView: FC<Props> = ({
}}
onError={
(/**error*/) => {
if (rtcSession.isJoined()) onLeave("error");
// TODO this should not be "user". It needs a new case
if (rtcSession.isJoined()) onLeft("user");
}
}
>

View File

@@ -15,7 +15,6 @@ import {
} from "vitest";
import { act, render, type RenderResult } from "@testing-library/react";
import { type MatrixClient, JoinRule, type RoomState } from "matrix-js-sdk";
import { type MatrixRTCSession } from "matrix-js-sdk/lib/matrixrtc";
import { type RelationsContainer } from "matrix-js-sdk/lib/models/relations-container";
import { type LocalParticipant } from "livekit-client";
import { of } from "rxjs";
@@ -24,7 +23,6 @@ import { TooltipProvider } from "@vector-im/compound-web";
import { RoomContext, useLocalParticipant } from "@livekit/components-react";
import { RoomAndToDeviceEvents } from "matrix-js-sdk/lib/matrixrtc/RoomAndToDeviceKeyTransport";
import { type MuteStates } from "./MuteStates";
import { InCallView } from "./InCallView";
import {
mockLivekitRoom,
@@ -32,6 +30,7 @@ import {
mockMatrixRoom,
mockMatrixRoomMember,
mockMediaDevices,
mockMuteStates,
mockRemoteParticipant,
mockRtcMembership,
type MockRTCSession,
@@ -45,7 +44,7 @@ import {
} from "../settings/settings";
import { ReactionsSenderProvider } from "../reactions/useReactionsSender";
import { useRoomEncryptionSystem } from "../e2ee/sharedKeyManagement";
import { MatrixAudioRenderer } from "../livekit/MatrixAudioRenderer";
import { LivekitRoomAudioRenderer } from "../livekit/MatrixAudioRenderer";
import { MediaDevicesContext } from "../MediaDevicesContext";
import { HeaderStyle } from "../UrlParams";
@@ -88,7 +87,7 @@ beforeEach(() => {
// MatrixAudioRenderer is tested separately.
(
MatrixAudioRenderer as MockedFunction<typeof MatrixAudioRenderer>
LivekitRoomAudioRenderer as MockedFunction<typeof LivekitRoomAudioRenderer>
).mockImplementation((_props) => {
return <div>mocked: MatrixAudioRenderer</div>;
});
@@ -111,7 +110,7 @@ function createInCallView(): RenderResult & {
} {
const client = {
getUser: () => null,
getUserId: () => localRtcMember.sender,
getUserId: () => localRtcMember.userId,
getDeviceId: () => localRtcMember.deviceId,
getRoom: (rId) => (rId === roomId ? room : null),
} as Partial<MatrixClient> as MatrixClient;
@@ -133,10 +132,7 @@ function createInCallView(): RenderResult & {
} as Partial<RoomState> as RoomState,
});
const muteState = {
audio: { enabled: false },
video: { enabled: false },
} as MuteStates;
const muteState = mockMuteStates();
const livekitRoom = mockLivekitRoom(
{
localParticipant,
@@ -153,14 +149,14 @@ function createInCallView(): RenderResult & {
<MediaDevicesContext value={mockMediaDevices({})}>
<ReactionsSenderProvider
vm={vm}
rtcSession={rtcSession as unknown as MatrixRTCSession}
rtcSession={rtcSession.asMockedSession()}
>
<TooltipProvider>
<RoomContext value={livekitRoom}>
<InCallView
client={client}
header={HeaderStyle.Standard}
rtcSession={rtcSession as unknown as MatrixRTCSession}
rtcSession={rtcSession.asMockedSession()}
muteStates={muteState}
vm={vm}
matrixInfo={{
@@ -176,10 +172,6 @@ function createInCallView(): RenderResult & {
},
}}
matrixRoom={room}
livekitRoom={livekitRoom}
onLeave={function (): void {
throw new Error("Function not implemented.");
}}
onShareClick={null}
/>
</RoomContext>

View File

@@ -5,9 +5,7 @@ SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { RoomContext, useLocalParticipant } from "@livekit/components-react";
import { IconButton, Text, Tooltip } from "@vector-im/compound-web";
import { ConnectionState, type Room as LivekitRoom } from "livekit-client";
import { type MatrixClient, type Room as MatrixRoom } from "matrix-js-sdk";
import {
type FC,
@@ -25,11 +23,7 @@ import useMeasure from "react-use-measure";
import { type MatrixRTCSession } from "matrix-js-sdk/lib/matrixrtc";
import classNames from "classnames";
import { BehaviorSubject, map } from "rxjs";
import {
useObservable,
useObservableEagerState,
useSubscription,
} from "observable-hooks";
import { useObservable } from "observable-hooks";
import { logger } from "matrix-js-sdk/lib/logger";
import { RoomAndToDeviceEvents } from "matrix-js-sdk/lib/matrixrtc/RoomAndToDeviceKeyTransport";
import {
@@ -59,19 +53,13 @@ import { type OTelGroupCallMembership } from "../otel/OTelGroupCallMembership";
import { SettingsModal, defaultSettingsTab } from "../settings/SettingsModal";
import { useRageshakeRequestModal } from "../settings/submit-rageshake";
import { RageshakeRequestModal } from "./RageshakeRequestModal";
import { useLivekit } from "../livekit/useLivekit.ts";
import { useWakeLock } from "../useWakeLock";
import { useMergedRefs } from "../useMergedRefs";
import { type MuteStates } from "./MuteStates";
import { type MuteStates } from "../state/MuteStates";
import { type MatrixInfo } from "./VideoPreview";
import { InviteButton } from "../button/InviteButton";
import { LayoutToggle } from "./LayoutToggle";
import { useOpenIDSFU } from "../livekit/openIDSFU";
import {
CallViewModel,
type GridMode,
type Layout,
} from "../state/CallViewModel";
import { CallViewModel, type GridMode } from "../state/CallViewModel";
import { Grid, type TileProps } from "../grid/Grid";
import { useInitial } from "../useInitial";
import { SpotlightTile } from "../tile/SpotlightTile";
@@ -97,10 +85,7 @@ import {
} from "../reactions/useReactionsSender";
import { ReactionsAudioRenderer } from "./ReactionAudioRenderer";
import { ReactionsOverlay } from "./ReactionsOverlay";
import {
CallEventAudioRenderer,
type CallEventSounds,
} from "./CallEventAudioRenderer";
import { CallEventAudioRenderer } from "./CallEventAudioRenderer";
import {
debugTileLayout as debugTileLayoutSetting,
useExperimentalToDeviceTransport as useExperimentalToDeviceTransportSetting,
@@ -108,11 +93,9 @@ import {
useSetting,
} from "../settings/settings";
import { ReactionsReader } from "../reactions/ReactionsReader";
import { ConnectionLostError } from "../utils/errors.ts";
import { useTypedEventEmitter } from "../useEvents.ts";
import { MatrixAudioRenderer } from "../livekit/MatrixAudioRenderer.tsx";
import { LivekitRoomAudioRenderer } from "../livekit/MatrixAudioRenderer.tsx";
import { muteAllAudio$ } from "../state/MuteAllAudioModel.ts";
import { useMatrixRTCSessionMemberships } from "../useMatrixRTCSessionMemberships.ts";
import { useMediaDevices } from "../MediaDevicesContext.ts";
import { EarpieceOverlay } from "./EarpieceOverlay.tsx";
import { useAppBarHidden, useAppBarSecondaryButton } from "../AppBar.tsx";
@@ -125,99 +108,69 @@ import { prefetchSounds } from "../soundUtils";
import { useAudioContext } from "../useAudioContext";
import ringtoneMp3 from "../sound/ringtone.mp3?url";
import ringtoneOgg from "../sound/ringtone.ogg?url";
const canScreenshare = "getDisplayMedia" in (navigator.mediaDevices ?? {});
import { useTrackProcessorObservable$ } from "../livekit/TrackProcessorContext.tsx";
import { type Layout } from "../state/layout-types.ts";
const maxTapDurationMs = 400;
export interface ActiveCallProps
extends Omit<InCallViewProps, "vm" | "livekitRoom" | "connState"> {
e2eeSystem: EncryptionSystem;
// TODO refactor those reasons into an enum
onLeft: (reason: "user" | "timeout" | "decline" | "allOthersLeft") => void;
}
export const ActiveCall: FC<ActiveCallProps> = (props) => {
const mediaDevices = useMediaDevices();
const sfuConfig = useOpenIDSFU(props.client, props.rtcSession);
const { livekitRoom, connState } = useLivekit(
props.rtcSession,
props.muteStates,
sfuConfig,
props.e2eeSystem,
);
const connStateObservable$ = useObservable(
(inputs$) => inputs$.pipe(map(([connState]) => connState)),
[connState],
);
const [vm, setVm] = useState<CallViewModel | null>(null);
useEffect(() => {
logger.info(
`[Lifecycle] InCallView Component mounted, livekit room state ${livekitRoom?.state}`,
);
return (): void => {
logger.info(
`[Lifecycle] InCallView Component unmounted, livekit room state ${livekitRoom?.state}`,
);
livekitRoom
?.disconnect()
.then(() => {
logger.info(
`[Lifecycle] Disconnected from livekit room, state:${livekitRoom?.state}`,
);
})
.catch((e) => {
logger.error("[Lifecycle] Failed to disconnect from livekit room", e);
});
};
}, [livekitRoom]);
const { autoLeaveWhenOthersLeft, sendNotificationType, waitForCallPickup } =
const { autoLeaveWhenOthersLeft, waitForCallPickup, sendNotificationType } =
useUrlParams();
const trackProcessorState$ = useTrackProcessorObservable$();
useEffect(() => {
if (livekitRoom !== undefined) {
const reactionsReader = new ReactionsReader(props.rtcSession);
const vm = new CallViewModel(
props.rtcSession,
props.matrixRoom,
livekitRoom,
mediaDevices,
{
encryptionSystem: props.e2eeSystem,
autoLeaveWhenOthersLeft,
waitForCallPickup:
waitForCallPickup && sendNotificationType === "ring",
},
connStateObservable$,
reactionsReader.raisedHands$,
reactionsReader.reactions$,
);
setVm(vm);
return (): void => {
vm.destroy();
reactionsReader.destroy();
};
}
const reactionsReader = new ReactionsReader(props.rtcSession);
const vm = new CallViewModel(
props.rtcSession,
props.matrixRoom,
mediaDevices,
props.muteStates,
{
encryptionSystem: props.e2eeSystem,
autoLeaveWhenOthersLeft,
waitForCallPickup: waitForCallPickup && sendNotificationType === "ring",
},
reactionsReader.raisedHands$,
reactionsReader.reactions$,
trackProcessorState$,
);
setVm(vm);
const sub = vm.leave$.subscribe(props.onLeft);
return (): void => {
vm.destroy();
sub.unsubscribe();
reactionsReader.destroy();
};
}, [
props.rtcSession,
props.matrixRoom,
livekitRoom,
mediaDevices,
props.muteStates,
props.e2eeSystem,
connStateObservable$,
autoLeaveWhenOthersLeft,
sendNotificationType,
waitForCallPickup,
props.onLeft,
trackProcessorState$,
]);
if (livekitRoom === undefined || vm === null) return null;
if (vm === null) return null;
return (
<RoomContext value={livekitRoom}>
<ReactionsSenderProvider vm={vm} rtcSession={props.rtcSession}>
<InCallView {...props} vm={vm} livekitRoom={livekitRoom} />
</ReactionsSenderProvider>
</RoomContext>
<ReactionsSenderProvider vm={vm} rtcSession={props.rtcSession}>
<InCallView {...props} vm={vm} />
</ReactionsSenderProvider>
);
};
@@ -227,10 +180,7 @@ export interface InCallViewProps {
matrixInfo: MatrixInfo;
rtcSession: MatrixRTCSession;
matrixRoom: MatrixRoom;
livekitRoom: LivekitRoom;
muteStates: MuteStates;
/** Function to call when the user explicitly ends the call */
onLeave: (cause: "user", soundFile?: CallEventSounds) => void;
header: HeaderStyle;
otelGroupCallMembership?: OTelGroupCallMembership;
onShareClick: (() => void) | null;
@@ -242,9 +192,8 @@ export const InCallView: FC<InCallViewProps> = ({
matrixInfo,
rtcSession,
matrixRoom,
livekitRoom,
muteStates,
onLeave,
header: headerStyle,
onShareClick,
}) => {
@@ -253,23 +202,23 @@ export const InCallView: FC<InCallViewProps> = ({
useReactionsSender();
useWakeLock();
const connectionState = useObservableEagerState(vm.livekitConnectionState$);
// TODO-MULTI-SFU This is unused now??
// const connectionState = useObservableEagerState(vm.livekitConnectionState$);
// annoyingly we don't get the disconnection reason this way,
// only by listening for the emitted event
if (connectionState === ConnectionState.Disconnected)
throw new ConnectionLostError();
// This needs to be done differential. with the vm connection state we start with Disconnected.
// TODO-MULTI-SFU decide how to handle this properly
// @BillCarsonFr
// if (connectionState === ConnectionState.Disconnected)
// throw new ConnectionLostError();
const containerRef1 = useRef<HTMLDivElement | null>(null);
const [containerRef2, bounds] = useMeasure();
// Merge the refs so they can attach to the same element
const containerRef = useMergedRefs(containerRef1, containerRef2);
const { hideScreensharing, showControls } = useUrlParams();
const { isScreenShareEnabled, localParticipant } = useLocalParticipant({
room: livekitRoom,
});
const { showControls } = useUrlParams();
const muteAllAudio = useBehavior(muteAllAudio$);
// Call pickup state and display names are needed for waiting overlay/sounds
@@ -301,7 +250,6 @@ export const InCallView: FC<InCallViewProps> = ({
useExperimentalToDeviceTransportSetting,
);
const encryptionSystem = useRoomEncryptionSystem(matrixRoom.roomId);
const memberships = useMatrixRTCSessionMemberships(rtcSession);
const showToDeviceEncryption = useMemo(
() =>
@@ -317,26 +265,25 @@ export const InCallView: FC<InCallViewProps> = ({
],
);
const toggleMicrophone = useCallback(
() => muteStates.audio.setEnabled?.((e) => !e),
[muteStates],
);
const toggleCamera = useCallback(
() => muteStates.video.setEnabled?.((e) => !e),
[muteStates],
);
const audioEnabled = useBehavior(muteStates.audio.enabled$);
const videoEnabled = useBehavior(muteStates.video.enabled$);
const toggleAudio = useBehavior(muteStates.audio.toggle$);
const toggleVideo = useBehavior(muteStates.video.toggle$);
const setAudioEnabled = useBehavior(muteStates.audio.setEnabled$);
// This function incorrectly assumes that there is a camera and microphone, which is not always the case.
// TODO: Make sure that this module is resilient when it comes to camera/microphone availability!
useCallViewKeyboardShortcuts(
containerRef1,
toggleMicrophone,
toggleCamera,
(muted) => muteStates.audio.setEnabled?.(!muted),
toggleAudio,
toggleVideo,
setAudioEnabled,
(reaction) => void sendReaction(reaction),
() => void toggleRaisedHand(),
);
const allLivekitRooms = useBehavior(vm.allLivekitRooms$);
const audioParticipants = useBehavior(vm.audioParticipants$);
const participantCount = useBehavior(vm.participantCount$);
const reconnecting = useBehavior(vm.reconnecting$);
const windowMode = useBehavior(vm.windowMode$);
@@ -348,7 +295,11 @@ export const InCallView: FC<InCallViewProps> = ({
const showFooter = useBehavior(vm.showFooter$);
const earpieceMode = useBehavior(vm.earpieceMode$);
const audioOutputSwitcher = useBehavior(vm.audioOutputSwitcher$);
useSubscription(vm.autoLeave$, () => onLeave("user"));
const sharingScreen = useBehavior(vm.sharingScreen$);
const fatalCallError = useBehavior(vm.configError$);
// Stop the rendering and throw for the error boundary
if (fatalCallError) throw fatalCallError;
// We need to set the proper timings on the animation based upon the sound length.
const ringDuration = pickupPhaseAudio?.soundDuration["waiting"] ?? 1;
@@ -369,16 +320,6 @@ export const InCallView: FC<InCallViewProps> = ({
};
}, [pickupPhaseAudio?.soundDuration, ringDuration]);
// When we enter timeout or decline we will leave the call.
useEffect((): void | (() => void) => {
if (callPickupState === "timeout") {
onLeave("user", "timeout");
}
if (callPickupState === "decline") {
onLeave("user", "decline");
}
}, [callPickupState, onLeave, pickupPhaseAudio]);
// When waiting for pickup, loop a waiting sound
useEffect((): void | (() => void) => {
if (callPickupState !== "ringing" || !pickupPhaseAudio) return;
@@ -396,6 +337,7 @@ export const InCallView: FC<InCallViewProps> = ({
if (callPickupState !== "ringing") return null;
// Use room state for other participants data (the one that we likely want to reach)
// TODO: this screams it wants to be a behavior in the vm.
const roomOthers = [
...matrixRoom.getMembersWithMembership("join"),
...matrixRoom.getMembersWithMembership("invite"),
@@ -799,44 +741,33 @@ export const InCallView: FC<InCallViewProps> = ({
matrixRoom.roomId,
);
const toggleScreensharing = useCallback(() => {
localParticipant
.setScreenShareEnabled(!isScreenShareEnabled, {
audio: true,
selfBrowserSurface: "include",
surfaceSwitching: "include",
systemAudio: "include",
})
.catch(logger.error);
}, [localParticipant, isScreenShareEnabled]);
const buttons: JSX.Element[] = [];
buttons.push(
<MicButton
key="audio"
muted={!muteStates.audio.enabled}
onClick={toggleMicrophone}
muted={!audioEnabled}
onClick={toggleAudio ?? undefined}
onTouchEnd={onControlsTouchEnd}
disabled={muteStates.audio.setEnabled === null}
disabled={toggleAudio === null}
data-testid="incall_mute"
/>,
<VideoButton
key="video"
muted={!muteStates.video.enabled}
onClick={toggleCamera}
muted={!videoEnabled}
onClick={toggleVideo ?? undefined}
onTouchEnd={onControlsTouchEnd}
disabled={muteStates.video.setEnabled === null}
disabled={toggleVideo === null}
data-testid="incall_videomute"
/>,
);
if (canScreenshare && !hideScreensharing) {
if (vm.toggleScreenSharing !== null) {
buttons.push(
<ShareScreenButton
key="share_screen"
className={styles.shareScreen}
enabled={isScreenShareEnabled}
onClick={toggleScreensharing}
enabled={sharingScreen}
onClick={vm.toggleScreenSharing}
onTouchEnd={onControlsTouchEnd}
data-testid="incall_screenshare"
/>,
@@ -866,7 +797,7 @@ export const InCallView: FC<InCallViewProps> = ({
<EndCallButton
key="end_call"
onClick={function (): void {
onLeave("user");
vm.hangup();
}}
onTouchEnd={onControlsTouchEnd}
data-testid="incall_leave"
@@ -930,7 +861,15 @@ export const InCallView: FC<InCallViewProps> = ({
</Text>
)
}
<MatrixAudioRenderer members={memberships} muted={muteAllAudio} />
{audioParticipants.map(({ livekitRoom, url, participants }) => (
<LivekitRoomAudioRenderer
key={url}
url={url}
livekitRoom={livekitRoom}
validIdentities={participants.map((p) => p.identity)}
muted={muteAllAudio}
/>
))}
{renderContent()}
<CallEventAudioRenderer vm={vm} muted={muteAllAudio} />
<ReactionsAudioRenderer vm={vm} muted={muteAllAudio} />
@@ -949,7 +888,7 @@ export const InCallView: FC<InCallViewProps> = ({
onDismiss={closeSettings}
tab={settingsTab}
onTabChange={setSettingsTab}
livekitRoom={livekitRoom}
livekitRooms={allLivekitRooms}
/>
</>
)}

View File

@@ -31,7 +31,7 @@ import inCallStyles from "./InCallView.module.css";
import styles from "./LobbyView.module.css";
import { Header, LeftNav, RightNav, RoomHeaderInfo } from "../Header";
import { type MatrixInfo, VideoPreview } from "./VideoPreview";
import { type MuteStates } from "./MuteStates";
import { type MuteStates } from "../state/MuteStates";
import { InviteButton } from "../button/InviteButton";
import {
EndCallButton,
@@ -50,8 +50,8 @@ import {
useTrackProcessorSync,
} from "../livekit/TrackProcessorContext";
import { usePageTitle } from "../usePageTitle";
import { useLatest } from "../useLatest";
import { getValue } from "../utils/observable";
import { useBehavior } from "../useBehavior";
interface Props {
client: MatrixClient;
@@ -88,14 +88,10 @@ export const LobbyView: FC<Props> = ({
const { t } = useTranslation();
usePageTitle(matrixInfo.roomName);
const onAudioPress = useCallback(
() => muteStates.audio.setEnabled?.((e) => !e),
[muteStates],
);
const onVideoPress = useCallback(
() => muteStates.video.setEnabled?.((e) => !e),
[muteStates],
);
const audioEnabled = useBehavior(muteStates.audio.enabled$);
const videoEnabled = useBehavior(muteStates.video.enabled$);
const toggleAudio = useBehavior(muteStates.audio.toggle$);
const toggleVideo = useBehavior(muteStates.video.toggle$);
const [settingsModalOpen, setSettingsModalOpen] = useState(false);
const [settingsTab, setSettingsTab] = useState(defaultSettingsTab);
@@ -133,7 +129,7 @@ export const LobbyView: FC<Props> = ({
// re-open the devices when they change (see below).
const initialAudioOptions = useInitial(
() =>
muteStates.audio.enabled && {
audioEnabled && {
deviceId: getValue(devices.audioInput.selected$)?.id,
},
);
@@ -150,27 +146,21 @@ export const LobbyView: FC<Props> = ({
// We also pass in a clone because livekit mutates the object passed in,
// which would cause the devices to be re-opened on the next render.
audio: Object.assign({}, initialAudioOptions),
video: muteStates.video.enabled && {
video: videoEnabled && {
deviceId: videoInputId,
processor: initialProcessor,
},
}),
[
initialAudioOptions,
muteStates.video.enabled,
videoInputId,
initialProcessor,
],
[initialAudioOptions, videoEnabled, videoInputId, initialProcessor],
);
const latestMuteStates = useLatest(muteStates);
const onError = useCallback(
(error: Error) => {
logger.error("Error while creating preview Tracks:", error);
latestMuteStates.current.audio.setEnabled?.(false);
latestMuteStates.current.video.setEnabled?.(false);
muteStates.audio.setEnabled$.value?.(false);
muteStates.video.setEnabled$.value?.(false);
},
[latestMuteStates],
[muteStates],
);
const tracks = usePreviewTracks(localTrackOptions, onError);
@@ -217,7 +207,7 @@ export const LobbyView: FC<Props> = ({
<div className={styles.content}>
<VideoPreview
matrixInfo={matrixInfo}
muteStates={muteStates}
videoEnabled={videoEnabled}
videoTrack={videoTrack}
>
<Button
@@ -225,6 +215,7 @@ export const LobbyView: FC<Props> = ({
[styles.wait]: waitingForInvite,
})}
size={waitingForInvite ? "sm" : "lg"}
disabled={waitingForInvite}
onClick={() => {
if (!waitingForInvite) onEnter();
}}
@@ -239,14 +230,14 @@ export const LobbyView: FC<Props> = ({
{recentsButtonInFooter && recentsButton}
<div className={inCallStyles.buttons}>
<MicButton
muted={!muteStates.audio.enabled}
onClick={onAudioPress}
disabled={muteStates.audio.setEnabled === null}
muted={!audioEnabled}
onClick={toggleAudio ?? undefined}
disabled={toggleAudio === null}
/>
<VideoButton
muted={!muteStates.video.enabled}
onClick={onVideoPress}
disabled={muteStates.video.setEnabled === null}
muted={!videoEnabled}
onClick={toggleVideo ?? undefined}
disabled={toggleVideo === null}
/>
<SettingsButton onClick={openSettings} />
{!confineToRoom && <EndCallButton onClick={onLeaveClick} />}

View File

@@ -5,6 +5,9 @@ SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
// TODO-MULTI-SFU: These tests need to be ported to the new MuteStates class.
/*
import {
afterAll,
afterEach,
@@ -321,3 +324,4 @@ describe("useMuteStates in VITE_PACKAGE='embedded' (widget) mode", () => {
expect(screen.getByTestId("video-enabled").textContent).toBe("true");
});
});
*/

View File

@@ -1,178 +0,0 @@
/*
Copyright 2023, 2024 New Vector Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import {
type Dispatch,
type SetStateAction,
useCallback,
useEffect,
useMemo,
} from "react";
import { type IWidgetApiRequest } from "matrix-widget-api";
import { logger } from "matrix-js-sdk/lib/logger";
import { useObservableEagerState } from "observable-hooks";
import {
type DeviceLabel,
type SelectedDevice,
type MediaDevice,
} from "../state/MediaDevices";
import { useIsEarpiece, useMediaDevices } from "../MediaDevicesContext";
import { useReactiveState } from "../useReactiveState";
import { ElementWidgetActions, widget } from "../widget";
import { Config } from "../config/Config";
import { useUrlParams } from "../UrlParams";
/**
* If there already are this many participants in the call, we automatically mute
* the user.
*/
export const MUTE_PARTICIPANT_COUNT = 8;
interface DeviceAvailable {
enabled: boolean;
setEnabled: Dispatch<SetStateAction<boolean>>;
}
interface DeviceUnavailable {
enabled: false;
setEnabled: null;
}
const deviceUnavailable: DeviceUnavailable = {
enabled: false,
setEnabled: null,
};
type MuteState = DeviceAvailable | DeviceUnavailable;
export interface MuteStates {
audio: MuteState;
video: MuteState;
}
function useMuteState(
device: MediaDevice<DeviceLabel, SelectedDevice>,
enabledByDefault: () => boolean,
forceUnavailable: boolean = false,
): MuteState {
const available = useObservableEagerState(device.available$);
const [enabled, setEnabled] = useReactiveState<boolean | undefined>(
// Determine the default value once devices are actually connected
(prev) => prev ?? (available.size > 0 ? enabledByDefault() : undefined),
[available.size],
);
return useMemo(
() =>
available.size === 0 || forceUnavailable
? deviceUnavailable
: {
enabled: enabled ?? false,
setEnabled: setEnabled as Dispatch<SetStateAction<boolean>>,
},
[available.size, enabled, forceUnavailable, setEnabled],
);
}
export function useMuteStates(isJoined: boolean): MuteStates {
const devices = useMediaDevices();
const { skipLobby, defaultAudioEnabled, defaultVideoEnabled } =
useUrlParams();
const audio = useMuteState(
devices.audioInput,
() =>
(defaultAudioEnabled ?? Config.get().media_devices.enable_audio) &&
allowJoinUnmuted(skipLobby, isJoined),
);
useEffect(() => {
// If audio is enabled, we need to request the device names again,
// because iOS will not be able to switch to the correct device after un-muting.
// This is one of the main changes that makes iOS work with bluetooth audio devices.
if (audio.enabled) {
devices.requestDeviceNames();
}
}, [audio.enabled, devices]);
const isEarpiece = useIsEarpiece();
const video = useMuteState(
devices.videoInput,
() =>
(defaultVideoEnabled ?? Config.get().media_devices.enable_video) &&
allowJoinUnmuted(skipLobby, isJoined),
isEarpiece, // Force video to be unavailable if using earpiece
);
useEffect(() => {
widget?.api.transport
.send(ElementWidgetActions.DeviceMute, {
audio_enabled: audio.enabled,
video_enabled: video.enabled,
})
.catch((e) =>
logger.warn("Could not send DeviceMute action to widget", e),
);
}, [audio, video]);
const onMuteStateChangeRequest = useCallback(
(ev: CustomEvent<IWidgetApiRequest>) => {
// First copy the current state into our new state.
const newState = {
audio_enabled: audio.enabled,
video_enabled: video.enabled,
};
// Update new state if there are any requested changes from the widget action
// in `ev.detail.data`.
if (
ev.detail.data.audio_enabled != null &&
typeof ev.detail.data.audio_enabled === "boolean"
) {
audio.setEnabled?.(ev.detail.data.audio_enabled);
newState.audio_enabled = ev.detail.data.audio_enabled;
}
if (
ev.detail.data.video_enabled != null &&
typeof ev.detail.data.video_enabled === "boolean"
) {
video.setEnabled?.(ev.detail.data.video_enabled);
newState.video_enabled = ev.detail.data.video_enabled;
}
// Always reply with the new (now "current") state.
// This allows to also use this action to just get the unaltered current state
// by using a fromWidget request with: `ev.detail.data = {}`
widget!.api.transport.reply(ev.detail, newState);
},
[audio, video],
);
useEffect(() => {
// We setup a event listener for the widget action ElementWidgetActions.DeviceMute.
if (widget) {
// only setup the listener in widget mode
widget.lazyActions.on(
ElementWidgetActions.DeviceMute,
onMuteStateChangeRequest,
);
return (): void => {
// return a call to `off` so that we always clean up our listener.
widget?.lazyActions.off(
ElementWidgetActions.DeviceMute,
onMuteStateChangeRequest,
);
};
}
}, [onMuteStateChangeRequest]);
return useMemo(() => ({ audio, video }), [audio, video]);
}
function allowJoinUnmuted(skipLobby: boolean, isJoined: boolean): boolean {
return (
(!skipLobby && !isJoined) || import.meta.env.VITE_PACKAGE === "embedded"
);
}

View File

@@ -20,6 +20,8 @@ import {
CheckIcon,
UnknownSolidIcon,
} from "@vector-im/compound-design-tokens/assets/web/icons";
import { useObservable } from "observable-hooks";
import { map } from "rxjs";
import { useClientLegacy } from "../ClientContext";
import { ErrorPage, FullScreenView, LoadingPage } from "../FullScreenView";
@@ -35,12 +37,13 @@ import { CallTerminatedMessage, useLoadGroupCall } from "./useLoadGroupCall";
import { LobbyView } from "./LobbyView";
import { E2eeType } from "../e2ee/e2eeType";
import { useProfile } from "../profile/useProfile";
import { useMuteStates } from "./MuteStates";
import { useOptInAnalytics } from "../settings/settings";
import { Config } from "../config/Config";
import { Link } from "../button/Link";
import { ErrorView } from "../ErrorView";
import { useMatrixRTCSessionJoinState } from "../useMatrixRTCSessionJoinState";
import { useMediaDevices } from "../MediaDevicesContext";
import { MuteStates } from "../state/MuteStates";
import { ObservableScope } from "../state/ObservableScope";
export const RoomPage: FC = () => {
const { confineToRoom, appPrompt, preload, header, displayName, skipLobby } =
@@ -61,10 +64,19 @@ export const RoomPage: FC = () => {
const { avatarUrl, displayName: userDisplayName } = useProfile(client);
const groupCallState = useLoadGroupCall(client, roomIdOrAlias, viaServers);
const isJoined = useMatrixRTCSessionJoinState(
groupCallState.kind === "loaded" ? groupCallState.rtcSession : undefined,
const [joined, setJoined] = useState(false);
const devices = useMediaDevices();
const [muteStates, setMuteStates] = useState<MuteStates | null>(null);
const joined$ = useObservable(
(inputs$) => inputs$.pipe(map(([joined]) => joined)),
[joined],
);
const muteStates = useMuteStates(isJoined);
useEffect(() => {
const scope = new ObservableScope();
setMuteStates(new MuteStates(scope, devices, joined$));
return (): void => scope.end();
}, [devices, joined$]);
useEffect(() => {
// If we've finished loading, are not already authed and we've been given a display name as
@@ -101,22 +113,25 @@ export const RoomPage: FC = () => {
}
}, [groupCallState.kind]);
const groupCallView = (): JSX.Element => {
const groupCallView = (): ReactNode => {
switch (groupCallState.kind) {
case "loaded":
return (
<GroupCallView
widget={widget}
client={client!}
rtcSession={groupCallState.rtcSession}
isJoined={isJoined}
isPasswordlessUser={passwordlessUser}
confineToRoom={confineToRoom}
preload={preload}
skipLobby={skipLobby || wasInWaitForInviteState.current}
header={header}
muteStates={muteStates}
/>
muteStates && (
<GroupCallView
widget={widget}
client={client!}
rtcSession={groupCallState.rtcSession}
joined={joined}
setJoined={setJoined}
isPasswordlessUser={passwordlessUser}
confineToRoom={confineToRoom}
preload={preload}
skipLobby={skipLobby || wasInWaitForInviteState.current}
header={header}
muteStates={muteStates}
/>
)
);
case "waitForInvite":
case "canKnock": {
@@ -135,31 +150,35 @@ export const RoomPage: FC = () => {
</>
);
return (
<LobbyView
client={client!}
matrixInfo={{
userId: client!.getUserId() ?? "",
displayName: userDisplayName ?? "",
avatarUrl: avatarUrl ?? "",
roomAlias: null,
roomId: groupCallState.roomSummary.room_id,
roomName: groupCallState.roomSummary.name ?? "",
roomAvatar: groupCallState.roomSummary.avatar_url ?? null,
e2eeSystem: {
kind: groupCallState.roomSummary["im.nheko.summary.encryption"]
? E2eeType.PER_PARTICIPANT
: E2eeType.NONE,
},
}}
onEnter={(): void => knock?.()}
enterLabel={label}
waitingForInvite={groupCallState.kind === "waitForInvite"}
confineToRoom={confineToRoom}
hideHeader={header !== "standard"}
participantCount={null}
muteStates={muteStates}
onShareClick={null}
/>
muteStates && (
<LobbyView
client={client!}
matrixInfo={{
userId: client!.getUserId() ?? "",
displayName: userDisplayName ?? "",
avatarUrl: avatarUrl ?? "",
roomAlias: null,
roomId: groupCallState.roomSummary.room_id,
roomName: groupCallState.roomSummary.name ?? "",
roomAvatar: groupCallState.roomSummary.avatar_url ?? null,
e2eeSystem: {
kind: groupCallState.roomSummary[
"im.nheko.summary.encryption"
]
? E2eeType.PER_PARTICIPANT
: E2eeType.NONE,
},
}}
onEnter={(): void => knock?.()}
enterLabel={label}
waitingForInvite={groupCallState.kind === "waitForInvite"}
confineToRoom={confineToRoom}
hideHeader={header !== "standard"}
participantCount={null}
muteStates={muteStates}
onShareClick={null}
/>
)
);
}
case "loading":

View File

@@ -5,20 +5,12 @@ SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { expect, describe, it, vi, beforeAll } from "vitest";
import { expect, describe, it, beforeAll } from "vitest";
import { render } from "@testing-library/react";
import { type MatrixInfo, VideoPreview } from "./VideoPreview";
import { type MuteStates } from "./MuteStates";
import { E2eeType } from "../e2ee/e2eeType";
function mockMuteStates({ audio = true, video = true } = {}): MuteStates {
return {
audio: { enabled: audio, setEnabled: vi.fn() },
video: { enabled: video, setEnabled: vi.fn() },
};
}
describe("VideoPreview", () => {
const matrixInfo: MatrixInfo = {
userId: "@a:example.org",
@@ -49,7 +41,7 @@ describe("VideoPreview", () => {
const { queryByRole } = render(
<VideoPreview
matrixInfo={matrixInfo}
muteStates={mockMuteStates({ video: false })}
videoEnabled={false}
videoTrack={null}
children={<></>}
/>,
@@ -61,7 +53,7 @@ describe("VideoPreview", () => {
const { queryByRole } = render(
<VideoPreview
matrixInfo={matrixInfo}
muteStates={mockMuteStates({ video: true })}
videoEnabled
videoTrack={null}
children={<></>}
/>,

View File

@@ -13,7 +13,6 @@ import { useTranslation } from "react-i18next";
import { TileAvatar } from "../tile/TileAvatar";
import styles from "./VideoPreview.module.css";
import { type MuteStates } from "./MuteStates";
import { type EncryptionSystem } from "../e2ee/sharedKeyManagement";
export type MatrixInfo = {
@@ -29,14 +28,14 @@ export type MatrixInfo = {
interface Props {
matrixInfo: MatrixInfo;
muteStates: MuteStates;
videoEnabled: boolean;
videoTrack: LocalVideoTrack | null;
children: ReactNode;
}
export const VideoPreview: FC<Props> = ({
matrixInfo,
muteStates,
videoEnabled,
videoTrack,
children,
}) => {
@@ -56,8 +55,8 @@ export const VideoPreview: FC<Props> = ({
}, [videoTrack]);
const cameraIsStarting = useMemo(
() => muteStates.video.enabled && !videoTrack,
[muteStates.video.enabled, videoTrack],
() => videoEnabled && !videoTrack,
[videoEnabled, videoTrack],
);
return (
@@ -76,7 +75,7 @@ export const VideoPreview: FC<Props> = ({
tabIndex={-1}
disablePictureInPicture
/>
{(!muteStates.video.enabled || cameraIsStarting) && (
{(!videoEnabled || cameraIsStarting) && (
<>
<div className={styles.avatarContainer}>
{cameraIsStarting && (

View File

@@ -292,7 +292,7 @@ exports[`should have a close button in widget mode 1`] = `
Call is not supported
</h1>
<p>
The server is not configured to work with Element Call. Please contact your server admin (Domain: example.com, Error Code: MISSING_MATRIX_RTC_FOCUS).
The server is not configured to work with Element Call. Please contact your server admin (Domain: example.com, Error Code: MISSING_MATRIX_RTC_TRANSPORT).
</p>
<button
class="_button_vczzf_8"
@@ -445,7 +445,7 @@ exports[`should render the error page with link back to home 1`] = `
Call is not supported
</h1>
<p>
The server is not configured to work with Element Call. Please contact your server admin (Domain: example.com, Error Code: MISSING_MATRIX_RTC_FOCUS).
The server is not configured to work with Element Call. Please contact your server admin (Domain: example.com, Error Code: MISSING_MATRIX_RTC_TRANSPORT).
</p>
<button
class="_button_vczzf_8 homeLink"
@@ -598,7 +598,7 @@ exports[`should report correct error for 'Call is not supported' 1`] = `
Call is not supported
</h1>
<p>
The server is not configured to work with Element Call. Please contact your server admin (Domain: example.com, Error Code: MISSING_MATRIX_RTC_FOCUS).
The server is not configured to work with Element Call. Please contact your server admin (Domain: example.com, Error Code: MISSING_MATRIX_RTC_TRANSPORT).
</p>
<button
class="_button_vczzf_8 homeLink"

View File

@@ -1,45 +0,0 @@
/*
Copyright 2023, 2024 New Vector Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import {
type MatrixRTCSession,
MatrixRTCSessionEvent,
} from "matrix-js-sdk/lib/matrixrtc";
import { useCallback, useRef } from "react";
import { deepCompare } from "matrix-js-sdk/lib/utils";
import { logger } from "matrix-js-sdk/lib/logger";
import { type LivekitFocus, isLivekitFocus } from "matrix-js-sdk/lib/matrixrtc";
import { useTypedEventEmitterState } from "../useEvents";
/**
* Gets the currently active (livekit) focus for a MatrixRTC session
* This logic is specific to livekit foci where the whole call must use one
* and the same focus.
*/
export function useActiveLivekitFocus(
rtcSession: MatrixRTCSession,
): LivekitFocus | undefined {
const prevActiveFocus = useRef<LivekitFocus | undefined>(undefined);
return useTypedEventEmitterState(
rtcSession,
MatrixRTCSessionEvent.MembershipsChanged,
useCallback(() => {
const f = rtcSession.getActiveFocus();
// Only handle foci with type="livekit" for now.
if (f && isLivekitFocus(f) && !deepCompare(f, prevActiveFocus.current)) {
const oldestMembership = rtcSession.getOldestMembership();
logger.info(
`Got new active focus from membership: ${oldestMembership?.sender}/${oldestMembership?.deviceId}.
Updated focus (focus switch) from ${JSON.stringify(prevActiveFocus.current)} to ${JSON.stringify(f)}`,
);
prevActiveFocus.current = f;
}
return prevActiveFocus.current;
}, [rtcSession]),
);
}

View File

@@ -13,8 +13,8 @@ import EventEmitter from "events";
import { enterRTCSession, leaveRTCSession } from "../src/rtcSessionHelpers";
import { mockConfig } from "./utils/test";
import { ElementWidgetActions, widget } from "./widget";
import { ErrorCode } from "./utils/errors.ts";
const USE_MUTI_SFU = false;
const getUrlParams = vi.hoisted(() => vi.fn(() => ({})));
vi.mock("./UrlParams", () => ({ getUrlParams }));
@@ -85,41 +85,35 @@ test("It joins the correct Session", async () => {
}),
joinRoomSession: vi.fn(),
}) as unknown as MatrixRTCSession;
await enterRTCSession(mockedSession, false);
await enterRTCSession(
mockedSession,
{
livekit_alias: "roomId",
livekit_service_url: "http://my-well-known-service-url.com",
type: "livekit",
},
{
encryptMedia: true,
useMultiSfu: USE_MUTI_SFU,
preferStickyEvents: false,
},
);
expect(mockedSession.joinRoomSession).toHaveBeenLastCalledWith(
[
{
livekit_alias: "my-oldest-member-service-alias",
livekit_service_url: "http://my-oldest-member-service-url.com",
type: "livekit",
},
{
livekit_alias: "roomId",
livekit_service_url: "http://my-well-known-service-url.com",
type: "livekit",
},
{
livekit_alias: "roomId",
livekit_service_url: "http://my-well-known-service-url2.com",
type: "livekit",
},
{
livekit_alias: "roomId",
livekit_service_url: "http://my-default-service-url.com",
type: "livekit",
},
],
{
focus_selection: "oldest_membership",
type: "livekit",
},
{
manageMediaKeys: false,
undefined,
expect.objectContaining({
manageMediaKeys: true,
useLegacyMemberEvents: false,
useNewMembershipManager: true,
useExperimentalToDeviceTransport: false,
},
}),
);
});
@@ -164,27 +158,6 @@ test("leaveRTCSession doesn't close the widget when returning to lobby", async (
await testLeaveRTCSession("user", false);
});
test("It fails with configuration error if no live kit url config is set in fallback", async () => {
mockConfig({});
vi.spyOn(AutoDiscovery, "getRawClientConfig").mockResolvedValue({});
const mockedSession = vi.mocked({
room: {
roomId: "roomId",
client: {
getDomain: vi.fn().mockReturnValue("example.org"),
},
},
memberships: [],
getFocusInUse: vi.fn(),
joinRoomSession: vi.fn(),
}) as unknown as MatrixRTCSession;
await expect(enterRTCSession(mockedSession, false)).rejects.toThrowError(
expect.objectContaining({ code: ErrorCode.MISSING_MATRIX_RTC_FOCUS }),
);
});
test("It should not fail with configuration error if homeserver config has livekit url but not fallback", async () => {
mockConfig({});
vi.spyOn(AutoDiscovery, "getRawClientConfig").mockResolvedValue({
@@ -214,5 +187,17 @@ test("It should not fail with configuration error if homeserver config has livek
joinRoomSession: vi.fn(),
}) as unknown as MatrixRTCSession;
await enterRTCSession(mockedSession, false);
await enterRTCSession(
mockedSession,
{
livekit_alias: "roomId",
livekit_service_url: "http://my-well-known-service-url.com",
type: "livekit",
},
{
encryptMedia: true,
useMultiSfu: USE_MUTI_SFU,
preferStickyEvents: false,
},
);
});

View File

@@ -6,11 +6,10 @@ Please see LICENSE in the repository root for full details.
*/
import {
isLivekitFocus,
isLivekitFocusConfig,
type LivekitFocus,
type LivekitFocusActive,
type MatrixRTCSession,
isLivekitTransportConfig,
type LivekitTransportConfig,
type LivekitTransport,
} from "matrix-js-sdk/lib/matrixrtc";
import { logger } from "matrix-js-sdk/lib/logger";
import { AutoDiscovery } from "matrix-js-sdk/lib/autodiscovery";
@@ -18,38 +17,40 @@ import { AutoDiscovery } from "matrix-js-sdk/lib/autodiscovery";
import { PosthogAnalytics } from "./analytics/PosthogAnalytics";
import { Config } from "./config/Config";
import { ElementWidgetActions, widget, type WidgetHelpers } from "./widget";
import { MatrixRTCFocusMissingError } from "./utils/errors";
import { MatrixRTCTransportMissingError } from "./utils/errors";
import { getUrlParams } from "./UrlParams";
import { getSFUConfigWithOpenID } from "./livekit/openIDSFU.ts";
const FOCI_WK_KEY = "org.matrix.msc4143.rtc_foci";
export function makeActiveFocus(): LivekitFocusActive {
return {
type: "livekit",
focus_selection: "oldest_membership",
};
export function getLivekitAlias(rtcSession: MatrixRTCSession): string {
// For now we assume everything is a room-scoped call
return rtcSession.room.roomId;
}
async function makePreferredLivekitFoci(
async function makeTransportInternal(
rtcSession: MatrixRTCSession,
livekitAlias: string,
): Promise<LivekitFocus[]> {
logger.log("Start building foci_preferred list: ", rtcSession.room.roomId);
): Promise<LivekitTransport> {
logger.log("Searching for a preferred transport");
const livekitAlias = getLivekitAlias(rtcSession);
const preferredFoci: LivekitFocus[] = [];
// Make the Focus from the running rtc session the highest priority one
// This minimizes how often we need to switch foci during a call.
const focusInUse = rtcSession.getFocusInUse();
if (focusInUse && isLivekitFocus(focusInUse)) {
logger.log("Adding livekit focus from oldest member: ", focusInUse);
preferredFoci.push(focusInUse);
// TODO-MULTI-SFU: Either remove this dev tool or make it more official
const urlFromStorage =
localStorage.getItem("robin-matrixrtc-auth") ??
localStorage.getItem("timo-focus-url");
if (urlFromStorage !== null) {
const transportFromStorage: LivekitTransport = {
type: "livekit",
livekit_service_url: urlFromStorage,
livekit_alias: livekitAlias,
};
logger.log(
"Using LiveKit transport from local storage: ",
transportFromStorage,
);
return transportFromStorage;
}
// Warm up the first focus we owned, to ensure livekit room is created before any state event sent.
let toWarmUp: LivekitFocus | undefined;
// Prioritize the .well-known/matrix/client, if available, over the configured SFU
const domain = rtcSession.room.client.getDomain();
if (domain) {
@@ -59,54 +60,67 @@ async function makePreferredLivekitFoci(
FOCI_WK_KEY
];
if (Array.isArray(wellKnownFoci)) {
const validWellKnownFoci = wellKnownFoci
.filter((f) => !!f)
.filter(isLivekitFocusConfig)
.map((wellKnownFocus) => {
logger.log("Adding livekit focus from well known: ", wellKnownFocus);
return { ...wellKnownFocus, livekit_alias: livekitAlias };
});
if (validWellKnownFoci.length > 0) {
toWarmUp = validWellKnownFoci[0];
const transport: LivekitTransportConfig | undefined = wellKnownFoci.find(
(f) => f && isLivekitTransportConfig(f),
);
if (transport !== undefined) {
logger.log("Using LiveKit transport from .well-known: ", transport);
return { ...transport, livekit_alias: livekitAlias };
}
preferredFoci.push(...validWellKnownFoci);
}
}
const urlFromConf = Config.get().livekit?.livekit_service_url;
if (urlFromConf) {
const focusFormConf: LivekitFocus = {
const transportFromConf: LivekitTransport = {
type: "livekit",
livekit_service_url: urlFromConf,
livekit_alias: livekitAlias,
};
toWarmUp = toWarmUp ?? focusFormConf;
logger.log("Adding livekit focus from config: ", focusFormConf);
preferredFoci.push(focusFormConf);
logger.log("Using LiveKit transport from config: ", transportFromConf);
return transportFromConf;
}
if (toWarmUp) {
// this will call the jwt/sfu/get endpoint to pre create the livekit room.
await getSFUConfigWithOpenID(rtcSession.room.client, toWarmUp);
}
if (preferredFoci.length === 0)
throw new MatrixRTCFocusMissingError(domain ?? "");
return Promise.resolve(preferredFoci);
// TODO: we want to do something like this:
//
// const focusOtherMembers = await focusFromOtherMembers(
// rtcSession,
// livekitAlias,
// );
// if (focusOtherMembers) preferredFoci.push(focusOtherMembers);
throw new MatrixRTCTransportMissingError(domain ?? "");
}
export async function makeTransport(
rtcSession: MatrixRTCSession,
): Promise<LivekitTransport> {
const transport = await makeTransportInternal(rtcSession);
// this will call the jwt/sfu/get endpoint to pre create the livekit room.
await getSFUConfigWithOpenID(
rtcSession.room.client,
transport.livekit_service_url,
transport.livekit_alias,
);
return transport;
}
export interface EnterRTCSessionOptions {
encryptMedia: boolean;
// TODO: remove this flag, to-device transport is stable enough now
useExperimentalToDeviceTransport?: boolean;
/** EXPERIMENTAL: If true, will use the multi-sfu codepath where each member connects to its SFU instead of everyone connecting to an elected on. */
useMultiSfu: boolean;
preferStickyEvents: boolean;
}
/**
* TODO! document this function properly
* @param rtcSession
* @param transport
* @param options
*/
export async function enterRTCSession(
rtcSession: MatrixRTCSession,
encryptMedia: boolean,
useNewMembershipManager = true,
useExperimentalToDeviceTransport = false,
transport: LivekitTransport,
{
encryptMedia,
useExperimentalToDeviceTransport = false,
useMultiSfu,
preferStickyEvents,
}: EnterRTCSessionOptions,
): Promise<void> {
PosthogAnalytics.instance.eventCallEnded.cacheStartCall(new Date());
PosthogAnalytics.instance.eventCallStarted.track(rtcSession.room.roomId);
@@ -115,17 +129,17 @@ export async function enterRTCSession(
// have started tracking by the time calls start getting created.
// groupCallOTelMembership?.onJoinCall();
// right now we assume everything is a room-scoped call
const livekitAlias = rtcSession.room.roomId;
const { features, matrix_rtc_session: matrixRtcSessionConfig } = Config.get();
const useDeviceSessionMemberEvents =
features?.feature_use_device_session_member_events;
const { sendNotificationType: notificationType, callIntent } = getUrlParams();
// Multi-sfu does not need a preferred foci list. just the focus that is actually used.
rtcSession.joinRoomSession(
await makePreferredLivekitFoci(rtcSession, livekitAlias),
makeActiveFocus(),
useMultiSfu ? [] : [transport],
useMultiSfu ? transport : undefined,
{
notificationType: getUrlParams().sendNotificationType,
useNewMembershipManager,
notificationType,
callIntent,
manageMediaKeys: encryptMedia,
...(useDeviceSessionMemberEvents !== undefined && {
useLegacyMemberEvents: !useDeviceSessionMemberEvents,
@@ -141,6 +155,7 @@ export async function enterRTCSession(
membershipEventExpiryMs:
matrixRtcSessionConfig?.membership_event_expiry_ms,
useExperimentalToDeviceTransport,
unstableSendStickyEvents: preferStickyEvents,
},
);
if (widget) {

View File

@@ -5,48 +5,69 @@ SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { type ChangeEvent, type FC, useCallback, useMemo } from "react";
import {
type ChangeEvent,
type FC,
useCallback,
useEffect,
useMemo,
useState,
} from "react";
import { useTranslation } from "react-i18next";
import {
UNSTABLE_MSC4354_STICKY_EVENTS,
type MatrixClient,
} from "matrix-js-sdk";
import { logger } from "matrix-js-sdk/lib/logger";
import { FieldRow, InputField } from "../input/Input";
import {
useSetting,
duplicateTiles as duplicateTilesSetting,
debugTileLayout as debugTileLayoutSetting,
showNonMemberTiles as showNonMemberTilesSetting,
showConnectionStats as showConnectionStatsSetting,
useNewMembershipManager as useNewMembershipManagerSetting,
useExperimentalToDeviceTransport as useExperimentalToDeviceTransportSetting,
multiSfu as multiSfuSetting,
muteAllAudio as muteAllAudioSetting,
alwaysShowIphoneEarpiece as alwaysShowIphoneEarpieceSetting,
preferStickyEvents as preferStickyEventsSetting,
} from "./settings";
import type { MatrixClient } from "matrix-js-sdk";
import type { Room as LivekitRoom } from "livekit-client";
import styles from "./DeveloperSettingsTab.module.css";
import { useUrlParams } from "../UrlParams";
interface Props {
client: MatrixClient;
livekitRoom?: LivekitRoom;
livekitRooms?: { room: LivekitRoom; url: string; isLocal?: boolean }[];
}
export const DeveloperSettingsTab: FC<Props> = ({ client, livekitRoom }) => {
export const DeveloperSettingsTab: FC<Props> = ({ client, livekitRooms }) => {
const { t } = useTranslation();
const [duplicateTiles, setDuplicateTiles] = useSetting(duplicateTilesSetting);
const [debugTileLayout, setDebugTileLayout] = useSetting(
debugTileLayoutSetting,
);
const [showNonMemberTiles, setShowNonMemberTiles] = useSetting(
showNonMemberTilesSetting,
const [stickyEventsSupported, setStickyEventsSupported] = useState(false);
useEffect(() => {
client
.doesServerSupportUnstableFeature(UNSTABLE_MSC4354_STICKY_EVENTS)
.then((result) => {
setStickyEventsSupported(result);
})
.catch((ex) => {
logger.warn("Failed to check if sticky events are supported", ex);
});
}, [client]);
const [preferStickyEvents, setPreferStickyEvents] = useSetting(
preferStickyEventsSetting,
);
const [showConnectionStats, setShowConnectionStats] = useSetting(
showConnectionStatsSetting,
);
const [useNewMembershipManager, setNewMembershipManager] = useSetting(
useNewMembershipManagerSetting,
);
const [alwaysShowIphoneEarpiece, setAlwaysShowIphoneEarpiece] = useSetting(
alwaysShowIphoneEarpieceSetting,
);
@@ -54,20 +75,22 @@ export const DeveloperSettingsTab: FC<Props> = ({ client, livekitRoom }) => {
useExperimentalToDeviceTransport,
setUseExperimentalToDeviceTransport,
] = useSetting(useExperimentalToDeviceTransportSetting);
const [multiSfu, setMultiSfu] = useSetting(multiSfuSetting);
const [muteAllAudio, setMuteAllAudio] = useSetting(muteAllAudioSetting);
const urlParams = useUrlParams();
const sfuUrl = useMemo((): URL | null => {
if (livekitRoom?.engine.client.ws?.url) {
const localSfuUrl = useMemo((): URL | null => {
const localRoom = livekitRooms?.find((r) => r.isLocal)?.room;
if (localRoom?.engine.client.ws?.url) {
// strip the URL params
const url = new URL(livekitRoom.engine.client.ws.url);
const url = new URL(localRoom.engine.client.ws.url);
url.search = "";
return url;
}
return null;
}, [livekitRoom]);
}, [livekitRooms]);
return (
<>
@@ -129,15 +152,17 @@ export const DeveloperSettingsTab: FC<Props> = ({ client, livekitRoom }) => {
</FieldRow>
<FieldRow>
<InputField
id="showNonMemberTiles"
id="preferStickyEvents"
type="checkbox"
label={t("developer_mode.show_non_member_tiles")}
checked={!!showNonMemberTiles}
label={t("developer_mode.prefer_sticky_events.label")}
disabled={!stickyEventsSupported}
description={t("developer_mode.prefer_sticky_events.description")}
checked={!!preferStickyEvents}
onChange={useCallback(
(event: ChangeEvent<HTMLInputElement>): void => {
setShowNonMemberTiles(event.target.checked);
setPreferStickyEvents(event.target.checked);
},
[setShowNonMemberTiles],
[setPreferStickyEvents],
)}
/>
</FieldRow>
@@ -155,20 +180,6 @@ export const DeveloperSettingsTab: FC<Props> = ({ client, livekitRoom }) => {
)}
/>
</FieldRow>
<FieldRow>
<InputField
id="useNewMembershipManager"
type="checkbox"
label={t("developer_mode.use_new_membership_manager")}
checked={!!useNewMembershipManager}
onChange={useCallback(
(event: ChangeEvent<HTMLInputElement>): void => {
setNewMembershipManager(event.target.checked);
},
[setNewMembershipManager],
)}
/>
</FieldRow>
<FieldRow>
<InputField
id="useToDeviceKeyTransport"
@@ -183,6 +194,22 @@ export const DeveloperSettingsTab: FC<Props> = ({ client, livekitRoom }) => {
)}
/>
</FieldRow>
<FieldRow>
<InputField
id="multiSfu"
type="checkbox"
label={t("developer_mode.multi_sfu")}
// If using sticky events we implicitly prefer use multi-sfu
checked={multiSfu || preferStickyEvents}
disabled={preferStickyEvents}
onChange={useCallback(
(event: ChangeEvent<HTMLInputElement>): void => {
setMultiSfu(event.target.checked);
},
[setMultiSfu],
)}
/>
</FieldRow>
<FieldRow>
<InputField
id="muteAllAudio"
@@ -211,22 +238,26 @@ export const DeveloperSettingsTab: FC<Props> = ({ client, livekitRoom }) => {
)}
/>{" "}
</FieldRow>
{livekitRoom ? (
{livekitRooms?.map((livekitRoom) => (
<>
<p>
<h3>
{t("developer_mode.livekit_sfu", {
url: sfuUrl?.href || "unknown",
url: livekitRoom.url || "unknown",
})}
</h3>
{livekitRoom.isLocal && <p>ws-url: {localSfuUrl?.href}</p>}
<p>
{t("developer_mode.livekit_server_info")}(
{livekitRoom.isLocal ? "local" : "remote"})
</p>
<p>{t("developer_mode.livekit_server_info")}</p>
<pre className={styles.pre}>
{livekitRoom.serverInfo
? JSON.stringify(livekitRoom.serverInfo, null, 2)
{livekitRoom.room.serverInfo
? JSON.stringify(livekitRoom.room.serverInfo, null, 2)
: "undefined"}
{livekitRoom.metadata}
{livekitRoom.room.metadata}
</pre>
</>
) : null}
))}
<p>{t("developer_mode.environment_variables")}</p>
<pre>{JSON.stringify(import.meta.env, null, 2)}</pre>
<p>{t("developer_mode.url_params")}</p>

View File

@@ -51,7 +51,11 @@ interface Props {
onTabChange: (tab: SettingsTab) => void;
client: MatrixClient;
roomId?: string;
livekitRoom?: LivekitRoom;
livekitRooms?: {
room: LivekitRoom;
url: string;
isLocal?: boolean;
}[];
}
export const defaultSettingsTab: SettingsTab = "audio";
@@ -63,7 +67,7 @@ export const SettingsModal: FC<Props> = ({
onTabChange,
client,
roomId,
livekitRoom,
livekitRooms,
}) => {
const { t } = useTranslation();
@@ -204,7 +208,9 @@ export const SettingsModal: FC<Props> = ({
const developerTab: Tab<SettingsTab> = {
key: "developer",
name: t("settings.developer_tab_title"),
content: <DeveloperSettingsTab client={client} livekitRoom={livekitRoom} />,
content: (
<DeveloperSettingsTab client={client} livekitRooms={livekitRooms} />
),
};
const tabs = [audioTab, videoTab];

View File

@@ -76,10 +76,6 @@ export const developerMode = new Setting("developer-settings-tab", false);
export const duplicateTiles = new Setting("duplicate-tiles", 0);
export const showNonMemberTiles = new Setting<boolean>(
"show-non-member-tiles",
false,
);
export const debugTileLayout = new Setting("debug-tile-layout", false);
export const showConnectionStats = new Setting<boolean>(
@@ -87,6 +83,11 @@ export const showConnectionStats = new Setting<boolean>(
false,
);
export const preferStickyEvents = new Setting<boolean>(
"prefer-sticky-events",
false,
);
export const audioInput = new Setting<string | undefined>(
"audio-input",
undefined,
@@ -119,16 +120,13 @@ export const soundEffectVolume = new Setting<number>(
0.5,
);
export const useNewMembershipManager = new Setting<boolean>(
"new-membership-manager",
true,
);
export const useExperimentalToDeviceTransport = new Setting<boolean>(
"experimental-to-device-transport",
true,
);
export const multiSfu = new Setting<boolean>("multi-sfu", false);
export const muteAllAudio = new Setting<boolean>("mute-all-audio", false);
export const alwaysShowSelf = new Setting<boolean>("always-show-self", true);

53
src/state/Async.ts Normal file
View File

@@ -0,0 +1,53 @@
/*
Copyright 2025 New Vector Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { catchError, from, map, type Observable, of, startWith } from "rxjs";
/**
* Data that may need to be loaded asynchronously.
*
* This type is for when you need to represent the current state of an operation
* involving Promises as **immutable data**. See the async$ function below.
*/
export type Async<A> =
| { state: "loading" }
| { state: "error"; value: Error }
| { state: "ready"; value: A };
export const loading: Async<never> = { state: "loading" };
export function error(value: Error): Async<never> {
return { state: "error", value };
}
export function ready<A>(value: A): Async<A> {
return { state: "ready", value };
}
/**
* Turn a Promise into an Observable async value. The Observable will have the
* value "loading" while the Promise is pending, "ready" when the Promise
* resolves, and "error" when the Promise rejects.
*/
export function async$<A>(promise: Promise<A>): Observable<Async<A>> {
return from(promise).pipe(
map(ready),
startWith(loading),
catchError((e: unknown) =>
of(error((e as Error) ?? new Error("Unknown error"))),
),
);
}
/**
* If the async value is ready, apply the given function to the inner value.
*/
export function mapAsync<A, B>(
async: Async<A>,
project: (value: A) => B,
): Async<B> {
return async.state === "ready" ? ready(project(async.value)) : async;
}

View File

@@ -5,7 +5,7 @@ SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { test, vi, onTestFinished, it, describe } from "vitest";
import { test, vi, onTestFinished, it, describe, expect } from "vitest";
import EventEmitter from "events";
import {
BehaviorSubject,
@@ -40,20 +40,16 @@ import * as ComponentsCore from "@livekit/components-core";
import {
Status,
type CallMembership,
type MatrixRTCSession,
type IRTCNotificationContent,
type ICallNotifyContent,
MatrixRTCSessionEvent,
} from "matrix-js-sdk/lib/matrixrtc";
import { deepCompare } from "matrix-js-sdk/lib/utils";
import { AutoDiscovery } from "matrix-js-sdk/lib/autodiscovery";
import { CallViewModel, type CallViewModelOptions } from "./CallViewModel";
import { type Layout } from "./layout-types";
import {
CallViewModel,
type CallViewModelOptions,
type Layout,
} from "./CallViewModel";
import {
mockLivekitRoom,
mockLocalParticipant,
mockMatrixRoom,
mockMatrixRoomMember,
@@ -62,14 +58,15 @@ import {
mockRtcMembership,
MockRTCSession,
mockMediaDevices,
mockMuteStates,
mockConfig,
} from "../utils/test";
import {
ECAddonConnectionState,
type ECConnectionState,
} from "../livekit/useECConnectionState";
import { E2eeType } from "../e2ee/e2eeType";
import type { RaisedHandInfo } from "../reactions";
import { showNonMemberTiles } from "../settings/settings";
import type { RaisedHandInfo, ReactionInfo } from "../reactions";
import {
alice,
aliceDoppelganger,
@@ -96,6 +93,11 @@ import { ObservableScope } from "./ObservableScope";
import { MediaDevices } from "./MediaDevices";
import { getValue } from "../utils/observable";
import { type Behavior, constant } from "./Behavior";
import type { ProcessorState } from "../livekit/TrackProcessorContext.tsx";
import {
type ElementCallError,
MatrixRTCTransportMissingError,
} from "../utils/errors.ts";
const getUrlParams = vi.hoisted(() => vi.fn(() => ({})));
vi.mock("../UrlParams", () => ({ getUrlParams }));
@@ -266,7 +268,7 @@ const mockLegacyRingEvent = {} as { event_id: string } & ICallNotifyContent;
interface CallViewModelInputs {
remoteParticipants$: Behavior<RemoteParticipant[]>;
rtcMembers$: Behavior<Partial<CallMembership>[]>;
connectionState$: Observable<ECConnectionState>;
livekitConnectionState$: Behavior<ECConnectionState>;
speaking: Map<Participant, Observable<boolean>>;
mediaDevices: MediaDevices;
initialSyncState: SyncState;
@@ -276,7 +278,9 @@ function withCallViewModel(
{
remoteParticipants$ = constant([]),
rtcMembers$ = constant([localRtcMember]),
connectionState$ = of(ConnectionState.Connected),
livekitConnectionState$: connectionState$ = constant(
ConnectionState.Connected,
),
speaking = new Map(),
mediaDevices = mockMediaDevices({}),
initialSyncState = SyncState.Syncing,
@@ -301,7 +305,7 @@ function withCallViewModel(
const room = mockMatrixRoom({
client: new (class extends EventEmitter {
public getUserId(): string | undefined {
return localRtcMember.sender;
return localRtcMember.userId;
}
public getDeviceId(): string {
return localRtcMember.deviceId;
@@ -338,23 +342,22 @@ function withCallViewModel(
const roomEventSelectorSpy = vi
.spyOn(ComponentsCore, "roomEventSelector")
.mockImplementation((_room, _eventType) => of());
const livekitRoom = mockLivekitRoom(
{ localParticipant },
{ remoteParticipants$ },
);
const muteStates = mockMuteStates();
const raisedHands$ = new BehaviorSubject<Record<string, RaisedHandInfo>>({});
const reactions$ = new BehaviorSubject<Record<string, ReactionInfo>>({});
const vm = new CallViewModel(
rtcSession as unknown as MatrixRTCSession,
rtcSession.asMockedSession(),
room,
livekitRoom,
mediaDevices,
muteStates,
options,
connectionState$,
raisedHands$,
new BehaviorSubject({}),
reactions$,
new BehaviorSubject<ProcessorState>({
processor: undefined,
supported: undefined,
}),
);
onTestFinished(() => {
@@ -368,6 +371,61 @@ function withCallViewModel(
continuation(vm, rtcSession, { raisedHands$: raisedHands$ }, setSyncState);
}
test("test missing RTC config error", async () => {
const rtcMemberships$ = new BehaviorSubject<CallMembership[]>([]);
const emitter = new EventEmitter();
const client = vi.mocked<MatrixClient>({
on: emitter.on.bind(emitter),
off: emitter.off.bind(emitter),
getSyncState: vi.fn().mockReturnValue(SyncState.Syncing),
getUserId: vi.fn().mockReturnValue("@user:localhost"),
getUser: vi.fn().mockReturnValue(null),
getDeviceId: vi.fn().mockReturnValue("DEVICE"),
credentials: {
userId: "@user:localhost",
},
getCrypto: vi.fn().mockReturnValue(undefined),
getDomain: vi.fn().mockReturnValue("example.org"),
} as unknown as MatrixClient);
const matrixRoom = mockMatrixRoom({
roomId: "!myRoomId:example.com",
client,
getMember: vi.fn().mockReturnValue(undefined),
});
const fakeRtcSession = new MockRTCSession(matrixRoom).withMemberships(
rtcMemberships$,
);
mockConfig({});
vi.spyOn(AutoDiscovery, "getRawClientConfig").mockResolvedValue({});
const callVM = new CallViewModel(
fakeRtcSession.asMockedSession(),
matrixRoom,
mockMediaDevices({}),
mockMuteStates(),
{
encryptionSystem: { kind: E2eeType.PER_PARTICIPANT },
autoLeaveWhenOthersLeft: false,
},
new BehaviorSubject({} as Record<string, RaisedHandInfo>),
new BehaviorSubject({} as Record<string, ReactionInfo>),
of({ processor: undefined, supported: false }),
);
const failPromise = Promise.withResolvers<ElementCallError>();
callVM.configError$.subscribe((error) => {
if (error) {
failPromise.resolve(error);
}
});
const error = await failPromise.promise;
expect(error).toBeInstanceOf(MatrixRTCTransportMissingError);
});
test("participants are retained during a focus switch", () => {
withTestScheduler(({ behavior, expectObservable }) => {
// Participants disappear on frame 2 and come back on frame 3
@@ -384,7 +442,7 @@ test("participants are retained during a focus switch", () => {
b: [],
}),
rtcMembers$: constant([localRtcMember, aliceRtcMember, bobRtcMember]),
connectionState$: behavior(connectionInputMarbles, {
livekitConnectionState$: behavior(connectionInputMarbles, {
c: ConnectionState.Connected,
s: ECAddonConnectionState.ECSwitchingFocus,
}),
@@ -828,53 +886,6 @@ test("participants must have a MatrixRTCSession to be visible", () => {
});
});
test("shows participants without MatrixRTCSession when enabled in settings", () => {
try {
// enable the setting:
showNonMemberTiles.setValue(true);
withTestScheduler(({ behavior, expectObservable }) => {
const scenarioInputMarbles = " abc";
const expectedLayoutMarbles = "abc";
withCallViewModel(
{
remoteParticipants$: behavior(scenarioInputMarbles, {
a: [],
b: [aliceParticipant],
c: [aliceParticipant, bobParticipant],
}),
rtcMembers$: constant([localRtcMember]), // No one else joins the MatrixRTC session
},
(vm) => {
vm.setGridMode("grid");
expectObservable(summarizeLayout$(vm.layout$)).toBe(
expectedLayoutMarbles,
{
a: {
type: "grid",
spotlight: undefined,
grid: ["local:0"],
},
b: {
type: "one-on-one",
local: "local:0",
remote: `${aliceId}:0`,
},
c: {
type: "grid",
spotlight: undefined,
grid: ["local:0", `${aliceId}:0`, `${bobId}:0`],
},
},
);
},
);
});
} finally {
showNonMemberTiles.setValue(showNonMemberTiles.defaultValue);
}
});
it("should show at least one tile per MatrixRTCSession", () => {
withTestScheduler(({ behavior, expectObservable }) => {
// iterate through some combinations of MatrixRTC memberships
@@ -1059,7 +1070,7 @@ it("should rank raised hands above video feeds and below speakers and presenters
},
b: () => {
raisedHands$.next({
[`${bobRtcMember.sender}:${bobRtcMember.deviceId}`]: {
[`${bobRtcMember.userId}:${bobRtcMember.deviceId}`]: {
time: new Date(),
reactionEventId: "",
membershipEventId: "",
@@ -1251,6 +1262,41 @@ describe("waitForCallPickup$", () => {
});
});
test("regression test: does stop ringing in case livekitConnectionState$ emits after didSendCallNotification$ has already emitted", () => {
withTestScheduler(({ schedule, expectObservable, behavior }) => {
withCallViewModel(
{
livekitConnectionState$: behavior("d 9ms c", {
d: ConnectionState.Disconnected,
c: ConnectionState.Connected,
}),
},
(vm, rtcSession) => {
// Fire a call notification IMMEDIATELY (its important for this test, that this happens before the livekitConnectionState$ emits)
schedule("n", {
n: () => {
rtcSession.emit(
MatrixRTCSessionEvent.DidSendCallNotification,
mockRingEvent("$notif1", 30),
mockLegacyRingEvent,
);
},
});
expectObservable(vm.callPickupState$).toBe("a 9ms b 19ms c", {
a: "unknown",
b: "ringing",
c: "timeout",
});
},
{
waitForCallPickup: true,
encryptionSystem: { kind: E2eeType.PER_PARTICIPANT },
},
);
});
});
test("ringing -> success if someone joins before timeout", () => {
withTestScheduler(({ behavior, schedule, expectObservable }) => {
// Someone joins at 20ms (both LiveKit participant and MatrixRTC member)
@@ -1305,7 +1351,7 @@ describe("waitForCallPickup$", () => {
a: [localRtcMember],
b: [localRtcMember, aliceRtcMember],
}),
connectionState$,
livekitConnectionState$: connectionState$,
},
(vm, rtcSession) => {
// Notify at 5ms so we enter ringing, then get disconnected 5ms later

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,703 @@
/*
Copyright 2025 New Vector Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import {
afterEach,
describe,
expect,
it,
type Mock,
type MockedObject,
onTestFinished,
vi,
} from "vitest";
import { BehaviorSubject, of } from "rxjs";
import {
ConnectionState,
type LocalParticipant,
type RemoteParticipant,
type Room as LivekitRoom,
RoomEvent,
type RoomOptions,
} from "livekit-client";
import fetchMock from "fetch-mock";
import EventEmitter from "events";
import { type IOpenIDToken } from "matrix-js-sdk";
import type {
CallMembership,
LivekitTransport,
} from "matrix-js-sdk/lib/matrixrtc";
import {
type ConnectionOpts,
type TransportState,
type PublishingParticipant,
RemoteConnection,
} from "./Connection.ts";
import { ObservableScope } from "./ObservableScope.ts";
import { type OpenIDClientParts } from "../livekit/openIDSFU.ts";
import { FailToGetOpenIdToken } from "../utils/errors.ts";
import { PublishConnection } from "./PublishConnection.ts";
import { mockMediaDevices, mockMuteStates } from "../utils/test.ts";
import type { ProcessorState } from "../livekit/TrackProcessorContext.tsx";
import { type MuteStates } from "./MuteStates.ts";
let testScope: ObservableScope;
let client: MockedObject<OpenIDClientParts>;
let fakeLivekitRoom: MockedObject<LivekitRoom>;
let localParticipantEventEmiter: EventEmitter;
let fakeLocalParticipant: MockedObject<LocalParticipant>;
let fakeRoomEventEmiter: EventEmitter;
let fakeMembershipsFocusMap$: BehaviorSubject<
{ membership: CallMembership; transport: LivekitTransport }[]
>;
const livekitFocus: LivekitTransport = {
livekit_alias: "!roomID:example.org",
livekit_service_url: "https://matrix-rtc.example.org/livekit/jwt",
type: "livekit",
};
function setupTest(): void {
testScope = new ObservableScope();
client = vi.mocked<OpenIDClientParts>({
getOpenIdToken: vi.fn().mockResolvedValue({
access_token: "rYsmGUEwNjKgJYyeNUkZseJN",
token_type: "Bearer",
matrix_server_name: "example.org",
expires_in: 3600,
}),
getDeviceId: vi.fn().mockReturnValue("ABCDEF"),
} as unknown as OpenIDClientParts);
fakeMembershipsFocusMap$ = new BehaviorSubject<
{ membership: CallMembership; transport: LivekitTransport }[]
>([]);
localParticipantEventEmiter = new EventEmitter();
fakeLocalParticipant = vi.mocked<LocalParticipant>({
identity: "@me:example.org",
isMicrophoneEnabled: vi.fn().mockReturnValue(true),
getTrackPublication: vi.fn().mockReturnValue(undefined),
on: localParticipantEventEmiter.on.bind(localParticipantEventEmiter),
off: localParticipantEventEmiter.off.bind(localParticipantEventEmiter),
addListener: localParticipantEventEmiter.addListener.bind(
localParticipantEventEmiter,
),
removeListener: localParticipantEventEmiter.removeListener.bind(
localParticipantEventEmiter,
),
removeAllListeners: localParticipantEventEmiter.removeAllListeners.bind(
localParticipantEventEmiter,
),
} as unknown as LocalParticipant);
fakeRoomEventEmiter = new EventEmitter();
fakeLivekitRoom = vi.mocked<LivekitRoom>({
connect: vi.fn(),
disconnect: vi.fn(),
remoteParticipants: new Map(),
localParticipant: fakeLocalParticipant,
state: ConnectionState.Disconnected,
on: fakeRoomEventEmiter.on.bind(fakeRoomEventEmiter),
off: fakeRoomEventEmiter.off.bind(fakeRoomEventEmiter),
addListener: fakeRoomEventEmiter.addListener.bind(fakeRoomEventEmiter),
removeListener:
fakeRoomEventEmiter.removeListener.bind(fakeRoomEventEmiter),
removeAllListeners:
fakeRoomEventEmiter.removeAllListeners.bind(fakeRoomEventEmiter),
setE2EEEnabled: vi.fn().mockResolvedValue(undefined),
} as unknown as LivekitRoom);
}
function setupRemoteConnection(): RemoteConnection {
const opts: ConnectionOpts = {
client: client,
transport: livekitFocus,
remoteTransports$: fakeMembershipsFocusMap$,
scope: testScope,
livekitRoomFactory: () => fakeLivekitRoom,
};
fetchMock.post(`${livekitFocus.livekit_service_url}/sfu/get`, () => {
return {
status: 200,
body: {
url: "wss://matrix-rtc.m.localhost/livekit/sfu",
jwt: "ATOKEN",
},
};
});
fakeLivekitRoom.connect.mockResolvedValue(undefined);
return new RemoteConnection(opts, undefined);
}
afterEach(() => {
vi.useRealTimers();
vi.clearAllMocks();
fetchMock.reset();
});
describe("Start connection states", () => {
it("start in initialized state", () => {
setupTest();
const opts: ConnectionOpts = {
client: client,
transport: livekitFocus,
remoteTransports$: fakeMembershipsFocusMap$,
scope: testScope,
livekitRoomFactory: () => fakeLivekitRoom,
};
const connection = new RemoteConnection(opts, undefined);
expect(connection.transportState$.getValue().state).toEqual("Initialized");
});
it("fail to getOpenId token then error state", async () => {
setupTest();
vi.useFakeTimers();
const opts: ConnectionOpts = {
client: client,
transport: livekitFocus,
remoteTransports$: fakeMembershipsFocusMap$,
scope: testScope,
livekitRoomFactory: () => fakeLivekitRoom,
};
const connection = new RemoteConnection(opts, undefined);
const capturedStates: TransportState[] = [];
const s = connection.transportState$.subscribe((value) => {
capturedStates.push(value);
});
onTestFinished(() => s.unsubscribe());
const deferred = Promise.withResolvers<IOpenIDToken>();
client.getOpenIdToken.mockImplementation(
async (): Promise<IOpenIDToken> => {
return await deferred.promise;
},
);
connection.start().catch(() => {
// expected to throw
});
let capturedState = capturedStates.pop();
expect(capturedState).toBeDefined();
expect(capturedState!.state).toEqual("FetchingConfig");
deferred.reject(new FailToGetOpenIdToken(new Error("Failed to get token")));
await vi.runAllTimersAsync();
capturedState = capturedStates.pop();
if (capturedState!.state === "FailedToStart") {
expect(capturedState!.error.message).toEqual("Something went wrong");
expect(capturedState!.transport.livekit_alias).toEqual(
livekitFocus.livekit_alias,
);
} else {
expect.fail(
"Expected FailedToStart state but got " + capturedState?.state,
);
}
});
it("fail to get JWT token and error state", async () => {
setupTest();
vi.useFakeTimers();
const opts: ConnectionOpts = {
client: client,
transport: livekitFocus,
remoteTransports$: fakeMembershipsFocusMap$,
scope: testScope,
livekitRoomFactory: () => fakeLivekitRoom,
};
const connection = new RemoteConnection(opts, undefined);
const capturedStates: TransportState[] = [];
const s = connection.transportState$.subscribe((value) => {
capturedStates.push(value);
});
onTestFinished(() => s.unsubscribe());
const deferredSFU = Promise.withResolvers<void>();
// mock the /sfu/get call
fetchMock.post(`${livekitFocus.livekit_service_url}/sfu/get`, async () => {
await deferredSFU.promise;
return {
status: 500,
body: "Internal Server Error",
};
});
connection.start().catch(() => {
// expected to throw
});
let capturedState = capturedStates.pop();
expect(capturedState).toBeDefined();
expect(capturedState?.state).toEqual("FetchingConfig");
deferredSFU.resolve();
await vi.runAllTimersAsync();
capturedState = capturedStates.pop();
if (capturedState?.state === "FailedToStart") {
expect(capturedState?.error.message).toContain(
"SFU Config fetch failed with exception Error",
);
expect(capturedState?.transport.livekit_alias).toEqual(
livekitFocus.livekit_alias,
);
} else {
expect.fail(
"Expected FailedToStart state but got " + capturedState?.state,
);
}
});
it("fail to connect to livekit error state", async () => {
setupTest();
vi.useFakeTimers();
const opts: ConnectionOpts = {
client: client,
transport: livekitFocus,
remoteTransports$: fakeMembershipsFocusMap$,
scope: testScope,
livekitRoomFactory: () => fakeLivekitRoom,
};
const connection = new RemoteConnection(opts, undefined);
const capturedStates: TransportState[] = [];
const s = connection.transportState$.subscribe((value) => {
capturedStates.push(value);
});
onTestFinished(() => s.unsubscribe());
const deferredSFU = Promise.withResolvers<void>();
// mock the /sfu/get call
fetchMock.post(`${livekitFocus.livekit_service_url}/sfu/get`, () => {
return {
status: 200,
body: {
url: "wss://matrix-rtc.m.localhost/livekit/sfu",
jwt: "ATOKEN",
},
};
});
fakeLivekitRoom.connect.mockImplementation(async () => {
await deferredSFU.promise;
throw new Error("Failed to connect to livekit");
});
connection.start().catch(() => {
// expected to throw
});
let capturedState = capturedStates.pop();
expect(capturedState).toBeDefined();
expect(capturedState?.state).toEqual("FetchingConfig");
deferredSFU.resolve();
await vi.runAllTimersAsync();
capturedState = capturedStates.pop();
if (capturedState && capturedState?.state === "FailedToStart") {
expect(capturedState.error.message).toContain(
"Failed to connect to livekit",
);
expect(capturedState.transport.livekit_alias).toEqual(
livekitFocus.livekit_alias,
);
} else {
expect.fail(
"Expected FailedToStart state but got " + JSON.stringify(capturedState),
);
}
});
it("connection states happy path", async () => {
vi.useFakeTimers();
setupTest();
const connection = setupRemoteConnection();
const capturedStates: TransportState[] = [];
const s = connection.transportState$.subscribe((value) => {
capturedStates.push(value);
});
onTestFinished(() => s.unsubscribe());
await connection.start();
await vi.runAllTimersAsync();
const initialState = capturedStates.shift();
expect(initialState?.state).toEqual("Initialized");
const fetchingState = capturedStates.shift();
expect(fetchingState?.state).toEqual("FetchingConfig");
const connectingState = capturedStates.shift();
expect(connectingState?.state).toEqual("ConnectingToLkRoom");
const connectedState = capturedStates.shift();
expect(connectedState?.state).toEqual("ConnectedToLkRoom");
});
it("shutting down the scope should stop the connection", async () => {
setupTest();
vi.useFakeTimers();
const connection = setupRemoteConnection();
await connection.start();
const stopSpy = vi.spyOn(connection, "stop");
testScope.end();
expect(stopSpy).toHaveBeenCalled();
expect(fakeLivekitRoom.disconnect).toHaveBeenCalled();
});
});
function fakeRemoteLivekitParticipant(id: string): RemoteParticipant {
return {
identity: id,
} as unknown as RemoteParticipant;
}
function fakeRtcMemberShip(userId: string, deviceId: string): CallMembership {
return {
userId,
deviceId,
} as unknown as CallMembership;
}
describe("Publishing participants observations", () => {
it("should emit the list of publishing participants", async () => {
setupTest();
const connection = setupRemoteConnection();
const bobIsAPublisher = Promise.withResolvers<void>();
const danIsAPublisher = Promise.withResolvers<void>();
const observedPublishers: PublishingParticipant[][] = [];
const s = connection.publishingParticipants$.subscribe((publishers) => {
observedPublishers.push(publishers);
if (
publishers.some(
(p) => p.participant?.identity === "@bob:example.org:DEV111",
)
) {
bobIsAPublisher.resolve();
}
if (
publishers.some(
(p) => p.participant?.identity === "@dan:example.org:DEV333",
)
) {
danIsAPublisher.resolve();
}
});
onTestFinished(() => s.unsubscribe());
// The publishingParticipants$ observable is derived from the current members of the
// livekitRoom and the rtc membership in order to publish the members that are publishing
// on this connection.
let participants: RemoteParticipant[] = [
fakeRemoteLivekitParticipant("@alice:example.org:DEV000"),
fakeRemoteLivekitParticipant("@bob:example.org:DEV111"),
fakeRemoteLivekitParticipant("@carol:example.org:DEV222"),
fakeRemoteLivekitParticipant("@dan:example.org:DEV333"),
];
// Let's simulate 3 members on the livekitRoom
vi.spyOn(fakeLivekitRoom, "remoteParticipants", "get").mockReturnValue(
new Map(participants.map((p) => [p.identity, p])),
);
for (const participant of participants) {
fakeRoomEventEmiter.emit(RoomEvent.ParticipantConnected, participant);
}
// At this point there should be no publishers
expect(observedPublishers.pop()!.length).toEqual(0);
const otherFocus: LivekitTransport = {
livekit_alias: "!roomID:example.org",
livekit_service_url: "https://other-matrix-rtc.example.org/livekit/jwt",
type: "livekit",
};
const rtcMemberships = [
// Say bob is on the same focus
{
membership: fakeRtcMemberShip("@bob:example.org", "DEV111"),
transport: livekitFocus,
},
// Alice and carol is on a different focus
{
membership: fakeRtcMemberShip("@alice:example.org", "DEV000"),
transport: otherFocus,
},
{
membership: fakeRtcMemberShip("@carol:example.org", "DEV222"),
transport: otherFocus,
},
// NO DAVE YET
];
// signal this change in rtc memberships
fakeMembershipsFocusMap$.next(rtcMemberships);
// We should have bob has a publisher now
await bobIsAPublisher.promise;
const publishers = observedPublishers.pop();
expect(publishers?.length).toEqual(1);
expect(publishers?.[0].participant?.identity).toEqual(
"@bob:example.org:DEV111",
);
// Now let's make dan join the rtc memberships
rtcMemberships.push({
membership: fakeRtcMemberShip("@dan:example.org", "DEV333"),
transport: livekitFocus,
});
fakeMembershipsFocusMap$.next(rtcMemberships);
// We should have bob and dan has publishers now
await danIsAPublisher.promise;
const twoPublishers = observedPublishers.pop();
expect(twoPublishers?.length).toEqual(2);
expect(
twoPublishers?.some(
(p) => p.participant?.identity === "@bob:example.org:DEV111",
),
).toBeTruthy();
expect(
twoPublishers?.some(
(p) => p.participant?.identity === "@dan:example.org:DEV333",
),
).toBeTruthy();
// Now let's make bob leave the livekit room
participants = participants.filter(
(p) => p.identity !== "@bob:example.org:DEV111",
);
vi.spyOn(fakeLivekitRoom, "remoteParticipants", "get").mockReturnValue(
new Map(participants.map((p) => [p.identity, p])),
);
fakeRoomEventEmiter.emit(
RoomEvent.ParticipantDisconnected,
fakeRemoteLivekitParticipant("@bob:example.org:DEV111"),
);
const updatedPublishers = observedPublishers.pop();
// Bob is not connected to the room but he is still in the rtc memberships declaring that
// he is using that focus to publish, so he should still appear as a publisher
expect(updatedPublishers?.length).toEqual(2);
const pp = updatedPublishers?.find(
(p) => p.membership.userId == "@bob:example.org",
);
expect(pp).toBeDefined();
expect(pp!.participant).not.toBeDefined();
expect(
updatedPublishers?.some(
(p) => p.participant?.identity === "@dan:example.org:DEV333",
),
).toBeTruthy();
// Now if bob is not in the rtc memberships, he should disappear
const noBob = rtcMemberships.filter(
({ membership }) => membership.userId !== "@bob:example.org",
);
fakeMembershipsFocusMap$.next(noBob);
expect(observedPublishers.pop()?.length).toEqual(1);
});
it("should be scoped to parent scope", (): void => {
setupTest();
const connection = setupRemoteConnection();
let observedPublishers: PublishingParticipant[][] = [];
const s = connection.publishingParticipants$.subscribe((publishers) => {
observedPublishers.push(publishers);
});
onTestFinished(() => s.unsubscribe());
let participants: RemoteParticipant[] = [
fakeRemoteLivekitParticipant("@bob:example.org:DEV111"),
];
// Let's simulate 3 members on the livekitRoom
vi.spyOn(fakeLivekitRoom, "remoteParticipants", "get").mockReturnValue(
new Map(participants.map((p) => [p.identity, p])),
);
for (const participant of participants) {
fakeRoomEventEmiter.emit(RoomEvent.ParticipantConnected, participant);
}
// At this point there should be no publishers
expect(observedPublishers.pop()!.length).toEqual(0);
const rtcMemberships = [
// Say bob is on the same focus
{
membership: fakeRtcMemberShip("@bob:example.org", "DEV111"),
transport: livekitFocus,
},
];
// signal this change in rtc memberships
fakeMembershipsFocusMap$.next(rtcMemberships);
// We should have bob has a publisher now
const publishers = observedPublishers.pop();
expect(publishers?.length).toEqual(1);
expect(publishers?.[0].participant?.identity).toEqual(
"@bob:example.org:DEV111",
);
// end the parent scope
testScope.end();
observedPublishers = [];
// SHOULD NOT emit any more publishers as the scope is ended
participants = participants.filter(
(p) => p.identity !== "@bob:example.org:DEV111",
);
vi.spyOn(fakeLivekitRoom, "remoteParticipants", "get").mockReturnValue(
new Map(participants.map((p) => [p.identity, p])),
);
fakeRoomEventEmiter.emit(
RoomEvent.ParticipantDisconnected,
fakeRemoteLivekitParticipant("@bob:example.org:DEV111"),
);
expect(observedPublishers.length).toEqual(0);
});
});
describe("PublishConnection", () => {
// let fakeBlurProcessor: ProcessorWrapper<BackgroundOptions>;
let roomFactoryMock: Mock<() => LivekitRoom>;
let muteStates: MockedObject<MuteStates>;
function setUpPublishConnection(): void {
setupTest();
roomFactoryMock = vi.fn().mockReturnValue(fakeLivekitRoom);
muteStates = mockMuteStates();
// fakeBlurProcessor = vi.mocked<ProcessorWrapper<BackgroundOptions>>({
// name: "BackgroundBlur",
// restart: vi.fn().mockResolvedValue(undefined),
// setOptions: vi.fn().mockResolvedValue(undefined),
// getOptions: vi.fn().mockReturnValue({ strength: 0.5 }),
// isRunning: vi.fn().mockReturnValue(false)
// });
}
describe("Livekit room creation", () => {
function createSetup(): void {
setUpPublishConnection();
const fakeTrackProcessorSubject$ = new BehaviorSubject<ProcessorState>({
supported: true,
processor: undefined,
});
const opts: ConnectionOpts = {
client: client,
transport: livekitFocus,
remoteTransports$: fakeMembershipsFocusMap$,
scope: testScope,
livekitRoomFactory: roomFactoryMock,
};
const audioInput = {
available$: of(new Map([["mic1", { id: "mic1" }]])),
selected$: new BehaviorSubject({ id: "mic1" }),
select(): void {},
};
const videoInput = {
available$: of(new Map([["cam1", { id: "cam1" }]])),
selected$: new BehaviorSubject({ id: "cam1" }),
select(): void {},
};
const audioOutput = {
available$: of(new Map([["speaker", { id: "speaker" }]])),
selected$: new BehaviorSubject({ id: "speaker" }),
select(): void {},
};
// TODO understand what is wrong with our mocking that requires ts-expect-error
const fakeDevices = mockMediaDevices({
// @ts-expect-error Mocking only
audioInput,
// @ts-expect-error Mocking only
videoInput,
// @ts-expect-error Mocking only
audioOutput,
});
new PublishConnection(
opts,
fakeDevices,
muteStates,
undefined,
fakeTrackProcessorSubject$,
);
}
it("should create room with proper initial audio and video settings", () => {
createSetup();
expect(roomFactoryMock).toHaveBeenCalled();
const lastCallArgs =
roomFactoryMock.mock.calls[roomFactoryMock.mock.calls.length - 1];
const roomOptions = lastCallArgs.pop() as unknown as RoomOptions;
expect(roomOptions).toBeDefined();
expect(roomOptions!.videoCaptureDefaults?.deviceId).toEqual("cam1");
expect(roomOptions!.audioCaptureDefaults?.deviceId).toEqual("mic1");
expect(roomOptions!.audioOutput?.deviceId).toEqual("speaker");
});
it("respect controlledAudioDevices", () => {
// TODO: Refactor the code to make it testable.
// The UrlParams module is a singleton has a cache and is very hard to test.
// This breaks other tests as well if not handled properly.
// vi.mock(import("./../UrlParams"), () => {
// return {
// getUrlParams: vi.fn().mockReturnValue({
// controlledAudioDevices: true
// })
// };
// });
});
});
});

282
src/state/Connection.ts Normal file
View File

@@ -0,0 +1,282 @@
/*
Copyright 2025 New Vector Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import {
connectedParticipantsObserver,
connectionStateObserver,
} from "@livekit/components-core";
import {
ConnectionError,
type ConnectionState,
type E2EEOptions,
type RemoteParticipant,
Room as LivekitRoom,
type RoomOptions,
} from "livekit-client";
import {
type CallMembership,
type LivekitTransport,
} from "matrix-js-sdk/lib/matrixrtc";
import { BehaviorSubject, combineLatest, type Observable } from "rxjs";
import {
getSFUConfigWithOpenID,
type OpenIDClientParts,
type SFUConfig,
} from "../livekit/openIDSFU";
import { type Behavior } from "./Behavior";
import { type ObservableScope } from "./ObservableScope";
import { defaultLiveKitOptions } from "../livekit/options";
import {
InsufficientCapacityError,
SFURoomCreationRestrictedError,
} from "../utils/errors.ts";
export interface ConnectionOpts {
/** The media transport to connect to. */
transport: LivekitTransport;
/** The Matrix client to use for OpenID and SFU config requests. */
client: OpenIDClientParts;
/** The observable scope to use for this connection. */
scope: ObservableScope;
/** An observable of the current RTC call memberships and their associated transports. */
remoteTransports$: Behavior<
{ membership: CallMembership; transport: LivekitTransport }[]
>;
/** Optional factory to create the Livekit room, mainly for testing purposes. */
livekitRoomFactory?: (options?: RoomOptions) => LivekitRoom;
}
export type TransportState =
| { state: "Initialized" }
| { state: "FetchingConfig"; transport: LivekitTransport }
| { state: "ConnectingToLkRoom"; transport: LivekitTransport }
| { state: "PublishingTracks"; transport: LivekitTransport }
| { state: "FailedToStart"; error: Error; transport: LivekitTransport }
| {
state: "ConnectedToLkRoom";
connectionState$: Observable<ConnectionState>;
transport: LivekitTransport;
}
| { state: "Stopped"; transport: LivekitTransport };
/**
* Represents participant publishing or expected to publish on the connection.
* It is paired with its associated rtc membership.
*/
export type PublishingParticipant = {
/**
* The LiveKit participant publishing on this connection, or undefined if the participant is not currently (yet) connected to the livekit room.
*/
participant: RemoteParticipant | undefined;
/**
* The rtc call membership associated with this participant.
*/
membership: CallMembership;
};
/**
* A connection to a Matrix RTC LiveKit backend.
*
* Expose observables for participants and connection state.
*/
export class Connection {
// Private Behavior
private readonly _transportState$ = new BehaviorSubject<TransportState>({
state: "Initialized",
});
/**
* The current state of the connection to the media transport.
*/
public readonly transportState$: Behavior<TransportState> =
this._transportState$;
/**
* Whether the connection has been stopped.
* @see Connection.stop
* */
protected stopped = false;
/**
* Starts the connection.
*
* This will:
* 1. Request an OpenId token `request_token` (allows matrix users to verify their identity with a third-party service.)
* 2. Use this token to request the SFU config to the MatrixRtc authentication service.
* 3. Connect to the configured LiveKit room.
*
* @throws {InsufficientCapacityError} if the LiveKit server indicates that it has insufficient capacity to accept the connection.
* @throws {SFURoomCreationRestrictedError} if the LiveKit server indicates that the room does not exist and cannot be created.
*/
public async start(): Promise<void> {
this.stopped = false;
try {
this._transportState$.next({
state: "FetchingConfig",
transport: this.transport,
});
const { url, jwt } = await this.getSFUConfigWithOpenID();
// If we were stopped while fetching the config, don't proceed to connect
if (this.stopped) return;
this._transportState$.next({
state: "ConnectingToLkRoom",
transport: this.transport,
});
try {
await this.livekitRoom.connect(url, jwt);
} catch (e) {
// LiveKit uses 503 to indicate that the server has hit its track limits.
// https://github.com/livekit/livekit/blob/fcb05e97c5a31812ecf0ca6f7efa57c485cea9fb/pkg/service/rtcservice.go#L171
// It also errors with a status code of 200 (yes, really) for room
// participant limits.
// LiveKit Cloud uses 429 for connection limits.
// Either way, all these errors can be explained as "insufficient capacity".
if (e instanceof ConnectionError) {
if (e.status === 503 || e.status === 200 || e.status === 429) {
throw new InsufficientCapacityError();
}
if (e.status === 404) {
// error msg is "Could not establish signal connection: requested room does not exist"
// The room does not exist. There are two different modes of operation for the SFU:
// - the room is created on the fly when connecting (livekit `auto_create` option)
// - Only authorized users can create rooms, so the room must exist before connecting (done by the auth jwt service)
// In the first case there will not be a 404, so we are in the second case.
throw new SFURoomCreationRestrictedError();
}
}
throw e;
}
// If we were stopped while connecting, don't proceed to update state.
if (this.stopped) return;
this._transportState$.next({
state: "ConnectedToLkRoom",
transport: this.transport,
connectionState$: connectionStateObserver(this.livekitRoom),
});
} catch (error) {
this._transportState$.next({
state: "FailedToStart",
error: error instanceof Error ? error : new Error(`${error}`),
transport: this.transport,
});
throw error;
}
}
protected async getSFUConfigWithOpenID(): Promise<SFUConfig> {
return await getSFUConfigWithOpenID(
this.client,
this.transport.livekit_service_url,
this.transport.livekit_alias,
);
}
/**
* Stops the connection.
*
* This will disconnect from the LiveKit room.
* If the connection is already stopped, this is a no-op.
*/
public async stop(): Promise<void> {
if (this.stopped) return;
await this.livekitRoom.disconnect();
this._transportState$.next({
state: "Stopped",
transport: this.transport,
});
this.stopped = true;
}
/**
* An observable of the participants that are publishing on this connection.
* This is derived from `participantsIncludingSubscribers$` and `remoteTransports$`.
* It filters the participants to only those that are associated with a membership that claims to publish on this connection.
*/
public readonly publishingParticipants$: Behavior<PublishingParticipant[]>;
/**
* The media transport to connect to.
*/
public readonly transport: LivekitTransport;
private readonly client: OpenIDClientParts;
/**
* Creates a new connection to a matrix RTC LiveKit backend.
*
* @param livekitRoom - LiveKit room instance to use.
* @param opts - Connection options {@link ConnectionOpts}.
*
*/
protected constructor(
public readonly livekitRoom: LivekitRoom,
opts: ConnectionOpts,
) {
const { transport, client, scope, remoteTransports$ } = opts;
this.transport = transport;
this.client = client;
const participantsIncludingSubscribers$ = scope.behavior(
connectedParticipantsObserver(this.livekitRoom),
[],
);
this.publishingParticipants$ = scope.behavior(
combineLatest(
[participantsIncludingSubscribers$, remoteTransports$],
(participants, remoteTransports) =>
remoteTransports
// Find all members that claim to publish on this connection
.flatMap(({ membership, transport }) =>
transport.livekit_service_url ===
this.transport.livekit_service_url
? [membership]
: [],
)
// Pair with their associated LiveKit participant (if any)
.map((membership) => {
const id = `${membership.userId}:${membership.deviceId}`;
const participant = participants.find((p) => p.identity === id);
return { participant, membership };
}),
),
[],
);
scope.onEnd(() => void this.stop());
}
}
/**
* A remote connection to the Matrix RTC LiveKit backend.
*
* This connection is used for subscribing to remote participants.
* It does not publish any local tracks.
*/
export class RemoteConnection extends Connection {
/**
* Creates a new remote connection to a matrix RTC LiveKit backend.
* @param opts
* @param sharedE2eeOption - The shared E2EE options to use for the connection.
*/
public constructor(
opts: ConnectionOpts,
sharedE2eeOption: E2EEOptions | undefined,
) {
const factory =
opts.livekitRoomFactory ??
((options: RoomOptions): LivekitRoom => new LivekitRoom(options));
const livekitRoom = factory({
...defaultLiveKitOptions,
e2ee: sharedE2eeOption,
});
super(livekitRoom, opts);
}
}

View File

@@ -5,7 +5,7 @@ SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { type Layout, type LayoutMedia } from "./CallViewModel";
import { type Layout, type LayoutMedia } from "./layout-types.ts";
import { type TileStore } from "./TileStore";
export type GridLikeLayoutType =

View File

@@ -255,7 +255,7 @@ abstract class BaseMediaViewModel extends ViewModel {
*/
// TODO: Fully separate the data layer from the UI layer by keeping the
// member object internal
public readonly member: RoomMember | undefined,
public readonly member: RoomMember,
// We don't necessarily have a participant if a user connects via MatrixRTC but not (yet) through
// livekit.
protected readonly participant$: Observable<
@@ -266,6 +266,7 @@ abstract class BaseMediaViewModel extends ViewModel {
audioSource: AudioSource,
videoSource: VideoSource,
livekitRoom: LivekitRoom,
public readonly focusURL: string,
public readonly displayName$: Behavior<string>,
) {
super();
@@ -403,10 +404,11 @@ abstract class BaseUserMediaViewModel extends BaseMediaViewModel {
public constructor(
id: string,
member: RoomMember | undefined,
member: RoomMember,
participant$: Observable<LocalParticipant | RemoteParticipant | undefined>,
encryptionSystem: EncryptionSystem,
livekitRoom: LivekitRoom,
focusUrl: string,
displayName$: Behavior<string>,
public readonly handRaised$: Behavior<Date | null>,
public readonly reaction$: Behavior<ReactionOption | null>,
@@ -419,6 +421,7 @@ abstract class BaseUserMediaViewModel extends BaseMediaViewModel {
Track.Source.Microphone,
Track.Source.Camera,
livekitRoom,
focusUrl,
displayName$,
);
@@ -535,10 +538,11 @@ export class LocalUserMediaViewModel extends BaseUserMediaViewModel {
public constructor(
id: string,
member: RoomMember | undefined,
member: RoomMember,
participant$: Behavior<LocalParticipant | undefined>,
encryptionSystem: EncryptionSystem,
livekitRoom: LivekitRoom,
focusURL: string,
private readonly mediaDevices: MediaDevices,
displayName$: Behavior<string>,
handRaised$: Behavior<Date | null>,
@@ -550,6 +554,7 @@ export class LocalUserMediaViewModel extends BaseUserMediaViewModel {
participant$,
encryptionSystem,
livekitRoom,
focusURL,
displayName$,
handRaised$,
reaction$,
@@ -641,10 +646,11 @@ export class RemoteUserMediaViewModel extends BaseUserMediaViewModel {
public constructor(
id: string,
member: RoomMember | undefined,
member: RoomMember,
participant$: Observable<RemoteParticipant | undefined>,
encryptionSystem: EncryptionSystem,
livekitRoom: LivekitRoom,
focusUrl: string,
private readonly pretendToBeDisconnected$: Behavior<boolean>,
displayname$: Behavior<string>,
handRaised$: Behavior<Date | null>,
@@ -656,6 +662,7 @@ export class RemoteUserMediaViewModel extends BaseUserMediaViewModel {
participant$,
encryptionSystem,
livekitRoom,
focusUrl,
displayname$,
handRaised$,
reaction$,
@@ -736,10 +743,11 @@ export class ScreenShareViewModel extends BaseMediaViewModel {
public constructor(
id: string,
member: RoomMember | undefined,
member: RoomMember,
participant$: Observable<LocalParticipant | RemoteParticipant>,
encryptionSystem: EncryptionSystem,
livekitRoom: LivekitRoom,
focusUrl: string,
private readonly pretendToBeDisconnected$: Behavior<boolean>,
displayname$: Behavior<string>,
public readonly local: boolean,
@@ -752,6 +760,7 @@ export class ScreenShareViewModel extends BaseMediaViewModel {
Track.Source.ScreenShareAudio,
Track.Source.ScreenShare,
livekitRoom,
focusUrl,
displayname$,
);
}

213
src/state/MuteStates.ts Normal file
View File

@@ -0,0 +1,213 @@
/*
Copyright 2023-2025 New Vector Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { type IWidgetApiRequest } from "matrix-widget-api";
import { logger } from "matrix-js-sdk/lib/logger";
import {
BehaviorSubject,
combineLatest,
distinctUntilChanged,
firstValueFrom,
fromEvent,
map,
merge,
Observable,
of,
Subject,
switchMap,
withLatestFrom,
} from "rxjs";
import { type MediaDevices, type MediaDevice } from "../state/MediaDevices";
import { ElementWidgetActions, widget } from "../widget";
import { Config } from "../config/Config";
import { getUrlParams } from "../UrlParams";
import { type ObservableScope } from "./ObservableScope";
import { type Behavior } from "./Behavior";
interface MuteStateData {
enabled$: Observable<boolean>;
set: ((enabled: boolean) => void) | null;
toggle: (() => void) | null;
}
export type Handler = (desired: boolean) => Promise<boolean>;
const defaultHandler: Handler = async (desired) => Promise.resolve(desired);
class MuteState<Label, Selected> {
private readonly enabledByDefault$ =
this.enabledByConfig && !getUrlParams().skipLobby
? this.joined$.pipe(map((isJoined) => !isJoined))
: of(false);
private readonly handler$ = new BehaviorSubject(defaultHandler);
public setHandler(handler: Handler): void {
if (this.handler$.value !== defaultHandler)
throw new Error("Multiple mute state handlers are not supported");
this.handler$.next(handler);
}
public unsetHandler(): void {
this.handler$.next(defaultHandler);
}
private readonly data$ = this.scope.behavior<MuteStateData>(
this.device.available$.pipe(
map((available) => available.size > 0),
distinctUntilChanged(),
withLatestFrom(
this.enabledByDefault$,
(devicesConnected, enabledByDefault) => {
if (!devicesConnected)
return { enabled$: of(false), set: null, toggle: null };
// Assume the default value only once devices are actually connected
let enabled = enabledByDefault;
const set$ = new Subject<boolean>();
const toggle$ = new Subject<void>();
const desired$ = merge(set$, toggle$.pipe(map(() => !enabled)));
const enabled$ = new Observable<boolean>((subscriber) => {
subscriber.next(enabled);
let latestDesired = enabledByDefault;
let syncing = false;
const sync = async (): Promise<void> => {
if (enabled === latestDesired) syncing = false;
else {
const previouslyEnabled = enabled;
enabled = await firstValueFrom(
this.handler$.pipe(
switchMap(async (handler) => handler(latestDesired)),
),
);
if (enabled === previouslyEnabled) {
syncing = false;
} else {
subscriber.next(enabled);
syncing = true;
sync().catch((err) => {
// TODO: better error handling
logger.error("MuteState: handler error", err);
});
}
}
};
const s = desired$.subscribe((desired) => {
latestDesired = desired;
if (syncing === false) {
syncing = true;
sync().catch((err) => {
// TODO: better error handling
logger.error("MuteState: handler error", err);
});
}
});
return (): void => s.unsubscribe();
});
return {
set: (enabled: boolean): void => set$.next(enabled),
toggle: (): void => toggle$.next(),
enabled$,
};
},
),
),
);
public readonly enabled$: Behavior<boolean> = this.scope.behavior(
this.data$.pipe(switchMap(({ enabled$ }) => enabled$)),
);
public readonly setEnabled$: Behavior<((enabled: boolean) => void) | null> =
this.scope.behavior(this.data$.pipe(map(({ set }) => set)));
public readonly toggle$: Behavior<(() => void) | null> = this.scope.behavior(
this.data$.pipe(map(({ toggle }) => toggle)),
);
public constructor(
private readonly scope: ObservableScope,
private readonly device: MediaDevice<Label, Selected>,
private readonly joined$: Observable<boolean>,
private readonly enabledByConfig: boolean,
) {}
}
export class MuteStates {
public readonly audio = new MuteState(
this.scope,
this.mediaDevices.audioInput,
this.joined$,
Config.get().media_devices.enable_audio,
);
public readonly video = new MuteState(
this.scope,
this.mediaDevices.videoInput,
this.joined$,
Config.get().media_devices.enable_video,
);
public constructor(
private readonly scope: ObservableScope,
private readonly mediaDevices: MediaDevices,
private readonly joined$: Observable<boolean>,
) {
if (widget !== null) {
// Sync our mute states with the hosting client
const widgetApiState$ = combineLatest(
[this.audio.enabled$, this.video.enabled$],
(audio, video) => ({ audio_enabled: audio, video_enabled: video }),
);
widgetApiState$.pipe(this.scope.bind()).subscribe((state) => {
widget!.api.transport
.send(ElementWidgetActions.DeviceMute, state)
.catch((e) =>
logger.warn("Could not send DeviceMute action to widget", e),
);
});
// Also sync the hosting client's mute states back with ours
const muteActions$ = fromEvent(
widget.lazyActions,
ElementWidgetActions.DeviceMute,
) as Observable<CustomEvent<IWidgetApiRequest>>;
muteActions$
.pipe(
withLatestFrom(
widgetApiState$,
this.audio.setEnabled$,
this.video.setEnabled$,
),
this.scope.bind(),
)
.subscribe(([ev, state, setAudioEnabled, setVideoEnabled]) => {
// First copy the current state into our new state
const newState = { ...state };
// Update new state if there are any requested changes from the widget
// action in `ev.detail.data`.
if (
ev.detail.data.audio_enabled != null &&
typeof ev.detail.data.audio_enabled === "boolean" &&
setAudioEnabled !== null
) {
newState.audio_enabled = ev.detail.data.audio_enabled;
setAudioEnabled(newState.audio_enabled);
}
if (
ev.detail.data.video_enabled != null &&
typeof ev.detail.data.video_enabled === "boolean" &&
setVideoEnabled !== null
) {
newState.video_enabled = ev.detail.data.video_enabled;
setVideoEnabled(newState.video_enabled);
}
widget!.api.transport.reply(ev.detail, newState);
});
}
}
}

View File

@@ -7,9 +7,14 @@ Please see LICENSE in the repository root for full details.
import {
BehaviorSubject,
catchError,
distinctUntilChanged,
EMPTY,
endWith,
filter,
type Observable,
Subject,
share,
take,
takeUntil,
} from "rxjs";
@@ -23,9 +28,11 @@ const nothing = Symbol("nothing");
* A scope which limits the execution lifetime of its bound Observables.
*/
export class ObservableScope {
private readonly ended$ = new Subject<void>();
private readonly ended$ = new BehaviorSubject(false);
private readonly bindImpl: MonoTypeOperator = takeUntil(this.ended$);
private readonly bindImpl: MonoTypeOperator = takeUntil(
this.ended$.pipe(filter((ended) => ended)),
);
/**
* Binds an Observable to this scope, so that it completes when the scope
@@ -35,6 +42,17 @@ export class ObservableScope {
return this.bindImpl;
}
private readonly shareImpl: MonoTypeOperator = share({
resetOnError: false,
resetOnComplete: false,
resetOnRefCountZero: false,
});
/**
* Shares (multicasts) the Observable as a hot Observable.
*/
public readonly share: MonoTypeOperator = (input$) =>
input$.pipe(this.bindImpl, this.shareImpl);
/**
* Converts an Observable to a Behavior. If no initial value is specified, the
* Observable must synchronously emit an initial value.
@@ -66,8 +84,66 @@ export class ObservableScope {
* Ends the scope, causing any bound Observables to complete.
*/
public end(): void {
this.ended$.next();
this.ended$.complete();
this.ended$.next(true);
}
/**
* Register a callback to be executed when the scope is ended.
*/
public onEnd(callback: () => void): void {
this.ended$
.pipe(
filter((ended) => ended),
take(1),
)
.subscribe(callback);
}
/**
* For the duration of the scope, sync some external state with the value of
* the provided Behavior by way of an async function which attempts to update
* (reconcile) the external state. The reconciliation function may return a
* clean-up callback which will be called and awaited before the next change
* in value (or the end of the scope).
*
* All calls to the function and its clean-up callbacks are serialized. If the
* value changes faster than the handlers can keep up with, intermediate
* values may be skipped.
*
* Basically, this is like React's useEffect but async and for Behaviors.
*/
public reconcile<T>(
value$: Behavior<T>,
callback: (value: T) => Promise<(() => Promise<void>) | void>,
): void {
let latestValue: T | typeof nothing = nothing;
let reconciledValue: T | typeof nothing = nothing;
let cleanUp: (() => Promise<void>) | void = undefined;
value$
.pipe(
catchError(() => EMPTY), // Ignore errors
this.bind(), // Limit to the duration of the scope
endWith(nothing), // Clean up when the scope ends
)
.subscribe((value) => {
void (async (): Promise<void> => {
if (latestValue === nothing) {
latestValue = value;
while (latestValue !== reconciledValue) {
await cleanUp?.(); // Call the previous value's clean-up handler
reconciledValue = latestValue;
if (latestValue !== nothing)
cleanUp = await callback(latestValue); // Sync current value
}
// Reset to signal that reconciliation is done for now
latestValue = nothing;
} else {
// There's already an instance of the above 'while' loop running
// concurrently. Just update the latest value and let it be handled.
latestValue = value;
}
})();
});
}
}

View File

@@ -5,7 +5,7 @@ SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { type OneOnOneLayout, type OneOnOneLayoutMedia } from "./CallViewModel";
import { type OneOnOneLayout, type OneOnOneLayoutMedia } from "./layout-types";
import { type TileStore } from "./TileStore";
/**

View File

@@ -5,7 +5,7 @@ SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { type PipLayout, type PipLayoutMedia } from "./CallViewModel";
import { type PipLayout, type PipLayoutMedia } from "./layout-types.ts";
import { type TileStore } from "./TileStore";
/**

View File

@@ -0,0 +1,298 @@
/*
Copyright 2025 New Vector Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import {
ConnectionState,
type E2EEOptions,
LocalVideoTrack,
Room as LivekitRoom,
type RoomOptions,
Track,
} from "livekit-client";
import {
map,
NEVER,
type Observable,
type Subscription,
switchMap,
} from "rxjs";
import { logger } from "matrix-js-sdk/lib/logger";
import type { Behavior } from "./Behavior.ts";
import type { MediaDevices, SelectedDevice } from "./MediaDevices.ts";
import type { MuteStates } from "./MuteStates.ts";
import {
type ProcessorState,
trackProcessorSync,
} from "../livekit/TrackProcessorContext.tsx";
import { getUrlParams } from "../UrlParams.ts";
import { defaultLiveKitOptions } from "../livekit/options.ts";
import { getValue } from "../utils/observable.ts";
import { observeTrackReference$ } from "./MediaViewModel.ts";
import { Connection, type ConnectionOpts } from "./Connection.ts";
import { type ObservableScope } from "./ObservableScope.ts";
/**
* A connection to the local LiveKit room, the one the user is publishing to.
* This connection will publish the local user's audio and video tracks.
*/
export class PublishConnection extends Connection {
private readonly scope: ObservableScope;
/**
* Creates a new PublishConnection.
* @param args - The connection options. {@link ConnectionOpts}
* @param devices - The media devices to use for audio and video input.
* @param muteStates - The mute states for audio and video.
* @param e2eeLivekitOptions - The E2EE options to use for the LiveKit room. Use to share the same key provider across connections!.
* @param trackerProcessorState$ - The processor state for the video track processor (e.g. background blur).
*/
public constructor(
args: ConnectionOpts,
devices: MediaDevices,
private readonly muteStates: MuteStates,
e2eeLivekitOptions: E2EEOptions | undefined,
trackerProcessorState$: Behavior<ProcessorState>,
) {
const { scope } = args;
logger.info("[LivekitRoom] Create LiveKit room");
const { controlledAudioDevices } = getUrlParams();
const factory =
args.livekitRoomFactory ??
((options: RoomOptions): LivekitRoom => new LivekitRoom(options));
const room = factory(
generateRoomOption(
devices,
trackerProcessorState$.value,
controlledAudioDevices,
e2eeLivekitOptions,
),
);
room.setE2EEEnabled(e2eeLivekitOptions !== undefined).catch((e) => {
logger.error("Failed to set E2EE enabled on room", e);
});
super(room, args);
this.scope = scope;
// Setup track processor syncing (blur)
this.observeTrackProcessors(scope, room, trackerProcessorState$);
// Observe media device changes and update LiveKit active devices accordingly
this.observeMediaDevices(scope, devices, controlledAudioDevices);
this.workaroundRestartAudioInputTrackChrome(devices, scope);
}
/**
* Start the connection to LiveKit and publish local tracks.
*
* This will:
* 1. Request an OpenId token `request_token` (allows matrix users to verify their identity with a third-party service.)
* 2. Use this token to request the SFU config to the MatrixRtc authentication service.
* 3. Connect to the configured LiveKit room.
* 4. Create local audio and video tracks based on the current mute states and publish them to the room.
*
* @throws {InsufficientCapacityError} if the LiveKit server indicates that it has insufficient capacity to accept the connection.
* @throws {SFURoomCreationRestrictedError} if the LiveKit server indicates that the room does not exist and cannot be created.
*/
public async start(): Promise<void> {
this.stopped = false;
// Observe mute state changes and update LiveKit microphone/camera states accordingly
this.observeMuteStates(this.scope);
// TODO: This will fetch the JWT token. Perhaps we could keep it preloaded
// instead? This optimization would only be safe for a publish connection,
// because we don't want to leak the user's intent to perhaps join a call to
// remote servers before they actually commit to it.
await super.start();
if (this.stopped) return;
// TODO-MULTI-SFU: Prepublish a microphone track
const audio = this.muteStates.audio.enabled$.value;
const video = this.muteStates.video.enabled$.value;
// createTracks throws if called with audio=false and video=false
if (audio || video) {
// TODO this can still throw errors? It will also prompt for permissions if not already granted
const tracks = await this.livekitRoom.localParticipant.createTracks({
audio,
video,
});
if (this.stopped) return;
for (const track of tracks) {
// TODO: handle errors? Needs the signaling connection to be up, but it has some retries internally
// with a timeout.
await this.livekitRoom.localParticipant.publishTrack(track);
if (this.stopped) return;
// TODO: check if the connection is still active? and break the loop if not?
}
}
}
public async stop(): Promise<void> {
// TODO-MULTI-SFU: Move these calls back to ObservableScope.onEnd once scope
// actually has the right lifetime
this.muteStates.audio.unsetHandler();
this.muteStates.video.unsetHandler();
await super.stop();
}
/// Private methods
// Restart the audio input track whenever we detect that the active media
// device has changed to refer to a different hardware device. We do this
// for the sake of Chrome, which provides a "default" device that is meant
// to match the system's default audio input, whatever that may be.
// This is special-cased for only audio inputs because we need to dig around
// in the LocalParticipant object for the track object and there's not a nice
// way to do that generically. There is usually no OS-level default video capture
// device anyway, and audio outputs work differently.
private workaroundRestartAudioInputTrackChrome(
devices: MediaDevices,
scope: ObservableScope,
): void {
devices.audioInput.selected$
.pipe(
switchMap((device) => device?.hardwareDeviceChange$ ?? NEVER),
scope.bind(),
)
.subscribe(() => {
if (this.livekitRoom.state != ConnectionState.Connected) return;
const activeMicTrack = Array.from(
this.livekitRoom.localParticipant.audioTrackPublications.values(),
).find((d) => d.source === Track.Source.Microphone)?.track;
if (
activeMicTrack &&
// only restart if the stream is still running: LiveKit will detect
// when a track stops & restart appropriately, so this is not our job.
// Plus, we need to avoid restarting again if the track is already in
// the process of being restarted.
activeMicTrack.mediaStreamTrack.readyState !== "ended"
) {
// Restart the track, which will cause Livekit to do another
// getUserMedia() call with deviceId: default to get the *new* default device.
// Note that room.switchActiveDevice() won't work: Livekit will ignore it because
// the deviceId hasn't changed (was & still is default).
this.livekitRoom.localParticipant
.getTrackPublication(Track.Source.Microphone)
?.audioTrack?.restartTrack()
.catch((e) => {
logger.error(`Failed to restart audio device track`, e);
});
}
});
}
// Observe changes in the selected media devices and update the LiveKit room accordingly.
private observeMediaDevices(
scope: ObservableScope,
devices: MediaDevices,
controlledAudioDevices: boolean,
): void {
const syncDevice = (
kind: MediaDeviceKind,
selected$: Observable<SelectedDevice | undefined>,
): Subscription =>
selected$.pipe(scope.bind()).subscribe((device) => {
if (this.livekitRoom.state != ConnectionState.Connected) return;
// if (this.connectionState$.value !== ConnectionState.Connected) return;
logger.info(
"[LivekitRoom] syncDevice room.getActiveDevice(kind) !== d.id :",
this.livekitRoom.getActiveDevice(kind),
" !== ",
device?.id,
);
if (
device !== undefined &&
this.livekitRoom.getActiveDevice(kind) !== device.id
) {
this.livekitRoom
.switchActiveDevice(kind, device.id)
.catch((e) =>
logger.error(`Failed to sync ${kind} device with LiveKit`, e),
);
}
});
syncDevice("audioinput", devices.audioInput.selected$);
if (!controlledAudioDevices)
syncDevice("audiooutput", devices.audioOutput.selected$);
syncDevice("videoinput", devices.videoInput.selected$);
}
/**
* Observe changes in the mute states and update the LiveKit room accordingly.
* @param scope
* @private
*/
private observeMuteStates(scope: ObservableScope): void {
this.muteStates.audio.setHandler(async (desired) => {
try {
await this.livekitRoom.localParticipant.setMicrophoneEnabled(desired);
} catch (e) {
logger.error("Failed to update LiveKit audio input mute state", e);
}
return this.livekitRoom.localParticipant.isMicrophoneEnabled;
});
this.muteStates.video.setHandler(async (desired) => {
try {
await this.livekitRoom.localParticipant.setCameraEnabled(desired);
} catch (e) {
logger.error("Failed to update LiveKit video input mute state", e);
}
return this.livekitRoom.localParticipant.isCameraEnabled;
});
}
private observeTrackProcessors(
scope: ObservableScope,
room: LivekitRoom,
trackerProcessorState$: Behavior<ProcessorState>,
): void {
const track$ = scope.behavior(
observeTrackReference$(room.localParticipant, Track.Source.Camera).pipe(
map((trackRef) => {
const track = trackRef?.publication?.track;
return track instanceof LocalVideoTrack ? track : null;
}),
),
);
trackProcessorSync(track$, trackerProcessorState$);
}
}
// Generate the initial LiveKit RoomOptions based on the current media devices and processor state.
function generateRoomOption(
devices: MediaDevices,
processorState: ProcessorState,
controlledAudioDevices: boolean,
e2eeLivekitOptions: E2EEOptions | undefined,
): RoomOptions {
return {
...defaultLiveKitOptions,
videoCaptureDefaults: {
...defaultLiveKitOptions.videoCaptureDefaults,
deviceId: devices.videoInput.selected$.value?.id,
processor: processorState.processor,
},
audioCaptureDefaults: {
...defaultLiveKitOptions.audioCaptureDefaults,
deviceId: devices.audioInput.selected$.value?.id,
},
audioOutput: {
// When using controlled audio devices, we don't want to set the
// deviceId here, because it will be set by the native app.
// (also the id does not need to match a browser device id)
deviceId: controlledAudioDevices
? undefined
: getValue(devices.audioOutput.selected$)?.id,
},
e2ee: e2eeLivekitOptions,
};
}

56
src/state/ScreenShare.ts Normal file
View File

@@ -0,0 +1,56 @@
/*
Copyright 2025 New Vector Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { of, type Observable } from "rxjs";
import {
type LocalParticipant,
type RemoteParticipant,
type Room as LivekitRoom,
} from "livekit-client";
import { ObservableScope } from "./ObservableScope.ts";
import { ScreenShareViewModel } from "./MediaViewModel.ts";
import type { RoomMember } from "matrix-js-sdk";
import type { EncryptionSystem } from "../e2ee/sharedKeyManagement.ts";
import type { Behavior } from "./Behavior.ts";
/**
* A screen share media item to be presented in a tile. This is a thin wrapper
* around ScreenShareViewModel which essentially just establishes an
* ObservableScope for behaviors that the view model depends on.
*/
export class ScreenShare {
private readonly scope = new ObservableScope();
public readonly vm: ScreenShareViewModel;
public constructor(
id: string,
member: RoomMember,
participant: LocalParticipant | RemoteParticipant,
encryptionSystem: EncryptionSystem,
livekitRoom: LivekitRoom,
focusUrl: string,
pretendToBeDisconnected$: Behavior<boolean>,
displayName$: Observable<string>,
) {
this.vm = new ScreenShareViewModel(
id,
member,
of(participant),
encryptionSystem,
livekitRoom,
focusUrl,
pretendToBeDisconnected$,
this.scope.behavior(displayName$),
participant.isLocal,
);
}
public destroy(): void {
this.scope.end();
this.vm.destroy();
}
}

View File

@@ -8,7 +8,7 @@ Please see LICENSE in the repository root for full details.
import {
type SpotlightExpandedLayout,
type SpotlightExpandedLayoutMedia,
} from "./CallViewModel";
} from "./layout-types";
import { type TileStore } from "./TileStore";
/**

186
src/state/UserMedia.ts Normal file
View File

@@ -0,0 +1,186 @@
/*
Copyright 2025 New Vector Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import {
BehaviorSubject,
combineLatest,
map,
type Observable,
of,
switchMap,
} from "rxjs";
import {
type LocalParticipant,
type Participant,
ParticipantEvent,
type RemoteParticipant,
type Room as LivekitRoom,
} from "livekit-client";
import { observeParticipantEvents } from "@livekit/components-core";
import { ObservableScope } from "./ObservableScope.ts";
import {
LocalUserMediaViewModel,
RemoteUserMediaViewModel,
type UserMediaViewModel,
} from "./MediaViewModel.ts";
import type { Behavior } from "./Behavior.ts";
import type { RoomMember } from "matrix-js-sdk";
import type { EncryptionSystem } from "../e2ee/sharedKeyManagement.ts";
import type { MediaDevices } from "./MediaDevices.ts";
import type { ReactionOption } from "../reactions";
import { observeSpeaker$ } from "./observeSpeaker.ts";
/**
* Sorting bins defining the order in which media tiles appear in the layout.
*/
enum SortingBin {
/**
* Yourself, when the "always show self" option is on.
*/
SelfAlwaysShown,
/**
* Participants that are sharing their screen.
*/
Presenters,
/**
* Participants that have been speaking recently.
*/
Speakers,
/**
* Participants that have their hand raised.
*/
HandRaised,
/**
* Participants with video.
*/
Video,
/**
* Participants not sharing any video.
*/
NoVideo,
/**
* Yourself, when the "always show self" option is off.
*/
SelfNotAlwaysShown,
}
/**
* A user media item to be presented in a tile. This is a thin wrapper around
* UserMediaViewModel which additionally determines the media item's sorting bin
* for inclusion in the call layout.
*/
export class UserMedia {
private readonly scope = new ObservableScope();
private readonly participant$ = new BehaviorSubject(this.initialParticipant);
public readonly vm: UserMediaViewModel = this.participant$.value?.isLocal
? new LocalUserMediaViewModel(
this.id,
this.member,
this.participant$ as Behavior<LocalParticipant>,
this.encryptionSystem,
this.livekitRoom,
this.focusURL,
this.mediaDevices,
this.scope.behavior(this.displayname$),
this.scope.behavior(this.handRaised$),
this.scope.behavior(this.reaction$),
)
: new RemoteUserMediaViewModel(
this.id,
this.member,
this.participant$ as Observable<RemoteParticipant | undefined>,
this.encryptionSystem,
this.livekitRoom,
this.focusURL,
this.pretendToBeDisconnected$,
this.scope.behavior(this.displayname$),
this.scope.behavior(this.handRaised$),
this.scope.behavior(this.reaction$),
);
private readonly speaker$ = this.scope.behavior(
observeSpeaker$(this.vm.speaking$),
);
private readonly presenter$ = this.scope.behavior(
this.participant$.pipe(
switchMap((p) => (p === undefined ? of(false) : sharingScreen$(p))),
),
);
/**
* Which sorting bin the media item should be placed in.
*/
// This is exposed here rather than by UserMediaViewModel because it's only
// relevant to the layout algorithms; the MediaView component should be
// ignorant of this value.
public readonly bin$ = combineLatest(
[
this.speaker$,
this.presenter$,
this.vm.videoEnabled$,
this.vm.handRaised$,
this.vm instanceof LocalUserMediaViewModel
? this.vm.alwaysShow$
: of(false),
],
(speaker, presenter, video, handRaised, alwaysShow) => {
if (this.vm.local)
return alwaysShow
? SortingBin.SelfAlwaysShown
: SortingBin.SelfNotAlwaysShown;
else if (presenter) return SortingBin.Presenters;
else if (speaker) return SortingBin.Speakers;
else if (handRaised) return SortingBin.HandRaised;
else if (video) return SortingBin.Video;
else return SortingBin.NoVideo;
},
);
public constructor(
public readonly id: string,
private readonly member: RoomMember,
private readonly initialParticipant:
| LocalParticipant
| RemoteParticipant
| undefined,
private readonly encryptionSystem: EncryptionSystem,
private readonly livekitRoom: LivekitRoom,
private readonly focusURL: string,
private readonly mediaDevices: MediaDevices,
private readonly pretendToBeDisconnected$: Behavior<boolean>,
private readonly displayname$: Observable<string>,
private readonly handRaised$: Observable<Date | null>,
private readonly reaction$: Observable<ReactionOption | null>,
) {}
public updateParticipant(
newParticipant: LocalParticipant | RemoteParticipant | undefined,
): void {
if (this.participant$.value !== newParticipant) {
// Update the BehaviourSubject in the UserMedia.
this.participant$.next(newParticipant);
}
}
public destroy(): void {
this.scope.end();
this.vm.destroy();
}
}
export function sharingScreen$(p: Participant): Observable<boolean> {
return observeParticipantEvents(
p,
ParticipantEvent.TrackPublished,
ParticipantEvent.TrackUnpublished,
ParticipantEvent.LocalTrackPublished,
ParticipantEvent.LocalTrackUnpublished,
).pipe(map((p) => p.isScreenShareEnabled));
}

108
src/state/layout-types.ts Normal file
View File

@@ -0,0 +1,108 @@
/*
Copyright 2025 New Vector Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import {
type GridTileViewModel,
type SpotlightTileViewModel,
} from "./TileViewModel.ts";
import {
type MediaViewModel,
type UserMediaViewModel,
} from "./MediaViewModel.ts";
export interface GridLayoutMedia {
type: "grid";
spotlight?: MediaViewModel[];
grid: UserMediaViewModel[];
}
export interface SpotlightLandscapeLayoutMedia {
type: "spotlight-landscape";
spotlight: MediaViewModel[];
grid: UserMediaViewModel[];
}
export interface SpotlightPortraitLayoutMedia {
type: "spotlight-portrait";
spotlight: MediaViewModel[];
grid: UserMediaViewModel[];
}
export interface SpotlightExpandedLayoutMedia {
type: "spotlight-expanded";
spotlight: MediaViewModel[];
pip?: UserMediaViewModel;
}
export interface OneOnOneLayoutMedia {
type: "one-on-one";
local: UserMediaViewModel;
remote: UserMediaViewModel;
}
export interface PipLayoutMedia {
type: "pip";
spotlight: MediaViewModel[];
}
export type LayoutMedia =
| GridLayoutMedia
| SpotlightLandscapeLayoutMedia
| SpotlightPortraitLayoutMedia
| SpotlightExpandedLayoutMedia
| OneOnOneLayoutMedia
| PipLayoutMedia;
export interface GridLayout {
type: "grid";
spotlight?: SpotlightTileViewModel;
grid: GridTileViewModel[];
setVisibleTiles: (value: number) => void;
}
export interface SpotlightLandscapeLayout {
type: "spotlight-landscape";
spotlight: SpotlightTileViewModel;
grid: GridTileViewModel[];
setVisibleTiles: (value: number) => void;
}
export interface SpotlightPortraitLayout {
type: "spotlight-portrait";
spotlight: SpotlightTileViewModel;
grid: GridTileViewModel[];
setVisibleTiles: (value: number) => void;
}
export interface SpotlightExpandedLayout {
type: "spotlight-expanded";
spotlight: SpotlightTileViewModel;
pip?: GridTileViewModel;
}
export interface OneOnOneLayout {
type: "one-on-one";
local: GridTileViewModel;
remote: GridTileViewModel;
}
export interface PipLayout {
type: "pip";
spotlight: SpotlightTileViewModel;
}
/**
* A layout defining the media tiles present on screen and their visual
* arrangement.
*/
export type Layout =
| GridLayout
| SpotlightLandscapeLayout
| SpotlightPortraitLayout
| SpotlightExpandedLayout
| OneOnOneLayout
| PipLayout;

View File

@@ -190,6 +190,7 @@ const UserMediaTile: FC<UserMediaTileProps> = ({
currentReaction={reaction ?? undefined}
raisedHandOnClick={raisedHandOnClick}
localParticipant={vm.local}
focusUrl={vm.focusURL}
audioStreamStats={audioStreamStats}
videoStreamStats={videoStreamStats}
{...props}

View File

@@ -5,7 +5,7 @@ SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { describe, expect, it, test } from "vitest";
import { describe, expect, it, test, vi } from "vitest";
import { render, screen } from "@testing-library/react";
import { axe } from "vitest-axe";
import { TooltipProvider } from "@vector-im/compound-web";
@@ -16,6 +16,7 @@ import {
import { LocalTrackPublication, Track } from "livekit-client";
import { TrackInfo } from "@livekit/protocol";
import { type ComponentProps } from "react";
import { type RoomMember } from "matrix-js-sdk";
import { MediaView } from "./MediaView";
import { EncryptionStatus } from "../state/MediaViewModel";
@@ -45,7 +46,10 @@ describe("MediaView", () => {
mirror: false,
unencryptedWarning: false,
video: trackReference,
member: undefined,
member: vi.mocked<RoomMember>({
userId: "@alice:example.com",
getMxcAvatarUrl: vi.fn().mockReturnValue(undefined),
} as unknown as RoomMember),
localParticipant: false,
focusable: true,
};
@@ -59,9 +63,9 @@ describe("MediaView", () => {
test("neither video nor avatar are shown", () => {
render(<MediaView {...baseProps} video={trackReferencePlaceholder} />);
expect(screen.queryByTestId("video")).toBeNull();
expect(screen.queryAllByRole("img", { name: "some name" }).length).toBe(
0,
);
expect(
screen.queryAllByRole("img", { name: "@alice:example.com" }).length,
).toBe(0);
});
});
@@ -70,14 +74,18 @@ describe("MediaView", () => {
render(
<MediaView {...baseProps} video={undefined} localParticipant={true} />,
);
expect(screen.getByRole("img", { name: "some name" })).toBeVisible();
expect(
screen.getByRole("img", { name: "@alice:example.com" }),
).toBeVisible();
expect(screen.queryAllByText("Waiting for media...").length).toBe(0);
});
it("shows avatar and label for remote user", () => {
render(
<MediaView {...baseProps} video={undefined} localParticipant={false} />,
);
expect(screen.getByRole("img", { name: "some name" })).toBeVisible();
expect(
screen.getByRole("img", { name: "@alice:example.com" }),
).toBeVisible();
expect(screen.getByText("Waiting for media...")).toBeVisible();
});
});
@@ -131,7 +139,9 @@ describe("MediaView", () => {
<MediaView {...baseProps} videoEnabled={false} />
</TooltipProvider>,
);
expect(screen.getByRole("img", { name: "some name" })).toBeVisible();
expect(
screen.getByRole("img", { name: "@alice:example.com" }),
).toBeVisible();
expect(screen.getByTestId("video")).not.toBeVisible();
});
});

View File

@@ -32,7 +32,7 @@ interface Props extends ComponentProps<typeof animated.div> {
video: TrackReferenceOrPlaceholder | undefined;
videoFit: "cover" | "contain";
mirror: boolean;
member: RoomMember | undefined;
member: RoomMember;
videoEnabled: boolean;
unencryptedWarning: boolean;
encryptionStatus: EncryptionStatus;
@@ -46,6 +46,8 @@ interface Props extends ComponentProps<typeof animated.div> {
localParticipant: boolean;
audioStreamStats?: RTCInboundRtpStreamStats | RTCOutboundRtpStreamStats;
videoStreamStats?: RTCInboundRtpStreamStats | RTCOutboundRtpStreamStats;
// The focus url, mainly for debugging purposes
focusUrl?: string;
}
export const MediaView: FC<Props> = ({
@@ -71,6 +73,7 @@ export const MediaView: FC<Props> = ({
localParticipant,
audioStreamStats,
videoStreamStats,
focusUrl,
...props
}) => {
const { t } = useTranslation();
@@ -134,6 +137,7 @@ export const MediaView: FC<Props> = ({
<RTCConnectionStats
audio={audioStreamStats}
video={videoStreamStats}
focusUrl={focusUrl}
/>
)}
{/* TODO: Bring this back once encryption status is less broken */}

View File

@@ -55,7 +55,7 @@ interface SpotlightItemBaseProps {
targetHeight: number;
video: TrackReferenceOrPlaceholder | undefined;
videoEnabled: boolean;
member: RoomMember | undefined;
member: RoomMember;
unencryptedWarning: boolean;
encryptionStatus: EncryptionStatus;
displayName: string;
@@ -78,7 +78,7 @@ const SpotlightLocalUserMediaItem: FC<SpotlightLocalUserMediaItemProps> = ({
...props
}) => {
const mirror = useBehavior(vm.mirror$);
return <MediaView mirror={mirror} {...props} />;
return <MediaView mirror={mirror} focusUrl={vm.focusURL} {...props} />;
};
SpotlightLocalUserMediaItem.displayName = "SpotlightLocalUserMediaItem";

View File

@@ -23,14 +23,14 @@ import {
// The TestComponent just wraps a button around that hook.
interface TestComponentProps {
setMicrophoneMuted?: (muted: boolean) => void;
setAudioEnabled?: (enabled: boolean) => void;
onButtonClick?: () => void;
sendReaction?: () => void;
toggleHandRaised?: () => void;
}
const TestComponent: FC<TestComponentProps> = ({
setMicrophoneMuted = (): void => {},
setAudioEnabled = (): void => {},
onButtonClick = (): void => {},
sendReaction = (reaction: ReactionOption): void => {},
toggleHandRaised = (): void => {},
@@ -40,7 +40,7 @@ const TestComponent: FC<TestComponentProps> = ({
ref,
() => {},
() => {},
setMicrophoneMuted,
setAudioEnabled,
sendReaction,
toggleHandRaised,
);
@@ -57,12 +57,13 @@ test("spacebar unmutes", async () => {
render(
<TestComponent
onButtonClick={() => (muted = false)}
setMicrophoneMuted={(m) => {
muted = m;
setAudioEnabled={(m) => {
muted = !m;
}}
/>,
);
expect(muted).toBe(true);
await user.keyboard("[Space>]");
expect(muted).toBe(false);
await user.keyboard("[/Space]");
@@ -73,15 +74,15 @@ test("spacebar unmutes", async () => {
test("spacebar prioritizes pressing a button", async () => {
const user = userEvent.setup();
const setMuted = vi.fn();
const setAudioEnabled = vi.fn();
const onClick = vi.fn();
render(
<TestComponent setMicrophoneMuted={setMuted} onButtonClick={onClick} />,
<TestComponent setAudioEnabled={setAudioEnabled} onButtonClick={onClick} />,
);
await user.tab(); // Focus the button
await user.keyboard("[Space]");
expect(setMuted).not.toBeCalled();
expect(setAudioEnabled).not.toBeCalled();
expect(onClick).toBeCalled();
});
@@ -129,7 +130,7 @@ test("unmuting happens in place of the default action", async () => {
tabIndex={0}
onKeyDown={(e) => defaultPrevented(e.isDefaultPrevented())}
>
<TestComponent setMicrophoneMuted={() => {}} />
<TestComponent setAudioEnabled={() => {}} />
</video>,
);

View File

@@ -29,9 +29,9 @@ const KeyToReactionMap: Record<string, ReactionOption> = Object.fromEntries(
export function useCallViewKeyboardShortcuts(
focusElement: RefObject<HTMLElement | null>,
toggleMicrophoneMuted: () => void,
toggleLocalVideoMuted: () => void,
setMicrophoneMuted: (muted: boolean) => void,
toggleAudio: (() => void) | null,
toggleVideo: (() => void) | null,
setAudioEnabled: ((enabled: boolean) => void) | null,
sendReaction: (reaction: ReactionOption) => void,
toggleHandRaised: () => void,
): void {
@@ -52,15 +52,15 @@ export function useCallViewKeyboardShortcuts(
if (event.key === "m") {
event.preventDefault();
toggleMicrophoneMuted();
} else if (event.key == "v") {
toggleAudio?.();
} else if (event.key === "v") {
event.preventDefault();
toggleLocalVideoMuted();
toggleVideo?.();
} else if (event.key === " ") {
event.preventDefault();
if (!spacebarHeld.current) {
spacebarHeld.current = true;
setMicrophoneMuted(false);
setAudioEnabled?.(true);
}
} else if (event.key === "h") {
event.preventDefault();
@@ -72,9 +72,9 @@ export function useCallViewKeyboardShortcuts(
},
[
focusElement,
toggleLocalVideoMuted,
toggleMicrophoneMuted,
setMicrophoneMuted,
toggleVideo,
toggleAudio,
setAudioEnabled,
sendReaction,
toggleHandRaised,
],
@@ -95,10 +95,10 @@ export function useCallViewKeyboardShortcuts(
if (event.key === " ") {
spacebarHeld.current = false;
setMicrophoneMuted(true);
setAudioEnabled?.(false);
}
},
[focusElement, setMicrophoneMuted],
[focusElement, setAudioEnabled],
),
);
@@ -108,8 +108,8 @@ export function useCallViewKeyboardShortcuts(
useCallback(() => {
if (spacebarHeld.current) {
spacebarHeld.current = false;
setMicrophoneMuted(true);
setAudioEnabled?.(true);
}
}, [setMicrophoneMuted, spacebarHeld]),
}, [setAudioEnabled, spacebarHeld]),
);
}

View File

@@ -1,40 +0,0 @@
/*
Copyright 2023, 2024 New Vector Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { logger } from "matrix-js-sdk/lib/logger";
import {
type MatrixRTCSession,
MatrixRTCSessionEvent,
} from "matrix-js-sdk/lib/matrixrtc";
import { TypedEventEmitter } from "matrix-js-sdk";
import { useCallback, useEffect } from "react";
import { useTypedEventEmitterState } from "./useEvents";
const dummySession = new TypedEventEmitter();
export function useMatrixRTCSessionJoinState(
rtcSession: MatrixRTCSession | undefined,
): boolean {
// React doesn't allow you to run a hook conditionally, so we have to plug in
// a dummy event emitter in case there is no rtcSession yet
const isJoined = useTypedEventEmitterState(
rtcSession ?? dummySession,
MatrixRTCSessionEvent.JoinStateChanged,
useCallback(() => rtcSession?.isJoined() ?? false, [rtcSession]),
);
useEffect(() => {
logger.info(
`Session in room ${rtcSession?.room.roomId} changed to ${
isJoined ? "joined" : "left"
}`,
);
}, [rtcSession, isJoined]);
return isJoined;
}

View File

@@ -65,7 +65,7 @@ export function shouldDisambiguate(
// displayname, after hidden character removal.
return (
memberships
.map((m) => m.sender && room.getMember(m.sender))
.map((m) => m.userId && room.getMember(m.userId))
// NOTE: We *should* have a room member for everyone.
.filter((m) => !!m)
.filter((m) => m.userId !== userId)

View File

@@ -11,7 +11,7 @@ export enum ErrorCode {
/**
* Configuration problem due to no MatrixRTC backend/SFU is exposed via .well-known and no fallback configured.
*/
MISSING_MATRIX_RTC_FOCUS = "MISSING_MATRIX_RTC_FOCUS",
MISSING_MATRIX_RTC_TRANSPORT = "MISSING_MATRIX_RTC_TRANSPORT",
CONNECTION_LOST_ERROR = "CONNECTION_LOST_ERROR",
/** LiveKit indicates that the server has hit its track limits */
INSUFFICIENT_CAPACITY_ERROR = "INSUFFICIENT_CAPACITY_ERROR",
@@ -54,18 +54,18 @@ export class ElementCallError extends Error {
}
}
export class MatrixRTCFocusMissingError extends ElementCallError {
export class MatrixRTCTransportMissingError extends ElementCallError {
public domain: string;
public constructor(domain: string) {
super(
t("error.call_is_not_supported"),
ErrorCode.MISSING_MATRIX_RTC_FOCUS,
ErrorCode.MISSING_MATRIX_RTC_TRANSPORT,
ErrorCategory.CONFIGURATION_ISSUE,
t("error.matrix_rtc_focus_missing", {
t("error.matrix_rtc_transport_missing", {
domain,
brand: import.meta.env.VITE_PRODUCT_NAME || "Element Call",
errorCode: ErrorCode.MISSING_MATRIX_RTC_FOCUS,
errorCode: ErrorCode.MISSING_MATRIX_RTC_TRANSPORT,
}),
);
this.domain = domain;

View File

@@ -9,7 +9,6 @@ import {
mockRtcMembership,
mockMatrixRoomMember,
mockRemoteParticipant,
mockLocalParticipant,
} from "./test";
export const localRtcMember = mockRtcMembership("@carol:example.org", "1111");
@@ -18,7 +17,7 @@ export const localRtcMemberDevice2 = mockRtcMembership(
"2222",
);
export const local = mockMatrixRoomMember(localRtcMember);
export const localParticipant = mockLocalParticipant({ identity: "" });
// export const localParticipant = mockLocalParticipant({ identity: "" });
export const localId = `${local.userId}:${localRtcMember.deviceId}`;
export const aliceRtcMember = mockRtcMembership("@alice:example.org", "AAAA");

View File

@@ -5,11 +5,7 @@ SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { ConnectionState } from "livekit-client";
import {
type CallMembership,
type MatrixRTCSession,
} from "matrix-js-sdk/lib/matrixrtc";
import { type CallMembership } from "matrix-js-sdk/lib/matrixrtc";
import { BehaviorSubject, of } from "rxjs";
import { vitest } from "vitest";
import { type RelationsContainer } from "matrix-js-sdk/lib/models/relations-container";
@@ -27,17 +23,12 @@ import {
type CallViewModelOptions,
} from "../state/CallViewModel";
import {
mockLivekitRoom,
mockMatrixRoom,
mockMediaDevices,
mockMuteStates,
MockRTCSession,
} from "./test";
import {
aliceRtcMember,
aliceParticipant,
localParticipant,
localRtcMember,
} from "./test-fixtures";
import { aliceRtcMember, localRtcMember } from "./test-fixtures";
import { type RaisedHandInfo, type ReactionInfo } from "../reactions";
export function getBasicRTCSession(
@@ -58,7 +49,7 @@ export function getBasicRTCSession(
getChildEventsForEvent: vitest.fn(),
} as Partial<RelationsContainer> as RelationsContainer,
client: {
getUserId: () => localRtcMember.sender,
getUserId: () => localRtcMember.userId,
getDeviceId: () => localRtcMember.deviceId,
getSyncState: () => SyncState.Syncing,
sendEvent: vitest.fn().mockResolvedValue({ event_id: "$fake:event" }),
@@ -105,12 +96,12 @@ export function getBasicRTCSession(
initialRtcMemberships,
);
const rtcSession = new MockRTCSession(matrixRoom).withMemberships(
const fakeRtcSession = new MockRTCSession(matrixRoom).withMemberships(
rtcMemberships$,
);
return {
rtcSession,
rtcSession: fakeRtcSession,
matrixRoom,
rtcMemberships$,
};
@@ -140,23 +131,20 @@ export function getBasicCallViewModelEnvironment(
const handRaisedSubject$ = new BehaviorSubject({});
const reactionsSubject$ = new BehaviorSubject({});
const remoteParticipants$ = of([aliceParticipant]);
const livekitRoom = mockLivekitRoom(
{ localParticipant },
{ remoteParticipants$ },
);
// const remoteParticipants$ = of([aliceParticipant]);
const vm = new CallViewModel(
rtcSession as unknown as MatrixRTCSession,
rtcSession.asMockedSession(),
matrixRoom,
livekitRoom,
mockMediaDevices({}),
mockMuteStates(),
{
encryptionSystem: { kind: E2eeType.PER_PARTICIPANT },
...callViewModelOptions,
},
of(ConnectionState.Connected),
handRaisedSubject$,
reactionsSubject$,
of({ processor: undefined, supported: false }),
);
return {
vm,

View File

@@ -6,7 +6,7 @@ Please see LICENSE in the repository root for full details.
*/
import { map, type Observable, of, type SchedulerLike } from "rxjs";
import { type RunHelpers, TestScheduler } from "rxjs/testing";
import { expect, vi, vitest } from "vitest";
import { expect, type MockedObject, vi, vitest } from "vitest";
import {
type RoomMember,
type Room as MatrixRoom,
@@ -16,17 +16,20 @@ import {
} from "matrix-js-sdk";
import {
CallMembership,
type Focus,
type Transport,
MatrixRTCSessionEvent,
type MatrixRTCSessionEventHandlerMap,
MembershipManagerEvent,
type SessionMembershipData,
Status,
type LivekitFocusSelection,
type MatrixRTCSession,
} from "matrix-js-sdk/lib/matrixrtc";
import { type MembershipManagerEventHandlerMap } from "matrix-js-sdk/lib/matrixrtc/IMembershipManager";
import {
type LocalParticipant,
type LocalTrackPublication,
type Participant,
type RemoteParticipant,
type RemoteTrackPublication,
type Room as LivekitRoom,
@@ -53,6 +56,7 @@ import { Config } from "../config/Config";
import { type MediaDevices } from "../state/MediaDevices";
import { type Behavior, constant } from "../state/Behavior";
import { ObservableScope } from "../state/ObservableScope";
import { MuteStates } from "../state/MuteStates";
export function withFakeTimers(continuation: () => void): void {
vi.useFakeTimers();
@@ -171,8 +175,11 @@ export function mockRtcMembership(
user: string | RoomMember,
deviceId: string,
callId = "",
fociPreferred: Focus[] = [],
focusActive: Focus = { type: "oldest_membership" },
fociPreferred: Transport[] = [],
focusActive: LivekitFocusSelection = {
type: "livekit",
focus_selection: "oldest_membership",
},
membership: Partial<SessionMembershipData> = {},
): CallMembership {
const data: SessionMembershipData = {
@@ -186,8 +193,12 @@ export function mockRtcMembership(
const event = new MatrixEvent({
sender: typeof user === "string" ? user : user.userId,
event_id: `$-ev-${randomUUID()}:example.org`,
content: data,
});
return new CallMembership(event, data);
const cms = new CallMembership(event, data);
vi.mocked(cms).getTransport = vi.fn().mockReturnValue(fociPreferred[0]);
return cms;
}
// Maybe it'd be good to move this to matrix-js-sdk? Our testing needs are
@@ -199,7 +210,11 @@ export function mockMatrixRoomMember(
): RoomMember {
return {
...mockEmitter(),
userId: rtcMembership.sender,
userId: rtcMembership.userId,
getMxcAvatarUrl(): string | undefined {
return undefined;
},
rawDisplayName: rtcMembership.userId,
...member,
} as RoomMember;
}
@@ -259,6 +274,7 @@ export async function withLocalMedia(
kind: E2eeType.PER_PARTICIPANT,
},
mockLivekitRoom({ localParticipant }),
"https://rtc-example.org",
mediaDevices,
constant(roomMember.rawDisplayName ?? "nodisplayname"),
constant(null),
@@ -299,6 +315,7 @@ export async function withRemoteMedia(
kind: E2eeType.PER_PARTICIPANT,
},
mockLivekitRoom({}, { remoteParticipants$: of([remoteParticipant]) }),
"https://rtc-example.org",
constant(false),
constant(roomMember.rawDisplayName ?? "nodisplayname"),
constant(null),
@@ -326,6 +343,19 @@ export class MockRTCSession extends TypedEventEmitter<
RoomAndToDeviceEventsHandlerMap &
MembershipManagerEventHandlerMap
> {
public asMockedSession(): MockedObject<MatrixRTCSession> {
const session = this as unknown as MockedObject<MatrixRTCSession>;
vi.mocked(session).reemitEncryptionKeys = vi
.fn<() => void>()
.mockReturnValue(undefined);
vi.mocked(session).getOldestMembership = vi
.fn<() => CallMembership | undefined>()
.mockReturnValue(this.memberships[0]);
return session;
}
public readonly statistics = {
counters: {},
};
@@ -360,6 +390,8 @@ export class MockRTCSession extends TypedEventEmitter<
return this;
}
public updateCallIntent = vitest.fn();
private _membershipStatus = Status.Connected;
public get membershipStatus(): Status {
return this._membershipStatus;
@@ -382,15 +414,17 @@ export class MockRTCSession extends TypedEventEmitter<
}
}
export const mockTrack = (identity: string): TrackReference =>
export const mockTrack = (
participant: Participant,
kind?: Track.Kind,
source?: Track.Source,
): TrackReference =>
({
participant: {
identity,
},
participant,
publication: {
kind: Track.Kind.Audio,
source: "mic",
trackSid: "123",
kind: kind ?? Track.Kind.Audio,
source: source ?? Track.Source.Microphone,
trackSid: `123##${participant.identity}`,
track: {
attach: vi.fn(),
detach: vi.fn(),
@@ -417,3 +451,10 @@ export function mockMediaDevices(data: Partial<MediaDevices>): MediaDevices {
...data,
} as MediaDevices;
}
export function mockMuteStates(
joined$: Observable<boolean> = of(true),
): MuteStates {
const observableScope = new ObservableScope();
return new MuteStates(observableScope, mockMediaDevices({}), joined$);
}

View File

@@ -97,6 +97,9 @@ export default ({
cert: fs.readFileSync("./backend/dev_tls_m.localhost.crt"),
},
},
worker: {
format: "es",
},
build: {
minify: mode === "production" ? true : false,
sourcemap: true,

111
yarn.lock
View File

@@ -2793,10 +2793,10 @@ __metadata:
languageName: node
linkType: hard
"@matrix-org/matrix-sdk-crypto-wasm@npm:^15.1.0":
version: 15.1.0
resolution: "@matrix-org/matrix-sdk-crypto-wasm@npm:15.1.0"
checksum: 10c0/19edc6d0045ff49fad8d77b6e561cee994f7513f8c18a7176ae2d3f0116c1a91980e02d10300b09c2b72dea4da4a8c3392f2bf1752057f2d6b53030a056d76d8
"@matrix-org/matrix-sdk-crypto-wasm@npm:^15.3.0":
version: 15.3.0
resolution: "@matrix-org/matrix-sdk-crypto-wasm@npm:15.3.0"
checksum: 10c0/45628f36b7b0e54a8777ae67a7233dbdf3e3cf14e0d95d21f62f89a7ea7e3f907232f1eb7b1262193b1e227759fad47af829dcccc103ded89011f13c66f01d76
languageName: node
linkType: hard
@@ -5176,6 +5176,13 @@ __metadata:
languageName: node
linkType: hard
"@types/glob-to-regexp@npm:^0.4.4":
version: 0.4.4
resolution: "@types/glob-to-regexp@npm:0.4.4"
checksum: 10c0/7288ff853850d8302a8770a3698b187fc3970ad12ee6427f0b3758a3e7a0ebb0bd993abc6ebaaa979d09695b4194157d2bfaa7601b0fb9ed72c688b4c1298b88
languageName: node
linkType: hard
"@types/grecaptcha@npm:^3.0.9":
version: 3.0.9
resolution: "@types/grecaptcha@npm:3.0.9"
@@ -5290,13 +5297,6 @@ __metadata:
languageName: node
linkType: hard
"@types/retry@npm:0.12.0":
version: 0.12.0
resolution: "@types/retry@npm:0.12.0"
checksum: 10c0/7c5c9086369826f569b83a4683661557cab1361bac0897a1cefa1a915ff739acd10ca0d62b01071046fe3f5a3f7f2aec80785fe283b75602dc6726781ea3e328
languageName: node
linkType: hard
"@types/sdp-transform@npm:^2.4.5":
version: 2.4.10
resolution: "@types/sdp-transform@npm:2.4.10"
@@ -7535,6 +7535,7 @@ __metadata:
eslint-plugin-react-hooks: "npm:^5.0.0"
eslint-plugin-rxjs: "npm:^5.0.3"
eslint-plugin-unicorn: "npm:^56.0.0"
fetch-mock: "npm:11.1.5"
global-jsdom: "npm:^26.0.0"
i18next: "npm:^24.0.0"
i18next-browser-languagedetector: "npm:^8.0.0"
@@ -7544,7 +7545,7 @@ __metadata:
livekit-client: "npm:^2.13.0"
lodash-es: "npm:^4.17.21"
loglevel: "npm:^1.9.1"
matrix-js-sdk: "github:matrix-org/matrix-js-sdk#head=develop"
matrix-js-sdk: "github:matrix-org/matrix-js-sdk#head=toger5/sticky-events&commit=e7f5bec51b6f70501a025b79fe5021c933385b21"
matrix-widget-api: "npm:^1.13.0"
normalize.css: "npm:^8.0.1"
observable-hooks: "npm:^4.2.3"
@@ -8502,6 +8503,22 @@ __metadata:
languageName: node
linkType: hard
"fetch-mock@npm:11.1.5":
version: 11.1.5
resolution: "fetch-mock@npm:11.1.5"
dependencies:
"@types/glob-to-regexp": "npm:^0.4.4"
dequal: "npm:^2.0.3"
glob-to-regexp: "npm:^0.4.1"
is-subset: "npm:^0.1.1"
regexparam: "npm:^3.0.0"
peerDependenciesMeta:
node-fetch:
optional: true
checksum: 10c0/f32f1d7879b654a3fab7c3576901193ddd4c63cb9aeae2ed66ff42062400c0937d4696b1a5171e739d5f62470e6554e190f14816789f5e3b2bf1ad90208222e6
languageName: node
linkType: hard
"fflate@npm:^0.4.8":
version: 0.4.8
resolution: "fflate@npm:0.4.8"
@@ -8883,6 +8900,13 @@ __metadata:
languageName: node
linkType: hard
"glob-to-regexp@npm:^0.4.1":
version: 0.4.1
resolution: "glob-to-regexp@npm:0.4.1"
checksum: 10c0/0486925072d7a916f052842772b61c3e86247f0a80cc0deb9b5a3e8a1a9faad5b04fb6f58986a09f34d3e96cd2a22a24b7e9882fb1cf904c31e9a310de96c429
languageName: node
linkType: hard
"glob@npm:^10.2.2, glob@npm:^10.3.10, glob@npm:^10.3.7, glob@npm:^10.4.1":
version: 10.4.5
resolution: "glob@npm:10.4.5"
@@ -9528,6 +9552,13 @@ __metadata:
languageName: node
linkType: hard
"is-network-error@npm:^1.1.0":
version: 1.3.0
resolution: "is-network-error@npm:1.3.0"
checksum: 10c0/3e85a69e957988db66d5af5412efdd531a5a63e150d1bdd5647cfd4dc54fd89b1dbdd472621f8915233c3176ba1e6922afa8a51a9e363ba4693edf96a294f898
languageName: node
linkType: hard
"is-number-object@npm:^1.1.1":
version: 1.1.1
resolution: "is-number-object@npm:1.1.1"
@@ -9611,6 +9642,13 @@ __metadata:
languageName: node
linkType: hard
"is-subset@npm:^0.1.1":
version: 0.1.1
resolution: "is-subset@npm:0.1.1"
checksum: 10c0/d8125598ab9077a76684e18726fb915f5cea7a7358ed0c6ff723f4484d71a0a9981ee5aae06c44de99cfdef0fefce37438c6257ab129e53c82045ea0c2acdebf
languageName: node
linkType: hard
"is-symbol@npm:^1.0.4, is-symbol@npm:^1.1.1":
version: 1.1.1
resolution: "is-symbol@npm:1.1.1"
@@ -10297,12 +10335,12 @@ __metadata:
languageName: node
linkType: hard
"matrix-js-sdk@github:matrix-org/matrix-js-sdk#head=develop":
version: 37.13.0
resolution: "matrix-js-sdk@https://github.com/matrix-org/matrix-js-sdk.git#commit=2f1d654f14be8dd03896e9e76f12017b6f9eec1c"
"matrix-js-sdk@github:matrix-org/matrix-js-sdk#head=toger5/sticky-events&commit=e7f5bec51b6f70501a025b79fe5021c933385b21":
version: 38.4.0
resolution: "matrix-js-sdk@https://github.com/matrix-org/matrix-js-sdk.git#commit=e7f5bec51b6f70501a025b79fe5021c933385b21"
dependencies:
"@babel/runtime": "npm:^7.12.5"
"@matrix-org/matrix-sdk-crypto-wasm": "npm:^15.1.0"
"@matrix-org/matrix-sdk-crypto-wasm": "npm:^15.3.0"
another-json: "npm:^0.2.0"
bs58: "npm:^6.0.0"
content-type: "npm:^1.0.4"
@@ -10311,11 +10349,11 @@ __metadata:
matrix-events-sdk: "npm:0.0.1"
matrix-widget-api: "npm:^1.10.0"
oidc-client-ts: "npm:^3.0.1"
p-retry: "npm:4"
p-retry: "npm:7"
sdp-transform: "npm:^2.14.1"
unhomoglyph: "npm:^1.0.6"
uuid: "npm:11"
checksum: 10c0/ecd019c677c272c5598617dcde407dbe4b1b11460863b2a577e33f3fd8732c9d9073ec0221b471ec1eb24e2839eec20728db7f92c9348be83126547286e50805
uuid: "npm:13"
checksum: 10c0/7adffdc183affd2d3ee1e8497cad6ca7904a37f98328ff7bc15aa6c1829dc9f9a92f8e1bd6260432a33626ff2a839644de938270163e73438b7294675cd954e4
languageName: node
linkType: hard
@@ -10922,13 +10960,12 @@ __metadata:
languageName: node
linkType: hard
"p-retry@npm:4":
version: 4.6.2
resolution: "p-retry@npm:4.6.2"
"p-retry@npm:7":
version: 7.0.0
resolution: "p-retry@npm:7.0.0"
dependencies:
"@types/retry": "npm:0.12.0"
retry: "npm:^0.13.1"
checksum: 10c0/d58512f120f1590cfedb4c2e0c42cb3fa66f3cea8a4646632fcb834c56055bb7a6f138aa57b20cc236fb207c9d694e362e0b5c2b14d9b062f67e8925580c73b0
is-network-error: "npm:^1.1.0"
checksum: 10c0/3c090ac72bbe00fd2f062ee6178c44f7302f298936ab2290a458575e73650e7834b556beb2b09fa9fbebedab2ec3358cb474c09710cf828972b670c3c0cb89e4
languageName: node
linkType: hard
@@ -12044,6 +12081,13 @@ __metadata:
languageName: node
linkType: hard
"regexparam@npm:^3.0.0":
version: 3.0.0
resolution: "regexparam@npm:3.0.0"
checksum: 10c0/a6430d7b97d5a7d5518f37a850b6b73aab479029d02f46af4fa0e8e4a1d7aad05b7a0d2d10c86ded21a14d5f0fa4c68525f873a5fca2efeefcccd93c36627459
languageName: node
linkType: hard
"regexpu-core@npm:^6.2.0":
version: 6.2.0
resolution: "regexpu-core@npm:6.2.0"
@@ -12230,13 +12274,6 @@ __metadata:
languageName: node
linkType: hard
"retry@npm:^0.13.1":
version: 0.13.1
resolution: "retry@npm:0.13.1"
checksum: 10c0/9ae822ee19db2163497e074ea919780b1efa00431d197c7afdb950e42bf109196774b92a49fc9821f0b8b328a98eea6017410bfc5e8a0fc19c85c6d11adb3772
languageName: node
linkType: hard
"reusify@npm:^1.0.4":
version: 1.1.0
resolution: "reusify@npm:1.1.0"
@@ -13766,12 +13803,12 @@ __metadata:
languageName: node
linkType: hard
"uuid@npm:11":
version: 11.0.5
resolution: "uuid@npm:11.0.5"
"uuid@npm:13":
version: 13.0.0
resolution: "uuid@npm:13.0.0"
bin:
uuid: dist/esm/bin/uuid
checksum: 10c0/6f59f0c605e02c14515401084ca124b9cb462b4dcac866916a49862bcf831874508a308588c23a7718269226ad11a92da29b39d761ad2b86e736623e3a33b6e7
uuid: dist-node/bin/uuid
checksum: 10c0/950e4c18d57fef6c69675344f5700a08af21e26b9eff2bf2180427564297368c538ea11ac9fb2e6528b17fc3966a9fd2c5049361b0b63c7d654f3c550c9b3d67
languageName: node
linkType: hard