Run the media backend inside a combination of backends

The participation now builds its default backend by combining every backend
the SDK has, publishing on the type of the transport the host names (LiveKit
without one) and handing each remote member to the backend of its transport.
LiveKit is the only member of the combination for now. A backend serves a
list of transport types, so the combination routes every type of every
backend.

The LiveKit backend can run without a local transport, so it subscribes only
while another backend publishes, and probes the homeserver for delayed leave
delegation only when it first publishes. The combination tells the backends
that don't publish, with a null transport, and until a backend knows either
way it connects nowhere: a subscriber connection to the transport it is
about to publish on would only be replaced.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

sdk/SdkArchitecture.md

@@ -66,9 +66,13 @@ media backend: whatever carries the media, behind the `MediaBackend` interface i
 `sdk/src/media-backend/api.ts`. The client is the join of the two: a member exists
 once its membership does, and gets its media once the backend has something
 for that membership on its transport; `state$` is the membership state combined
-with the local media connection. LiveKit is the one backend today; a
-cascading SFU or full mesh is another folder under `sdk/src/media-backend/`, handed
-in through `options.backend`, and nothing in the left box changes.
+with the local media connection. LiveKit (`livekit/`) is the one backend today,
+and the client runs it inside a combination (`combineBackends.ts`). It publishes
+on the backend of the type of `options.transport`, LiveKit without one, and
+hands every remote member to the backend of that member's transport. A backend
+nobody is on connects nowhere. A cascading SFU or full mesh is another folder
+under `sdk/src/media-backend/`, added to the combination, and nothing in the
+left box changes; `options.backend` replaces the combination altogether.
 
 ### How Element Call uses it
 
@@ -154,7 +158,7 @@ the sections below still describe `MatrixRTCClient` until that lands.
 | publish request             | a `PublishRequest`: a source, where to capture it from and how to encode it. Given at the join in `publish`, or later to `publish` on the participation                                                                                                |                                                                                                             |
 | attach, detach              | handing a `<video>` or `<audio>` element to a track and taking it back. The SDK owns the stream and the observers on the element                                                                                                                       | LiveKit `Track.attach`                                                                                      |
 | transport                   | an `RTCTransport`: where media is exchanged, a LiveKit service url today. Carries the raw js-sdk `Transport` from the membership and, in `resolved$`, what the backend fetched to connect to it                                                        | MSC4143 "focus"                                                                                             |
-| media backend               | a `MediaBackend`: what carries the media, behind `sdk/src/media-backend/api.ts`. One per participation, chosen with `backend` in the join options; LiveKit under `sdk/src/media-backend/livekit/` is the one that exists                               | "SFU", a LiveKit room, a peer connection                                                                    |
+| media backend               | a `MediaBackend`: what carries the media, behind `sdk/src/media-backend/api.ts`. One per participation: the SDK's backends combined, publishing on the type of `transport` in the join options, unless `backend` replaces them                         | "SFU", a LiveKit room, a peer connection                                                                    |
 
 Names use `RTC` in capitals, as the js-sdk does: `RTCSlot`, `RTCParticipation`,
 `RTCMember`, `matrixRTCMode`.

sdk/src/RTCParticipation.ts

@@ -50,6 +50,7 @@ import {
   type MediaBackendContext,
   type MediaKey,
 } from "./media-backend/api";
+import { combineBackends } from "./media-backend/combineBackends";
 import { createLivekitBackend } from "./media-backend/livekit/LivekitBackend";
 import { findRemotePublishLivekitTransport } from "./media-backend/livekit/RemotePublishTransport";
 import { createHomeserverConnected$ } from "./matrixrtc/HomeserverConnected";
@@ -311,15 +312,23 @@ export function createRTCParticipation(
   };
 }
 
-/** The host's backend, or LiveKit configured from the participation options. */
+/**
+ * The host's backend, or every backend the SDK has, combined: we publish on
+ * the type of the transport the host named (LiveKit without one) and receive
+ * from members on any of them.
+ */
 function createBackend(
   scope: ObservableScope,
   client: MatrixClient,
-  { backend, mediaQuality }: RTCParticipationOptions,
+  { backend, mediaQuality, transport }: RTCParticipationOptions,
   context: MediaBackendContext,
 ): MediaBackend {
   if (backend) return backend(scope, context);
-  return createLivekitBackend(scope, context, { client, mediaQuality });
+  return combineBackends(
+    scope,
+    [createLivekitBackend(scope, context, { client, mediaQuality })],
+    transport?.type ?? "m.livekit",
+  );
 }
 
 /** The slot's per-participant keys: the ones it already holds on subscribe, then each new one. */

sdk/src/media-backend/api.ts

@@ -125,8 +125,15 @@ export interface MediaBackend {
    * session found it; the backend does whatever has to succeed before it is
    * advertised in the membership. Rejects with the error the session reports
    * as fatal. Called once per session.
+   *
+   * `null` tells the backend that the local member publishes on another
+   * backend: it only subscribes from then on. Until either call, a backend
+   * that would connect for its publisher connects nowhere, so that it doesn't
+   * subscribe to a transport it is about to publish on.
    */
-  prepareLocalTransport(transport: Transport): Promise<TransportCapabilities>;
+  prepareLocalTransport(
+    transport: Transport | null,
+  ): Promise<TransportCapabilities>;
 
   /**
    * Hands the delayed leave event to the backend, which restarts it from

sdk/src/media-backend/combineBackends.test.ts

@@ -0,0 +1,103 @@
+/*
+Copyright 2026 Element Creations Ltd.
+
+SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
+Please see LICENSE in the repository root for full details.
+*/
+
+import { describe, expect, it, vi } from "vitest";
+import { BehaviorSubject, NEVER } from "rxjs";
+
+import { type RTCMember, type TransportMetadata } from "../api";
+import { type MemberMedia } from "../media-api";
+import { constant } from "../reactive/Behavior";
+import { ObservableScope } from "../reactive/ObservableScope";
+import { type MediaBackend } from "./api";
+import { combineBackends } from "./combineBackends";
+
+function fakeBackend(
+  type: string,
+  ...moreTypes: string[]
+): MediaBackend & { ended: string[] } {
+  const ended: string[] = [];
+  return {
+    ended,
+    transportTypes: [type, ...moreTypes],
+    prepareLocalTransport: vi.fn(async () =>
+      Promise.resolve({ canDelegateDelayedLeave: false }),
+    ),
+    delegateDelayedLeave: vi.fn(),
+    local: {} as MediaBackend["local"],
+    mediaFor: (scope, member): MemberMedia => {
+      scope.onEnd(() => ended.push(member.memberId));
+      return {
+        tracks$: constant([{ id: `${type}:${member.memberId}` }] as never),
+        encryptionError$: NEVER,
+      };
+    },
+    connections$: constant([
+      { transport: { type }, state: "connected" as never, resolved: undefined },
+    ]),
+    setAudioOutputDeviceId: vi.fn(),
+    sendData: vi.fn(),
+    data$: NEVER,
+  };
+}
+
+function transport(type: string): TransportMetadata {
+  return { type, id: type, raw: { type }, resolved$: constant(undefined) };
+}
+
+describe("combineBackends", () => {
+  it("routes each member to the backend of its transport type", () => {
+    const scope = new ObservableScope();
+    const livekit = fakeBackend("m.livekit", "livekit");
+    const other = fakeBackend("other");
+    const combined = combineBackends(scope, [livekit, other], "livekit");
+    expect(combined.transportTypes).toEqual(["m.livekit", "livekit"]);
+    expect(combined.connections$.value).toHaveLength(2);
+
+    const transport$ = new BehaviorSubject<TransportMetadata | undefined>(
+      transport("other"),
+    );
+    const memberScope = new ObservableScope();
+    const media = combined.mediaFor(memberScope, {
+      memberId: "m1",
+      transport$,
+    } as unknown as RTCMember);
+    expect(media.tracks$.value).toEqual([{ id: "other:m1" }]);
+
+    // Either type of the publishing backend stays with it
+    transport$.next(transport("livekit"));
+    expect(other.ended).toEqual(["m1"]);
+    expect(media.tracks$.value).toEqual([{ id: "m.livekit:m1" }]);
+    transport$.next(transport("m.livekit"));
+    expect(media.tracks$.value).toEqual([{ id: "m.livekit:m1" }]);
+
+    transport$.next(transport("unknown"));
+    expect(media.tracks$.value).toBeNull();
+    // The publishing backend keeps every member for as long as it is one
+    expect(livekit.ended).toEqual([]);
+    memberScope.end();
+    expect(livekit.ended).toEqual(["m1"]);
+    scope.end();
+  });
+
+  it("publishes on the backend of the publish type, telling the others", async () => {
+    const scope = new ObservableScope();
+    const livekit = fakeBackend("livekit");
+    const other = fakeBackend("other");
+    const combined = combineBackends(scope, [livekit, other], "other");
+    expect(combined.transportTypes).toEqual(["other"]);
+    const transport = { type: "other" };
+    await combined.prepareLocalTransport(transport);
+    expect(other.prepareLocalTransport).toHaveBeenCalledExactlyOnceWith(
+      transport,
+    );
+    expect(livekit.prepareLocalTransport).toHaveBeenCalledExactlyOnceWith(null);
+    expect(() => combineBackends(scope, [livekit, other], "unknown")).toThrow(
+      "No media backend for transport unknown",
+    );
+    scope.end();
+  });
+});

sdk/src/media-backend/combineBackends.ts

@@ -0,0 +1,90 @@
+/*
+Copyright 2026 Element Creations Ltd.
+
+SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
+Please see LICENSE in the repository root for full details.
+*/
+
+import { combineLatest, map, merge, NEVER, of, switchMap } from "rxjs";
+
+import { type RTCMember } from "../api";
+import { MatrixRTCError } from "../errors";
+import { type MemberMedia } from "../media-api";
+import { type ObservableScope } from "../reactive/ObservableScope";
+import { mapScoped } from "../utils/mapScoped";
+import { type MediaBackend } from "./api";
+
+/**
+ * Several backends as one: we publish on the backend of `publishType`, and
+ * receive from every member on a transport any of the backends serves. A
+ * member is handed to the backend of its current transport type. A backend
+ * nobody is on costs nothing: it connects only for the members handed to it.
+ */
+export function combineBackends(
+  scope: ObservableScope,
+  all: MediaBackend[],
+  publishType: string,
+): MediaBackend {
+  const byType = new Map(
+    all.flatMap((b) => b.transportTypes.map((type) => [type, b] as const)),
+  );
+  const publish = byType.get(publishType);
+  if (publish === undefined)
+    throw new MatrixRTCError(`No media backend for transport ${publishType}`);
+
+  const mediaFor = (
+    memberScope: ObservableScope,
+    member: RTCMember,
+  ): MemberMedia => {
+    // The publishing backend sees every member, whatever its transport, so
+    // that it can tell members from strangers when a backend is asked for our
+    // media by members on other transports. It connects nowhere for them
+    const publishMedia = publish.mediaFor(memberScope, member);
+    const type$ = memberScope.behavior(
+      member.transport$.pipe(map((transport) => transport?.type)),
+    );
+    const media$ = mapScoped(memberScope, type$, (s, type) =>
+      type !== undefined && publish.transportTypes.includes(type)
+        ? publishMedia
+        : byType.get(type)?.mediaFor(s, member),
+    );
+    return {
+      tracks$: memberScope.behavior(
+        media$.pipe(switchMap((media) => media?.tracks$ ?? of(null))),
+      ),
+      encryptionError$: media$.pipe(
+        switchMap((media) => media?.encryptionError$ ?? NEVER),
+      ),
+    };
+  };
+
+  return {
+    transportTypes: publish.transportTypes,
+    // The others learn that they only subscribe
+    prepareLocalTransport: async (transport) => {
+      const [capabilities] = await Promise.all([
+        publish.prepareLocalTransport(transport),
+        ...all
+          .filter((b) => b !== publish)
+          .map(async (b) => b.prepareLocalTransport(null)),
+      ]);
+      return capabilities;
+    },
+    delegateDelayedLeave: async (delayId) =>
+      publish.delegateDelayedLeave(delayId),
+    local: publish.local,
+    mediaFor,
+    connections$: scope.behavior(
+      combineLatest(all.map((b) => b.connections$)).pipe(
+        map((lists) => lists.flat()),
+      ),
+    ),
+    setAudioOutputDeviceId: async (deviceId) => {
+      await Promise.all(
+        all.map(async (b) => b.setAudioOutputDeviceId(deviceId)),
+      );
+    },
+    sendData: async (topic, text) => publish.sendData(topic, text),
+    data$: merge(...all.map((b) => b.data$)),
+  };
+}

sdk/src/media-backend/livekit/ConnectionMap.ts

@@ -108,8 +108,11 @@ export class ConnectionMap {
 interface Props {
   scope: ObservableScope;
   connectionFactory: ConnectionFactory;
-  /** The transport we publish on; the one connection with the publisher role. */
-  localTransport: TransportLocator;
+  /**
+   * The transport we publish on; the one connection with the publisher role.
+   * Null for a backend that only subscribes, while another backend publishes.
+   */
+  localTransport: TransportLocator | null;
   /** Every transport some member is on, deduplicated against the local one. */
   remoteTransports$: Behavior<Epoch<TransportLocator[]>>;
   logger: Logger;
@@ -135,7 +138,11 @@ export function createConnectionMap$({
     scope.behavior(
       remoteTransports$.pipe(
         mapEpoch((remoteTransports) =>
-          removeDuplicateTransports([localTransport, ...remoteTransports]),
+          removeDuplicateTransports(
+            localTransport
+              ? [localTransport, ...remoteTransports]
+              : remoteTransports,
+          ),
         ),
         tap((transports) =>
           logger.debug("localAndRemoteTransports$ = ", transports),
@@ -143,7 +150,7 @@ export function createConnectionMap$({
       ),
     );
 
-  const localTransportKey = keyFor(localTransport);
+  const localTransportKey = localTransport && keyFor(localTransport);
 
   const connections$ = scope.behavior(
     localAndRemoteTransports$.pipe(

sdk/src/media-backend/livekit/LivekitBackend.test.ts

@@ -475,6 +475,38 @@ describe("createLivekitBackend", () => {
       ]);
     });
 
+    it("connects nowhere until it knows whether it publishes", async () => {
+      const b = backend();
+      const media = b.mediaFor(testScope(), bob(bothTransport, [legacyId]));
+      await flushPromises();
+      expect(media.tracks$.value).toBeNull();
+      // A subscriber connection now would be replaced by the publisher one
+      expect(connections).toHaveLength(0);
+
+      fetchMock.post(homeserverEndpoint, 404);
+      fetchMock.post(`${exampleServiceUrl}/delegate_delayed_leave`, 404);
+      await b.prepareLocalTransport(bothTransport);
+      await flushPromises();
+      expect(connections.map((c) => c.connection.role)).toEqual(["publisher"]);
+    });
+
+    it("only subscribes once told that another backend publishes", async () => {
+      const b = backend();
+      b.mediaFor(testScope(), bob(bothTransport, [legacyId]));
+      await flushPromises();
+      expect(connections).toHaveLength(0);
+
+      await expect(b.prepareLocalTransport(null)).resolves.toEqual({
+        canDelegateDelayedLeave: false,
+      });
+      await flushPromises();
+      expect(connections.map((c) => c.connection.role)).toEqual(["subscriber"]);
+      expect(connections[0].connection.transport).toEqual(bothTransport);
+      await expect(b.delegateDelayedLeave("id")).rejects.toThrow(
+        "No local transport",
+      );
+    });
+
     it("has no media for a member without a transport, and opens no connection", async () => {
       const b = await prepared();
       const media = b.mediaFor(testScope(), {

sdk/src/media-backend/livekit/LivekitBackend.ts

@@ -106,7 +106,11 @@ export function createLivekitBackend(
     : [requestsOrMedia.requests, requestsOrMedia];
   const keyProvider = createKeyProvider(encryptionSystem, mediaKeys$, logger);
   const transports = createTransportRegistry(scope);
-  const localTransport$ = new BehaviorSubject<TransportLocator | null>(null);
+  // Undefined until the session says whether we publish here: null where
+  // another backend publishes
+  const localTransport$ = new BehaviorSubject<
+    TransportLocator | null | undefined
+  >(undefined);
   const audioOutputDeviceId$ = new BehaviorSubject<string | undefined>(
     undefined,
   );
@@ -121,9 +125,20 @@ export function createLivekitBackend(
   );
 
   // A new map, in a scope of its own, whenever the local transport changes:
-  // the publisher role sits on one connection, and which one is fixed per map
+  // the publisher role sits on one connection, and which one is fixed per map.
+  // Without a local transport the map only subscribes, for when another
+  // backend publishes; boxed so that `mapScoped` builds a map for null too.
+  // None at all while it isn't known yet: a subscriber connection to the
+  // transport we are about to publish on would only be torn down again
+  const boxedLocalTransport$ = scope.behavior(
+    localTransport$.pipe(
+      map((localTransport) =>
+        localTransport === undefined ? undefined : { localTransport },
+      ),
+    ),
+  );
   const connectionMap$ = scope.behavior<Epoch<ConnectionMap>>(
-    mapScoped(scope, localTransport$, (mapScope, localTransport) =>
+    mapScoped(scope, boxedLocalTransport$, (mapScope, { localTransport }) =>
       createConnectionMap$({
         scope: mapScope,
         connectionFactory,
@@ -138,7 +153,9 @@ export function createLivekitBackend(
   const localMedia_ = createLivekitLocalMedia({
     scope,
     connectionMap$,
-    localTransport$,
+    localTransport$: scope.behavior(
+      localTransport$.pipe(map((localTransport) => localTransport ?? null)),
+    ),
     publish,
     media,
     audioOutputDeviceId$,
@@ -146,18 +163,25 @@ export function createLivekitBackend(
     logger,
   });
 
-  // Probed once: the answer cannot change within a session
-  const homeserverSupportsDelegation = supportsDelegation(
-    `${client.baseUrl}/_matrix/client/unstable/io.element.msc4195/rtc/livekit/delegate_delayed_leave`,
-    "homeserver",
-    logger,
-  );
+  // Probed once, when first needed: the answer cannot change within a
+  // session, and a backend that only subscribes never needs it
+  let homeserverProbe: Promise<boolean> | undefined;
+  const homeserverSupportsDelegation = async (): Promise<boolean> =>
+    (homeserverProbe ??= supportsDelegation(
+      `${client.baseUrl}/_matrix/client/unstable/io.element.msc4195/rtc/livekit/delegate_delayed_leave`,
+      "homeserver",
+      logger,
+    ));
 
   const ownServerName = serverNameOf(ownMembershipIdentity.userId);
 
   const prepareLocalTransport = async (
-    transport: Transport,
+    transport: Transport | null,
   ): Promise<TransportCapabilities> => {
+    if (transport === null) {
+      localTransport$.next(null);
+      return { canDelegateDelayedLeave: false };
+    }
     if (!isUnstableLivekitTransport(transport))
       throw new MatrixRTCError(`Not a LiveKit transport: ${transport.type}`);
     const serverName = serverNameOfTransport(
@@ -182,7 +206,7 @@ export function createLivekitBackend(
               `transport ${flow.serviceUrl}`,
               logger,
             )
-          : homeserverSupportsDelegation;
+          : homeserverSupportsDelegation();
     }
     localTransport$.next(locator);
     // The membership is only sent once the publisher connection holds its
@@ -198,7 +222,7 @@ export function createLivekitBackend(
 
   const delegateLeave = async (delayId: string): Promise<void> => {
     const local = localTransport$.value;
-    if (local === null)
+    if (!local)
       throw new MatrixRTCError("No local transport to delegate the leave to");
     await delegateDelayedLeave({
       client,