Media & SFU (mediasoup)

TL;DR -A live room is one broadcaster streaming to many viewers. A media server called an SFU takes the host's video once and relays it out to everyone, so hosting a room with 5 or 5,000 viewers costs the host the same upload. This media travels its own path over UDP and does not go through nginx.

Who this is for -Product & Business: read "Why an SFU" for the concept. Engineering, Backend & DevOps: the session flow diagram, permissions, simulcast, the exact ports, and the producer/consumer lifecycle.

Why an SFU

Imagine a host with 500 viewers. If the host had to send their video directly to each viewer, they'd need 500 copies of their upload -impossible on a phone. Instead, the host sends their stream once to a server, and that server forwards a copy to each viewer. That server is an SFU (Selective Forwarding Unit): a media server that receives one broadcaster's audio/video and relays it out to many viewers. The host's upload cost stays constant no matter how big the audience gets.

Social Live's SFU is mediasoup, running inside the backend.

Live rooms are one-to-many broadcasts. A Selective Forwarding Unit (SFU) receives each publisher's media once and forwards it to every viewer, so a host's upload cost is constant regardless of audience size.

Components

Three pieces cooperate: the media server on the backend, the media client in the app, and the signaling channel that lets them set up a connection.

• Server: mediasoup workers inside the backend container, managed by LiveMediaService (backend/src/modules/live/live-media.service.ts), initialized at module startup.
• Client: flutter_webrtc + mediasfu_mediasoup_client in lib/modules/media_webrtc/, driven by the live-room controller.
• Signaling: all mediasoup negotiation (router capabilities, transport creation, produce/consume) rides over the Socket.IO live gateway.

The media session flow

Before any video flows, the app and the SFU perform a handshake to set up a connection (a "transport") and agree on what's being sent. In mediasoup terms, a publisher creates a producer (an outgoing track) and a viewer creates a consumer (an incoming track). The diagram below traces that handshake, from a host starting a broadcast to a viewer receiving it.

Rendering diagram…

Publish permissions

Not everyone in a room may broadcast. Only the host and guests the host has invited onto the stage can send media; everyone else is receive-only.

LiveService decides who may produce media: hosts and approved stage guests only. Viewers get receive-only sessions. Guests choose audio-only or audio+video when accepting a stage invite; the choice constrains which producers they may create.

Simulcast (adaptive quality)

Viewers have wildly different connections -some on fast Wi-Fi, some on a weak cell signal. Simulcast lets a broadcaster send several quality levels of their video at once so the SFU can hand each viewer the best one their connection can actually carry, rather than freezing everyone at the top quality.

Video publishers can send a 3-layer simulcast ladder so the SFU can serve each viewer the spatial layer their connection can carry -low-bandwidth viewers get a watchable stream instead of a frozen top layer.

• Admin toggle: simulcast_enabled app setting (default off), flipped from the admin portal Settings page or PUT /api/admin/settings/simulcast. Runtime setting -no deploy needed. Enabled in production and staging since 2026-07-16.
• Flow: GET /live/:id/media/rtp-capabilities returns simulcastEnabled; the client then publishes 3 encodings (or one flat encoding when off). The backend defensively clips multi-layer encodings from stale clients while the toggle is off.
• Layer selection: simulcast consumers start on the middle spatial layer and step up/down from mediasoup's RTP-quality score with a 5 s dwell time to avoid flapping.

Network specifics

Media takes a different network path than everything else, and it needs specific ports open. UDP is the lightweight, low-latency protocol real-time media uses (unlike the reliable-but-slower TCP behind web requests). These settings tell the SFU what address to bind to and which ports to use.

Media does not pass through nginx -only HTTPS/WSS does. If viewers connect but see black video, the UDP range or SFU_ANNOUNCED_IP is the first thing to check.

Video orientation (CVO disabled)

Getting video to appear right-side-up across iOS and Android took a deliberate fix. Phones can send video "sideways" along with a flag telling the viewer how to rotate it -a header extension called CVO (Coordination of Video Orientation). Our consume path didn't reliably apply that flag, so viewers saw rotated video. The fix: turn CVO off and have the sender physically rotate each frame upright before encoding.

The client strips the urn:3gpp:video-orientation (CVO) RTP header extension from the router capabilities before loading the mediasoup device (WebRtcService._withoutVideoOrientationExtension). With CVO negotiated, libwebrtc ships raw sensor frames plus a rotation flag that our consume path did not reliably apply, so viewers saw sideways (iOS hosts) or upside-down (Android hosts) video while the host's local preview looked fine. Without CVO the sender rotates every frame upright before encoding, so streams render correctly on every platform with no viewer-side rotation or flip.

Consequences to keep in mind:

• Viewer-side orientation compensation was removed from the live room UI (_LiveRendererView is a plain cover-fit RTCVideoView); do not add per-platform rotation hacks back.
• Clients built before this change (iOS ≤ 1.0, build 9) send unrotated frames; on fixed clients those render sideways until the fleet is on the fixed build. Ship iOS and Android together.
• A mid-stream device rotation now costs a re-encode instead of a flag flip -irrelevant while the app is portrait-locked.

Producer lifecycle & resync

Live connections drop and apps restart, so the system has to handle a broadcaster republishing without breaking. The rule: a fresh producer for the same track replaces the old one instead of being rejected, and viewers are told to re-sync onto the replacement. A producer is an outgoing media track; a consumer is a viewer's incoming copy of it.

Producing the same owner + kind + source again supersedes the previous producer (closed with reason superseded_by_reproduce) instead of being rejected: a killed/restarted app resuming its live, or a republish after transport recovery, must not brick on its own stale producer. Attached consumers see producer_closed and pick up the replacement via resync.

When a producer appears/disappears (guest joins the stage, host toggles camera), the gateway broadcasts a resync. Clients diff the producer list, consume new ones, and drop closed ones -skipping any producer they own (you never consume yourself). This resync path is also what heals clients after a reconnect.

Consumer lifecycle mirrors this. Re-consuming the same owner + producer replaces the previous consumer (closed, logged as duplicateReplaced) instead of 400ing -the retry only ever happens because the first consumer never came up. On the client, every consume claim is tracked (ConsumeClaims): if the consume RPC succeeds but the consumer attach stalls (no track callback), an 8 s watchdog reclaims the claim and re-consumes, up to 3 attempts per producer. Consumer resume (server consumers start paused) also retries with backoff. Together these close the "viewer joins but hears nothing until they request the stage" race — before, a half-completed consume was claimed forever client-side and rejected as a duplicate server-side, so only a full media re-init healed it.

Related pages

• Realtime (Socket.IO) -the signaling channel that sets up every media session.
• System Architecture -the media path in the overall topology, and the UDP ports.
• Backend Module Map -the live module and LiveMediaService.