Media & streaming, an interactive architecture walkthrough
Video conferencing: how the architecture works
How a video conferencing system connects clients with WebRTC, forwards streams through an SFU, and adapts quality to each viewer's bandwidth.
This is a learning explainer based on public engineering writing and general design practice. It is simplified, it is not the actual architecture of any company, and it may be inaccurate.
Interactive diagram
Overview
A video conferencing system lets a group see and hear each other with a delay low enough for conversation, usually under a few hundred milliseconds. Clients use WebRTC: they negotiate codecs and encryption through a signaling channel the application provides, find a network path with STUN, and fall back to a TURN relay when firewalls block direct UDP.
Media goes through a selective forwarding unit (SFU) rather than a full mesh or a mixing server. Each participant uploads one stream, and the SFU forwards it to everyone else, choosing per viewer which of several simulcast qualities to send based on their bandwidth and what is on screen. Meetings spanning regions cascade between SFUs, and recording and captions join as hidden participants. This is a simplified learning design, not any product's actual architecture.
Trade-offs and what changes at scale
An SFU keeps upload constant and server CPU low, but every participant still downloads several streams, so large meetings forward only the active speakers' video. A mixing server would send one stream per viewer but costs much more CPU and adds delay. TURN relays make calls work on restrictive networks at a real bandwidth cost. Server-side recording and captions need access to media, which conflicts with end-to-end encryption. At larger scale capacity is planned per region, with meetings placed on SFUs close to most of their participants.
How it works, step by step
The walkthrough above, as text. Each step lights up the components named under it.
A participant opens the meeting link, and the Meeting Service checks their role and the meeting's settings, holding them in a waiting room if the host requires it. Control actions go over HTTPS; media never touches this path.
The SFU Allocator picks a media server with spare capacity close to the first participants and records the choice in Room State. Everyone else in the meeting is then sent to the same server.
The client opens a WebSocket to the Signaling Service and exchanges session descriptions with the SFU: which codecs, tracks and encryption to use. WebRTC leaves signaling to the application, so this channel is the product's own protocol.
The client asks the STUN Server for its public address and gathers candidate paths to the SFU. Most connections then go directly over UDP, which is best for latency.
When a firewall or strict NAT blocks every direct path, media goes through the TURN Relay, often over TCP or TLS on port 443. Relaying costs bandwidth and adds latency, so it is only the fallback.
Each client sends its encrypted audio and video once to the SFU Node, which forwards it to the other participants without decoding or mixing it. Forwarding instead of mixing keeps server CPU low and latency short.
Cameras send three qualities at once (simulcast), and the SFU Node forwards the high layer for the active speaker and low layers for thumbnails, based on each viewer's bandwidth. A viewer on a weak network gets a smaller stream instead of freezing everyone else.
When participants join from another continent, a Remote SFU Node near them receives one copy of each stream from the first SFU and serves local viewers. Each viewer then connects to a nearby server instead of crossing an ocean.
Chat and reactions go to In-meeting Chat over the regular connection. Text needs reliable delivery and history, which the media path does not provide.
For Live Captions, the SFU forwards audio to a speech recognizer, and the text is sent to participants through signaling. Because the server can read media for this, end-to-end encrypted meetings disable it.
Recording adds a Recording Bot that joins like a hidden participant and receives every track, and the Recording Compositor mixes them into one file in the Recording Store. Mixing happens only for recordings, not for the live meeting.
Clients and SFUs report packet loss, jitter and frozen frames to Quality Telemetry. Most call problems are on the user's network, and these numbers show where.
Sequence diagram
The same walkthrough as a UML sequence diagram, one lane per layer or group (18 components in all), left to right in the order a request reaches them, numbered by step. A loop on a lane is a step that stays inside that group. Solid arrows are synchronous calls, dashed are asynchronous.
Key components
Meeting Controldomain
- API Gateway: auth, meeting links, rate limits
- Meeting DB: meetings, invites, settings (postgresql)
- Meeting Service: rooms, roles, waiting room
- Room State: who is in which room, on which SFU (redis)
- Signaling Service: SDP offers/answers, ICE candidates (websocket)
Participantslayer
- Browser Client: WebRTC camera, mic, screen share
- Desktop & Mobile Apps: same media stack, native
Media Planeboundary
selective forwarding, no mixing
- Live Captions: speech to text on the audio
- Recording Bot: joins as a hidden participant
- Remote SFU Node: same meeting, another region
- SFU Allocator: picks a media server near participants
- SFU Node: forwards streams, picks layers per viewer
Connectivitydomain
- STUN Server: tells clients their public address
- TURN Relay: relays media when direct paths fail
Storagedomain
- In-meeting Chat: messages and reactions
- Recording Compositor: mixes tracks into one video file
- Recording Store: recordings and transcripts (s3)
Quality Telemetry
- Quality Telemetry: packet loss, jitter, freezes per call
Sources
Public writing this explainer draws on. Read these for what the real systems actually do.
Make it yours
Copy this diagram and its walkthrough into your own workspace, then change anything: swap Postgres for DynamoDB, add a cache, rewrite the narration. Free, and it uses no AI credits.
Related walkthroughs
- Netflix-style video streamingHow a streaming service encodes each title into many bitrates, caches it near viewers on a CDN, and personalises what each person sees.
- Video sharing platform (YouTube-style)How a video platform takes resumable uploads, encodes segments in parallel into several renditions, and serves them via a CDN with adaptive bitrate.
- Photo sharing platformHow a photo sharing platform takes uploads straight to object storage, resizes them in the background, and builds follower feeds from cached post ids.
This is a learning explainer based on public engineering writing and general design practice. It is simplified, it is not the actual architecture of any company, and it may be inaccurate. Real systems differ in the details. Browse the library.