Bridging WebRTC and HLS with mediasoup and FFmpeg

GitHub

I built this project to experiment with connecting two very different kinds of video delivery: WebRTC for interactive calls and HLS for passive live-stream viewers.

The idea was simple:

Let participants communicate through WebRTC, then take those live media streams on the server and turn them into an HLS broadcast.

The implementation ended up involving mediasoup, RTP, SDP, FFmpeg, Socket.IO, Next.js, and quite a lot of debugging.

Architecture

The media pipeline looks roughly like this:

WebRTC Participants
        │
        ▼
   mediasoup SFU
        │
        │ RTP / RTCP
        ▼
      FFmpeg
        │
        │ H.264 + AAC
        ▼
        HLS
        │
        ▼
    HLS Viewers

The frontend is built with Next.js and TypeScript, while a separate Node.js server handles mediasoup, Socket.IO, FFmpeg, and HLS delivery.

There are two main pages:

• /stream for WebRTC participants
• /watch for HLS playback

WebRTC with mediasoup

Rather than connecting every participant directly to every other participant, the project uses mediasoup as an SFU.

Each participant sends their audio and video to mediasoup. The server can then create consumers that forward those media streams to other participants.

Socket.IO handles the coordination needed to create WebRTC transports, producers, and consumers.

This gives the server access to the media routing layer, which is important because the same streams can then be forwarded into the HLS pipeline.

Bridging mediasoup to FFmpeg

This was the most interesting part of the project.

The browser's media arrives at mediasoup through WebRTC, but FFmpeg needs a format it can consume independently.

For each selected producer, the server creates a mediasoup PlainTransport.

WebRTC Producer
      │
      ▼
mediasoup Router
      │
      ▼
 PlainTransport
      │
    RTP
      ▼
   FFmpeg

The PlainTransport sends RTP and RTCP locally to ports that FFmpeg listens on.

The server then generates an SDP file describing those RTP streams, including their codecs, payload types, and ports.

FFmpeg opens that SDP description and begins receiving the live media.

Getting the RTP ports, RTCP ports, payload types, and SDP configuration to agree was one of the trickier parts of building the pipeline.

Turning WebRTC Media into HLS

The mediasoup router currently uses VP8 video and Opus audio.

FFmpeg receives those streams and converts them into formats suitable for the HLS output:

VP8  → H.264
Opus → AAC

Video is encoded using libx264 with the ultrafast preset and zerolatency tuning.

The HLS playlist uses short 1-second segments and keeps a small rolling playlist to reduce latency compared with a more traditional HLS configuration.

This is still normal HLS rather than a true ultra-low-latency broadcast, but it keeps the delay reasonably small for the experiment.

Multiple Participants

The current HLS pipeline supports up to two video producers and two audio producers.

With one participant, FFmpeg scales the video to the output resolution.

With two participants, both videos are scaled and combined side-by-side:

┌─────────────────────────────┐
│              │              │
│ Participant  │ Participant  │
│      A       │      B       │
│              │              │
└─────────────────────────────┘

Their audio streams are mixed together before being encoded to AAC.

This produces a single composed broadcast instead of requiring HLS viewers to understand the individual WebRTC streams.

HLS Playback

FFmpeg writes an .m3u8 playlist and MPEG-TS segments into a local HLS directory.

The Express server exposes those files over HTTP.

On /watch, hls.js loads the playlist and attaches it to a normal HTML video element.

So the viewer doesn't participate in the WebRTC session at all.

They simply consume the generated HLS stream:

FFmpeg
   │
   ▼
.m3u8 + .ts segments
   │
   ▼
Express
   │
   ▼
hls.js
   │
   ▼
<video>

That separation is the main idea behind the project.

Interactive participants use WebRTC, while passive viewers receive a conventional HTTP-based stream.

Resource Cleanup

Media applications create a surprising amount of temporary state.

A participant can have:

• WebRTC transports
• mediasoup producers
• mediasoup consumers
• PlainTransports
• RTP consumers
• an FFmpeg process
• a temporary SDP file
• generated HLS segments

The server tracks these resources and closes them when clients disconnect or an HLS stream stops.

FFmpeg is terminated, mediasoup transports and consumers are closed, and temporary files are removed.

Without that cleanup, repeatedly starting streams would quickly leave behind dead media resources and processes.

Tech Stack

• Next.js + React + TypeScript
• mediasoup / mediasoup-client
• WebRTC
• Socket.IO
• FFmpeg
• Express
• HLS + hls.js
• Tailwind CSS

The project runs the Next.js frontend and Node.js media server together during development using concurrently.

Current Limitations

This is an experimental media pipeline rather than a production streaming service.

The mediasoup transport configuration currently contains a LAN-specific announced IP, the HLS viewer connects directly to the local Express server, and there is no CDN or production TURN deployment.

The HLS pipeline is also limited to two participants and produces a single rendition rather than an adaptive bitrate ladder.

FFmpeg software transcoding is another obvious bottleneck. Encoding H.264 in real time becomes expensive as the number of streams grows.

A production version would need proper room isolation, dynamic RTP port allocation, TURN infrastructure, authentication, CDN-backed HLS delivery, adaptive bitrate output, and probably hardware-accelerated encoding.

What I Learned

The most useful part of this project wasn't simply getting a video call working.

It was understanding what happens between media systems.

WebRTC, RTP, SDP, codecs, mediasoup, FFmpeg, and HLS all solve different parts of the problem.

The interesting part was connecting them:

WebRTC
   ↓
mediasoup
   ↓
RTP
   ↓
FFmpeg
   ↓
HLS

Once the RTP streams leave mediasoup correctly and FFmpeg understands their SDP description, the WebRTC session can effectively become the source for a conventional live broadcast.

That bridge was the entire point of the project.

tl;dr

I built a prototype that takes live WebRTC audio/video routed through mediasoup, forwards selected streams to FFmpeg over RTP, composites and transcodes them, and exposes the result as an HLS stream that can be watched through hls.js.

WebRTC handles the interactive side.

HLS handles the broadcast side.

FFmpeg is the slightly terrifying glue in the middle.