APIs around the WebRTC

2020/04/15 PWA Night vol.15

About me

About me 1/2

  • Yuji Sugiura
  • NTT Communications
    • Web Engineer

About me 2/2

Today's theme

What is WebRTC?

is WebRTC...?

๐Ÿ™†๐Ÿปโ€โ™‚๏ธ ๐Ÿ™…๐Ÿปโ€โ™€๏ธ

WebRTC: as protocol suite

WebRTC: as W3C API

WebRTC: as APIs around(includes) the WebRTC API

  • RTCPeerConnection
  • navigator.mediaDevices
  • MediaStream, MediaStreamTrack
  • MediaRecorder
  • WebAudio
  • HTMLMediaElement
  • etc...

โ˜๐Ÿป Today's main theme

What can WebRTC do?

There are many different use-cases for WebRTC, from basic web apps that uses the camera or microphone, to more advanced video-calling applications and screen sharing.

from https://webrtc.org/

WebRTC ๐Ÿค PWA ?

  • RTCPeerConnection requires network connection ๐Ÿ“ถ
    • Not suitable for offline app
  • But other APIs are available ๐Ÿ˜‰

Table of Contents

  1. About W3C WebRTC API
  2. APIs around the WebRTC
  3. RTC APIs in the future

About W3C WebRTC API

โš ๏ธ Simplified explanation

  • WebRTC protocols are huge and complicated ๐Ÿคฏ
  • And W3C WebRTC also has a lot of APIs ๐Ÿ˜‡

So today, I'll introduce the concepts only ๐ŸŽฏ

Example scenario

  • P2P video-calling application
  • Bidirectional communication between Alice ๐Ÿ‘ฉ and Bob ๐Ÿ‘จ

Procedure 1/4

  • Alice & Bob: Set up
    • Get user media device and permission to send
    • Create RTCPeerConnection instance

Procedure 2/4

  • Alice: Create and send OFFER via Signaling server
    • I want to send & receive video+audio w/ codec A, params B
    • My global IP address and port is x.x.x.x:yyyyy
    • etc...

Procedure 3/4

  • Bob: Receive OFFER via Signaling server
  • Bob: Apply OFFER to own RTCPeerConnection instance
  • Bob: Create and send back ANSWER via Signaling server
    • I also want to send & receive video+audio w/ codec A, params B
    • My global IP address and port is x.x.x.x:yyyyy
    • etc...

Procedure 4/4

  • Alice: Receive ANSWER via Signaling server
  • Alice: Apply ANSWER to own RTCPeerConnection instance
  • Alice & Bob: Media flows ๐Ÿ‘

Summary

  • Exchange OFFER and ANSWER to establish P2P connection
    • OFFER may be accepted or declined = negotiation
  • The 3rd person something like Signaling server is needed
    • WebSocket, REST, etc...
    • No required format
  • You may need SDK for client
    • Instead of handling raw RTCPeerConnection

How to try WebRTC?

  • Find demo sites through the Internet
    • "webrtc demo" ๐Ÿ”
  • Create your own demo in 1 page(tab)
    • Do not need signaling server
    • "webrtc handson" ๐Ÿ”
  • Deploy your own demo
    • Signaling server is needed

APIs around the WebRTC

Establish P2P connection

// sender side
const pc1 = new RTCPeerConnection();
// RTCRtpSender
const sender = pc1.addTrack(track, stream);

/* ... negotiation flows are omitted ... */

// receiver side
pc2.addEventListener("track", ev => {
  // RTCRtpReceiver
  const { receiver } = ev;
  $video.srcObject = new MediaStream([receiver.track]);
});
  • It just worksโ„ข๏ธ
    • For most common cases, but in detail...๐Ÿ™ˆ

https://dontcallmedom.github.io/webrtc-impl-tracker/?webrtc

Get MediaStream from user media devices

// for camera, microphone
await navigator.mediaDevices.getUserMedia({ video: true, audio: true });

// for screen sharing
await navigator.mediaDevices.getDisplayMedia({ video: true });
  • Safari 13.1 now supports getDisplayMedia() โœŒ๏ธ
  • Mobile browsers does not support getDisplayMedia()

Render MediaStream

// from MediaStream
videoElement.srcObject = stream;
// deprecated
// videoElement.src = URL.createObjectURL(stream);

// from MediaStreamTrack
audioElement.srcObject = new MediaStream([audioTrack]);
  • Be careful with autoplay restriction
  • And be sure to add playsinline for mobile browsers ๐Ÿ“ฑ

Process MediaStream from other sources

// from canvas or other HTMLMediaElement
canvasElement.captureStream(fps);

// from WebAudio
audioContext.createMediaStreamDestination();
  • captureStream() is still experimental ๐Ÿงช

Techniques using MediaStream

List media devices

const devices = await navigator.mediaDevices.enumerateDevices();

const [audioIn] = devices.filter(d => d.kind === "audioinput");
const [videoIn] = devices.filter(d => d.kind === "videoinput");
// Chrome, ChromiumEdge only
const [audioOut] = devices.filter(d => d.kind === "audiooutput");
  • Return value may differ from each browsers ๐Ÿƒ
    • Chrome has default devices
    • Firefox and Safari does not expose label property until calling getUserMedia()
  • Behavior may be changed by its privacy policy

Specify device to getUserMedia()

const [videoIn] = devices.filter(d => d.kind === "videoinput");
await navigator.mediaDevices.getUserMedia({
  video: { deviceId: { exact: videoIn.deviceId } }
});
  • Be sure to add exact keyword

Specify output device

const [audioOut] = devices.filter(d => d.kind === "audiooutput");
await audioElement.setSinkId(audioOut.deviceId);
  • Experimental
  • Chrome, ChromiumEdge only

Record MediaStream

const mediaRecorder = new MediaRecorder(stream);

const chunks = [];
mediaRecorder.ondataavailable = ev => chunks.push(ev.data);
mediaRecorder.onstop = () =>
  finalize(new Blob(chunks, { type: "audio/ogg; codecs=opus" }));

mediaRecorder.start();
// mediaRecorder.stop();
  • Chrome, ChromiumEdge, Firefox only
    • Available codecs depend on browsers
  • You may consider server side recording w/ WebRTC SFU

RTC APIs in the future

[Proposal] Insertable Streams API for WebRTC

https://github.com/alvestrand/webrtc-media-streams

const pc = new RTCPeerConnection({ forceEncodedVideoInsertableStreams: true });

const senderStreams = videoSender.getEncodedVideoStreams();
senderStreams.readable
  .pipeThrough(senderTransform) // your pipeline here
  .pipeTo(senderStreams.writable);

[Behind a flag] WebSocketStream

https://github.com/ricea/websocketstream-explainer

const wss = new WebSocketStream(url);
const { readable } = await wss.connection;

const reader = readable.getReader();
while (true) {
  const { value, done } = await reader.read();
  if (done) break;

  await process(value);
}

[In development] WebTransport(QuicTransport)

https://github.com/WICG/web-transport

const transport = new QuicTransport("example.com", 10001);

setInterval(async () => {
  const stream = await transport.createSendStream();
  const writer = stream.writable.getWriter();
  writer.write(getSerializedGameState());
  writer.close();
}, 100);

[In development] WebCodecs

https://github.com/WICG/web-codecs

const videoDecoder = new VideoDecoder({
  output(videoFrame) {
    const bitmap = videoFrame.transferToImageBitmap();
    canvasContext.transferFromImageBitmap(bitmap);
  }
});
await videoDecoder.configure({ codec: "vp8" });

// provides stream of encoded chunks to decoder
streamEncodedChunks(chunk => videoDecoder.decode(chunk));

Thank you!

References ๐Ÿ“š