Allow data flow from QUIC agent to video agent. (#1065)

* WebTransportStream sends frames to internal server. * Allow data flow between QUIC agent and video agent. * Add WebTransportFrameSource to associate multiple streams to a single publication.
open-webrtc-toolkit · Aug 31, 2021 · e4cf5f4 · e4cf5f4
1 parent ff9cfd1
commit e4cf5f4
Show file tree

Hide file tree

Showing 26 changed files with 1,167 additions and 194 deletions.
diff --git a/doc/Client-Portal Protocol.md b/doc/Client-Portal Protocol.md
@@ -362,36 +362,41 @@ This a format for client reconnects.
 ```
   object(PublicationRequest)::
     {
-     media: object(WebRTCMediaOptions) | null,
+     media: object(MediaOptions) | null,
      data: true | false,
      transport: object(TransportOptions),
      attributes: object(ClientDefinedAttributes) | null
     }
 ```
 
-A publication can send either media or data, but a QUIC *transport* channel can support multiple stream for both media and data. Setting `media:null` and `data:false` is meaningless, so it should be rejected by server. Protocol itself doesn't forbit to create WebRTC connection for data. However, SCTP data channel is not implemented at server side, so currently `data:true` is only support by QUIC transport channels. 
+A publication can send either media or data. Setting `media:null` and `data:false` is meaningless, so it should be rejected by server. Protocol itself doesn't forbid to create WebRTC connection for data. However, SCTP data channel is not implemented at server side, so currently `data:true` is only support by WebTransport channels.
 
 ```
-  object(WebRTCMediaOptions)::
+  object(MediaOptions)::
       {
         tracks: [
           {
             type: "audio" | "video",
-            mid: string(MID),
+            mid: string(MID) | undefined,  /* undefined if transport's type is "quic" */
             source: "mic" | "screen-cast" | ... | "encoded-file",
+            format: object(AudioFormat) | object(VideoFormat) | undefined /* undefined if transport's type is "webrtc" */
           }
         ]
       }
     }
 ```
 
+
 **ResponseData**: The PublicationResult object with following definition if **ResponseStatus** is “ok”:
 
+```
   object(PublicationResult)::
     {
       transportId: string(transportId),  // Can be reused in the following publication or subscription.
      id: string(SessionId) //will be used as the stream id when it gets ready.
     }
+```
+
 ### 3.3.8 Participant Stops Publishing a Stream to Room
 **RequestName**: “unpublish”<br>
 

diff --git a/doc/design/pics/.gitkeep b/doc/design/pics/.gitkeep
diff --git a/doc/design/pics/quic_agent_data_flow.svg b/doc/design/pics/quic_agent_data_flow.svg
diff --git a/doc/design/quic-agent.md b/doc/design/quic-agent.md
@@ -0,0 +1,115 @@
+# QUIC agent
+
+## Overview
+QUIC agents are designed for [WebTransport](https://w3c.github.io/webtransport/) over HTTP/3 connections. A WebTransport connection could send and receive arbitrary data, as well as media data encoded or can be decoded by [WebCodecs](https://www.w3.org/TR/webcodecs/).
+
+## Architecture and dataflow
+
+![data flow](./pics/quic_agent_data_flow.svg)
+
+A WebTransportFrameSource handles all audio and video frames for a publication. A WebTransportFrameDestination dispatches audio and video frames to different WebTransport streams or a datagram sender.
+
+A DatagramSource processes datagrams (RTP packets) received from client side, depacketizes them to create audio or video frames, and dispatches media frames to a WebTransportFrameSource. It also handles FEC and NACK, similar to RTCRtpReceiver in WebRTC. A DatagramDestination is similar to RTCRtpSender.
+
+## WebTransport payload and message format
+
+This section defines the payload and message format for data transmitted over WebTransport.
+
+### Streams
+
+Both server and client can initialize a stream. When a stream is created, initial side sends a session ID, which is a 128 bit length message to the remote side. Session ID could be a publication ID or subscription ID as defined in [Client-Portal Protocol](https://github.com/open-webrtc-toolkit/owt-server/blob/master/doc/Client-Portal%20Protocol.md). As the session ID issued by server may less than 128 bit right now, fill it with 0 in most significant bits. Session ID 0 is reserved for signaling. When remote side receives the session ID, it should check whether session ID is valid. Terminate the stream if session ID is invalid, or send the same session ID to client if it is valid. Depends on the type of stream it created, one side or both sides are ready to send data.
+
+### Datagram
+
+Each package has a 128 bit header for session ID.
+
+```
+    0                   1                   2                   3
+    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+   |                                                               |
+   |                      Session Identifier                       |
+   |                             ....                              |
+   |                                                               |
+   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+   |                      Datagram Data (*)                      ...
+   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+```
+
+It may increase about 2% network cost.
+
+### Signaling Session
+
+After creating a WebTransport, a stream with session 0 should be created for authentication and signaling. Every signaling message is followed by a 32 bit length integer that indicates the body's length.
+
+```
+    0                   1                   2                   3
+    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+   |                      Message length                           |
+   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+   |                          Message                            ...
+   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+```
+
+### Media Stream
+
+After sending 128 bit length session ID, a 128 bit length track ID is sent to remote side to indicates the track of a stream. Since audio track and video track of a single stream shares the same track ID at this time, track 1 is for audio and track 2 is for video.
+
+When a WebTransport stream is used for transmitting data of a media stream track (e.g.: H.264 bitstream), a 32 (8+24) bit length header is added to indicate frame size.
+
+```
+    0                   1                   2                   3
+    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+   |     Reserved    |               Message length                |
+   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+   |                          Message                            ...
+   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+```
+
+### Authentication
+
+If signaling messages are transmitted over WebTransport, authentication follows the regular process defined by [Client-Portal Protocol](https://github.com/open-webrtc-toolkit/owt-server/blob/master/doc/Client-Portal%20Protocol.md). Otherwise, client sends a token for WebTransport as a signaling message. WebTransport token is issued during joining a conference. If the token is valid, server sends a 128 bit length zeros to client.
+
+## Build conference server with QUIC agent
+
+Because we don't have a good place to store pre-built QUIC SDK for public access, QUIC agent is not enabled by default. Additional flags are required to enable QUIC agent.
+
+1. Download QUIC SDK from the URL specified [here](https://github.com/open-webrtc-toolkit/owt-server/blob/master/source/agent/addons/quic/quic_sdk_url). QUIC SDK is hosted on GitHub as an artifact. You will need to follow [this description](https://docs.github.com/en/rest/reference/actions#download-an-artifact) to make a REST request to GitHub. Or you can download the latest QUIC SDK from [GitHub Actions](https://github.com/open-webrtc-toolkit/owt-sdk-quic/actions) tab. Commits pushed to main branch have artifact for downloading.
+1. After running `installDeps.sh`, put headers to build/libdeps/build/include, and put libraries(.so file) to build/libdeps/build/lib.
+1. Append `-t quic` to the arguments for build.js.
+1. Append `-t quic-agent` to the arguments for pack.js.
+
+## Certificate for QUIC
+
+OWT Conference Server is using a self-signed certificate during development phase, which would be only valid for 14 days. You can use a CA-signed certificate to avoid refreshing the certificate periodically. A CA-signed certificate is recommended for production environment. WebTransport connection will fail if certificate is not valid or expires.
+
+### Certificates signed by a trusted CA
+
+- Copy your PKCS12 format certificate to `quic_agent/cert/` directory to replace the one there.
+- Restart Conference Server QUIC agent to apply the change. 
+- Don't provide any fingerprint in client applications.
+
+### Generate self-signed certificates
+
+#### Precondition
+- Make sure you are running the tool under Linux and,
+- Openssl tool is correctly setup in your system.
+- Download the tool under chromium/src/net/tools/quic/certs/ from chromium project ([v93.0.4575.1](https://chromium.googlesource.com/chromium/src/+archive/refs/tags/93.0.4575.1/net/tools/quic/certs.tar.gz.)) to local directory named `tool`. This contains three files: `ca.cnf`, `generate-certs.sh` and `leaf.cnf`.
+
+#### Certificate Generation
+
+- Modify leaf.cnf, adding an entry into `other_hosts` section.
+- Make sure generate-certs.sh is executable. If not, run `chmod +x generate-certs.sh`;
+- Remove the `out` dir in case it exists.
+- Under the downloaded tool dir, run `./generate-certs.sh`. It is expected to generate a series of files under out dir.
+- Under the downloaded tool dir, run `openssl pkcs12 -inkey out/leaf_cert.key -in out/leaf_cert.pem -export -out out/certificate.pfx`. This will prompt for password for the pfx. Please type the certificate password of your conference server. The default password is `abc123`.
+- Under the downloaded tool dir, run `openssl x509 -noout -fingerprint -sha256 -inform pem -in out/leaf_cert.pem`. You will get the fingerprint string in the form of "XX:XX:XX....XX:XX".
+
+#### Use the Certificate
+
+- Copy the generated certificate.pfx under `out` dir to `quic_agent/cert/` dir to replace the one there.
+- Restart Conference Server QUIC agent to apply the change.
+- If you're using JavaScript sample for QUIC, make sure you also update JS sample with the new fingerprint.
+- In your native client sample, make sure you include the fingerprint of new cert in the `ConferenceClientConfiguration.trusted_quic_certificate_fingerprints` you passed to `ConferenceClient` ctor. See more details in the conference sample.
diff --git a/doc/design/quic-programming-guide.md b/doc/design/quic-programming-guide.md
@@ -123,15 +123,6 @@ Please see the conference sample application for more detailed usage.
 
 Please follow [Conference Server build instructions](https://github.com/open-webrtc-toolkit/owt-server/blob/master/README.md) on how to build and deploy the conference server.
 
-## Build Conference Server with QUIC agent
-
-Because we don't have a good place to store pre-built QUIC SDK for public access, QUIC agent is not enabled by default. Additional flags are required to enable QUIC agent.
-
-1. Download QUIC SDK from the URL specified [here](https://github.com/open-webrtc-toolkit/owt-server/blob/master/source/agent/addons/quic/quic_sdk_url). QUIC SDK is hosted on GitHub as an artifact. You will need to follow [this description](https://docs.github.com/en/rest/reference/actions#download-an-artifact) to make a REST request to GitHub. Or you can download the latest QUIC SDK from [GitHub Actions](https://github.com/open-webrtc-toolkit/owt-sdk-quic/actions) tab. Commits pushed to main branch have artifact for downloading.
-1. After running `installDeps.sh`, put headers to build/libdeps/build/include, and put libraries(.so file) to build/libdeps/build/lib.
-1. Append `-t quic` to the arguments for build.js.
-1. Append `-t quic-agent` to the arguments for pack.js.
-
 ## How to use Pre-built Conference Server Binary
 
 Steps to run Conference Server with pre-built binary:
@@ -146,30 +137,4 @@ Steps to run Conference Server with pre-built binary:
 
 # OWT QUIC Windows Sample
 
-The Windows sample will be provided in OWT repo separately. More details will be provided later.
-
-# How to Replace the Certificate for QUIC
-
-OWT Conference Server is using a self-signed certificate during development phase, which would be only valid for 14 days. You can use a CA-signed certificate to avoid refreshing the certificate periodically. WebTransport connection will fail if certificate is not valid or expires.
-
-## Precondition
-
-- Make sure you are running the tool under Linux and,
-- Openssl tool is correctly setup in your system.
-- Download the tool under chromium/src/net/tools/quic/certs/ from chromium project to local dir named `tool`. This contains three files: `ca.cnf`, `generate-certs.sh` and `leaf.cnf`.
-
-## Certificate Generation
-
-- Modify leaf.cnf, adding an entry into `other_hosts` section.
-- Make sure generate-certs.sh is exectuable. If not, run `chmod +x generate-certs.sh`;
-- Remove the `out` dir in case it exists.
-- Under the downloaded tool dir, run `./generate-certs.sh`. It is expected to generate a series of files under out dir.
-- Under the downloaded tool dir, run `openssl pkcs12 -inkey out/leaf_cert.key -in out/leaf_cert.pem -export -out out/certificate.pfx`. This will prompt for password for the pfx. Make sure you always use `abc123` as the password.
-- Under the downloaded tool dir, run `openssl x509 -noout -fingerprint -sha256 -inform pem -in out/leaf_cert.pem`. You will get the fingerprint string in the form of "XX:XX:XX....XX:XX".
-
-## Use the Certificate
-
-- Copy the generated certificate.pfx under `out` dir to `quic_agent/cert/` dir to replace the one there.
-- Restart Conference Server QUIC agent to apply the change. If you're using JS sample for QUIC, make sure you also update JS sample with the new fingerprint.
-- In your native client sample, make sure you include the fingerprint of new cert in the `ConferenceClientConfiguration.trusted_quic_certificate_fingerprints` you passed to `ConferenceClient` ctor. See more details in the conference sample.
-
+The Windows sample will be provided in OWT repo separately. More details will be provided later.
diff --git a/doc/design/quic-transport-payload-format.md b/doc/design/quic-transport-payload-format.md
diff --git a/source/agent/addons/common/MediaFramePipelineWrapper.h b/source/agent/addons/common/MediaFramePipelineWrapper.h
@@ -6,10 +6,10 @@
 #ifndef MEDIAFRAMEPIPELINEWRAPPER_H
 #define MEDIAFRAMEPIPELINEWRAPPER_H
 
+#include <MediaFramePipeline.h>
+#include <nan.h>
 #include <node.h>
 #include <node_object_wrap.h>
-#include <MediaFramePipeline.h>
-
 
 /*
  * Wrapper class of owt_base::FrameDestination
@@ -30,5 +30,13 @@ class FrameSource : public node::ObjectWrap{
   owt_base::FrameSource* src;
 };
 
+/*
+ * Nan::ObjectWrap of owt_base::FrameSource and owt_base::FrameDestination, represents a node in the media or data pipeline.
+ */
+class NanFrameNode : public Nan::ObjectWrap {
+public:
+    virtual owt_base::FrameSource* FrameSource() = 0;
+    virtual owt_base::FrameDestination* FrameDestination() = 0;
+};
 
 #endif
diff --git a/source/agent/addons/internalIO/InternalClientWrapper.cc b/source/agent/addons/internalIO/InternalClientWrapper.cc
@@ -88,23 +88,36 @@ NAN_METHOD(InternalClient::close) {
   obj->me = nullptr;
 }
 
-NAN_METHOD(InternalClient::addDestination) {
-  InternalClient* obj = ObjectWrap::Unwrap<InternalClient>(info.Holder());
-  owt_base::InternalClient* me = obj->me;
+NAN_METHOD(InternalClient::addDestination)
+{
+    InternalClient* obj = ObjectWrap::Unwrap<InternalClient>(info.Holder());
+    owt_base::InternalClient* me = obj->me;
 
-  Nan::Utf8String param0(Nan::To<v8::String>(info[0]).ToLocalChecked());
-  std::string track = std::string(*param0);
+    Nan::Utf8String param0(Nan::To<v8::String>(info[0]).ToLocalChecked());
+    std::string track = std::string(*param0);
 
-  FrameDestination* param =
-    ObjectWrap::Unwrap<FrameDestination>(
-      info[1]->ToObject(Nan::GetCurrentContext()).ToLocalChecked());
-  owt_base::FrameDestination* dest = param->dest;
+    bool isNanDestination(false);
+    if (info.Length() >= 3) {
+        isNanDestination = info[2]->ToBoolean(Nan::GetCurrentContext()).ToLocalChecked()->Value();
+    }
 
-  if (track == "audio") {
-    me->addAudioDestination(dest);
-  } else if (track == "video") {
-    me->addVideoDestination(dest);
-  }
+    owt_base::FrameDestination* dest(nullptr);
+    if (isNanDestination) {
+        NanFrameNode* param = Nan::ObjectWrap::Unwrap<NanFrameNode>(info[1]->ToObject());
+        dest = param->FrameDestination();
+    } else {
+        FrameDestination* param = ObjectWrap::Unwrap<FrameDestination>(
+            info[1]->ToObject(Nan::GetCurrentContext()).ToLocalChecked());
+        dest = param->dest;
+    }
+
+    if (track == "audio") {
+        me->addAudioDestination(dest);
+    } else if (track == "video") {
+        me->addVideoDestination(dest);
+    } else if (track == "data") {
+        me->addDataDestination(dest);
+    }
 }
 
 NAN_METHOD(InternalClient::removeDestination) {