IPFS Live Streaming

This project is started by @ASoTNetworks and @darkdrgn2k to stream videos over IPFS, which overlapped with the need to live stream the Our Networks 2018 conference in Toronto. We will document here the components and processes necessary to run live streams throughout the conference and archive the video assets on the IPFS network, that is suitable for a small conference with an audience size of less than 100 people.

Here is a presentation and demo video about this project, recorded from the IPFS All Hands: November 26, 2018. Slides are available here.

Set Up

We will record live streams in 720p (3 Mbps) and archive mp4 of each talk in 1080p.

Equipment List

• HD Video camera that supports HDMI live feed (e.g. Panasonic HC V500)
• Tripod for video camera
• Wireless microphones (e.g. presenter, handheld, audience, room tone, etc.)
• Microphone mixer (e.g. Shure M67)
• On-premise laptop or desktop running OBS Studio and other software
• Two different USB HDMI capture cards with HDMI pass-through and 3.5 mm audio input (e.g. one Elgato HD60 and one AVerMedia LGP Lite)
• 1 TB external hard disk drive
• HDMI, XLR, 3.5 mm audio, USB, ethernet, power cables

On-Premise Setup

    Microphones
         |
  (xlr / wireless)
         |
         v
+------------------+
| Audio Mixer      | --(3.5 mm audio)----+
+------------------+                     |
                                         v
+------------------+             +---------------+            +---------------------------+
| HD Video Camera  | --(hdmi)--> | Elgato HD60   | --(usb)--> | Laptop running OBS Studio |
+------------------+             +---------------+            | ↳ Streams to RTMP server  |
                                                              |   for HTTP & IPFS streams |
+------------------+             +---------------+            | ↳ Records mp4 files       |
| Presenter Laptop | --(hdmi)--> | AVerMedia LGP | --(usb)--> |   for local archiving     |
+------------------+             +---------------+            +---------------------------+
                                         |                        |               |
                                (hdmi pass-through)             (usb)         (ethernet)
                                         |                        |               |
                                         v                        v               v
                                     Projector                 1 TB HDD    Gigabit Internet

The laptop is the control centre. It has two USB capture cards, connected to separate USB buses (e.g. if it has a USB2 and USB3 interface) if possible to avoid bandwidth issues. These will be the video and audio inputs. The capture cards are of two different brands because cards like the Elgato have problems when running two in parallel. At least one card should take a 3.5 mm audio input so we can mix the audio into the stream via the audio mixer.

The laptop runs the following software:

• OBS Studio locally to stream to our servers
• OpenVPN or Yggdrasil to authenticate the streaming device

For Yggdrasil, you should compile at tag v0.3.2, and when streaming, run it with the configurations that will be downloaded from the streaming server at a later step.

OBS Studio is used throughout the conference to toggle between the two video feeds (i.e. the slides and the presenter video). Using the Start Streaming function in OBS Studio, the stream is published at 720p to a RTMP server we will set up in the next step. Using the Start Recording function in OBS Studio, the operator will also record each talk as a separate 1080p mp4 file to the external hard disk to be published after the event.

Remote Server Architecture

      OBS Studio              Website Embedded     Viewer with
        Source                    Video Player     IPFS Client
          |                              ^             ^                   ^             ^
    (rtmp-publish)                       |             |                   |             |
          |                            (http)        (ipfs)              (http)        (ipfs)
          v                              |             |                   |             |
+-------------------+                +---------------------+           +---------------------+
| rtmp-server       |                | ipfs-server         |           | ipfs-mirror         |
| ↳ nginx-rtmp      |                | ↳ ipfs with pubsub  |           | ↳ ipfs with pubsub  |
| ↳ openvpn         |<--(rtmp-pull)--| ↳ ipfs-http gateway |<--(ipfs)--| ↳ ipfs-http gateway |
| ↳ yggdrasil       |                | ↳ ffmpeg            |           |- - - - - - - - - - -|
|- - - - - - - - - -|                |- - - - - - - - - - -|           | Pins IPFS hashes    |
| Runs RTMP server  |                | Encodes HLS ts+m3u8 |           | learnt from IPNS id |
| publishable from  |                | pins on IPFS and    |           | of ipfs-server      |
| authenticated IPs |                | publishes to IPNS   |           +---------------------+
+-------------------+                +---------------------+
          |
   (rtmp-pull/push)
          |
          v
Other Streaming Services

The on-premise laptop running OBS Studio pushes to the rtmp-server, which through IP-pinning of the OpenVPN or Yggdrasil-generated IP address will allow only that device to publish. The ipfs-server pulls that RTMP stream, encodes ts chunks in a live m3u8 file using ffmpeg, then IPFS adds and pins those files and uses IPNS to publish the m3u8 to its node ID. The built-in ipfs-http gateway allow those content to be accessed via HTTP, which is what the embedded player on the website will use. However, viewers running a IPFS client (with pubsub enabled) can directly view the streams over IPFS. Optionally, we can run one or more ipfs-mirror servers that pin the live streaming content and run additional gateways.

All the servers described above are provisioned using Terraform on Digital Ocean. In addition, the RTMP stream can be consumed by other services to provide a parallel stream that does not involve IPFS.

Provision Streaming Servers

We will be using the following tools and services:

• Digital Ocean as the virtual machine provider
• Terraform to provision the cloud servers
• Let's Encrypt to get SSL certificates for HTTPS

The following steps assume you have a Digital Ocean account and the above listed software installed on your local machine, which can be the same device running OBS Studio.

1. Clone this repository and work from the terraform directory:

    git clone https://github.com/tomeshnet/ipfs-live-streaming.git
    cd ipfs-live-streaming/terraform
   

2. From your domain name registrar, point name servers to Digital Ocean's name servers:

    ns1.digitalocean.com
    ns2.digitalocean.com
    ns3.digitalocean.com
   

   Then store the domain name in your local environment:

    echo -n YOUR_DOMAIN_NAME > .keys/domain_name
   

3. Set an email address to use as contact email for Let's Encrypt:

    echo -n YOUR_EMAIL_ADDRESS > .keys/email_address
   

   DEVELOPER TIP Set dryrun to use Let's Encrypt staging servers for testing

    echo -n true > .keys/dryrun
   

4. Obtain a read-write access token from your Digital Ocean account's API tab, then store it in your local environment:

    echo -n YOUR_DIGITAL_OCEAN_ACCESS_TOKEN > .keys/do_token
   

5. Generate RSA keys to access your Digital Ocean VMs:

    ssh-keygen -t rsa -f .keys/id_rsa
   

   Add the SSH key to your Digital Ocean account under Settings > Security, then copy the SSH fingerprint to your local environment:

    echo -n YOUR_SSH_FINGERPRINT > .keys/ssh_fingerprint
   

6. Download Terraform, add it to your path. On Linux it would look something like this:

    https://releases.hashicorp.com/terraform/0.11.7/terraform_0.11.7_linux_amd64.zip
    unzip terraform_0.11.7_linux_amd64.zip
    mv terraform /usr/bin
   

   Then run initialization from our terraform working directory:

    terraform init
   

7. Provision the streaming servers by running:

    terraform apply
   

   By default, this will create one instance of each server type. You may choose to create multiple instances of ipfs-mirror by overriding the mirror variable.

   You may also choose to include an external HTTP (non-IPFS) stream source by adding one or more URLs to a m3u8 playlist.

   For example:

    terraform apply \
      -var "mirror=2" \
      -var "m3u8_http_urls=\'https://HLS_SOURCE_0/live.m3u8\',\'https://HLS_SOURCE_1/live.m3u8\'"
   

   From your browser, login to your Digital Ocean dashboard and find your new VMs tagged with ipfs-live-streaming.

8. You will find a couple new files in your .keys folder:

    client.conf    (for OpenVPN on Linux)
    client.ovpn    (for OpenVPN on MacOS and Windows)
    yggdrasil.conf (for Yggdrasil)
   

   To authenticate using OpenVPN, connect with your OpenVPN client using client.conf or client.ovpn, then publish your OBS Studio stream to:

    rtmp://10.10.10.1:1935/live/stream1
   

   To authenticate using Yggdrasil, start it with yggdrasil.conf and note the last line of output like this:

    sudo yggdrasil --useconf < ./keys/yggdrasil.conf
    ...
    2018/12/14 15:16:22 Connected: 203:4bb0:9ff1:2312:e7f3:b8c4:852:[email protected] source 192.168.1.173
   

   Then publish your OBS Studio stream to the IPv6:

    rtmp://[203:4bb0:9ff1:2312:e7f3:b8c4:852:a8b1]:1935/live/stream1
   

9. When your streaming session is done, you can stop OpenVPN or Yggdrasil and destroy the servers with:

    terraform destroy
   

Video Playback

The following streams become publicly available about 30 seconds after you start publishing to the RTMP server:

Stream	URL
RTMP stream	rtmp://rtmp-server.YOUR_DOMAIN_NAME/live
HLS stream (origin)	https://YOUR_DOMAIN_NAME/live.m3u8
HLS stream (mirror-N)	https://ipfs-mirror-N.YOUR_DOMAIN_NAME/live.m3u8
IPNS HLS stream (origin)	https://ipfs-gateway.YOUR_DOMAIN_NAME/ipns/IPNS_ID (disabled)
IPNS HLS stream (mirror-N)	https://ipfs-gateway-N.YOUR_DOMAIN_NAME/ipns/IPNS_ID (disabled)

The origin ipfs-server and each ipfs-mirror also host an embedded video player, publicly available at:

Site	URL
Video player (origin)	https://YOUR_DOMAIN_NAME
Video player (mirror-N)	https://ipfs-mirror-N.YOUR_DOMAIN_NAME

The embedded video player is tested on common desktop and mobile browsers, and support the following optional URL query parameters:

Parameter	Description
gw	Set custom IPFS gateway
m3u8	Set m3u8 file URL to override IPFS live stream
vod	Set IPFS content hash of mp4 file to play IPFS on-demand video stream
from	Set IPFS content hash or timecode to start video playback from

Attribution

The video player uses code from Video.js, graphics from ipfs/artwork, and loading animation from jxnblk/loading.