How to correctly append buffers to play multiple media files in sequence?

[guest271314]

Have been attempting to implement, for lack of a more descriptive reference, an "offline media context". The basic concept is to be able to use the tools available at the most recent browsers alone to record or request media fragments capable of independent playback and to be able to concatenate those discrete media fragments into a single stream of media playback at an HTMLMediaElement. A brief summary of the progression of the proof of concept Proposal: Implement OfflineMediaContext #2824.

From the outset have had the sense that MediaSource could possibly be utilized to achieve part if not all of the requirement. However, had not located an existing or configured an appropriate pattern during own testing to realize the concatenation of discrete files using MediaSource.

Found this question and answer How do i append two video files data to a source buffer using media source api? which appeared to indicate that setting the .timestampOffset property of MediaSource could result in sequencing media playback of discrete buffers appended to SourceBuffer. Following the question led to a 2012 Editor's Draft Media Source Extensions
W3C Editor's Draft 8 October 2012 which states at 2.11. Applying Timestamp Offsets

Here is a simple example to clarify how timestampOffset can be used. Say I have two sounds I want to play in sequence. The first sound is 5 seconds long and the second one is 10 seconds. Both sound files have timestamps that start at 0. First append the initialization segment and all media segments for the first sound. Now set timestampOffset to 5 seconds. Finally append the initialization segment and media segments for the second sound. This will result in a 15 second presentation that plays the two sounds in sequence.

Which tried dozens of times using different patterns over the past several days. Interestingly all attempts using .webm video files failed; generally resulting in the following being logged at console at plnkr

Uncaught DOMException: Failed to execute 'appendBuffer' on 'SourceBuffer': This SourceBuffer has been removed from the parent media source.

Uncaught DOMException: Failed to set the 'timestampOffset' property on 'SourceBuffer': This SourceBuffer has been removed from the parent media source.

All of attempts using .mp4 video files failed save for a single .mp4 file which is a downloaded copy of "Big Buck Bunny" trailer. Not entirely sure where downloaded the file from during testing, though may have been "https://nickdesaulniers.github.io/netfix/demo/frag_bunny.mp4". Is this fact related to what is revealed at FFmpeg FAQ

A few multimedia containers (MPEG-1, MPEG-2 PS, DV) allow one to concatenate video by merely concatenating the files containing them.

?

Made a copy of the original file in the same directory. Used <input type="file"> with multiple attribute set to upload the files. Converted the File objects to ArrayBuffer using FileReader and used, in pertinent part, this pattern

           // code within a loop, within `Promise` constructor
           // `reader` is a `FileReader` instance
           reader.onload = () => {
              reader.onload = null;
              sourceBuffer.appendBuffer(reader.result);
              sourceBuffer.onupdateend = e => {
                sourceBuffer.onupdateend = null;
                // `chunk.mediaDuration` is `.duration` of media retrieved from `loadedmetadata` event
                sourceBuffer.timestampOffset += chunk.mediaDuration;
                // read next `File` object
                resolve()
              }
            }
            reader.readAsArrayBuffer(chunk.mediaBuffer);

The questions that have for authors and contributors to the specification are

1. What is the correct code pattern (as clear and definitive as possible) to use to append array buffers from discrete files or media fragments to one or more SourceBuffers of MediaSource, where the HTMLMediaElement renders playback of each of the files or media fragments?

2. Why was a single .mp4 which was copied the only two files which MediaSource correctly set the .duration of the to total time of the two files and rendered playback?

[guest271314]

Just tried with https://github.com/w3c/web-platform-tests/blob/master/media-source/mp4/test.mp4 and the .mp4 mentioned at OP, which properly plays the two files in sequence.

[guest271314]

However, if we place the array buffer of media having shortest duration before array buffer of media having longest duration the media playback stops rendering after first media file playback completes, though .currentTime of media is three seconds longer than shortest media file.

[guest271314]

Followup to previous post, if longest media is placed after shortest media within SourceBuffer and after shortest media stops playback we can seek to longest part of media, which plays the remainder of expected media.

[paulbrucecotton]

Have you experimented with different MSE implementations? Is there any chance this is an implementation bug in the browser you are using?

/paulc
HME WG Chair

[guest271314]

@paulbrucecotton Trying at Chromium 60.0.3112.78 (Developer Build). Have not tried at Firefox, yet. Seeking guidance on what the recommended or working pattern is to achieve expected result.

Can include the code tried here if that will be helpful.

[guest271314]

@paulbrucecotton Just tried at Firefox 55, though the codec video/mp4; codecs=avc1.42E01E, mp4a.40.2 does not appear to be supported.

[guest271314]

@paulbrucecotton Code tried using fetch() instead of <input type="file" multiple> http://plnkr.co/edit/KBbopiad1wR25nqtrvxw?p=preview

<!DOCTYPE html>
<html>

<head>
</head>

<body>
  <br>
  <video controls="true" autoplay="true"></video>

  <script>
    (async() => {


      const mediaSource = new MediaSource();

      const video = document.querySelector("video");

      // video.oncanplay = e => video.play();

      const urls = ["https://nickdesaulniers.github.io/netfix/demo/frag_bunny.mp4", "https://raw.githubusercontent.com/w3c/web-platform-tests/master/media-source/mp4/test.mp4"];

      const request = url => fetch(url).then(response => response.arrayBuffer());

      // `urls.reverse()` stops at `.currentTime` : `9`
      const files = await Promise.all(urls.map(request));

      /*
       `.webm` files
       Uncaught DOMException: Failed to execute 'appendBuffer' on 'SourceBuffer': This SourceBuffer has been removed from the parent media source.
       Uncaught DOMException: Failed to set the 'timestampOffset' property on 'SourceBuffer': This SourceBuffer has been removed from the parent media source.
      */
      // const mimeCodec = "video/webm; codecs=opus";
      // https://stackoverflow.com/questions/14108536/how-do-i-append-two-video-files-data-to-a-source-buffer-using-media-source-api/
      const mimeCodec = "video/mp4; codecs=avc1.42E01E, mp4a.40.2";


      const media = await Promise.all(files.map(file => {
        return new Promise(resolve => {
          let media = document.createElement("video");
          let blobURL = URL.createObjectURL(new Blob([file]));
          media.onloadedmetadata = async e => {
            resolve({
              mediaDuration: media.duration,
              mediaBuffer: file
            })
          }
          media.src = blobURL;
        })
      }));

      console.log(media);

      mediaSource.addEventListener("sourceopen", sourceOpen);

      video.src = URL.createObjectURL(mediaSource);

      async function sourceOpen(event) {

        if (MediaSource.isTypeSupported(mimeCodec)) {
          const sourceBuffer = mediaSource.addSourceBuffer(mimeCodec);

          for (let chunk of media) {
            await new Promise(resolve => {
              sourceBuffer.appendBuffer(chunk.mediaBuffer);
              sourceBuffer.onupdateend = e => {
                sourceBuffer.onupdateend = null;
                sourceBuffer.timestampOffset += chunk.mediaDuration;
                console.log(mediaSource.duration);
                resolve()
              }
            })

          }

          mediaSource.endOfStream();

        }  
        else {
          console.warn(mimeCodec + " not supported");
        }
      };

    })()
  </script>


</body>

</html>

One issue that have been facing is testing sourcing and using properly encoded media files served with CORS headers. Not sure if files created using MediaRecorder are encoded accurately.

[jyavenard]

@guest271314 wrote:

@paulbrucecotton Just tried at Firefox 55, though the codec video/mp4; codecs=avc1.42E01E, mp4a.40.2 does not appear to be supported.

What do you mean it's not supported? it certainly is
https://jsfiddle.net/hcfvyx9k/

[removed off topic comments]

[guest271314]

@jyavenard

What do you mean it's not supported? it certainly is
https://jsfiddle.net/hcfvyx9k/

The quotes surrounding value of codec key are the difference. Including approproiate quotes

'video/mp4; codecs="avc1.42E01E,mp4a.40.2"'

returns expected result at Firefox 55.

Still have yet to try pattern with more than two media files.

[guest271314]

@jyavenard Just tried with four requests for media files. Chromium 60 plays the files in sequence without an issue using same code pattern. Firefox 55 logged an apparent fetch() error

TypeError: NetworkError when attempting to fetch resource. (unknown)

http://plnkr.co/edit/9FYe4cJ6d4BC0B0LyOmN?p=preview, https://jsfiddle.net/hcfvyx9k/1/

[guest271314]

@jyavenard The media finally rendered playback at Firefox 55 with four requests. Not sure about the reason for previous errors? Timeout?

[jyavenard]

@guest271314 you had a space in your original mimetype separating the two codecs, so yes, quotes are required then. If you remove the space, there's no need for quotes.

Your fiddle worked first go here.
Glad it's working, but that's not using sequence mode as far as I can tell.
As far as network errors are concerned, looking into the network devtools (Command+Option+E on mac) would give you more details

[guest271314]

@jyavenard

Glad it's working, but that's not using sequence mode as far as I can tell.

Not sure what you mean as to sequence mode? What is the effective rendering difference between sequence and segments mode relevant to the expected result described at OP? Can you illuminate?

[jyavenard]

Oops my bad. I read the title and understood it as you using sequence mode. My first comment was about using the source buffer in sequence mode (as opposed to segment mode)

[guest271314]

@jyavenard

My first comment was about using the source buffer in sequence mode (as opposed to segment mode)

What is the effective difference between the two modes as to processing or rendering the media? Is "segments" the default mode?

[jyavenard]

http://w3c.github.io/media-source/#dom-mediasource-addsourcebuffer step 7 for the default value.

For an explanation of the different modes

http://w3c.github.io/media-source/#idl-def-appendmode

[guest271314]

@jyavenard Utilizing https://github.com/legokichi/ts-ebml was able to convert a Blob retrieved at dataavialable event of MediaRecorder recording the playback of a media fragment to a Blob where when passed to URL.createObjectURL() and set at an HTMLMediaElement has .duration set. Had the sense that that would then allow MediaSource to render the ArrayBuffer representation of Blob having metadata written. However, errors are thrown emitted from SourceBuffer Uncaught DOMException: Failed to set the 'timestampOffset' property on 'SourceBuffer': This SourceBuffer has been removed from the parent media source. at SourceBuffer.sourceBuffer.onupdateend.e.

The use case is trying to play media fragments at MediaSource, which appears to have the capability to render media playback with minimal apparent gap between discrete buffers.

At step 6. at first link does the algorithm expect the metadata within the file to be at a certain location within the file being read?

[guest271314]

RE: https://bugs.chromium.org/p/chromium/issues/detail?id=820489

I'm also a bit confused on the use case: is it meant to accommodate varying mime-type/codec media fragements' concatenation into a single resulting stream for later use? Or is there some other purpose.

MediaRecorder was used to (presumably) create a single type of file structure (EBML, webm) which could be parsed by MediaSource.

MediaSource was used because we cannot currently render multiple tracks at a <video> or <audio> element "seamlessly". The closest have been able to come is with audio, either concatenating Blobs to a single webm file or using Web Audio API OfflineAudioContext (which have not yet done so consistently).

Where we want to play specific segments of media content as a single stream to convey the resulting communication when such media is "concatenated" or "merged", and not rendered individually, or with no noticeable (to perception) gaps in in the media playback.

The concept is to, ideally

1. Request only media fragments (using for example Media Fragment URI);
2. Server responds with only media fragments (this does not currently occur and is, perhaps, outside the scope of this issue);
3. We should now have the options to
   a. render playback of media fragments individually
   b. render playback of media fragments as a single ("seamless") media stream ("media stream" should be clearly defined here; are referring to MediaSource or MediaStream or rendering at <canvas>, etc.)
   c. create a single file of the media fragments as a single media file (for example, for download);
   which incorporates both immediate usage of the media and later usage of the media;
4. The codecs, etc. of the original files are arbitrary; we cannot guarantee that the original individual media files having the content intended to be rendered as a single "stream" have the same file type; as long as the media is served with appropriate CORS headers an the <video> element at the browser can play the file type, we should be able to re-arrange the content in any conceivable manner at the browser - without using third-party polyfils or libraries;
5. We should be able to get, re-arrange and concatenate the media fragments in any order for 3.a. into a single file - without having to first play the files - and possibly faster than the time that would be necessary to play the media fragments individually.
6. The functionality of 1-5 should be able to be processed outside of a document; for example at a Worker thread; as we want to splice, re-arrange, concatenate, merge the media content as raw data, then either offer the file for download, stream the media as the normalization is occurring, or post the resulting media to a document for rendering.

The above is a brief synopsis of the concept. Based on the combined implementations of OfflineAudioContext, MediaSource and MediaRecorder; a conceptual OfflineMediaContext.

To a an approeciable extent the use case is possible using <canvas> and AudioContext and OfflineAudioContext, (see https://stackoverflow.com/q/40570114 ; https://bugs.chromium.org/p/chromium/issues/attachmentText?aid=328544) though the video media presently needs to be rendered first.

Am not sure if there is interest in a unified specification to merge the disparate APIs into a single "media editing" API, or if there is any interest at all in the use case outside of creating a "playlist" of media.

[guest271314]

The code could look something like

<!DOCTYPE html>
<html>
<head>
</head>
<body>
  <script>
    (async() => {
      let audioContext = new AudioContext();
      let urls = [{
        src: "https://upload.wikimedia.org/wikipedia/commons/a/a4/Xacti-AC8EX-Sample_video-001.ogv",
        from: 0,
        to: 4
      }, {
        src: "https://mirrors.creativecommons.org/movingimages/webm/ScienceCommonsJesseDylan_240p.webm",
        from: 10,
        to: 14
      }, {
        from: 55,
        to: 60,
        src: "https://nickdesaulniers.github.io/netfix/demo/frag_bunny.mp4"
      }, {
        from: 0,
        to: 5,
        src: "https://raw.githubusercontent.com/w3c/web-platform-tests/master/media-source/mp4/test.mp4"
      }, {
        from: 0,
        to: 5,
        src: "https://commondatastorage.googleapis.com/gtv-videos-bucket/sample/ForBiggerBlazes.mp4"
      }, {
        from: 0,
        to: 5,
        src: "https://commondatastorage.googleapis.com/gtv-videos-bucket/sample/ForBiggerJoyrides.mp4"
      }, {
        src: "https://commondatastorage.googleapis.com/gtv-videos-bucket/sample/ForBiggerMeltdowns.mp4",
        from: 0,
        to: 6
      }];

      let duration = urls.reduce((n, {
        from, to
      }) => n + (to - from), 0);
      let offlineAudioContext = new OfflineAudioContext(2, 44100 * 60, 44100);
      offlineAudioContext.onstatechange = e => console.log(e, e.target.state);

      let audioBuffers = await Promise.all(
        urls.map(({
          src
        }) => fetch(src).then(async(response) => audioContext.decodeAudioData(await response.arrayBuffer())))
      );

      let sources = audioBuffers.map(ab => {
        let source = offlineAudioContext.createBufferSource();
        source.buffer = ab;
        source.connect(offlineAudioContext.destination);
        return source;
      });
      let started = false;
      sources.reduce((promise, source, index) => { 
        return promise.then(() => new Promise(resolve => {
        source.start(0, urls[index].from, urls[index].to);
        if (started) {
          offlineAudioContext.resume();
        }
        source.onended = () => {
          resolve();
          if (!started) {
            started = true;
            offlineAudioContext.suspend(offlineAudioContext.currentTime);
          }
        };
      }))}, Promise.resolve());

      let rendering = offlineAudioContext.startRendering()
        .then(ab => {
          let source = audioContext.createBufferSource();
          source.buffer = ab;
          source.onended = e => console.log(e, source, offlineAudioContext, audioContext);
          source.connect(audioContext.destination);
          source.start(0);
        })
    })();
  </script>
</body>
</html>

or without using the resume method, which is not mentioned in the current specification

<!DOCTYPE html>
<html>
<head>
</head>
<body>
  <script>
    (async() => {
      let audioContext = new AudioContext();
      let urls = [{
        src: "https://upload.wikimedia.org/wikipedia/commons/a/a4/Xacti-AC8EX-Sample_video-001.ogv",
        from: 0,
        to: 4
      }, {
        src: "https://mirrors.creativecommons.org/movingimages/webm/ScienceCommonsJesseDylan_240p.webm",
        from: 10,
        to: 15
      }, {
        from: 55,
        to: 60,
        src: "https://nickdesaulniers.github.io/netfix/demo/frag_bunny.mp4"
      }, {
        from: 0,
        to: 5,
        src: "https://raw.githubusercontent.com/w3c/web-platform-tests/master/media-source/mp4/test.mp4"
      }, {
        from: 0,
        to: 5,
        src: "https://commondatastorage.googleapis.com/gtv-videos-bucket/sample/ForBiggerBlazes.mp4"
      }, {
        from: 0,
        to: 5,
        src: "https://commondatastorage.googleapis.com/gtv-videos-bucket/sample/ForBiggerJoyrides.mp4"
      }, {
        src: "https://commondatastorage.googleapis.com/gtv-videos-bucket/sample/ForBiggerMeltdowns.mp4",
        from: 0,
        to: 6
      }];

      let duration = urls.reduce((n, {
        from, to
      }) => n + (to - from), 0);
      let offlineAudioContext = new OfflineAudioContext(2, 44100 * duration, 44100);

      let audioBuffers = await Promise.all(
        urls.map(({
          src
        }) => fetch(src).then(async(response) => audioContext.decodeAudioData(await response.arrayBuffer())))
      );

      let sources = audioBuffers.map(ab => {
        let source = offlineAudioContext.createBufferSource();
        source.buffer = ab;
        source.connect(offlineAudioContext.destination);
        return source;
      });

      sources.reduce((promise, source, index) => { return promise.then(() => new Promise(resolve => {
        source.start(0, urls[index].from, urls[index].to);
        source.onended = resolve;
      }))}, Promise.resolve());

      let rendering = offlineAudioContext.startRendering()
        .then(ab => {
          let source = audioContext.createBufferSource();
          source.buffer = ab;
          source.onended = e => console.log(e, source, offlineAudioContext, audioContext);
          source.connect(audioContext.destination);
          source.start(0);
        })
    })();
  </script>
</body>
</html>

though the result from the above is inconsistent as to gaps in playback or playback at all (no video or audio track?) and we would be using a decodeMediaData method to get both audio and video data, perhaps separately, and concatenated into a single buffer.

[guest271314]

Finally composed a pattern using "segments" mode with .timestampOffset and .abort()

const sourceOpen = e => {
          console.log(e);
          const sourceBuffer = mediaSource.addSourceBuffer(mimeCodec);
          sourceBuffer.addEventListener("updateend", e => {
            if (mediaSource.duration === Infinity) {
              mediaSource.duration = duration;
            }
            console.log(e, video.currentTime, mediaSource.sourceBuffers[0].updating);
          });
          sourceBuffer.appendBuffer(chunks.shift());
        }
        const handleWaiting = e => {
          console.log(e, video.currentTime);
          if (chunks.length) {
            mediaSource.sourceBuffers[0].abort();
            mediaSource.sourceBuffers[0].timestampOffset = video.currentTime;
            mediaSource.sourceBuffers[0].appendBuffer(chunks.shift());
          } else {
            console.log(e);
            video.autoplay = false;
            video.removeEventListener("waiting", handleWaiting);
          }
        }

The audio and video are not precisely synchronized, though a start at using the https://plnkr.co/edit/E8OlvwiUmCwIUTOKSNkv at version 5.

Have not yet tried with MediaRecorder.

[guest271314]

@wolenetz

Was able to put together code which records MediaSource at Firefox 60

<!DOCTYPE html>
<html>

<head>
  <title>Record media fragments of any MIME type to single video using HTMLMediaElement.captureStream(), MediaRecorder, and MediaSource</title>
</head>

<body>
  <script>
    const captureStream = mediaElement =>
      !!mediaElement.mozCaptureStream ? mediaElement.mozCaptureStream() : mediaElement.captureStream();

    class MediaFragmentRecorder {
      constructor({
        urls = [], video = document.createElement("video"), width = 320, height = 280
      } = {}) {
        if (urls.length === 0) {
          throw new TypeError("no urls passed to MediaFragmentRecorder");
        }
        return (async() => {
          video.height = height;
          video.width = width;
          video.autoplay = true;
          video.preload = "auto";
          video.controls = true;
          // video.crossOrigin = "anonymous";
          const chunks = [];
          let duration = 0;
          let media = await Promise.all(
            urls.map(async({
              from, to, src
            }, index) => {
              const url = new URL(src);
              // get media fragment hash from `src`
              if (url.hash.length) {
                [from, to] = url.hash.match(/\d+/g);
              }
              return {
                blob: await fetch(src).then(response => response.blob()),
                from,
                to
              }
            }));
          // using `Promise.all()` here apparently is not the same
          // as recording the media in sequence as the next video
          // is not played when a buffer is added to `MediaSource`
          for (let {
              from, to, blob
            }
            of media) {
            await new Promise(async(resolve) => {
              let recorder;
              const blobURL = URL.createObjectURL(blob);
              video.addEventListener("playing", e => {
                const mediaStream = captureStream(video);
                recorder = new MediaRecorder(mediaStream, {
                  mimeType: "video/webm;codecs=vp8,opus"
                });
                recorder.start();
                recorder.addEventListener("stop", e => {
                  resolve();
                  console.log(e);
                }, {
                  once: true
                });
                recorder.addEventListener("dataavailable", async(e) => {
                  console.log(e);
                  chunks.push(await new Response(e.data).arrayBuffer());
                  URL.revokeObjectURL(blobURL);
                });
                video.addEventListener("pause", e => {
                  if (recorder.state === "recording") {
                    recorder.stop();
                  } else {
                    recorder.requestData();
                  }
                  console.log(video.played.end(0) - video.played.start(0), video.currentTime - from, video.currentTime);
                  duration += video.currentTime - from;
                }, {
                  once: true
                });

              }, {
                once: true
              });
              video.addEventListener("canplay", e => video.play(), {
                once: true
              });
              video.src = `${blobURL}#t=${from},${to}`;
            })
          };
          // using same `<video>` element at `.then()` does not
          // return expected result as to `autoplay`
          video.load();
          return {
            chunks, duration, width, height, video
          }
        })()
      }
    }
    let urls = [{
      src: "https://upload.wikimedia.org/wikipedia/commons/a/a4/Xacti-AC8EX-Sample_video-001.ogv",
      from: 0,
      to: 4
    }, {
      src: "https://mirrors.creativecommons.org/movingimages/webm/ScienceCommonsJesseDylan_240p.webm#t=10,20"
    }, {
      from: 55,
      to: 60,
      src: "https://nickdesaulniers.github.io/netfix/demo/frag_bunny.mp4"
    }, {
      from: 0,
      to: 5,
      src: "https://raw.githubusercontent.com/w3c/web-platform-tests/master/media-source/mp4/test.mp4"
    }, {
      from: 0,
      to: 5,
      src: "https://commondatastorage.googleapis.com/gtv-videos-bucket/sample/ForBiggerBlazes.mp4"
    }, {
      from: 0,
      to: 5,
      src: "https://commondatastorage.googleapis.com/gtv-videos-bucket/sample/ForBiggerJoyrides.mp4"
    }, {
      src: "https://commondatastorage.googleapis.com/gtv-videos-bucket/sample/ForBiggerMeltdowns.mp4#t=0,6"
    }];

    new MediaFragmentRecorder({
        urls
      })
      .then(({
        chunks, duration, width, height, video
      }) => {
        let recorder, mediaStream;
        console.log(chunks, duration);
        document.body.appendChild(video);
        const mediaSource = new MediaSource();
        const mimeCodec = "video/webm;codecs=vp8,opus";
        const sourceOpen = e => {
          console.log(e);
          const sourceBuffer = mediaSource.addSourceBuffer(mimeCodec);
          sourceBuffer.addEventListener("updateend", e => {
            console.log(e, video.currentTime, mediaSource);
          });
          sourceBuffer.appendBuffer(chunks.shift());
        }
        const handleWaiting = e => {
          console.log(e, video.currentTime, recorder && recorder.state);
          if (chunks.length) {
            // mediaSource.sourceBuffers[0].abort();
            mediaSource.sourceBuffers[0].timestampOffset = video.currentTime;
            mediaSource.sourceBuffers[0].appendBuffer(chunks.shift());
          } else {
            video.removeEventListener("waiting", handleWaiting);
            mediaSource.duration = video.currentTime;
            mediaSource.endOfStream();
            console.log(video.currentTime, duration, mediaSource.duration);
            try {
              if (recorder.state === "recording") {
                recorder.stop();
                video.removeEventListener("waiting", handleWaiting);
                mediaSource.duration = video.currentTime;
                mediaSource.endOfStream();
                console.log(video.currentTime, duration, mediaSource.duration);
              }
            } catch (e) {
              console.error(e.stack);
              console.trace();
            }
          }
        }
        mediaSource.sourceBuffers.addEventListener("addsourcebuffer", e => console.log(e));
        video.addEventListener("canplay", e => {
          console.log(video.readyState, video.paused);
          if (video.paused) {
            video.play();
          }
          console.log(e, duration, video.buffered.end(0), video.seekable.end(0), video.duration, mediaSource.duration);
        });
        video.addEventListener("playing", e => {
          
          mediaStream = captureStream(video);
          mediaStream.addEventListener("inactive", e => console.log(e));
          recorder = new MediaRecorder(mediaStream, {
            mimeType: "video/webm;codecs=vp8,opus"
          });
          recorder.addEventListener("dataavailable", e => {
            let media = document.createElement("video");
            media.width = width;
            media.height = height;
            media.controls = true;
            document.body.appendChild(media);
            media.src = URL.createObjectURL(e.data);
          });
          recorder.addEventListener("stop", e => console.log(e));
          recorder.start();
          
          console.log(e, duration, video.buffered.end(0), video.seekable.end(0), video.duration, mediaSource.duration);
        }, {
          once: true
        });
        video.addEventListener("waiting", handleWaiting);
        video.addEventListener("pause", e => console.log(e));
        video.addEventListener("stalled", e => console.log(e));
        video.addEventListener("loadedmetadata", e => console.log(e));
        video.addEventListener("loadeddata", e => console.log(e));
        video.addEventListener("seeking", e => console.log(e));
        video.addEventListener("seeked", e => console.log(e));
        video.addEventListener("durationchange", e => console.log(e));
        video.addEventListener("abort", e => console.log(e));
        video.addEventListener("emptied", e => console.log(e));
        video.addEventListener("suspend", e => console.log(e));
        mediaSource.addEventListener("sourceopen", sourceOpen);
        video.src = URL.createObjectURL(mediaSource);
      })
  </script>
</body>

</html>

Observations:

The implementations of MediaSource at Chromium and Firefox are different in several ways.

Firefox 60 using "segments" mode

1. Audio is not played at either
   a. recording of individual media fragments using mozCaptureStream();
   b. when ArrayBuffer representations of recorded Blobs of media fragments are passed to .appendBuffer();
2. Audio is played at the resulting recording of the playing <video> with MediaSource set at src;
3. .abort() does not need to be executed before setting .timestampOffset
4. autoplay set as either a property video.autoplay = true or video.setAttribute("autoplay", true) has no effect; the <video> does not begin playback when the attribute is set, canplay event needs to be used to execute video.play(), although autoplay does begin playback when using a src other than the recorded Blob converted to ArrayBuffer

<!DOCTYPE html>
<html>

<head>
</head>

<body>
  <video autoplay controls></video>
  <script>
    const video = document.querySelector("video");
    const mediaSource = new MediaSource();
    const mimeCodec = "video/webm;codecs=vp8,opus";
    const sourceOpen = async(e) => {
      console.log(e);
      const sourceBuffer = mediaSource.addSourceBuffer(mimeCodec);
      sourceBuffer.addEventListener("updateend", e => {
        console.log(e, video.currentTime, mediaSource.sourceBuffers[0].updating, mediaSource.duration);
      });
      sourceBuffer.appendBuffer(
        await fetch("https://mirrors.creativecommons.org/movingimages/webm/ScienceCommonsJesseDylan_240p.webm")
              .then(response => response.arrayBuffer())
      );
    }
    mediaSource.onsourceopen = sourceOpen;
    video.src = URL.createObjectURL(mediaSource);
  </script>
</body>

</html>

5. An error is thrown at waiting event handler when an object is referenced when no ArrayBuffers exist to append to the SourceBuffer; not certain if the error is due to a reference to MediaRecorder, MediaStream or MediaSource;
6. Infrequently the tab crashes; or else statement at waiting event handler is not reached

Chromium 64 using "segments" mode

Since presently we know the MediaRecorder usage crashes the tab, we comment those portions of code

1. .abort() needs to be executed before .timestampOffset is set;
2. the tab freezes during media playback at <video> element with MediaSource set as src
3. the tab crashes several times using only MediaSource code, where MediaRecorder code is commented

<!DOCTYPE html>
<html>

<head>
  <title>Record media fragments of any MIME type to single video using HTMLMediaElement.captureStream(), MediaRecorder, and MediaSource</title>
</head>

<body>
  <script>
    const captureStream = mediaElement =>
      !!mediaElement.mozCaptureStream ? mediaElement.mozCaptureStream() : mediaElement.captureStream();

    class MediaFragmentRecorder {
      constructor({
        urls = [], video = document.createElement("video"), width = 320, height = 280
      } = {}) {
        if (urls.length === 0) {
          throw new TypeError("no urls passed to MediaFragmentRecorder");
        }
        return (async() => {
          video.height = height;
          video.width = width;
          video.autoplay = true;
          video.preload = "auto";
          video.controls = true;
          // video.crossOrigin = "anonymous";
          const chunks = [];
          let duration = 0;
          let media = await Promise.all(
            urls.map(async({
              from, to, src
            }, index) => {
              const url = new URL(src);
              // get media fragment hash from `src`
              if (url.hash.length) {
                [from, to] = url.hash.match(/\d+/g);
              }
              return {
                blob: await fetch(src).then(response => response.blob()),
                from,
                to
              }
            }));
          // using `Promise.all()` here apparently is not the same
          // as recording the media in sequence as the next video
          // is not played when a buffer is added to `MediaSource`
          for (let {
              from, to, blob
            }
            of media) {
            await new Promise(async(resolve) => {
              let recorder;
              const blobURL = URL.createObjectURL(blob);
              video.addEventListener("playing", e => {
                const mediaStream = captureStream(video);
                recorder = new MediaRecorder(mediaStream, {
                  mimeType: "video/webm;codecs=vp8,opus"
                });
                recorder.start();
                recorder.addEventListener("stop", e => {
                  resolve();
                  console.log(e);
                }, {
                  once: true
                });
                recorder.addEventListener("dataavailable", async(e) => {
                  console.log(e);
                  chunks.push(await new Response(e.data).arrayBuffer());
                  URL.revokeObjectURL(blobURL);
                });
                video.addEventListener("pause", e => {
                  if (recorder.state === "recording") {
                    recorder.stop();
                  } else {
                    recorder.requestData();
                  }
                  console.log(video.played.end(0) - video.played.start(0), video.currentTime - from, video.currentTime);
                  duration += video.currentTime - from;
                }, {
                  once: true
                });

              }, {
                once: true
              });
              video.addEventListener("canplay", e => video.play(), {
                once: true
              });
              video.src = `${blobURL}#t=${from},${to}`;
            })
          };
          // using same `<video>` element at `.then()` does not
          // return expected result as to `autoplay`
          video.load();
          return {
            chunks, duration, width, height, video
          }
        })()
      }
    }
    let urls = [{
      src: "https://upload.wikimedia.org/wikipedia/commons/a/a4/Xacti-AC8EX-Sample_video-001.ogv",
      from: 0,
      to: 4
    }, {
      src: "https://mirrors.creativecommons.org/movingimages/webm/ScienceCommonsJesseDylan_240p.webm#t=10,20"
    }, {
      from: 55,
      to: 60,
      src: "https://nickdesaulniers.github.io/netfix/demo/frag_bunny.mp4"
    }, {
      from: 0,
      to: 5,
      src: "https://raw.githubusercontent.com/w3c/web-platform-tests/master/media-source/mp4/test.mp4"
    }, {
      from: 0,
      to: 5,
      src: "https://commondatastorage.googleapis.com/gtv-videos-bucket/sample/ForBiggerBlazes.mp4"
    }, {
      from: 0,
      to: 5,
      src: "https://commondatastorage.googleapis.com/gtv-videos-bucket/sample/ForBiggerJoyrides.mp4"
    }, {
      src: "https://commondatastorage.googleapis.com/gtv-videos-bucket/sample/ForBiggerMeltdowns.mp4#t=0,6"
    }];

    new MediaFragmentRecorder({
        urls
      })
      .then(({
        chunks, duration, width, height, video
      }) => {
        let recorder, mediaStream;
        console.log(chunks, duration);
        document.body.appendChild(video);
        const mediaSource = new MediaSource();
        const mimeCodec = "video/webm;codecs=vp8,opus";
        const sourceOpen = e => {
          console.log(e);
          const sourceBuffer = mediaSource.addSourceBuffer(mimeCodec);
          sourceBuffer.addEventListener("updateend", e => {
            console.log(e, video.currentTime, mediaSource);
          });
          sourceBuffer.appendBuffer(chunks.shift());
        }
        const handleWaiting = e => {
          console.log(e, video.currentTime);
          if (chunks.length) {
            mediaSource.sourceBuffers[0].abort();
            mediaSource.sourceBuffers[0].timestampOffset = video.currentTime;
            mediaSource.sourceBuffers[0].appendBuffer(chunks.shift());
          } else {
            video.removeEventListener("waiting", handleWaiting);
            mediaSource.duration = video.currentTime;
            mediaSource.endOfStream();
            console.log(video.currentTime, duration, mediaSource.duration);
            try {
              if (recorder && recorder.state === "recording") {
                recorder.stop();
              }
              video.removeEventListener("waiting", handleWaiting);
              mediaSource.duration = video.currentTime;
              mediaSource.endOfStream();
              console.log(video.currentTime, duration, mediaSource.duration);

            } catch (e) {
              console.error(e.stack);
              console.trace();
            }
          }

        }
        mediaSource.sourceBuffers.addEventListener("addsourcebuffer", e => console.log(e));
        video.addEventListener("canplay", e => {
          console.log(video.readyState, video.paused);
          if (video.paused) {
            video.play();
          }
          console.log(e, duration, video.buffered.end(0), video.seekable.end(0), video.duration, mediaSource.duration);
        });
        video.addEventListener("playing", e => {
          /*
          mediaStream = captureStream(video);
          mediaStream.addEventListener("inactive", e => console.log(e));
          recorder = new MediaRecorder(mediaStream, {
            mimeType: "video/webm;codecs=vp8,opus"
          });
          recorder.addEventListener("dataavailable", e => {
            let media = document.createElement("video");
            media.width = width;
            media.height = height;
            media.controls = true;
            document.body.appendChild(media);
            media.src = URL.createObjectURL(e.data);
          });
          recorder.addEventListener("stop", e => console.log(e));
          recorder.start();
          */
          console.log(e, duration, video.buffered.end(0), video.seekable.end(0), video.duration, mediaSource.duration);
        }, {
          once: true
        });
        video.addEventListener("waiting", handleWaiting);
        video.addEventListener("pause", e => console.log(e));
        video.addEventListener("stalled", e => console.log(e));
        video.addEventListener("loadedmetadata", e => console.log(e));
        video.addEventListener("loadeddata", e => console.log(e));
        video.addEventListener("seeking", e => console.log(e));
        video.addEventListener("seeked", e => console.log(e));
        video.addEventListener("durationchange", e => console.log(e));
        video.addEventListener("abort", e => console.log(e));
        video.addEventListener("emptied", e => console.log(e));
        video.addEventListener("suspend", e => console.log(e));
        mediaSource.addEventListener("sourceopen", sourceOpen);
        video.src = URL.createObjectURL(mediaSource);
      })
  </script>
</body>

</html>

4. video playback stops at stalled event at either the first or second media of appended buffer

Even where MediaRecorder code is omitted from the pattern, not sure how to reconcile the differences between implementations of MediaSource at Firefox and Chromium as to actual results of using "sequence" mode or "segments" mode to achieve the same result at each browser with the same code.

Questions:

1. Should bug reports be filed for Firefox as to
   a. the autoplay attribute set to true having no effect?
   b. audio being muted when MediaRecorder is used?
   c. audio being muted when ArrayBuffer representation of recorded Blob is set at SourceBuffer; both during initial playback and after .endOfStream() is called and video is seeked?
2. How to resolve the inconsistencies between Firefox and Chromium implementations?

[guest271314]

@wolenetz The code at #190 (comment) results in a webm file which is seekable and comprises the recorded media fragments. The code below tries to record media fragments, append buffers to MediaSource and record the <video> element with Blob URL representation of MediaSource ostensibly at the same time (offset by waiting event being dispatched). The issue with the code below is 1.b. at the previous comment; that is, Firefox MediaRecorder implementation does not record the audio of captured stream from the MediaSource; or at least the audio is not played from the resulting Blob URL of the recorded Blob. At the prior code the resulting Blob URL does output audio. This is probably as far as can go, from perspective here, with this concept outside of recording the media fragments faster than real-time.

<!DOCTYPE html>
<html>

  <head>
    <title>Record media fragments of any MIME type to single video using HTMLMediaElement.captureStream(), MediaRecorder, and MediaSource</title>
  </head>

  <body>
    <script>
      const captureStream = mediaElement =>
        !!mediaElement.mozCaptureStream ? mediaElement.mozCaptureStream() : mediaElement.captureStream();
      class MediaFragmentRecorder {
        constructor({
          urls = [],
          video = document.createElement("video"),
          width = 320,
          height = 280
        } = {}) {
          if (urls.length === 0) {
            throw new TypeError("no urls passed to MediaFragmentRecorder");
          }
          return (async () => {
            video.height = height;
            video.width = width;
            video.preload = "auto";
            video.controls = true;
            let mediaStream, streamRecorder, promise = Promise.resolve();
            const mediaSource = new MediaSource();
            const mimeCodec = "video/webm;codecs=vp8,opus";
            const videoStream = document.createElement("video");
            video.crossOrigin = "anonymous";
            videoStream.crossOrigin = "anonymous";
            const sourceOpen = e => {
              console.log(e);
              const sourceBuffer = mediaSource.addSourceBuffer(mimeCodec);
              sourceBuffer.addEventListener("updateend", e => {
                console.log(e, videoStream.currentTime, mediaSource);
              });
            }
            const handleWaiting = e => {
              console.log(e, videoStream.currentTime, streamRecorder && streamRecorder.state);
            }
            mediaSource.sourceBuffers.addEventListener("addsourcebuffer", e => console.log(e));
            videoStream.addEventListener("canplay", e => {
              videoStream.play();
              if (streamRecorder && streamRecorder.state === "paused") {
                streamRecorder.resume();
              }
              console.log(e, duration, videoStream.buffered.end(0), videoStream.seekable.end(0), videoStream.duration, mediaSource.duration, videoStream.readyState, videoStream.paused);
            });
            videoStream.addEventListener("playing", e => {
              mediaStream = captureStream(videoStream);
              mediaStream.addEventListener("inactive", e => console.log(e));
              streamRecorder = new MediaRecorder(mediaStream, {
                mimeType: "video/webm;codecs=vp8,opus"
              });
              promise = new Promise(resolve => {
                let data;
                streamRecorder.addEventListener("dataavailable", e => {
                  data = e.data;
                });
                streamRecorder.addEventListener("stop", e => {
                  video.load();
                  videoStream.load()
                  console.log("streamRecorder", e);
                  resolve(data);
                });
              });
              streamRecorder.start();

              console.log(e, duration, videoStream.buffered.end(0), videoStream.seekable.end(0), videoStream.duration, mediaSource.duration);
            }, {
              once: true
            });
            videoStream.addEventListener("waiting", e => console.log(e));
            videoStream.addEventListener("pause", e => console.log(e));
            videoStream.addEventListener("loadedmetadata", e => console.log(e));
            videoStream.addEventListener("loadeddata", e => console.log(e));
            videoStream.addEventListener("seeking", e => console.log(e));
            videoStream.addEventListener("pause", e => console.log(e));

            videoStream.addEventListener("seeked", e => console.log(e));
            videoStream.addEventListener("durationchange", e => console.log(e));
            videoStream.addEventListener("abort", e => console.log(e));
            videoStream.addEventListener("emptied", e => console.log(e));
            videoStream.addEventListener("suspend", e => console.log(e));



            mediaSource.addEventListener("sourceopen", sourceOpen, {
              once: true
            });
            videoStream.src = URL.createObjectURL(mediaSource);
            const chunks = [];
            let duration = 0;
            let media = await Promise.all(
              urls.map(async ({
                from,
                to,
                src
              }, index) => {
                const url = new URL(src);
                // get media fragment hash from `src`
                if (url.hash.length) {
                  [from, to] = url.hash.match(/\d+/g);
                }
                return {
                  blob: await fetch(src).then(response => response.blob()),
                  from,
                  to
                }
              }));
            // using `Promise.all()` here apparently is not the same
            // as recording the media in sequence as the next video
            // is not played when a buffer is added to `MediaSource`
            let chunk;
            for (let {
                from,
                to,
                blob
              } of media) {
              await new Promise(async (resolve) => {
                let recorder;
                const blobURL = URL.createObjectURL(blob);
                video.addEventListener("playing", e => {
                  const stream = captureStream(video);
                  recorder = new MediaRecorder(stream, {
                    mimeType: "video/webm;codecs=vp8,opus"
                  });
                  recorder.start();
                  recorder.addEventListener("stop", e => console.log(e), {
                    once: true
                  });
                  recorder.addEventListener("dataavailable", async (e) => {
                    console.log(e);
                    // mediaSource.sourceBuffers[0].abort();
                    mediaSource.sourceBuffers[0].timestampOffset = videoStream.currentTime;
                    mediaSource.sourceBuffers[0].appendBuffer(await new Response(e.data).arrayBuffer());
                    videoStream.addEventListener("waiting", e => {
                      console.log(e);
                      if (streamRecorder && streamRecorder.state === "recording") {
                        streamRecorder.pause();
                      }
                      resolve();
                    }, {
                      once: true
                    });

                    URL.revokeObjectURL(blobURL);
                  });
                  video.addEventListener("pause", e => {
                    if (recorder.state === "recording") {
                      recorder.stop();
                    } else {
                      recorder.requestData();
                    }
                    console.log(video.played.end(0) - video.played.start(0), video.currentTime - from, video.currentTime);
                    duration += video.currentTime - from;

                  }, {
                    once: true
                  });

                }, {
                  once: true
                });
                video.addEventListener("canplay", e => video.play(), {
                  once: true
                });
                video.src = `${blobURL}#t=${from},${to}`;
              })
            };
            streamRecorder.stop();

            return {
              data: await promise,
              video
            };
          })()
        }
      }
      let urls = [{
        src: "https://upload.wikimedia.org/wikipedia/commons/a/a4/Xacti-AC8EX-Sample_video-001.ogv",
        from: 0,
        to: 4
      }, {
        src: "https://mirrors.creativecommons.org/movingimages/webm/ScienceCommonsJesseDylan_240p.webm#t=10,20"
      }, {
        from: 55,
        to: 60,
        src: "https://nickdesaulniers.github.io/netfix/demo/frag_bunny.mp4"
      }, {
        from: 0,
        to: 5,
        src: "https://raw.githubusercontent.com/w3c/web-platform-tests/master/media-source/mp4/test.mp4"
      }, {
        from: 0,
        to: 5,
        src: "https://commondatastorage.googleapis.com/gtv-videos-bucket/sample/ForBiggerBlazes.mp4"
      }, {
        from: 0,
        to: 5,
        src: "https://commondatastorage.googleapis.com/gtv-videos-bucket/sample/ForBiggerJoyrides.mp4"
      }, {
        src: "https://commondatastorage.googleapis.com/gtv-videos-bucket/sample/ForBiggerMeltdowns.mp4#t=0,6"
      }];

      new MediaFragmentRecorder({
          urls
        })
        .then(({
          data,
          video
        }) => {
          document.body.appendChild(video);
          video.src = URL.createObjectURL(data);
        })

    </script>
  </body>

</html>

[guest271314]

@wolenetz Just tried the code at #190 (comment) at Chromium without MediaRecorder and including .abort() at jsfiddle and the tab crashed. Will try several more times to verify, then file Chromium issue if the crash persists.

[guest271314]

https://bugs.chromium.org/p/chromium/issues/detail?id=820997

[guest271314]

@wolenetz How to remedy the various differences between Firefox and Chromium implementations of MediaSource?

Can we work towards creating a specification which decodes any media file - potentially faster than real-time - into a media container, i.e.g., webm?

[guest271314]

The tab still crashes at Chromium.

[wolenetz]

This looks like further investigation is needed to see if this is a spec or implementation issue. Note also that WebCodecs incubations may support some of the underlying use case better, and that MediaRecorder implementation issues may also be contributing to issues. Further note, MSE changeType() could be used to switch among bytestreams and codecs buffered within a single SourceBuffer.

[guest271314]

@wolenetz The use case has been largely solved by the experiments that were successful in meeting the requirement at branches here https://github.com/guest271314/MediaFragmentRecorder/branches/all. In particular master branch was crashing at Chromium because of https://bugs.chromium.org/p/chromium/issues/detail?id=992235. Have not retested the MediaSource version at Chromium 87 or Nightly 83. Can close the issue if that would be helpful for the specification repository.

[guest271314]

This code https://github.com/guest271314/MediaFragmentRecorder/blob/master/MediaFragmentRecorder.html currently does not output expected results at Chromium 87.0.4270.0 (Official Build) snap (64-bit) Revision | 6fc672b0fa6a30d0e3426e4a3f8d418290855a9c-refs/branch-heads/4270@{#1} Linux V8 8.7.142 or Nightly 83.0a1 (2020-09-27) (64-bit). At one point the code achieved the same result at both browsers. Will try again substituting changeType() for

          sourceBuffer.abort();
          sourceBuffer.timestampOffset = video.currentTime;
          if (chunks.length) {
            sourceBuffer.appendBuffer(chunks.shift());
          }

[guest271314]

Chromium audio is still 6 seconds ahead of video https://bugs.chromium.org/p/chromium/issues/detail?id=1006617, even when using ts-ebml to set duration of WebM Blob at Chromium. Recording variable width, height video has apparently regressed, the video is recorded with the same initial pixel dimensions, at least using MediaSource and captureStream(), back to crashing the tab. Nightly 83 does no fare better. This is with using changeType()