multitrack.md

Multitrack support

Starting with version 2.2.0, liquidsoap now supports track operations, making it possible to manipulate data at the track level, including demuxing, remuxing, encoding, decoding, applying filters and more!

Only the FFmpeg decoder and encoder supports multitrack. This means that you need to have liquidsoap compiled with the FFmpeg support to be able to decode or encode sources with multiple audio/video tracks.

Support for track muxing and demuxing and track-level operators, however, does not require the FFmpeg support but, without it, all decoders and outputs are limited to at most one audio and one video track.

Multitrack sources

Liquidsoap sources can have multiple tracks, such as an english language audio track and a french language audio track. The number of tracks in a source is determined by how you use it.

For instance, if you have a file movie.mkv with two audio tracks and one video track, you can create a source s with it using the single operator (or playlist, request.dynamic etc):

s = single("/path/to/movie.mkv")

By default, liquidsoap decodes only the track that you tell it to pick. So, if you output this source as an output with only one audio track, it will happily do so:

Resulting in the following logs:

[output_file:3] Content type is {audio=ffmpeg.copy,video=ffmpeg.copy}.
[decoder.ffmpeg:3] Requested content-type for "/path/to/movie.mkv": {audio=ffmpeg.copy,video=ffmpeg.copy}
[decoder.ffmpeg:3] FFmpeg recognizes "/path/to/movie.mkv" as video: {codec: h264, 1920x1038, yuv420p}, audio: {codec: aac, 48000Hz, 6 channel(s)}, audio_2: {codec: aac, 48000Hz, 6 channel(s)}
[decoder.ffmpeg:3] Decoded content-type for "/path/to/movie.mkv": {audio=ffmpeg.copy(codec="aac",channel_layout="5.1",sample_format=fltp,sample_rate=48000),video=ffmpeg.copy(codec="h264",width=1920,height=1038,aspect_ratio=1/1,pixel_format=yuv420p)}

This shows that the output.file source was initialized with expected content-type {audio=ffmpeg.copy,video=ffmpeg.copy}, i.e. one audio and one video track, both copied from the original file. This comes from the %ffmpeg encoder definition in our script.

Then, the single file was decoded with the same requested content-type and FFmpeg reported all the details about the file's content, including a second audio track named audio_2.

Eventually, we picked up only first audio and first video track and reported a more detailed content-type now that we know the actual content of each track.

Now, let's say that we want to also keep the second audio track but convert it to stereo and re-encode it into aac. We can then do:

And now we see the following logs:

[output_file:3] Content type is {audio=ffmpeg.copy,audio_2=pcm(stereo),video=ffmpeg.copy}.
[decoder.ffmpeg:3] Requested content-type for "/path/to/movie.mkv": {audio=ffmpeg.copy,audio_2=pcm(stereo),video=ffmpeg.copy}
[decoder.ffmpeg:3] FFmpeg recognizes "/path/to/movie.mkv" as video: {codec: h264, 1920x1038, yuv420p}, audio: {codec: aac, 48000Hz, 6 channel(s)}, audio_2: {codec: aac, 48000Hz, 6 channel(s)}
[decoder.ffmpeg:3] Decoded content-type for "/path/to/movie.mkv": {audio=ffmpeg.copy(codec="aac",channel_layout="5.1",sample_format=fltp,sample_rate=48000),audio_2=pcm(5.1),video=ffmpeg.copy(codec="h264",width=1920,height=1038,aspect_ratio=1/1,pixel_format=yuv420p)}

Now, we are actually using both audio tracks from movie.mkv and one of them is being converted to stereo audio!

One thing to keep in mind, however, is that expected content-type drives the input decoder. Typically, if, instead of a single, you use a playlist:

Then all the files in the playlist who do not have at least two audio tracks and one video track will be rejected by the decoder!

Lastly, it is important to keep in mind that decoders always assume a specific nomenclature for tracks. The convention when decoding is to name the first audio track audio, then audio_2, audio_3 and etc. Likewise for video: video, video_2, video_3.

Typically, the following will not work:

s = playlist("/path/to/playlist")

# Copy first audio track and video track
# and re-encode second audio track:
output.file(
  fallible=true,
  %ffmpeg(
    %audio_fr.copy,
    %audio_en(
      channels=2,
      codec="aac"
    ),
    %video.copy
  ),
  "/path/to/copy.mkv",
  s
)

This is because the decoder has no way of knowing which of the audio track present in the files in s should be matched to audio_en and audio_fr.copy.

One might think that the order in which they are declared in the encoder could be used but this would also be tricky as the decoder could report tracks in different order when decoding different files from the playlist.

Also, and perhaps more importantly, tracks can be demuxed and remuxed at will, which also makes us loose the notion of track order. Actually, let's talk about demuxing and remuxing next!

Tracks demuxing and muxing

For any given source, you can extract its tracks using the source.tracks operator:

s = playlist(...)

let {audio, video, metadata, track_marks} = source.tracks(s)

In the above, audio and video represent, resp., the audio and video track from the source s.

The metadata and track_marks tracks are special track type that are available in any source and hold, as the name suggests, the source's metadata and track marks. We will see later how this can be used to e.g. drop all tracks from a source (something that used to be done with the drop_tracks operator), or select metadata only from a specific source or track.

Internally, a track is a source restricted to a single content-type. This means that:

• When pulling data for a given track, the underlying source is used, potentially also pulling data for its other tracks
• Tracks are subject to the same limitations as sources w.r.t. clocks
• Tracks, like sources, always have a metadata and track_marks tracks. The track.metadata and track.track_marks operators can be used to retrieve them.

Tracks can be muxed using the source operator. The operator takes a record of tracks and creates a source with them. Tracks can have any name and type except metadata and track_marks that are reserved for their corresponding track types.

Add a video track

Here's how to add a video track to a source

The above example was purposely written in a longer form to make it more explicit. However, if you wish to just add/replace a track, you can also overload the existing tracks from the first source as follows:

Add a default video track

You can also check if a source has a certain track and do something accordingly:

Please note, however, that tracks available in the playlist sources are determined based on the first decoded file. If the first file in the playlist is audio-only then the playlist content-type is assumed to be audio-only for the whole playlist and the default video is added to all decoded files.

To decide on a case-by-case basis, you might need some more advanced coding!

Merge all tracks

As mentioned before, you can also remove the track marks from a source as follows:

s = playlist(...)

# Extract all tracks except track_marks:
let {track_marks=_, ...tracks} = source.tracks(s)

s = source(tracks)

Track-level operators

Some, but not all, operators have been updated to operate at the track level. They are documented under the Track section in the API documentation. More operators might be converted in the future (feel free to file a feature request for those!).

For instance, to convert an audio track to mono PCM audio, one can do:

mono_track = track.audio.mean(audio_track)

Likewise, inline encoders are now available at the track level, for instance:

encoded_audio_track = track.ffmpeg.encode.audio(%ffmpeg(%audio(codec="aac")), audio_track)

However, remember that tracks have the same limitations w.r.t. clocks that sources have. Here, in particular, encoded_audio_track is in a new clock (due to the fact that ffmpeg inline encoding is not synchronous). Therefore, the following will fail:

s = playlist(...)

let {audio, metadata, track_marks} = source.tracks(s)

encoded = source({
  audio       = track.ffmpeg.encode.audio(%ffmpeg(%audio(codec="aac")), audio),
  metadata    = metadata,
  track_marks = track_marks
})

This is because metadata and track_marks are tracks from the underlying s source, which belongs to a different clock. In this case, you should use the track marks and metadata from the encoded track:

s = playlist(...)

let {audio, metadata, track_marks} = source.tracks(s)

encoded_audio = track.ffmpeg.encode.audio(%ffmpeg(%audio(codec="aac")), audio)

encoded = source({
  audio       = encoded_audio,
  metadata    = track.metadata(encoded_audio),
  track_marks = track_marks(encoded_audio)
})

Conventions

Now that we have seen how we can create any collection of tracks with any possible name, in order to make things work, we need to assume a couple of conventions.

For decoders, the convention, as explained above, is, when decoding files, to name the first audio track audio, then audio_2, audio_3 and etc. Likewise for video: video, video_2, video_3.

This is the convention that you should use when demuxing tracks from request-based source:

s = playlist(...)

let {audio, audio_2, video, video_2, video_3} = source.tracks(s)

For encoders, to drive content-type at runtime, since tracks can be remuxed with any arbitrary name, we need to a way to decide what type of content a track contains, being audio, video or, potentially midi and, planned for later, subtitles. This is achieved using the following convention, by order of priority:

1. A copy track is any track named %<track_name>.copy. We do not need to know the track's content in this case.
2. If a track has audio_content or video_content as parameter (for instance %foo(audio_content, ...)) then it is considered, resp., audio or video.
3. If the track name has audio or video in it (for instance %dolby_audio_fr) then it is considered, resp., audio or video
4. If the track codec is hardcoded (for instance (%foo(codec="aac", ...)) then the codec is used to detect the content.

For instance, imagine that you want to encode a source with a fr and en audio track and a director_cut video track, you can do the following:

output.file(
  %ffmpeg(
    %en(codec="aac"),
    %fr(codec="aac"),
    %director_cut(codec="libx264")
  ),
  "/path/to/copy.mkv",
  s
)

This works because each of the codec used in these tracks can be mapped to a specific content-type.

Now, imagine that you actually want to use a variable for the codec of the en and director_cut track. In this case, you can do:

output.file(
  %ffmpeg(
    %en(audio_content, codec=audio_codec)
    %fr(codec="aac"),
    %director_cut(video_content, codec=video_codec)
  ),
  "/path/to/copy.mkv",
  s
)

This informs liquidsoap what type of content these tracks contain.

However, you might also opt for a more explicit track naming scheme. Something like:

output.file(
  %ffmpeg(
    %audio_en(codec=audio_codec),
    %audio_fr(codec="aac"),
    %director_cut_video(codec=video_codec)
  ),
  "/path/to/copy.mkv",
  s
)

In this case, liquidsoap assumes that the track with audio in their name are indeed audio track and the same goes for video tracks.

Lastly, these naming conventions have no bearing for the FFmpeg encoder. At the FFmpeg encoder level, tracks are identified by an integer and stored in the order they are declared in the %ffmpeg encoder. This means that, once encoded and saved to a file, track names internal to liquidsoap are not saved by the FFmpeg encoder and, instead, when decoding the file, you will get audio, audio_2, video and etc.