audio_index=all broken with Sintel since c170481

[Tjoppen]

Steps to reproduce with current master (fda7c02):

1. Download https://commons.wikimedia.org/wiki/File:Sintel_movie_720x306.ogv
2. Run melt -producer Sintel_movie_720x306.ogv audio_index=all
3. Notice that the audio is broken

Reverting c170481 fixes the issue, but probably brings back the issue that that commit is meant to fix.

Tested together with FFmpeg versions 5.1.4, 6.1.1 and 7.0.1.

[ddennedy]

1. This example file does not have multiple audio streams.
2. With my own non-shareable sample with 2 stereo audio streams for 2 different languages. when I play it with audio_index=all and -consumer sdl2 channels=4 I hear both at the same time during spoken footage.
3. You ought to describe "broken." Initially I was checking for silence or not all channels of all streams. However, I found it does have cracking artifacts, i.e. distorted.
4. Reverting the commit breaks the purpose of "all." It prevents the problem only because that makes it effectively ignore "all" and you did not test further back. It is not the root cause of the audio artifacts.

[Tjoppen]

I couldn't figure out a good word to describe the issue, sorry about that. Yes it sounds distorted/crackling.

I would expect audio_index=all to not arbitrarily stop working just because a file has only one stream. If it can flatten files with two or more streams then surely it should be able to do it with one stream even if it's a no-op in that case. Else workflows that want "flattened" audio out of any input need special attention for files with single audio streams.

I intend to investigate the issue further. There is no crackling with a file with one 8 channel stream. For reference, ffprobe output for Sintel (crackling):

Input #0, ogg, from 'Sintel_movie_720x306.ogv':
  Duration: 00:14:48.06, start: 0.000000, bitrate: 781 kb/s
  Stream #0:0: Data: none
  Stream #0:1: Video: theora, yuv420p(bt470bg/bt470bg/bt709), 720x306 [SAR 1:1 DAR 40:17], 24 tbr, 24 tbn
      Metadata:
        ENCODER         : ffmpeg2theora-0.24
  Stream #0:2: Audio: vorbis, 48000 Hz, stereo, fltp, 128 kb/s
      Metadata:
        ENCODER         : ffmpeg2theora-0.24

The 8 channel file (no crackling):

Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'V75UPPSNACK_LORDAG_P01_195335.mov':
  Metadata:
    major_brand     : qt  
    minor_version   : 512
    compatible_brands: qt  
    creation_time   : 2024-02-08T18:54:44.000000Z
  Duration: 00:07:36.12, start: 0.000000, bitrate: 139508 kb/s
  Stream #0:0[0x1](eng): Video: speedhq (SHQ2 / 0x32514853), yuv422p(bt470bg/unknown/unknown, progressive), 1920x1080, 124942 kb/s, SAR 1:1 DAR 16:9, 50 fps, 50 tbr, 12800 tbn (default)
      Metadata:
        creation_time   : 2024-02-08T18:54:44.000000Z
        handler_name    : VideoHandler
        vendor_id       : NewT
  Stream #0:1[0x2](eng): Audio: pcm_f32le (lpcm / 0x6D63706C), 48000 Hz, 8 channels, flt, 12288 kb/s (default)
      Metadata:
        creation_time   : 2024-02-08T18:54:44.000000Z
        handler_name    : SoundHandler
        vendor_id       : [0][0][0][0]

[Tjoppen]

The issue has to do with pick_audio_stream() not handling astream=all correctly. It is set to that further down in pick_audio_stream() when audio_index=all. This causes mlt_properties_get() to return non-NULL, which leads to mlt_properties_get_int() returning zero, which gets translated to absolute_index=2 which is returned. This in turn causes producer_set_up_audio() to avcodec_free_context() every frame for all audio streams because it thinks the audio stream has changed every frame.

I'm not quite sure what the logic of pick_audio_stream() should be when audio_index=all..