Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix ffmpeg error with bin_data streams #15846

Closed
wants to merge 2 commits into from
Closed

Fix ffmpeg error with bin_data streams #15846

wants to merge 2 commits into from

Conversation

Tatsh
Copy link
Contributor

@Tatsh Tatsh commented Mar 13, 2018

Before submitting a pull request make sure you have:

In order to be accepted and merged into youtube-dl each piece of code must be in public domain or released under Unlicense. Check one of the following options:

  • I am the original author of this code and I am willing to release it under Unlicense
  • I am not the original author of this code but it is in public domain or released under Unlicense (provide reliable evidence)

What is the purpose of your pull request?

  • Bug fix
  • Improvement
  • New extractor
  • New feature

Description of your pull request and other information

Some files coming from YouTube and probably others come with chapter data (or subtitle?) in a stream seen by ffmpeg as the bin_data codec, usually at 0:2. Without some help, ffmpeg will simply refuse to make an mp4 file containing this. This causes the subtitle post-processor to fail, which also stops any further processing (embed thumbnail, etc).

    Stream #0:2(eng): Data: bin_data (text / 0x74786574), 0 kb/s
    Metadata:
      handler_name    : SubtitleHandler

This PR adjusts the subtitle post-processor to 'disable' (but not really) all data streams in the ffmpeg command. The resulting files still end up having the bin_data stream, but the SRT subtitle is now properly embedded before it:

    Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p(tv, bt709), 1920x1080 [SAR 1:1 DAR 16:9], 1234 kb/s, 29.97 fps, 29.97 tbr, 90k tbn, 59.94 tbc (default)
    Metadata:
      handler_name    : VideoHandler
    Stream #0:1(und): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 125 kb/s (default)
    Metadata:
      handler_name    : SoundHandler
    Stream #0:2(eng): Subtitle: mov_text (tx3g / 0x67337874), 0 kb/s (default)
    Metadata:
      handler_name    : SubtitleHandler
    Stream #0:3(eng): Data: bin_data (text / 0x74786574), 0 kb/s
    Metadata:
      handler_name    : SubtitleHandler
    Stream #0:4: Video: mjpeg, yuvj420p(pc, bt470bg/unknown/unknown), 1280x720 [SAR 1:1 DAR 16:9], 90k tbr, 90k tbn, 90k tbc

Example YouTube ID: KaAdS8Oso9U

Related ffmpeg issue

This also disables "Last message repeated" messages which cause non-zero exit status. Most of the time these messages are programmer issues for the ffmpeg team and do not apply to users.

Example YouTube ID: KaAdS8Oso9U

Also disable "Last message repeated" messages which cause
non-zero exit status
@dstftw
Copy link
Collaborator

dstftw commented Mar 13, 2018

Provide complete verbose log demonstrating the issue this PR is supposed to fix.

@Tatsh
Copy link
Contributor Author

Tatsh commented Mar 13, 2018

$ youtube-dl -v 'KaAdS8Oso9U'
[debug] System config: []
[debug] User config: [u'--restrict-filenames', u'--no-overwrites', u'--embed-subs', u'--add-metadata', u'--netrc', u'--format', u'bestvideo[ext!=webm]+bestaudio[ext!=webm]/best[ext!=webm]/best', u'--merge-output-format', u'mp4', u'--prefer-ffmpeg', u'--output', u'%(title).128s___src=%(extractor)s___id=%(id)s.%(ext)s', u'--embed-thumbnail', u'--prefer-ffmpeg', u'--skip-unavailable-fragments', u'--geo-bypass', u'--write-info-json', u'--write-sub', u'--all-subs', u'--convert-subs', u'srt', u'--external-downloader', u'aria2c', u'--external-downloader-args', u'-j 16 -s 16 -x 16 -k 5M', u'--force-ipv4']
[debug] Custom config: []
[debug] Command-line args: [u'-v', u'KaAdS8Oso9U']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2018.03.10
[debug] Git HEAD: 08250b69c
[debug] Python version 2.7.14 (CPython) - Linux-4.15.1-gentoo-limelight-x86_64-Intel-R-_Core-TM-_i7-5930K_CPU_@_3.50GHz-with-gentoo-2.4.1
[debug] exe versions: ffmpeg 3.3.6, ffprobe 3.3.6, rtmpdump 2.4
[debug] Proxy map: {}
[youtube] Downloading login page
[youtube] Looking up account info
[youtube] Logging in
Type 2-step verification code and press [Return]:
[youtube] Submitting TFA code
[youtube] Checking cookie
[youtube] KaAdS8Oso9U: Downloading webpage
[youtube] KaAdS8Oso9U: Downloading video info webpage
[youtube] KaAdS8Oso9U: Extracting video information
[info] Writing video subtitles to: Awkward_Megyn_Kelly_Interviews_That_Will_Make_You_Cringe___src=youtube___id=KaAdS8Oso9U.en
.vtt
[info] Writing video description metadata as JSON to: Awkward_Megyn_Kelly_Interviews_That_Will_Make_You_Cringe___src=youtube___id=KaAdS8Oso9U.info.json
[youtube] KaAdS8Oso9U: Downloading thumbnail ...
[youtube] KaAdS8Oso9U: Writing thumbnail to: Awkward_Megyn_Kelly_Interviews_That_Will_Make_You_Cringe___src=youtube___id=KaAd
S8Oso9U.jpg
[debug] Invoking downloader on u'https://...
[download] Destination: Awkward_Megyn_Kelly_Interviews_That_Will_Make_You_Cringe___src=youtube___id=KaAdS8Oso9U.f137.mp4
[debug] aria2c command line: ...
[aria2c] Downloaded 43200782 bytes
[download] 100% of 41.20MiB
[debug] Invoking downloader on u'https://r3-...
[download] Destination: Awkward_Megyn_Kelly_Interviews_That_Will_Make_You_Cringe___src=youtube___id=KaAdS8Oso9U.f140.m4a
[debug] aria2c command line: ...
[aria2c] Downloaded 4437831 bytes
[download] 100% of 4.23MiB
[ffmpeg] Merging formats into "Awkward_Megyn_Kelly_Interviews_That_Will_Make_You_Cringe___src=youtube___id=KaAdS8Oso9U.mp4"
[debug] ffmpeg command line: ffmpeg -y -i 'file:Awkward_Megyn_Kelly_Interviews_That_Will_Make_You_Cringe___src=youtube___id=K
aAdS8Oso9U.f137.mp4' -i 'file:Awkward_Megyn_Kelly_Interviews_That_Will_Make_You_Cringe___src=youtube___id=KaAdS8Oso9U.f140.m4a' -c copy -map '0:v:0' -map '1:a:0' 'file:Awkward_Megyn_Kelly_Interviews_That_Will_Make_You_Cringe___src=youtube___id=KaAdS8Oso9U.temp.mp4'
Deleting original file Awkward_Megyn_Kelly_Interviews_That_Will_Make_You_Cringe___src=youtube___id=KaAdS8Oso9U.f137.mp4 (pass -k to keep)
Deleting original file Awkward_Megyn_Kelly_Interviews_That_Will_Make_You_Cringe___src=youtube___id=KaAdS8Oso9U.f140.m4a (pass
 -k to keep)
[ffmpeg] Adding metadata to 'Awkward_Megyn_Kelly_Interviews_That_Will_Make_You_Cringe___src=youtube___id=KaAdS8Oso9U.mp4'
[debug] ffmpeg command line: ffmpeg -y -i 'file:Awkward_Megyn_Kelly_Interviews_That_Will_Make_You_Cringe___src=youtube___id=K
aAdS8Oso9U.mp4' -i 'file:Awkward_Megyn_Kelly_Interviews_That_Will_Make_You_Cringe___src=youtube___id=KaAdS8Oso9U.meta' -c copy -metadata 'comment=...
[ffmpeg] Converting subtitles
[debug] ffmpeg command line: ffmpeg -y -i 'file:Awkward_Megyn_Kelly_Interviews_That_Will_Make_You_Cringe___src=youtube___id=KaAdS8Oso9U.en.vtt' -f srt 'file:Awkward_Megyn_Kelly_Interviews_That_Will_Make_You_Cringe___src=youtube___id=KaAdS8Oso9U.en.srt'
Deleting original file Awkward_Megyn_Kelly_Interviews_That_Will_Make_You_Cringe___src=youtube___id=KaAdS8Oso9U.en.vtt (pass -k to keep)
[ffmpeg] Embedding subtitles in 'Awkward_Megyn_Kelly_Interviews_That_Will_Make_You_Cringe___src=youtube___id=KaAdS8Oso9U.mp4'
[debug] ffmpeg command line: ffmpeg -y -i 'file:Awkward_Megyn_Kelly_Interviews_That_Will_Make_You_Cringe___src=youtube___id=KaAdS8Oso9U.mp4' -i 'file:Awkward_Megyn_Kelly_Interviews_That_Will_Make_You_Cringe___src=youtube___id=KaAdS8Oso9U.en.srt' -map 0 -c copy -map '-0:s' '-c:s' mov_text -map '1:0' '-metadata:s:s:0' 'language=eng' 'file:Awkward_Megyn_Kelly_Interviews_That_Will_Make_You_Cringe___src=youtube___id=KaAdS8Oso9U.temp.mp4'
ERROR:     Last message repeated 1 times
Traceback (most recent call last):
  File "/home/tatsh/dev/youtube-dl/youtube_dl/YoutubeDL.py", line 2035, in post_process
    files_to_delete, info = pp.run(info)
  File "/home/tatsh/dev/youtube-dl/youtube_dl/postprocessor/ffmpeg.py", line 393, in run
    self.run_ffmpeg_multiple_files(input_files, temp_filename, opts)
  File "/home/tatsh/dev/youtube-dl/youtube_dl/postprocessor/ffmpeg.py", line 204, in run_ffmpeg_multiple_files
    raise FFmpegPostProcessorError(msg)
FFmpegPostProcessorError:     Last message repeated 1 times

The above Last message repeated 1 times is the wrong message to report, and it is not important for users most of the time. So I have filtered it out with -loglevel repeat+info.

The real error is [mp4 @ 0x55f54b93fc00] Could not find tag for codec bin_data in stream #2, codec not currently supported in container Could not write header for output file #0 (incorrect codec parameters ?): Invalid argument.

 $ ffmpeg -y -i 'file:Awkward_Megyn_Kelly_Interviews_That_Will_Make_You_Cringe___src=youtube___id=KaAdS8Oso9U.mp4' -i 'file:A
wkward_Megyn_Kelly_Interviews_That_Will_Make_You_Cringe___src=youtube___id=KaAdS8Oso9U.en.srt' -map 0 -c copy -map '-0:s' '-c
:s' mov_text -map '1:0' '-metadata:s:s:0' 'language=eng' 'file:Awkward_Megyn_Kelly_Interviews_That_Will_Make_You_Cringe___src
=youtube___id=KaAdS8Oso9U.temp.mp4'
ffmpeg version 3.3.6 Copyright (c) 2000-2017 the FFmpeg developers
  built with gcc 7.3.0 (Gentoo 7.3.0 p1.0)
  configuration: --prefix=/usr --libdir=/usr/lib64 --shlibdir=/usr/lib64 --docdir=/usr/share/doc/ffmpeg-3.3.6/html --mandir=/usr/share/man --enable-shared --cc=x86_64-pc-linux-gnu-gcc --cxx=x86_64-pc-linux-gnu-g++ --ar=x86_64-pc-linux-gnu-ar --optflags='-O2 -pipe -march=native -fomit-frame-pointer -ggdb' --disable-static --enable-avfilter --enable-avresample --disable-stripping --enable-nonfree --enable-version3 --disable-indev=oss --disable-indev=jack --disable-outdev=oss --enable-version3 --en
able-nonfree --enable-bzlib --disable-runtime-cpudetect --disable-debug --disable-gcrypt --disable-gnutls --disable-gmp --enable-gpl --enable-hardcoded-tables --enable-iconv --disable-lzma --enable-network --enable-openssl --enable-postproc --enable-libsmbclient --enable-ffplay --enable-sdl2 --enable-vaapi --enable-vdpau --enable-xlib --enable-libxcb --enable-libxcb-shm --enable-libxcb-xfixes --enable-zlib --enable-libcdio --enable-libiec61883 --disable-libdc1394 --enable-libcaca --disable-opena
l --enable-opengl --disable-libv4l2 --disable-libpulse --enable-libopencore-amrwb --enable-libopencore-amrnb --enable-libfdk-aac --enable-libopenjpeg --enable-libbluray --disable-libcelt --enable-libgme --enable-libgsm --disable-mmal --enable-libmodplug --enable-libopus --disable-libilbc --enable-librtmp --enable-libssh --enable-libschroedinger --enable-libspeex --enable-libvorbis --enable-libvpx --enable-libzvbi --disable-libbs2b --enable-chromaprint --enable-libflite --disable-frei0r --enable-
libfribidi --disable-fontconfig --disable-ladspa --enable-libass --enable-libfreetype --enable-librubberband --disable-netcdf --disable-libzmq --enable-libzimg --disable-libsoxr --enable-pthreads --disable-libvo-amrwbenc --enable-libmp3lame --disable-libkvazaar --enable-nvenc --disable-libopenh264 --disable-libsnappy --enable-libtheora --enable-libtwolame --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxvid --disable-amd3dnow --disable-amd3dnowext --disable-fma
4 --disable-xop --cpu=host --disable-doc --disable-htmlpages --enable-manpages
  libavutil      55. 58.100 / 55. 58.100
  libavcodec     57. 89.100 / 57. 89.100
  libavformat    57. 71.100 / 57. 71.100
  libavdevice    57.  6.100 / 57.  6.100
  libavfilter     6. 82.100 /  6. 82.100
  libavresample   3.  5.  0 /  3.  5.  0
  libswscale      4.  6.100 /  4.  6.100
  libswresample   2.  7.100 /  2.  7.100
  libpostproc    54.  5.100 / 54.  5.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'file:Awkward_Megyn_Kelly_Interviews_That_Will_Make_You_Cringe___src=youtube___id=KaA
dS8Oso9U.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    title           : Awkward Megyn Kelly Interviews That Will Make You Cringe
    artist          : Nicki Swift
    date            : 20171219
    encoder         : Lavf57.71.100
  Duration: 00:04:39.36, start: 0.000000, bitrate: 1368 kb/s
    Chapter #0:0: start 0.000000, end 62.000000
    Metadata:
      title           : The cast of Will and Grace |
    Chapter #0:1: start 62.000000, end 103.000000
    Metadata:
      title           : Jane Fonda |
    Chapter #0:2: start 103.000000, end 134.000000
    Metadata:
      title           : The Menendez Brothers |
    Chapter #0:3: start 134.000000, end 173.000000
    Metadata:
      title           : Suzanne Somers |
    Chapter #0:4: start 173.000000, end 212.000000
    Metadata:
      title           : Carli Lloyd |
    Chapter #0:5: start 212.000000, end 279.000000
    Metadata:
      title           : Russell Brand |
    Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p(tv, bt709), 1920x1080 [SAR 1:1 DAR 16:9], 1234 kb/s, 29.97 fps, 29.97 tbr, 90k tbn, 59.94 tbc (default)
    Metadata:
      handler_name    : VideoHandler
    Stream #0:1(und): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 125 kb/s (default)
    Metadata:
      handler_name    : SoundHandler
    Stream #0:2(eng): Data: bin_data (text / 0x74786574), 0 kb/s
    Metadata:
      handler_name    : SubtitleHandler
Input #1, srt, from 'file:Awkward_Megyn_Kelly_Interviews_That_Will_Make_You_Cringe___src=youtube___id=KaAdS8Oso9U.en.srt':
  Duration: N/A, bitrate: N/A
    Stream #1:0: Subtitle: subrip
[mp4 @ 0x55f54b93fc00] Could not find tag for codec bin_data in stream #2, codec not currently supported in container
Could not write header for output file #0 (incorrect codec parameters ?): Invalid argument
Stream mapping:
  Stream #0:0 -> #0:0 (copy)
  Stream #0:1 -> #0:1 (copy)
  Stream #0:2 -> #0:2 (copy)
  Stream #1:0 -> #0:3 (subrip (srt) -> mov_text (native))
    Last message repeated 1 times

To fix the command, add -map '-0:d' after the -map 0:

ffmpeg -y -i 'file:Awkward_Megyn_Kelly_Interviews_That_Will_Make_You_Cringe___src=youtube___id=KaAdS8Oso9U.mp4' -i 'file:Awkward_Megyn_Kelly_Interviews_That_Will_Make_You_Cringe___src=youtube___id=KaAdS8Oso9U.en.srt' \
-map 0 -c copy \
-map '-0:s' \
-map '-0:d' \
'-c:s' mov_text -map '1:0' '-metadata:s:s:0' 'language=eng' 'file:Awkward_Megyn_Kelly_Interviews_That_Will_Make_You_Cringe___src=youtube___id=KaAdS8Oso9U.temp.mp4

@Tatsh
Copy link
Contributor Author

Tatsh commented Jan 26, 2019

@dstftw what are the fixes you are looking for?

@@ -379,6 +380,8 @@ def run(self, information):
# Don't copy the existing subtitles, we may be running the
# postprocessor a second time
'-map', '-0:s',
# Don't copy Apple TV subtitles format, bin_data
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a chapters track as explained in the ffmpeg issue you linked that appears after chapters metadata is embedded by --add-metadata.

@@ -190,6 +190,7 @@ def run_ffmpeg_multiple_files(self, input_paths, out_path, opts):
encodeFilename(self._ffmpeg_filename_argument(path), True)
])
cmd = ([encodeFilename(self.executable, True), encodeArgument('-y')] +
['-loglevel', 'repeat+info'] +
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't mix unrelated changes in single PR.

@Tatsh
Copy link
Contributor Author

Tatsh commented Jan 28, 2019

I cannot push to this branch anymore. I will open a new PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants