Cannot use VideoReader due to ffmpeg issue

[yiwen-song]

Hey experts,

Recently I've been trying to use the torchvision.io.VideoReader API on my MacOS machine.

Description

At first, I was trying to import the API directly, but hitting this issue:

  File "/Users/sallysyw/workspace/vision/gallery/plot_video_api.py", line 36, in <module>
    video = torchvision.io.VideoReader(video_path, stream)
  File "/Users/sallysyw/workspace/vision/torchvision/io/__init__.py", line 104, in __init__
    raise RuntimeError(
RuntimeError: Not compiled with video_reader support, to enable video_reader support, please install ffmpeg (version 4.2 is currently supported) andbuild torchvision from source.

Then I realized that I need to manually install ffmpeg package to get this work.
Firstly I tried conda install:

conda install ffmpeg=4.2=h677a3f5_0

After this is done, I re-built the torchvision package with

MACOSX_DEPLOYMENT_TARGET=10.9 CC=clang CXX=clang++ python setup.py develop

Then I tried

import torchvision

this time I'm hitting a new error:

libc++abi: terminating with uncaught exception of type c10::Error: Type c10::tagged_capsule<vision::video::Video> could not be converted to any of the known types.
Exception raised from operator() at /Users/sallysyw/opt/anaconda3/lib/python3.8/site-packages/torch/include/ATen/core/jit_type.h:1606 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >) + 98 (0x102646e12 in libc10.dylib)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) + 106 (0x10264552a in libc10.dylib)
frame #2: c10::detail::getTypePtr_<c10::tagged_capsule<vision::video::Video> >::call()::'lambda'()::operator()() const + 263 (0x129f49497 in video_reader.so)
frame #3: c10::detail::getTypePtr_<c10::tagged_capsule<vision::video::Video> >::call() + 27 (0x129f492eb in video_reader.so)
frame #4: c10::detail::infer_schema::(anonymous namespace)::createArgumentVector(c10::ArrayRef<c10::detail::infer_schema::ArgumentDef>) + 216 (0x10e27c508 in libtorch_cpu.dylib)
frame #5: c10::detail::infer_schema::make_function_schema(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&&, c10::ArrayRef<c10::detail::infer_schema::ArgumentDef>, c10::ArrayRef<c10::detail::infer_schema::ArgumentDef>) + 123 (0x10e27c2ab in libtorch_cpu.dylib)
frame #6: torch::jit::Function* torch::class_<vision::video::Video>::defineMethod<torch::class_<vision::video::Video>& torch::class_<vision::video::Video>::def<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >(torch::detail::types<void, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::initializer_list<torch::arg>)::'lambda'(c10::tagged_capsule<vision::video::Video>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >)>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::initializer_list<torch::arg>) + 477 (0x129f4872d in video_reader.so)
frame #7: torch::class_<vision::video::Video>& torch::class_<vision::video::Video>::def<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >(torch::detail::types<void, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::initializer_list<torch::arg>) + 114 (0x129f35242 in video_reader.so)
frame #8: _GLOBAL__sub_I_video.cpp + 226 (0x129f5b202 in video_reader.so)
frame #9: 0x0 + 4338874055 (0x1029df6c7 in ???)
frame #10: 0x0 + 4338875090 (0x1029dfad2 in ???)
frame #11: 0x0 + 4338853046 (0x1029da4b6 in ???)
frame #12: 0x0 + 4338844271 (0x1029d826f in ???)
frame #13: 0x0 + 4338844432 (0x1029d8310 in ???)
frame #14: 0x0 + 4338780690 (0x1029c8a12 in ???)
frame #15: 0x0 + 4338825966 (0x1029d3aee in ???)
frame #16: dlopen_internal(char const*, int, void*) + 185 (0x7fff203edcb4 in libdyld.dylib)
frame #17: dlopen + 28 (0x7fff203dc09e in libdyld.dylib)
frame #18: py_dl_open + 135 (0x102303ca7 in _ctypes.cpython-38-darwin.so)
<omitting python frames>

[1]    216 abort      python

I also tried the following options:

1. Download ffmpeg-4.2.4.tar.xz on their website and build it manually
2. Uninstall the conda package and use brew to install ffmpeg again

However, neither of them solves the issue, I'm still getting the python frames error.
Can anyone help? Thanks a lot!

Environment

wget https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py
# For security purposes, please check the contents of collect_env.py before running it.
python collect_env.py

Results:

Collecting environment information...
PyTorch version: 1.10.0.dev20210728
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 11.4 (x86_64)
GCC version: Could not collect
Clang version: 11.0.0 (clang-1100.0.33.17)
CMake version: Could not collect
Libc version: N/A

Python version: 3.8.8 (default, Apr 13 2021, 12:59:45)  [Clang 10.0.0 ] (64-bit runtime)
Python platform: macOS-10.16-x86_64-i386-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.20.1
[pip3] numpydoc==1.1.0
[pip3] pytorch-sphinx-theme==0.0.24
[pip3] torch==1.10.0.dev20210728
[pip3] torchvision==0.11.0a0+b29ed34
[conda] blas                      1.0                         mkl
[conda] mkl                       2021.2.0           hecd8cb5_269
[conda] mkl-service               2.3.0            py38h9ed2024_1
[conda] mkl_fft                   1.3.0            py38h4a7008c_2
[conda] mkl_random                1.2.1            py38hb2f4e1b_2
[conda] numpy                     1.20.1           py38hd6e1bb9_0
[conda] numpy-base                1.20.1           py38h585ceec_0
[conda] numpydoc                  1.1.0              pyhd3eb1b0_1
[conda] pytorch                   1.10.0.dev20210728         py3.8_0    pytorch-nightly
[conda] pytorch-sphinx-theme      0.0.24                    dev_0    <develop>
[conda] torch                     1.9.0                    pypi_0    pypi
[conda] torchvision               0.11.0a0+b29ed34           dev_0    <develop>

[NicolasHug]

I'm having the same issue. I don't know how to fix it for now :/

With some versions of ffmpeg, I even get a sefault in ffmpeg:

(pt) ➜  vision git:(develop) ✗ conda list ffmpeg
# packages in environment at /Users/nicolashug/opt/miniconda3/envs/pt:
#
# Name                    Version                   Build  Channel
ffmpeg                    4.3                  h0a44026_0    pytorch


(pt) ➜  vision git:(develop) ✗ ffmpeg
dyld: Library not loaded: @rpath/libopenh264.5.dylib
  Referenced from: /Users/nicolashug/opt/miniconda3/envs/pt/lib/libavcodec.58.91.100.dylib
  Reason: image not found
[1]    1719 abort      ffmpeg

[bjuncek]

@NicolasHug for FFMPEG, 4.3 is known to have a segfault, that's why we're only supporting 4.2 for now. I'll try to get my hands on a mac to investigate @sallysyw 's issue today

[vmoens]

One way to go around it is to install ffmpeg 4.4 (eg with brew or from source) and comment out the ffmpeg check part in the setup.py (l351-362). With that I can read videos flawlessly

[NicolasHug]

@bjuncek thanks for looking into this, did you find anything interesting?
Perhaps this is @andfoy can help with as well?

[bjuncek]

Not really - Edgar is on vacation, but I've booked him for Thursday to go over this and make a well documented issue that we can hopefully take to Francisco, Ralf and the pytorch builder team.

Basically, the errors (that I understand so far are the following)
0. pyav from conda-forge doesn't necessarily play nicely with custom built ffmpeg's

1. we cannot package FFMPEG as-is due to licensing; @andfoy had a PR in builder where he packaged pytorch-specific version, that never got merged and is at this point stale if I'm not mistaken
2. custom ffmpeg might (and do) break some other dependencies (such as pyav) if not updated
3. on top of all that several ffmpeg builds have a bug where they segfault in codec allocation I think; this is an issue as default conda-forge and apt versions are still 4.3.

My workaround is to install 4.2.x from conda-forge and try to link everything against it manually. @vmoens 's suggestion worked for me on a mac when 4.4. was installed from brew. But it's not the neatest solution.

[ahirner]

@andfoy had a PR in builder where he packaged pytorch-specific version

@bjuncek did you mean this pytorch/builder#554? What I glean from the current repo is that custom builds for windows/mac may never return.

[kilicsedat]

Hi;

I am having the same problem, when trying to work on some videos.This is what I get:

RuntimeError: Not compiled with video_reader support, to enable video_reader support, please install ffmpeg (version 4.2 is currently supported) and build torchvision from source.

I tried to install torchvision and latest version of ffmpeg from the source. I am still getting the same error.

[bjuncek]

Hi @kilicsedat : ATM there are a few gotchas (for more detailed info check the setup.py).

1. Building on a mac with python=3.9 causes a crash so it is disabled at the moment
2. If any of the following libraries is not installed (which might, but shouldn't happen when you install FFMPEG from source), it will not build: "libavcodec", "libavformat", "libavutil", "libswresample", "libswscale"

What OS are you building it on?