FFmpegFrameGrabber.grab() missing audio frames #1971

frankfliu · 2023-01-16T07:18:11Z

When I use FFmpegFrameGrabber.grab() audio frames, it stopped before hitting the end.

I'm using a flac file, it suppose contains 106 frame, but FFmpegFrameGrabber stopped at 94 frames:

URL url = new URL("https://github.com/openai/whisper/raw/main/tests/jfk.flac");
try (FFmpegFrameGrabber grabber = new FFmpegFrameGrabber(url)) {
    grabber.setAudioChannels(1);
    grabber.setSampleRate(16000);
    grabber.setAudioCodec(avutil.AV_SAMPLE_FMT_S16);
    grabber.start();
    int frameCount = 0;
    while (grabber.grab() != null) {
        ++frameCount;
    }
    System.out.println("Total frames: " + frameCount);
}

If I use ffmpeg command line it can read all 106 frames correctly:

ffmpeg -nostdin -threads 0 -i jfk.flac -f s16le -ac 1 -ar 16000 -y -loglevel debug out.bin

106 frames successfully decoded, 0 decoding errors

The text was updated successfully, but these errors were encountered:

…d by the codec (issue #1971) * Upgrade dependencies for OpenCV 4.7.0, librealsense2 2.53.1, Leptonica 1.83.0, Tesseract 5.3.0

saudet · 2023-01-16T12:16:15Z

Fixed in commit 53d19be. Thanks for reporting!

frankfliu · 2023-01-16T20:02:39Z

@saudet
Thanks for fixing the issue. I tried to use 1.5.9-SNAPSHOT version, but I get the following error:

java.lang.NoClassDefFoundError: Could not initialize class org.bytedeco.ffmpeg.global.avcodec

It looks like the error is caused by missing symbolic link:

~/.javacpp/cache/ffmpeg-5.1.2-1.5.9-SNAPSHOT-macosx-x86_64.jar/org/bytedeco/ffmpeg/macosx-x86_64/libavcodec.dylib

saudet · 2023-01-16T23:06:20Z

It's not a symbolic link, the libraries can't load for some reason. I'm guessing it's related to the recent addition of the AV1 codecs. @bradh @n-kai-cj Could you look into this?

frankfliu · 2023-01-16T23:14:28Z

In old version theres is a symbolic link from libavcodec.dylib -> libavcodec.59.dylib, this is missing in 1.5.9-SNAPSHOT version. And the Loader failed to find libavcodec.dylib

bradh · 2023-01-17T00:21:38Z

I just checked the readfewframes sample with the -SNAPSHOT version, and it worked fine. However that was on linux, and I have no test system for macosx.

My cache directory has the symlinks.

bradh@audax:~/.javacpp/cache/ffmpeg-5.1.2-1.5.9-SNAPSHOT-linux-x86_64-gpl.jar/org/bytedeco/ffmpeg/linux-x86_64-gpl$ ls -al
total 67652
drwxrwxr-x 2 bradh bradh     4096 Jan 17 11:15 .
drwxrwxr-x 3 bradh bradh     4096 Jan 17 11:15 ..
lrwxrwxrwx 1 bradh bradh       16 Jan 17 11:15 libavcodec.so -> libavcodec.so.59
-rw-rw-r-- 1 bradh bradh 54943288 Jan  8 20:08 libavcodec.so.59
lrwxrwxrwx 1 bradh bradh       17 Jan 17 11:15 libavformat.so -> libavformat.so.59
-rw-rw-r-- 1 bradh bradh  9568280 Jan  8 20:09 libavformat.so.59
lrwxrwxrwx 1 bradh bradh       15 Jan 17 11:15 libavutil.so -> libavutil.so.57
-rw-rw-r-- 1 bradh bradh   951488 Jan  8 20:09 libavutil.so.57
lrwxrwxrwx 1 bradh bradh       11 Jan 17 11:15 libdrm.so -> libdrm.so.2
-rw-rw-r-- 1 bradh bradh    70856 Jan  8 20:09 libdrm.so.2
-rw-rw-r-- 1 bradh bradh   699952 Jan  8 20:08 libjniavcodec.so
-rw-rw-r-- 1 bradh bradh   700128 Jan  8 20:09 libjniavformat.so
-rw-rw-r-- 1 bradh bradh  1219216 Jan  8 20:09 libjniavutil.so
-rw-rw-r-- 1 bradh bradh    88760 Jan  8 20:09 libjniswresample.so
-rw-rw-r-- 1 bradh bradh   113440 Jan  8 20:09 libjniswscale.so
lrwxrwxrwx 1 bradh bradh       18 Jan 17 11:15 libswresample.so -> libswresample.so.4
-rw-rw-r-- 1 bradh bradh   125168 Jan  8 20:09 libswresample.so.4
lrwxrwxrwx 1 bradh bradh       15 Jan 17 11:15 libswscale.so -> libswscale.so.6
-rw-rw-r-- 1 bradh bradh   616736 Jan  8 20:09 libswscale.so.6
lrwxrwxrwx 1 bradh bradh       14 Jan 17 11:15 libva-drm.so -> libva-drm.so.1
-rw-rw-r-- 1 bradh bradh    11400 Jan  8 20:09 libva-drm.so.1
lrwxrwxrwx 1 bradh bradh       10 Jan 17 11:15 libva.so -> libva.so.1
-rw-rw-r-- 1 bradh bradh   132144 Jan  8 20:09 libva.so.1

@frankfliu Can you try setting the org.bytedeco.javacpp.logger.debug property to true and providing the relevant parts?

frankfliu · 2023-01-17T00:33:34Z

    Debug: Loading class org.bytedeco.javacpp.presets.javacpp
    Debug: Loading class org.bytedeco.javacpp.Loader
    Debug: Loading library jnijavacpp
    Debug: Failed to load for jnijavacpp: java.lang.UnsatisfiedLinkError: no jnijavacpp in java.library.path: [/Users/ec2-user/Library/Java/Extensions, /Library/Java/Extensions, /Network/Library/Java/Extensions, /System/Library/Java/Extensions, /usr/lib/java, .]
    Debug: Could not load Loader: java.lang.UnsatisfiedLinkError: no jnijavacpp in java.library.path: [/Users/ec2-user/Library/Java/Extensions, /Library/Java/Extensions, /Network/Library/Java/Extensions, /System/Library/Java/Extensions, /usr/lib/java, .]
    Debug: Loading class org.bytedeco.javacpp.presets.javacpp
    Debug: Loading class org.bytedeco.ffmpeg.global.avutil
    Debug: Loading class org.bytedeco.javacpp.presets.javacpp
    Debug: Loading class org.bytedeco.ffmpeg.global.avutil
    Debug: Loading /Users/ec2-user/.javacpp/cache/ffmpeg-5.1.2-1.5.9-SNAPSHOT-macosx-x86_64.jar/org/bytedeco/ffmpeg/macosx-x86_64/libavutil.57.dylib
    Debug: Loading /Users/ec2-user/.javacpp/cache/ffmpeg-5.1.2-1.5.9-SNAPSHOT-macosx-x86_64.jar/org/bytedeco/ffmpeg/macosx-x86_64/libjniavutil.dylib
    Debug: Loading class org.bytedeco.javacpp.presets.javacpp
    Debug: Loading class org.bytedeco.javacpp.Pointer
    Debug: Loading class org.bytedeco.javacpp.presets.javacpp
    Debug: Loading class org.bytedeco.ffmpeg.global.avutil
    Debug: Loading class org.bytedeco.ffmpeg.avutil.AVChannelLayout
    Debug: Registering org.bytedeco.ffmpeg.avutil.AVChannelLayout[address=0x600003e4b7a0,position=0,limit=1,capacity=1,deallocator=org.bytedeco.javacpp.Pointer$NativeDeallocator[ownerAddress=0x600003e4b7a0,deallocatorAddress=0x12fafe860]]
    Debug: Loading class org.bytedeco.javacpp.presets.javacpp
    Debug: Loading class org.bytedeco.ffmpeg.global.avutil
    Debug: Loading class org.bytedeco.ffmpeg.global.swresample
    Debug: Loading class org.bytedeco.javacpp.presets.javacpp
    Debug: Loading class org.bytedeco.ffmpeg.global.avutil
    Debug: Loading class org.bytedeco.ffmpeg.global.swresample
    Debug: Loading /Users/ec2-user/.javacpp/cache/ffmpeg-5.1.2-1.5.9-SNAPSHOT-macosx-x86_64.jar/org/bytedeco/ffmpeg/macosx-x86_64/libswresample.4.dylib
    Debug: Loading /Users/ec2-user/.javacpp/cache/ffmpeg-5.1.2-1.5.9-SNAPSHOT-macosx-x86_64.jar/org/bytedeco/ffmpeg/macosx-x86_64/libjniswresample.dylib
    Debug: Loading class org.bytedeco.javacpp.presets.javacpp
    Debug: Loading class org.bytedeco.ffmpeg.global.avutil
    Debug: Loading class org.bytedeco.ffmpeg.global.swresample
    Debug: Loading class org.bytedeco.ffmpeg.global.avcodec
    Debug: Loading class org.bytedeco.javacpp.presets.javacpp
    Debug: Loading class org.bytedeco.ffmpeg.global.avutil
    Debug: Loading class org.bytedeco.ffmpeg.global.swresample
    Debug: Loading class org.bytedeco.ffmpeg.global.avcodec
    Debug: Loading /Users/ec2-user/.javacpp/cache/ffmpeg-5.1.2-1.5.9-SNAPSHOT-macosx-x86_64.jar/org/bytedeco/ffmpeg/macosx-x86_64/libavcodec.59.dylib
    Debug: Failed to load /Users/ec2-user/.javacpp/cache/ffmpeg-5.1.2-1.5.9-SNAPSHOT-macosx-x86_64.jar/org/bytedeco/ffmpeg/macosx-x86_64/libavcodec.59.dylib: java.lang.UnsatisfiedLinkError: Can't load library: /Users/ec2-user/.javacpp/cache/ffmpeg-5.1.2-1.5.9-SNAPSHOT-macosx-x86_64.jar/org/bytedeco/ffmpeg/macosx-x86_64/libavcodec.59.dylib
    Debug: Loading library avcodec
    Debug: Failed to load for [email protected]: java.lang.UnsatisfiedLinkError: no avcodec in java.library.path: [/Users/ec2-user/Library/Java/Extensions, /Library/Java/Extensions, /Network/Library/Java/Extensions, /System/Library/Java/Extensions, /usr/lib/java, .]
    Debug: Loading /Users/ec2-user/.javacpp/cache/ffmpeg-5.1.2-1.5.9-SNAPSHOT-macosx-x86_64.jar/org/bytedeco/ffmpeg/macosx-x86_64/libjniavcodec.dylib
    Debug: Failed to load /Users/ec2-user/.javacpp/cache/ffmpeg-5.1.2-1.5.9-SNAPSHOT-macosx-x86_64.jar/org/bytedeco/ffmpeg/macosx-x86_64/libjniavcodec.dylib: java.lang.UnsatisfiedLinkError: Can't load library: /Users/ec2-user/.javacpp/cache/ffmpeg-5.1.2-1.5.9-SNAPSHOT-macosx-x86_64.jar/org/bytedeco/ffmpeg/macosx-x86_64/libjniavcodec.dylib
    Debug: Loading library jniavcodec
    Debug: Failed to load for jniavcodec: java.lang.UnsatisfiedLinkError: no jniavcodec in java.library.path: [/Users/ec2-user/Library/Java/Extensions, /Library/Java/Extensions, /Network/Library/Java/Extensions, /System/Library/Java/Extensions, /usr/lib/java, .]
    Debug: Loading class org.bytedeco.javacpp.presets.javacpp
    Debug: Loading class org.bytedeco.ffmpeg.global.avutil
    Debug: Loading class org.bytedeco.ffmpeg.global.swresample
    Debug: Loading class org.bytedeco.ffmpeg.global.avcodec

frankfliu · 2023-01-17T00:35:37Z

Here is my cache folder:

[~/.javacpp/cache/ffmpeg-5.1.2-1.5.9-SNAPSHOT-macosx-x86_64.jar/org/bytedeco/ffmpeg/macosx-x86_64]>ls -la
total 62872
drwxr-xr-x  10 lufen  staff       320 Jan 16 11:41 .
drwxr-xr-x   3 lufen  staff        96 Jan 16 11:41 ..
-rw-r--r--   1 lufen  staff  29413456 Jan  8 01:45 libavcodec.59.dylib
-rw-r--r--   1 lufen  staff    718400 Jan  8 01:45 libavutil.57.dylib
lrwxr-xr-x   1 lufen  staff        18 Jan 16 11:41 libavutil.dylib -> libavutil.57.dylib
-rw-r--r--   1 lufen  staff    648560 Jan  8 01:45 libjniavcodec.dylib
-rw-r--r--   1 lufen  staff   1130728 Jan  8 01:45 libjniavutil.dylib
-rw-r--r--   1 lufen  staff    106920 Jan  8 01:45 libjniswresample.dylib
-rw-r--r--   1 lufen  staff    152768 Jan  8 01:45 libswresample.4.dylib
lrwxr-xr-x   1 lufen  staff        21 Jan 16 11:41 libswresample.dylib -> libswresample.4.dylib

frankfliu · 2023-01-17T00:48:16Z

Just tested on Linux, it works fine and missing audio frames issue is fixed.

bradh · 2023-01-17T00:58:10Z

@frankfliu that cache directory looks really incomplete (like, where is libavformat?). There are definitely more files than that in the jar file I downloaded.

Can you try removing and retrying it?

saudet · 2023-01-17T01:04:33Z

The problem is this error. JavaCPP bails out after that.

Debug: Failed to load /Users/ec2-user/.javacpp/cache/ffmpeg-5.1.2-1.5.9-SNAPSHOT-macosx-x86_64.jar/org/bytedeco/ffmpeg/macosx-x86_64/libavcodec.59.dylib: java.lang.UnsatisfiedLinkError: Can't load library: /Users/ec2-user/.javacpp/cache/ffmpeg-5.1.2-1.5.9-SNAPSHOT-macosx-x86_64.jar/org/bytedeco/ffmpeg/macosx-x86_64/libavcodec.59.dylib

Usually we get more useful errors on Mac, but not for this one it seems. There's something it doesn't like about AV1...

frankfliu · 2023-01-17T01:09:28Z

I cleaned the cached and still the same. I do see other files (e.g. libjniavformat.dylib) in the jar, but not in the cache folder.

otool -L libavcodec.59.dylib

libavcodec.59.dylib:
	@rpath/libavcodec.59.dylib (compatibility version 59.0.0, current version 59.37.100)
	@rpath/libswresample.4.dylib (compatibility version 4.0.0, current version 4.7.100)
	@rpath/libavutil.57.dylib (compatibility version 57.0.0, current version 57.28.100)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1311.0.0)
	/System/Library/Frameworks/AudioToolbox.framework/Versions/A/AudioToolbox (compatibility version 1.0.0, current version 1000.0.0)
	@rpath/libSvtAv1Enc.1.dylib (compatibility version 1.0.0, current version 1.4.1)
	/System/Library/Frameworks/VideoToolbox.framework/Versions/A/VideoToolbox (compatibility version 1.0.0, current version 1.0.0)
	/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1856.105.0)
	/System/Library/Frameworks/CoreMedia.framework/Versions/A/CoreMedia (compatibility version 1.0.0, current version 1.0.0)
	/System/Library/Frameworks/CoreVideo.framework/Versions/A/CoreVideo (compatibility version 1.2.0, current version 1.5.0)
	/System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices (compatibility version 1.0.0, current version 1141.1.0)
	/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 1200.3.0)

saudet · 2023-01-17T01:19:03Z

Ah, here's the problem:

@rpath/libSvtAv1Enc.1.dylib (compatibility version 1.0.0, current version 1.4.1)

@bradh Please make sure the build doesn't create a shared library for SVT-AV1.

bradh · 2023-01-17T02:17:37Z

@bradh Please make sure the build doesn't create a shared library for SVT-AV1.

Looks like we are missing a bunch of configuration options (including -DBUILD_SHARED_LIBS=OFF) that were meant to be in LIBSVTAV1_CONFIG but never got applied. Running a test build on that.

saudet · 2023-01-20T04:46:01Z

@frankfliu The builds have been fixed with pull bytedeco/javacpp-presets#1312

frankfliu · 2023-01-20T06:17:17Z

The snapshot build seems not atomic, I hit the following error:

Could not find ffmpeg-5.1.2-1.5.9-SNAPSHOT-linux-x86_64.jar (org.bytedeco:ffmpeg:5.1.2-1.5.9-SNAPSHOT:20230120.060711-153).
     Searched in the following locations:
         https://oss.sonatype.org/content/repositories/snapshots/org/bytedeco/ffmpeg/5.1.2-1.5.9-SNAPSHOT/ffmpeg-5.1.2-1.5.9-20230120.060711-153-linux-x86_64.jar

bradh · 2023-01-20T07:25:47Z

That is not the latest snapshot update at https://oss.sonatype.org/content/repositories/snapshots/org/bytedeco/ffmpeg/5.1.2-1.5.9-SNAPSHOT/

As of writing, I see https://oss.sonatype.org/content/repositories/snapshots/org/bytedeco/ffmpeg/5.1.2-1.5.9-SNAPSHOT/ffmpeg-5.1.2-1.5.9-20230120.070542-158-linux-x86_64.jar

Possibly you have something cached?

saudet · 2023-01-20T08:01:55Z

The snapshot build seems not atomic, I hit the following error:

By atomic if you mean something transactional like a staging repository, Nexus doesn't support that for snapshots, no.

frankfliu · 2023-01-20T16:20:50Z

I think what you are doing is publishing the SNAPSHOT version for each platform, and then re-publish them again as a whole.

The publishing takes more than one hour. If my build kick in during your publishing time, only one platform is available for the latest version. and gradle will fail to resolve other platform jars.

In the repo, -158.pom contain all the platform, but -175.pom only has on platform. If my build kick in at the time -158.pom is not ready, my build will fail.

saudet · 2023-01-21T02:27:43Z

Yes, I'm aware, that's the best we can do with Nexus unfortunately. We keep getting back to these things, but the answer is always the same. No one, not even Oracle nor Amazon, is investing in Java anymore, so we're stuck with poor tools like this and we have to come up with hacks like you're doing in DJL that don't work a lot better, that require a lot more maintenance just to keep them half working... I mean if DJL could offer an alternative to Nexus, that's great, but let's be honest, it's not going to happen.

saudet · 2023-06-06T08:53:04Z

The fix has been released with JavaCV 1.5.9! Enjoy and thanks again for reporting

saudet added the bug label Jan 16, 2023

saudet added a commit that referenced this issue Jan 16, 2023

* Fix FFmpegFrameGrabber.grab() not returning audio frames buffere…

53d19be

…d by the codec (issue #1971) * Upgrade dependencies for OpenCV 4.7.0, librealsense2 2.53.1, Leptonica 1.83.0, Tesseract 5.3.0

saudet closed this as completed Jun 6, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

FFmpegFrameGrabber.grab() missing audio frames #1971

FFmpegFrameGrabber.grab() missing audio frames #1971

frankfliu commented Jan 16, 2023

saudet commented Jan 16, 2023

frankfliu commented Jan 16, 2023

saudet commented Jan 16, 2023

frankfliu commented Jan 16, 2023

bradh commented Jan 17, 2023

frankfliu commented Jan 17, 2023

frankfliu commented Jan 17, 2023

frankfliu commented Jan 17, 2023

bradh commented Jan 17, 2023

saudet commented Jan 17, 2023

frankfliu commented Jan 17, 2023 •

edited

Loading

saudet commented Jan 17, 2023

bradh commented Jan 17, 2023

saudet commented Jan 20, 2023

frankfliu commented Jan 20, 2023

bradh commented Jan 20, 2023

saudet commented Jan 20, 2023

frankfliu commented Jan 20, 2023

saudet commented Jan 21, 2023

saudet commented Jun 6, 2023

FFmpegFrameGrabber.grab() missing audio frames #1971

FFmpegFrameGrabber.grab() missing audio frames #1971

Comments

frankfliu commented Jan 16, 2023

saudet commented Jan 16, 2023

frankfliu commented Jan 16, 2023

saudet commented Jan 16, 2023

frankfliu commented Jan 16, 2023

bradh commented Jan 17, 2023

frankfliu commented Jan 17, 2023

frankfliu commented Jan 17, 2023

frankfliu commented Jan 17, 2023

bradh commented Jan 17, 2023

saudet commented Jan 17, 2023

frankfliu commented Jan 17, 2023 • edited Loading

saudet commented Jan 17, 2023

bradh commented Jan 17, 2023

saudet commented Jan 20, 2023

frankfliu commented Jan 20, 2023

bradh commented Jan 20, 2023

saudet commented Jan 20, 2023

frankfliu commented Jan 20, 2023

saudet commented Jan 21, 2023

saudet commented Jun 6, 2023

frankfliu commented Jan 17, 2023 •

edited

Loading