Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

NO-GO: scoped storage JNI #1

Open
wants to merge 13 commits into
base: feature/scoped-storage
Choose a base branch
from

Conversation

alexcohn
Copy link
Owner

@alexcohn alexcohn commented Jul 19, 2020

This PR is not going anywhere, at least for now. The current approach in https://github.com/alexcohn/mobile-ffmpeg/tree/feature/scoped-storage is that it does not patch the ffmpeg core at all. The potential advantages of working with content: protocol in C dwarf in comparison with this.

alexcohn added 12 commits May 30, 2020 21:48
only .url_open function is different from file: protocol, and even that is almost identical
ffmpeg may need suffix and/or some other info that can be extracted from SAF intent data
also, use dup() to avoid [fdsan issues](https://developer.android.com/about/versions/10/features#fdsan)
with slight modifications to allow more flexible choice of output
see also https://stackoverflow.com/a/62252329/192373

The idea is to demonstrate how the saf: protocol can be used with ffmpeg (avformat) API.
using saf: protocol for input
working with ParcelFileDescriptor via JNI
done by receiving the file name from a content: Uri
also, adds error checking and required cleanup to JNI calls
cleanup JNI calls, error checking and logging
wrap the changes to file.c in #ifdef ANDROID
the previous method did not work when 'content:' Uri had provider path with dots
@JayParikh20
Copy link

i'm getting cpu features failed while building for LTS version

[ 66%] Linking C executable list_cpu_features
/mnt/d/NoSpaceFolder/Android/SAFJNI-mobile-ffmpeg/mobile-ffmpeg/src/cpu-features/src/utils/list_cpu_features.c:0: error: undefined reference to 'stderr'
/mnt/d/NoSpaceFolder/Android/SAFJNI-mobile-ffmpeg/mobile-ffmpeg/src/cpu-features/src/utils/list_cpu_features.c:0: error: undefined reference to 'stdout'
/mnt/d/NoSpaceFolder/Android/AndroidNDKL/toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/include/bits/fortify/fcntl.h:67: error: undefined reference to '__open_2'
/mnt/d/NoSpaceFolder/Android/AndroidNDKL/toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/include/bits/fortify/unistd.h:191: error: undefined reference to '__read_chk'
/mnt/d/NoSpaceFolder/Android/SAFJNI-mobile-ffmpeg/mobile-ffmpeg/src/cpu-features/src/hwcaps.c:43: error: undefined reference to 'getauxval'
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[2]: *** [CMakeFiles/list_cpu_features.dir/build.make:96: list_cpu_features] Error 1
make[2]: Leaving directory '/mnt/d/NoSpaceFolder/Android/SAFJNI-mobile-ffmpeg/mobile-ffmpeg/android/build/cpu-features/arm'
make[1]: *** [CMakeFiles/Makefile2:68: CMakeFiles/list_cpu_features.dir/all] Error 2
make[1]: Leaving directory '/mnt/d/NoSpaceFolder/Android/SAFJNI-mobile-ffmpeg/mobile-ffmpeg/android/build/cpu-features/arm'
make: *** [Makefile:130: all] Error 2
make: Leaving directory '/mnt/d/NoSpaceFolder/Android/SAFJNI-mobile-ffmpeg/mobile-ffmpeg/android/build/cpu-features/arm'

Build command:
./android.sh --lts --enable-gpl --enable-x264 --enable-android-zlib --enable-android-media-codec

@alexcohn
Copy link
Owner Author

Thanks, let me see what went wrong here.

keep latest filename for reuse
@alexcohn
Copy link
Owner Author

@JayParikh20 I tried your command line on a jpg file, and it produced correct results for regular output path, but empty jpg output on Android when I tried scoped storage for output.

The video did work correctly.

Note that I pushed a quick fix that removes the debug logging that had tons of recovered name 'IMG-20200714-WA0025.jpg' for '…'. I expect this fix to have a significant effect on performance of single-image transform.

@JayParikh20
Copy link

I expect this fix to have a significant effect on performance of single-image transform

@alexcohn hey, commit worked, images are now processing super fast.

it produced correct results for regular output path, but empty jpg output on Android when I tried scoped storage for output.

For some reason that command with hardcoded values does not work with scoped storage. You can ignore it, it is working with my original command, working fine for me in both cases (file path & scoped storage)

The video did work correctly.

video files are still not working for me.
Here is my video URI videoPath: content://com.android.providers.media.documents/document/video%3A37

What is your video selection method?
I'm using ACTION_OPEN_DOCUMENT intent and have also tried ACTION_GET_CONTENT.

@alexcohn
Copy link
Owner Author

Could you please try a different location (e.g. Downloads)? Are there any logs marked /mobile-ffmpeg ?

@JayParikh20
Copy link

JayParikh20 commented Jul 21, 2020

So I change my loglevel to AV_LOG_TRACE, and found out that the command stops working after preset option of x264. removing preset option everything works !! (using codec as libx264) I'm sorry I should have looked into this before

I/mobile-ffmpeg: Loading mobile-ffmpeg.
I/mobile-ffmpeg: Loaded mobile-ffmpeg-custom-x86-4.3.3-lts-20200720.
D/mobile-ffmpeg: Callback thread started.
I/mobile-ffmpeg: ffmpeg version v4.3-dev-2955
I/mobile-ffmpeg:  Copyright (c) 2000-2020 the FFmpeg developers
I/mobile-ffmpeg:   built with Android (6454773 based on r365631c2) clang version 9.0.8 (https://android.googlesource.com/toolchain/llvm-project 98c855489587874b2a325e7a516b99d838599c6f) (based on LLVM 9.0.8svn)
I/mobile-ffmpeg:   configuration: --cross-prefix=i686-linux-android- --sysroot=/mnt/d/NoSpaceFolder/Android/AndroidNDKL/toolchains/llvm/prebuilt/linux-x86_64/sysroot --prefix=/mnt/d/NoSpaceFolder/Android/SAFJNI-mobile-ffmpeg/mobile-f
I/mobile-ffmpeg:   libavutil      56. 42.102 / 56. 42.102
I/mobile-ffmpeg:   libavcodec     58. 78.102 / 58. 78.102
I/mobile-ffmpeg:   libavformat    58. 42.100 / 58. 42.100
I/mobile-ffmpeg:   libavdevice    58.  9.103 / 58.  9.103
I/mobile-ffmpeg:   libavfilter     7. 77.101 /  7. 77.101
I/mobile-ffmpeg:   libswscale      5.  6.101 /  5.  6.101
I/mobile-ffmpeg:   libswresample   3.  6.100 /  3.  6.100
D/mobile-ffmpeg: Splitting the commandline.
D/mobile-ffmpeg: Reading option '-y' ...
D/mobile-ffmpeg:  matched as option 'y' (overwrite output files) with argument '1'.
D/mobile-ffmpeg: Reading option '-f' ...
D/mobile-ffmpeg:  matched as option 'f' (force format) with argument 'lavfi'.
D/mobile-ffmpeg: Reading option '-i' ...
D/mobile-ffmpeg:  matched as input url with argument 'color=0xFFFFFF:1080x1350'.
D/mobile-ffmpeg: Reading option '-i' ...
D/mobile-ffmpeg:  matched as input url with argument 'content://com.android.providers.media.documents/document/video%3A37'.
D/mobile-ffmpeg: Reading option '-i' ...
D/mobile-ffmpeg:  matched as input url with argument '/data/user/0/com.jpi.captionAndFit/files/textOverlay.png'.
D/mobile-ffmpeg: Reading option '-filter_complex' ...
D/mobile-ffmpeg:  matched as option 'filter_complex' (create a complex filtergraph) with argument '[1:v]scale=673:1200[pvid];[0:v][pvid]overlay=204:150:shortest=1[vid];[2:v]scale=394:127[txt];[vid][txt]overlay=343:12'.
D/mobile-ffmpeg: Reading option '-c:v' ...
D/mobile-ffmpeg:  matched as option 'c' (codec name) with argument 'libx264'.
D/mobile-ffmpeg: Reading option '-preset' ...

@alexcohn
Copy link
Owner Author

Thanks for this update!

@alexcohn alexcohn force-pushed the feature/scoped-storage branch 2 times, most recently from 4ca25d6 to 609d0b9 Compare July 22, 2020 20:11
@alexcohn
Copy link
Owner Author

@JayParikh20 would you like to try the new https://github.com/alexcohn/mobile-ffmpeg/tree/feature/scoped-storage branch for this? I believe that now the balance between ease of use and minimum changes to ffmpeg core is better. The question is, does it effect the performance.

@JayParikh20
Copy link

@alexcohn hey, I tried the branch. Did not work for me

Successfully opened the file.
    Parsing a group of options: input url file:/proc/self/69/download.mp4.
    Successfully parsed a group of options.
    Opening an input file: file:/proc/self/69/download.mp4.
    [NULL @ 0x7f758bde00] Opening 'file:/proc/self/69/download.mp4' for reading
    [file @ 0x7f75991180] Setting default whitelist 'file,crypto,data'
    file:/proc/self/69/download.mp4: No such file or directory

Using this as an input, am I doing it right?

"-i", Config.getCommandParameter(context, selectedMediaUri), ...

@alexcohn
Copy link
Owner Author

alexcohn commented Jul 28, 2020

@JayParikh20 Sorry, I found it was my fault: it only worked to read documents. Fixed now.

But soon I hope to release another change that will work without modifying ffmpeg at all.

@JayParikh20
Copy link

JayParikh20 commented Jul 31, 2020

@alexcohn thanks, tried with the latest commit, I am still getting the same error. I also have tried it on multiple devices.

Successfully parsed a group of options.
    Opening an input file: file:/proc/self/194/bb.mp4.
    [NULL @ 0xc74a6400] Opening 'file:/proc/self/194/bb.mp4' for reading
    [file @ 0xbdd7bf00] Setting default whitelist 'file,crypto,data'
    file:/proc/self/194/bb.mp4: No such file or directory

But soon I hope to release another change that will work without modifying FFmpeg at all.

In which branch would this be?

@alexcohn
Copy link
Owner Author

alexcohn commented Jul 31, 2020

The (almost) final version is on https://github.com/alexcohn/mobile-ffmpeg/tree/feature/scoped-storage, it is in sync with https://github.com/tanersener/mobile-ffmpeg/tree/development and this removes my patch to ffmpeg core (libavformat/file.c). This means, you need a full rebuild after you pull this version.

@alexcohn alexcohn force-pushed the feature/scoped-storage branch from be9c88c to 97b2581 Compare August 6, 2020 12:07
@pgfsim
Copy link

pgfsim commented Aug 24, 2020

This isn't working for me on Android 9: mobile-ffmpeg: "saf:136/IMG_0.jpg": No such file or directory

My command is int rc = FFmpeg.execute("-framerate " + fps + " -i '" + Config.getSafParameterForReadAndWrite(mContext, imageList.get(0).getUri()) + "' -f matroska -c:v libx264 -vf fps=" + fps + " -pix_fmt yuv420p -g 2 " + Config.getSafParameterForReadAndWrite(mContext, outputVideo.getUri()));

@alexcohn
Copy link
Owner Author

@peter9870: you should use Config.getSafParameterForRead(…) for the input files (like IMG_0.jpg). Sorry for delay.

@pgfsim
Copy link

pgfsim commented Sep 2, 2020

@alexcohn no problem! The issue was the quotes around the input file. The other issue is it'll only take one image. Normally I'd just do mediaStorageDir.getAbsolutePath() + "/" + "IMG_%d.jpg' for the input and ffmpeg gets the lowest decimal in place of %d and loops through, but using SAF we need to specify an actual file, so it only processes one image.

@alexcohn alexcohn changed the title Feature/scoped storage jni NO-GO: scoped storage JNI Sep 2, 2020
@alexcohn
Copy link
Owner Author

alexcohn commented Sep 2, 2020

@peter9870: for assorted images, you can use EXTRA_ALLOW_MULTIPLE. For numbered images in the same directory, try ACTION_OPEN_DOCUMENT_TREE. But if these images were created by your app, you probably have full access to them without SAF intents.

@HBiSoft
Copy link

HBiSoft commented Sep 8, 2020

@alexcohn
One thing I've noticed is that we can not pass a String[] when using this.
If I pass a String, like this:

String videoPath = Config.getSafParameterForRead(this, mUri);
String testCommand = "-i "+videoPath+" /storage/emulated/0/test.mp4";

Then it works as expected. But when I use:

String videoPath = Config.getSafParameterForRead(this, mUri);
String[] testCommand = {"-i", videoPath, "/storage/emulated/0/test.mp4"};

Then I get "saf:73/test.mp4": No such file or directory.


Please note that passing a file path as an input works:

String[] testCommand = {"-i", "/storage/emulated/0/videoInput.mp4", "/storage/emulated/0/test.mp4"};

I'm not sure why this is because I could not see a difference between:

public static long executeAsync(final String[] arguments, final ExecuteCallback executeCallback) {
final long newExecutionId = executionIdCounter.incrementAndGet();
AsyncFFmpegExecuteTask asyncFFmpegExecuteTask = new AsyncFFmpegExecuteTask(newExecutionId, arguments, executeCallback);
asyncFFmpegExecuteTask.executeOnExecutor(AsyncTask.THREAD_POOL_EXECUTOR);
return newExecutionId;
}

public AsyncFFmpegExecuteTask(final String[] arguments, final ExecuteCallback executeCallback) {
this(FFmpeg.DEFAULT_EXECUTION_ID, arguments, executeCallback);
}

and:

public static long executeAsync(final String command, final ExecuteCallback executeCallback) {
final long newExecutionId = executionIdCounter.incrementAndGet();
AsyncFFmpegExecuteTask asyncFFmpegExecuteTask = new AsyncFFmpegExecuteTask(newExecutionId, command, executeCallback);
asyncFFmpegExecuteTask.executeOnExecutor(AsyncTask.THREAD_POOL_EXECUTOR);
return newExecutionId;
}

public AsyncFFmpegExecuteTask(final String command, final ExecuteCallback executeCallback) {
this(FFmpeg.parseArguments(command), executeCallback);
}


I'm not sure if it could be me doing something wrong, but I triple checked.

@alexcohn
Copy link
Owner Author

alexcohn commented Sep 8, 2020

@HBiSoft Thanks for testing. The cause is here

return "\"saf:" + fd + "/" + displayName + "\"";

I wanted to make getSafParameterForRead() easier to use, and wrapped it in quotes, to take care of names with spaces like "saf:73/another test.mp4". Unfortunately, the methods that take String[] consider these quotes as integral part of the argument.

It happens so, I didn't think about this use case. But I also want the most basic method to work correctly even for files that have spaces in their names, so I will provide a different workaround, at the price of mangling the original name a bit.

@HBiSoft
Copy link

HBiSoft commented Sep 8, 2020

@alexcohn
Ah, I see. It's not an issue, I just thought I would let you know since it was available and might cause confusion later on.

@alexcohn
Copy link
Owner Author

alexcohn commented Sep 8, 2020

@HBiSoft, you are welcome to try the fix.

@HBiSoft
Copy link

HBiSoft commented Sep 8, 2020

@alexcohn Thanks, that fixed it.

@HBiSoft
Copy link

HBiSoft commented Sep 10, 2020

@alexcohn
I found another issue. We can't select a file if it has spaces in the name. For example Video Name.mp4. If I do, I get the following:

Argument 'Name.mp4' provided as input filename, but 'saf:78/Video' was already specified.

Edit:
I see you did mention this in your previous comment.

@alexcohn
Copy link
Owner Author

@alexcohn
I found another issue. We can't select a file if it has spaces in the name. For example Video Name.mp4. If I do, I get the following:

Argument 'Name.mp4' provided as input filename, but 'saf:78/Video' was already specified.

This is strange. Did you use a string array API, or a single-string API? On the test app, you can see that it works correctly with Video Name.mp4.

@HBiSoft
Copy link

HBiSoft commented Sep 10, 2020

@alexcohn
I'm using the demo application to do my tests, with small changes.
I did a quick test to see if there is a difference between using a String or a String Array.
Using a String Array works. But when I use a String then I get the above.

@alexcohn
Copy link
Owner Author

alexcohn commented Sep 11, 2020

It's very strange indeed. The demo app uses String API, and for me it works. The command is logged before execution. Could you upload the log please?

UPDATE: as you use SAF for input, there may be some subtle difference. Could you also post your code that constructs the command String?

@alexcohn
Copy link
Owner Author

hi @HBiSoft what's the status with this Video Name.mp4? Is it still causing problems?

@HBiSoft
Copy link

HBiSoft commented Sep 13, 2020

@alexcohn
My apologies for the delay in response.

This only happens when calling FFprobe using the String API, as shown below:

final String ffprobeCommand = "-hide_banner " +Config.getSafParameterForRead(this, uri);
int rc = FFprobe.execute(ffprobeCommand);
if (rc == RETURN_CODE_SUCCESS) {
    //SUCCESS
}

When I change it to the following, then it works fine:

final String[] ffprobeCommand = {"-hide_banner", Config.getSafParameterForRead(this, uri)};
int rc = FFprobe.execute(ffprobeCommand);
if (rc == RETURN_CODE_SUCCESS) {
    //SUCCESS
}

Here is the entire log:

E: ffprobe version v4.4-dev-416
E:  Copyright (c) 2007-2020 the FFmpeg developers
E:   built with Android (5900059 based on r365631c) clang version 9.0.8 (https://android.googlesource.com/toolchain/llvm-project 207d7abc1a2abf3ef8d4301736d6a7ebc224a290) (based on LLVM 9.0.8svn)
E:   configuration: --cross-prefix=arm-linux-androideabi- --sysroot=/Users/Hagen/Library/Android/sdk/ndk-bundle/toolchains/llvm/prebuilt/darwin-x86_64/sysroot --prefix=/Users/Hagen/Desktop/mobile-ffmpeg-feature-scoped-storage/prebuilt/android-arm/ffmpeg --pkg-config=/usr/local/bin/pkg-config --enable-version3 --arch=armv7-a --cpu=armv7-a --cc=armv7a-linux-androideabi16-clang --cxx=armv7a-linux-androideabi16-clang++ --extra-libs='-L/Users/Hagen/Desktop/mobile-ffmpeg-feature-scoped-storage/prebuilt/android-arm/cpu-features/lib -lndk_compat' --target-os=android --disable-ffprobe --disable-neon --enable-asm --enable-inline-asm --enable-cross-compile --enable-pic --enable-jni --enable-optimizations --enable-swscale --enable-shared --enable-v4l2-m2m --disable-outdev=fbdev --disable-indev=fbdev --enable-small --disable-openssl --disable-xmm-clobber-test --disable-debug --enable-lto --disable-neon-clobber-test --disable-programs --disable-postproc --disable-doc --disable-htmlpages --disable-manpages --disable-podpages --disable-txtpages --disable-static --disable-sndio --disable-schannel --disable-securetransport --disable-xlib --disable-cuda --disable-cuvid --disable-nvenc --disable-vaapi --disable-vdpau --disable-videotoolbox --disable-audiotoolbox --disable-appkit --disable-alsa --disable-cuda --disable-cuvid --disable-nvenc --disable-vaapi --disable-vdpau --enable-libx264 --enable-gpl --disable-sdl2 --disable-zlib
E:   libavutil      56. 55.100 / 56. 55.100
E:   libavcodec     58. 96.100 / 58. 96.100
E:   libavformat    58. 48.100 / 58. 48.100
E:   libavdevice    58. 11.101 / 58. 11.101
E:   libavfilter     7. 87.100 /  7. 87.100
E:   libswscale      5.  8.100 /  5.  8.100
E:   libswresample   3.  8.100 /  3.  8.100
E: Argument 'Name.mp4' provided as input filename, but 'saf:73/Video' was already specified.
I: Command execution failed with rc=1 and the output below.

When I rename the file to VideoName.mp4, then it works.

Please let me know if there is anything else you need.

@HBiSoft
Copy link

HBiSoft commented Sep 14, 2020

@alexcohn
The problem seems to be with the return statement in getSafParameter.

Here are the two scenarios I found:


return "\"saf:" + fd + "/" + displayName + "\"";
  • Works perfectly using a String, but when passing a String[] I get the error above when selecting a file with a space in the name: ``saf:78/Video' was already specified`.

  • MediaInformation returns null.

return "saf:" + fd + "/" + displayName.replace(' ', ' ');
  • Works perfectly when using a String[], but get the error above when using a String.

What I've done to avoid this is use a String[] instead of a String, then everything works fine.


EDIT: So, I think the real issue is with parseArguments() (inside FFmpeg.java) when using return "saf:" + fd + "/" + displayName.replace(' ', ' '); and passing a String instead of a String[]?

@alexcohn
Copy link
Owner Author

alexcohn commented Sep 14, 2020

@HBiSoft I am afraid somewhere in your git+build process, the   character was destroyed in getSafParameter(). Let me try to set it in hexa: maybe it will be more robust.

@alexcohn
Copy link
Owner Author

alexcohn commented Sep 14, 2020

@HBiSoft I will be very much obliged if you can try again after this change

@HBiSoft
Copy link

HBiSoft commented Sep 14, 2020

@alexcohn
Awesome, it now works when using a String and String[], the 'saf:73/Video' was already specified issue is also gone.

tanersener pushed a commit to tanersener/mobile-ffmpeg that referenced this pull request Sep 21, 2020
* introducing saf_wrapper - a transparent API to use the file descriptor via custom AVIO for saf: Url

* demonstrate how use to use SAF with some demo tabs

SAF is enabled for API 19 or higher
SAF pickers are used for FFprobe on CommandTabFragment and for video output on PipeTabFragment

* Revert "demonstrate how use to use SAF with some demo tabs"

This reverts commit 97b2581

* add SAF tab with two buttons

* change the getSafParameter handles filenames with spaces

see alexcohn#1 (comment)

* take advantage of Cursor being Closeable

* provide NBSP in hexa, to make sure that git and editor don't destroy it

* Avoid "rw" because it may not be supported (e.g. Google Drive).

Also, some code cleanup in SafTabFragment.

* don't run FFprobe after video playback

* ACTION_GET_CONTENT is better than ACTION_OPEN_DOCUMENT: it gives us more options to choose.

* add closeParcelFileDescriptor to fix working with Google Drive

keep the temp files open, we must keep the ParcelFileDecriptor until the fd is closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants