Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

std::filesystem support #609

Closed
chenxiaolong opened this issue Dec 29, 2017 · 89 comments
Closed

std::filesystem support #609

chenxiaolong opened this issue Dec 29, 2017 · 89 comments
Assignees
Milestone

Comments

@chenxiaolong
Copy link

Description

The libc++ libraries (both shared and static) seem to be missing symbols needed to use <experimental/filesystem>. For example, trying to compile a simple program that calls std::experimental::filesystem::remove_all(path) will fail at link time with:

/stuff/Applications/Linux/android-ndk/sources/cxx-stl/llvm-libc++/include/experimental/filesystem:1695: error: undefined reference to 'std::experimental::filesystem::v1::__remove_all(std::experimental::filesystem::v1::path const&, std::__ndk1::error_code*)'
clang++: error: linker command failed with exit code 1 (use -v to see invocation)

Sample project: https://github.com/chenxiaolong/ndk_filesystem_test (run ./test.sh to reproduce issue)

Is this expected to happen? I couldn't find any details as to whether the filesystem library should work with the NDK.

Environment Details

  • NDK Version: r16b (16.1.4479499)
  • Build sytem: any
  • Host OS: Fedora 27 x86_64
  • Compiler: clang
  • ABI: any
  • STL: libc++
  • NDK API level: 21
  • Device API level: n/a
@jmgao jmgao changed the title [r16b] Symbols from <experimental/filesystem> not found in libc++ libraries Build/ship libc++experimental? Dec 29, 2017
@jmgao jmgao assigned jmgao and DanAlbert and unassigned jmgao Dec 29, 2017
@jmgao
Copy link
Contributor

jmgao commented Dec 29, 2017

libc++ compiles its experimental parts into libc++experimental, a library that you need to link in separately. Unfortunately, we don't seem to build it as part of the NDK. Retitling the bug to look into shipping it.

In the mean time, you might be able to just add the required files directly into your project (from $ANDROID_NDK_ROOT/sources/cxx-stl/llvm-libc++/src/experimental/filesystem) to work around this on your end, but I haven't tried it (and this is guaranteed to break in the future, when the files are moved into the standard or abandoned).

@chenxiaolong
Copy link
Author

Ah okay, that makes perfect sense. I didn't realize libc++ also had a separate library like gcc's libstdc++fs.

@DanAlbert DanAlbert added this to the r17 milestone Jan 2, 2018
@DanAlbert DanAlbert modified the milestones: r17, r18 Feb 14, 2018
@DanAlbert
Copy link
Member

The tests for std::filesystem are not going to be easy to run on Android (they require a python helper). Going to find a way to work around that, but it's not likely to happen during r17.

@DanAlbert DanAlbert modified the milestones: r18, r19 Jul 13, 2018
@DanAlbert
Copy link
Member

Unlikely to fit into r18. I want to get this done for the platform before the NDK since it should be a smaller problem to solve, and haven't done that yet.

@andreya108
Copy link

So, for now, can we suppose that full support for std::filesystem will appear in r19?

@DanAlbert
Copy link
Member

Given how strongly people feel about #463 I will be doing that before this, and I'm skeptical that I'll be able to accomplish that before r19 is complete (there's always a fair amount of overhead in coordinating with external products, and CMake is an unfamiliar code base to all of us here).

Going to move this to the r20 bucket to match the other bug, but odds are there will be some down time while waiting on reviews for #463 so I'll try to make some progress on this then.

@DanAlbert DanAlbert modified the milestones: r19, r20 Aug 27, 2018
@DanAlbert
Copy link
Member

DanAlbert commented Aug 27, 2018

So, for now, can we suppose that full support for std::filesystem will appear in r19?

Also, a disclaimer that when this does land, the NDK's support for std::filesystem will be as complete as libc++'s (since that's what we use). The libc++ docs still indicate that this is experimental (and not ABI stable), but it does at least appear to be complete.

@arcanis
Copy link

arcanis commented Sep 2, 2018

Of note: <filesystem> is now available without going through experimental (cf commit). Would that make it easier to integrate?

[edit] It seems like the symbols are still in a separate library, libc++fs

@DanAlbert
Copy link
Member

No. Like I've said, the difficulty here is getting the tests running.

@feliwir
Copy link

feliwir commented Sep 7, 2018

I'd like to see the filesystem support happening aswell 👍

@diegurrio
Copy link

Looking forward to the experimental file system libraries being available.

@KayEss
Copy link

KayEss commented Oct 27, 2018

In the current NDK the experimental filesystem is still not part of the libc++ library, and now the sources can no longer be installed through the SDK tools so our old work around suggested by @jmgao no longer works.

Any suggestions for what to do now?

@alexcohn
Copy link

@enh
Copy link
Contributor

enh commented Dec 20, 2018

https://issuetracker.google.com/121291977 was another request for std::filesystem...

@DanAlbert DanAlbert changed the title Build/ship libc++experimental? Build/ship libc++experimental/libc++fs? Jan 10, 2019
@arcanis
Copy link

arcanis commented Feb 4, 2019

@DanAlbert Is it still planned for r20?

@DanAlbert
Copy link
Member

Conveniently, upstream has actually fixed some of the issues that we were going to need to fix about a month ago, so there's a bit less work on our end.

After disabling a bunch of tests (everything relating to hard links, sockets, and fifos, since selinux prevents their use in /data/local/tmp), I do have a passing test run for Q devices.

There's at least one thing preventing me from running the tests on JB devices more or less at all though. I have an idea of how to fix it, but haven't had a chance to try yet. That could easily be where most of the trouble is so it's hard to say for sure whether this is close to done or not, but the few that do run have me optimistic.

@DanAlbert
Copy link
Member

Alright, managed to get a full test run on Jelly Bean. Everything looks good within a reasonable margin of error. Tests did uncover two bugs, but they're relatively minor:

  1. [BUG] fchmodat with AT_SYMLINK_NOFOLLOW is very inconsistent #1258 std::filesystem::permissions(symlink_path, p perms, std::filesystem::perm_options::nofollow) will not honor nofollow for devices older than API 23.
  2. [BUG] realpath doesn't fail on non-existent paths on old devices #1260 std::filesystem::canonical will incorrectly succeed for non-existent paths on devices older than API ~19 (not sure exactly when)

There are a fair number of other things that don't work, most of which depend on API level, but they are all related to Android security policy and std::filesystem signals the error appropriately:

  • hardlinks aren't allowed (at least in /data/local/tmp for the shell user)
  • no types of special files (mknod, named fifos, unix domain sockets, etc) are allowed (at least in /data/local/tmp for the shell user)
  • Ability to set the permissions of a symlink is inconsistent (see [BUG] fchmodat with AT_SYMLINK_NOFOLLOW is very inconsistent #1258)

It's going to take me some time to get the test fixes pushed upstream, but if that takes too long I don't think we need to wait for that before shipping it in r22 beta 1.

As for the two bugs, I don't think those are severe enough that we necessarily need to fix them before shipping. It'd be better to get them fixed if we can, but I don't think we need to hold back the feature for them.

@enh-google
Copy link
Collaborator

@alexcohn
Copy link

Unix domain sockets are supported. So, maybe the test should run in app private sandbox, and not in /data/local/tmp ?

@DanAlbert
Copy link
Member

Converting our tests to run in an app is something I've been working on. It's going to take some time, it seems, especially for the libc++ tests. Nothing untenable, just going to take longer than I want to hold this back.

Good to know that it's not wasted effort though :)

@DanAlbert
Copy link
Member

The implementation for this is now checked in to master and will be in r22. I've forked the remaining testing cleanup into #1265 so I can close this to make it obvious that the part you folks care about is complete without having to read the whole thing :)

rdbaris pushed a commit to rdbaris/OpenRCT2 that referenced this issue Jun 7, 2020
When attemping to build with ndk 19+ I get the following error: error: undefined reference to 'std::__ndk1::__fs::filesystem::path::__extension() const'

Looking at this issue here android/ndk#609 and https://android.googlesource.com/platform/ndk/+/master/docs/Roadmap.md#package-management it's stated that the ndk does not support std::filesystem.
@gulrak
Copy link

gulrak commented Jun 17, 2020

Yes the current implementation is irritating - either don't include it at all, or include it so it links!

As @Apeopex mentioned further up, there is a workable Android alternative which I am using and haven't had any issues with at all.

https://github.com/gulrak/filesystem

@oviano If I am reading docs right, this is equivalent of using boost::filesystem, right? It doesn't pretend to be std::?

These comments have been a few weeks ago, but I just found this discussion and want to add a small detail to the suggested workaround implementation in case someone might think about it:

It is comparable to boost::filesystem in that it is using its own namespace and so its not the same as std::, but it differs from boost::filesystem in that it pretends to be std::filesystem in questions of api and behavior. There are various differences between boost::fs and std::fs as for example joining absolute paths by operator/ and the above mentioned header-only implementation tries hard to implement std::fs behavior, and when it differs undocumented, I handle it as a bug. The one major documented difference is, it is utf-8 oriented, so if that's your encoding, you get what std::filesystem would do for C++17. Details of behavior differences are in the docs.

Still, to be clear, I'm currently not developing on Android myself, and I have no CI integration for it, so the current level of support is just based on feedback and fixed issues of users. And I am thankful for that! Any feedback or help to get a CI setup on Github going to enhance support would sure be appreciated.

@hcv1027
Copy link

hcv1027 commented Mar 3, 2021

The implementation for this is now checked in to master and will be in r22. I've forked the remaining testing cleanup into #1265 so I can close this to make it obvious that the part you folks care about is complete without having to read the whole thing :)

Hi @DanAlbert , can we use std::filesystem::create_directory in r22?

I still have the compile errors:

In function `std::__ndk1::__fs::filesystem::exists(std::__ndk1::__fs::filesystem::path const&)':
/root/workspace/3rd_party/android-ndk-r21d-linux-x86_64/toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/include/c++/v1/filesystem:1615: undefined reference to `std::__ndk1::__fs::filesystem::__status(std::__ndk1::__fs::filesystem::path const&, std::__ndk1::error_code*)'
In function `std::__ndk1::__fs::filesystem::create_directory(std::__ndk1::__fs::filesystem::path const&)':
/root/workspace/3rd_party/android-ndk-r21d-linux-x86_64/toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/include/c++/v1/filesystem:1555: undefined reference to `std::__ndk1::__fs::filesystem::__create_directory(std::__ndk1::__fs::filesystem::path const&, std::__ndk1::error_code*)'
clang++: error: linker command failed with exit code 1 (use -v to see invocation)

This is my cmake commad:

cmake -DCMAKE_TOOLCHAIN_FILE=$ANDROID_NDK/build/cmake/android.toolchain.cmake \
    -DANDROID_ABI=arm64-v8a \
    -DANDROID_PLATFORM=android-29 \
    -DANDROID_NATIVE_API_LEVEL=29 \
    -DANDROID_STL=c++_shared \
    -DBUILD_SHARED_LIBS=ON \
    -DCMAKE_BUILD_TYPE=RELEASE \
    -DCMAKE_INSTALL_PREFIX=$DEST \
    ..

Accroding to this link
I also tries to add -lc++experimental or -lc++fs in my CMakeList.txt

target_link_libraries( vmc_io_lite
  c++fs
)

But this will cause below error message:

/root/workspace/3rd_party/android-ndk-r21d-linux-x86_64/toolchains/llvm/prebuilt/linux-x86_64/lib/gcc/aarch64-linux-android/4.9.x/../../../../aarch64-linux-android/bin/ld: cannot find -lc++fs
clang++: error: linker command failed with exit code 1 (use -v to see invocation)

Can we use these function in r22?
std::filesystem::create_directory, std::filesystem::exists, std::filesystem::remove_all

@rprichard
Copy link
Collaborator

This path is for r21d not r22:

/root/workspace/3rd_party/android-ndk-r21d-linux-x86_64/toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/include/c++/v1/filesystem

G10h4ck added a commit to G10h4ck/libretroshare that referenced this issue Apr 8, 2022
sepehrst pushed a commit to spsforks/android-bionic-libc that referenced this issue Apr 22, 2024
Split statfs and statvfs. The former has been available forever, and the
latter is implemented in terms of the former. The implementation has
been moved into headers so that it can be used at low API levels.

There's no reason for any Android or Linux code to use statvfs rather
than statfs, but code that needs to build on Darwin too will want to use
statvfs because Darwin's statfs is very spartan.

Bug: android/ndk#609
Test: treehugger
Change-Id: Icf3d5723a260099fddb2d9f902e3047b0f041647
sepehrst pushed a commit to spsforks/android-bionic-libc that referenced this issue Apr 22, 2024
Split statfs and statvfs. The former has been available forever, and the
latter is implemented in terms of the former. The implementation has
been moved into headers so that it can be used at low API levels.

There's no reason for any Android or Linux code to use statvfs rather
than statfs, but code that needs to build on Darwin too will want to use
statvfs because Darwin's statfs is very spartan.

Bug: android/ndk#609
Test: treehugger
Change-Id: Icf3d5723a260099fddb2d9f902e3047b0f041647
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests