Update FUSE kernel ABI

[zargony]

Update to use FUSE kernel ABI 7.19 which opens up the possibility to support more desired features like better optimization, ioctl and polling/notifications. This will raise the requirements to FUSE 2.9.1 or later on Linux and OSXFUSE 3.0.0 or later on macOS.
There are even newer ABI versions, but 7.19 seems to be the one most commonly supported by various systems.

[Mic92]

Splice would be really neat!

[Mic92]

Do you need help with the implementation?

[zargony]

Help would be very welcome of course. I'm just pondering because I'd like to overhaul the overall interface to be more idiomatic and more flexible (especially considering async io) and I don't know if that shouldn't be done first. Batch forget might be more or less easy, but I don't know if I would want the whole notification stuff in the current state

[Mic92]

I currently selectively implement some fuse methods as needed by my implementation.
Feel free to pickup code, if you find something useful: https://github.com/Mic92/rust-fuse/commits/cntrfs

[Mic92]

I also working on a feature-complete passthrough filesystem: https://github.com/Mic92/cntr/blob/master/src/fs.rs
This could serve as an example for rust-fuse maybe. I want to pass as much as possible tests from xfstests.

[Mic92]

Do you see any use for this? Mic92@9c53b9f I made libfuse optional, since I needed low-level access to mount stuff across container namespaces. It will also make my project easier to cross-compile.

[zargony]

Thanks! Making libfuse optional sounds like a nice idea. It would be nice if we could get rid of the libfuse dependency completely one day, but that'd require to reimplement the mount function (which turned out to be more complicated that I thought when I had a look last time).

A passthrough filesystem would be a great example but also a great reference for people to see how it works. Plus we could perform benchmarks comparisons of the passthrough vs direct filesystem to determine the FUSE overhead (maybe even compare it to a libfuse implementation to see how it performs)

[Mic92]

I will do benchmarks anyway for my paper in the next two weeks. Implementing mount was easy in my case. The main purpose of fusemount on linux is to have a dedicated setuid wrapper for mount. However in my case I have to start as root anyway for other reasons.

[Mic92]

Batch forget is also now implemented.

[Mic92]

I partly support Fuse API 26, the following is missing: FUSE_INTERRUPT, FUSE_POLL, FUSE_NOTIFY_REPLY, CUSE_INIT

[Mic92]

Some preliminary performance results from ec2. Performance still sucks since I have not finished multi-threading / splice read/write https://openbenchmarking.org/result/1710208-AL-MERGE427034

Currently the full benchmark suite is running. That discussion might be also of interest: hanwen/go-fuse#192 (comment)

[Mic92]

I have now some optional hacky support for splice read/write. (hacky in the sense that there could be less code duplication)

[hanwen]

I would avoid support for FUSE_POLL. If rust ever gets built-in support for polling file handles (which Go has), this may complicate writing tests. See hanwen/go-fuse#165

[zargony]

Rebased and updated.

After splitting kernel ABI and libfuse FFI stuff into a fuse-sys crate, I was able to select different ABI versions by using cargo features. It's not the most beatiful code (especially the version constant definition looks ugly now), but hopefully it's still readable. Using cargo features works ok to some degree for versioning types and constants and doesn't require huge code duplications like with separate modules or crates per ABI version.
This should probably be better covered by tests, but creating tests for every ABI version is a very tedious task.

This should allow us to have at least ABI and FFI definitions for more recent FUSE versions. How different versions can/should be supported in the fuse crate, is however still open.

[Mic92]

In the evaluation part there a performance analysis: https://www.usenix.org/conference/atc18/presentation/thalheim

[ghost]

I found this thread while looking for docs around implementing notify functions.... any progress on this front?

[piranna]

I'm also interested on that, specially the new support to mount filesystems by plain users added in Linux 4.18. Is that supported?

[dns2utf8]

I am using Arch Linux with the 4.19 and soon 4.20 kernel.
Any news on this? Or is no change needed?
Since I am here and ask questions, I see the development is still going on do you plan a release soon?

[Mic92]

The fuse protocol is versioned and the linux kernel will supports older versions practically for ever. Newer version of the protocol will only add new features, which are then not accessible to the user of this library.

[Mic92]

Since rust-fuse uses fusermount it should be also possible to mount filesystems as unprivileged user.

[Mic92]

Mhm. Rust-fuse still uses the old symbol from fuse2 to mount the filesystem, so I am not sure if the fuse3 unprivileged mode would work. In my fork I removed the need for libfuse when mounting that but I require in my project root for different reason, which is why I did not implement unprivileged mounting with linux namespaces.

[zargony]

There's ongoing effort to take advantage of a newer ABI and support notify and other functions in #122.

About fuse3 unprivileged mode: fuse_mount_compat25 is used because it returns a plain fd while newer functions return structs like fuse_chan that I wanted to avoid because I'm not sure if their memory layout is guaranteed. As far as I remember, fuse_mount calls fuse_mount_compat25 internally, so there shouldn't be a difference. I'm not very familiar with the mounting process since I always tried to actually get rid of it to become independent from libfuse, but if there's an advantage in using fuse3 functions, I'm open to it (please create a separate issue)