Bugs on OSX regarding extended attributes and file system updates

[retrography]

I have been using bindfs on OSX for a while now, and there seems to be two lingering and daunting bugs in the way bindfs works on OSX, mounting folders from hfs+ partitions:

Extended file attributes won't work. Unless xattr-none option is mentioned in the mount command, bindfs will fail to start. This is quite problematic given that without xattr, OSX keeps creating dot files to save its badly needed metadata.

Also, file system updates go often undetected by bindfs, meaning that a file added to or deleted from the folder monitored and mounted (to another address) with bindfs is not detected.

I grouped these two issues here, because I figured out second issue may also have to do with handling file attributes. Don't hesitate to separate the two issues if this is not the case.

[mpartel]

How does bindfs fail to start if xattr-none is not given? Is there an error message? I'm not a mac user and my access to a mac with dev tools is a rare occurrence these days, so I'm afraid I can't debug this directly, but I'll happily accept pull requests. I can also make xattr-none the default on OS X if that seems like the best solution.

I believe the second bug is the same as #7. Essentially bindfs would have to (1) monitor the source dir itself and (2) emit similar file change events in the mounted dir. (1) is doable but requires a fair bit of work to do completely and reliably, but I'm not aware of any way to do (2). There doesn't seem to be any API for sending such events in FUSE or elsewhere. I pondered "simulating" an approximation of the events in a comment in #7 but that didn't really work out either.

[retrography]

Hi Martin,

Here is the error I get:

The operation can’t be completed because an unexpected error occurred (error code -50).

If there is a way to get more precise debugging information, please let me know and I will provide further feedback. If anything the differences between xattrs under Linux, BSD, and OSX should be minimal and it must not be difficult to make this feature work. Most important multi-platform programs (including tar, dropbox and rsync) support syncing attributes among these seamlessly. I am not a system programmer, but if you guide me on how to debug the issue, I don't mind writing a few lines of code. Meanwhile, your suggestion on disabling xattr by default on OSX/HFS is probably the safe way to go.

With regards to the second issue, I think you are right, it is probably a FUSE issue. I think I have even read about it somewhere before (lack of a general framework for FS notifications). The issue is that file systems differ in the way they handle even notification. For instance I don't think NFS has that mechanism implemented into it at all. But most local file systems get these capabilities through the operating system (on Linux, OSX, and Windows at least) have well-defined mechanisms for such that (see here for OSX's).

How much access do the FUSE file systems have to the FS underlying FUSE? Is there any way bindfs can latch to these OS-specific APIs, without bothering to go through FUSE?

[mpartel]

You can do ./configure --enable-debug-output when compiling bindfs and then run it with -d to get it to stay in the foreground and print debug information.

It may be useful to add print statements to print out errno in bindfs_getxattr and bindfs_setxattr. I'm curious if that's where the 50 comes from. Then it'd be interesting to know what that error number means, either by looking it up in OS X's errno.h (which I wasn't able to google) or with strerror().

When I once tried to get bindfs to work under Travis OS X, I got some compiler warnings about there apparently being an extra parameter to the xattr callbacks on OS X: https://travis-ci.org/mpartel/bindfs/builds/27010766#L791. Not sure what that is or what to do with it, but it might be a useful lead. On Linux there is no such warning.

About the second issue: a FUSE FS is a regular userspace process and it has pretty much the same limitations i.e. no special access to the underlying FS or other kernel code. There's only the FUSE API and what it provides. I don't think there's any fundamental reason why FUSE couldn't have an API for sending file change events (just like NFS could in principle send them if it wanted). It's just a non-trivial amount of work to build and nobody's chosen to do it.

[retrography]

Well, this is kind of funny: Error number 50 is a network error. See for yourself here.

I have actually never compiled bindfs manually, as I always try the homebrew builds first. There I don't seem to get any warning. I am leaving town for a conference tomorrow, but once I am back I will follow up on this, and will produce the output for you

[retrography]

By the way, I just rebuilt bindfs using homebrew with verbose on to see if I get any warning. Nothing of that kind. You can see the full output here. I wall try a manual build once I am back.

[mpartel]

The FUSE library might talks to the kernel over some kind of socket, so a protocol error there being signalled with a "network error" might make sense.

Compiling bindfs manually should be fairly easy: download and extract the .tar.gz, ./configure --enable-debug-output, make and then use src/bindfs or do make install to put it in /usr/local/bin. If you use a git clone instead of the tar file, you need to do ./autogen.sh before the ./configure, and that may require you to install some subset of autoreconf, autoconf, automake and libtool.

[retrography]

I'm back, @mpartel!

I compiled bindfs with debugging on, and then tested different cases and recorded the debug log for you. You can find the outputs in this gist. The files are pretty large, so Github fails to display them properly. You will have to clone the gist to your computer instead.

New finding: Unlike at the startup time, if I run bindfs manually from the command line it manages to mount successfully regardless of xattr state. But then when I try to do file operations (like copying) within the mounted directory I received the same old error -50. Interestingly the file gets copied despite the error, but I still receive the error. xattrs are not recorded correctly no matter the state of xattr option.

Below is a list of output files with some description.

• error_screen_shot.png

The error dialogue referencing error -50. This particular one showed up when I was trying to duplicate a file in the mounted directory.

• make_output.txt

This is ./make's output with debugging enabled during configuration (as you suggested).

Below is the list of bindfs debug output logs. In each of the below cases the file with _fg suffix contains the log for running bindfs in foreground and then immediately shutting it down. The file with _bg suffix contains the log for running bindfs in background with stderr redirected to a file to capture all the debug output as I entered the mounted directory in the finder, copied or duplicated a file and tried to set a xattr (comment) on another file. You can find the command that produced each log at the beginning of the log.

• bindfs_output_proven_working_options_bg.txt
• bindfs_output_proven_working_options_fg.txt

Running bindfs with the normal options I have found stable over time (which means xattrs are disabled).

• bindfs_output_proven_working_options_with_xattr_bg.txt
• bindfs_output_proven_working_options_with_xattr_fg.txt

Running bindfs with the normal options I have found stable over time, except leaving xattrs enabled.

• bindfs_output_minimum_working_options_bg.txt
• bindfs_output_minimum_working_options_fg.txt

Running bindfs with the minimum working options (all options removed except the one that disables xattrs).

• bindfs_output_with_xattr_bg.txt
• bindfs_output_with_xattr_fg.txt

Running bindfs with no options (xattrs are left enabled).

[mpartel]

Thanks for the thorough logs. I still don't have a solid explanation for why the error code is -50 instead of the -93 that bindfs returns to FUSE. The function signature mismatch that the compiler warns about seemed fishy though, so I looked at that in some more detail.

It finally occurred to me to look at the example code that comes with osxfuse. I copied over their (horribly verbose) Apple-specific implementations and pushed them to the mac-xattrs branch (diff).

Could you give that branch a try? Without a mac I couldn't check if it even compiled, so you may need to make some small tweaks.

[retrography]

Great job!

Now bindfs can read the xattrs it seems (I actually only tried comments and they worked). Writing back the xattrs to disk seems to be still broken. I don't receive the error dialog anymore, and in the debug log we can see error -93 now.

I had to make some minimal changes to the file for it to compile. I have made a pull request on the branch for you to see. All the build outputs and the debug log are here: https://gist.github.com/retrography/228eb420522e60a6b88d

I guess it needs only one more push!

[mpartel]

Thanks!

Strange, not sure what's wrong with setting xattrs. I didn't see any setxattr calls in your debug log.

Would you like to investigate that further or shall I put out a release?

[retrography]

Let me run it once again. This time I make sure I mark where xattr setter is invoked.

[retrography]

I did a thorough investigation of the matter this time.

Overall, bindfs seems to behave correctly on OSX now. Tags, which are the most important xattrs for me seem to work with no visible issue, read and write.

The situation with comment xattr is pretty fishy though...

When mounting a folder with bindfs, OSX finder does not display the comments saved in the file metadata at all. But more interestingly, it does not display correctly the comments on the source directory's files either, unless they have been directly set through it.

I explain:

A user can access file metadata in OSX at three levels:

1. xattr command
2. mdls command
3. Finder info

xattr command, reads directly from the file attributes. mdls reflects the information cached by the OS's mdworkers -- a part of the indexing service. Finder seems to rely on yet another level of caching of the same information.

When setting xattrs through finder on the bindfs mounted volumes, the changes take effect immediately on both source folder and mounted volume as long as the xattr command is concerned. mdls won't report the xattrs at all on bindfs volumes. It only shows the normal attributes. It shows the attributes from source folders correctly. When setting xattrs on mounted volumes through finder, sometimes the information in mdls go out of sync, but most of the time it is right on. Finder, on the other hand, remains completely out of sync with comments if anything other than itself changes the attributes. This can be bindfs mediating between the finder and the file system. But it can as well be the xattr command. If I set the comments using xattr, Finder won't recognize the new values and will keep showing the old ones!

So, I think most of the job is done. Setting and getting xattrs seems to work correctly. But that doesn't mean bindfs is working nicely on OSX. The main remaining issues are:

1. mdls does not display the xattrs correctly on mounted volumes, although it recognizes the changes in the mounted volumes as reflected in the underlying folders
2. Finder goes completely out of sync with comments, and possibly with other xattrs that I haven't tested. Tags/keywords are not affected by this.

I have uploaded a partial debug log from bindfs as well as a bunch of outputs from xattr and mdls in this gist. I continue investigating the matter to see what causes these behaviors - specially why the xattrs are not reported by mdls on mounted volumes.

By the way, I read osxfuse's documentation once again and came up with the following switches for a working solution. Some of them are directly related to xattrs, and the xattrs wouldn't work as well without using them. For instance mdls definitely needs allow_other to be set.

local,allow_other,extended_security,noappledouble

[mpartel]

Great, thanks!

I added the options you mentioned to the man page and the readme and released 1.12.7.

[retrography]

Thank you for all this work!

[retrography]

By the way, you may (or may not) want to add in the readme/man files that while tagging works properly now, some xattrs like comments may still show inconsistent behavior due to OSX's peculiarities. It took me a while to understand that my issues emanated from running bindfs on OSX. Others don't have to repeat the same experiments.

[mpartel]

Good idea, done (for the next release).