not ok 71 - stat watcher invoked once for size chnage

[chu11]

not ok 71 - stat watcher invoked once for size chnage
#   Failed test 'stat watcher invoked once for size chnage'
#   at ../../../../src/common/libflux/test/reactor.c line 644.

Just hit this in #996 in travis. Appears to have happened atleast three other times (#884, #822, #799), apparently always with clang compiler. Should try and figure out what's going on causing this to intermittently fail.

As an aside, there is a typo in the above that needs fixing too.

[garlick]

I hit that one last week during testing of PR #1000

[grondo]

Hit again in test of #1124 -- I'll take a brief look at it and see if we can get it to fail outside of travis.

[chu11]

perhaps it's worth noting that with #1124 it was with gcc, not with clang

[chu11]

Decided to look into this old issue and I think I figured it out. The libev stat watcher is a simple polling loop that runs lstat() on a file. The test in question appends data to a file, then unlinks it. With a little bit of racy behavior, an append + unlink can occur in between lstat() checks. Thus, the unlink is recognized but the change in the size of the file is missed.

[grondo]

The libev stat watcher is a simple polling loop that runs lstat() on a file.

Good find! However, I thought libev used inotify by default if it was present, so I'm surprised it has reverted to polling. I wonder why.

[chu11]

I thought libev used inotify by default if it was present, so I'm surprised it has reverted to polling

Ahhh, you are right. I had missed that subtlety when looking through libev. So the polling part of my comment was incorrect.

The reason I thought it was polling is b/c (AFAICT) libev calls lstat() every time it has been notified. It doesn't check the flags from inotify_event results. So I suspect the race exists either way, as the issue is the append & unlink can happen in between consecutive lstat() calls.

[grondo]

Yep, I think I just came to the same conclusion. There is still a race because libev only uses inotify to avoid polling, not for actual processing of events. Nice find!