Add file notifier integration into the #17
Conversation
In order to use file watching, it is necessary to be able to share data between threads. Therefore, adding the `Send` requirement to the `input_stream` and `output_stream` methods of `Opts` lets us share the streams.
There was a problem hiding this comment.
Are you on Linux? If so, please run this and the original under perf stat to see how much CPU each one uses. It's possible for this approach to be much worse, especially in the case where the file gets new input every few milliseconds.
Without any tests for this functionality, it's also possible it doesn't work. :-\
Co-authored-by: Nathan Stocks <[email protected]>
|
@jorendorff It works on my machine. 🤣 Seriously, though. I measured it on macOS and the CPU usage dropped to a stable 0.0%, compared to ~0.2% before this change. It appears to actually be interrupt-driven now. And my manual tests still showed normal function. So that's one data point. |
CleanCut
force-pushed
the
file-notifier
branch
from
a87987f
to
10cc684
Compare
We will later want to better handle files moving and related changes in the file system.
|
I did a large refactor last night. I think it works well (I mean it seems great when I manually test it out), but I'm struggling to write an integration test. If anyone wants to help me figure out what's going on, that'd be great. Otherwise I'll return to the problem tonight or tomorrow and keep tinkering away. |
|
Okay, so the root of the problem is that Rust doesn't block on filesystem operations. I needed to add some delays to let file writes, file removes, and headtail's reads to all happen sequentially instead of in parallel. |
|
CI does not appear to have inotify available on Linux. The fallback polling implementation doesn't seem to detect file deletion and recreation. So it doesn't appear we can test this feature on Linux CI. 😢 I'm going to try the other OS's... |
|
This PR is included in |
Uses the
notifycrate in order to use the file watching/notification functionality often provided by operating systems (eginotify(7)on Linux.Also wraps
io::Errorandnotify::Errorinto a custom error type usingthiserror.Closes #16