Provide access to posix_fadvise

[rdiez]

Copying large files effectively flushes the Linux filesystem cache, which has an impact on the overall system performance. I have written about this in the past:

The Linux Filesystem Cache is Braindead

One way to workaround this issue is to call posix_fadvise( POSIX_FADV_NOREUSE ) . Unfortunately, Perl does not provide easy access to this routine. You need to download an extra module like Sys::PageCache. But that module uses mincore, which looks specific to Linux.

I would like to see this routine accessible through the POSIX core module. After all, that call is defined in the POSIX standard. This way, it should be available on many more platforms than just Linux.

[khwilliamson]

I'm willing to work to do this, unless someone tells me why it is inadvisable

[Leont]

I think this is useful functionality, yet I'm also not sure if it should be core (literally, I'm not saying it shouldn't). Portability may be an argument against it though.

Possible implementation locations include IO::Handle, POSIX or a PerlIO::layer.

[khwilliamson]

I would think it would just go trivially
into POSIX

[dur-randir]

While posix_fadvise can be useful in it's own ways, the desired task of not thrashing page cache can be achieved with by opening file handle with O_DIRECT in most use cases.

[rdiez]

O_DIRECT is problematic. First of all, Linus himself says "There really is no valid reason for EVER using O_DIRECT". A second consideration is that O_DIRECT bypasses caching, while POSIX_FADV_NOREUSE still allows for some caching, which may actually be better, depending on the situation. Yet another aspect is that O_DIRECT does no read-ahead, which can easily kill performance in many simple applications. Furthermore, O_DIRECT is specific to Linux, while posix_fadvise is a POSIX standard that should be more portable. Finally, O_DIRECT demands a certain memory alignment, which can be difficult to achieve in a scripting language like Perl.

[dur-randir]

First of all, Linus himself says
Since he's a proponent of "kernel must to proper thing, you shouldn't workaround it", the same argument applies to posix_fadivise - yet both are in.

O_DIRECT demands a certain memory alignment, which can be difficult to achieve in a scripting language like Perl
All allocated buffers are already properly aligned, so by default it just Does The Right Thing.

So, it comes down to do you care for read cache or not. But even then, manpage for posix_fadvise reads

In kernels before 2.6.18, POSIX_FADV_NOREUSE had the same semantics as POSIX_FADV_WILLNEED.  
This was  probably a bug; since kernel 2.6.18, this flag is a no-op.

which kind of makes O_DIRECT a winner, since at least it does something.

[rdiez]

After the long list of issues I mentioned, calling O_DIRECT "a winner", especially in the Perl context, is... interesting.

There is also flag POSIX_FADV_DONTNEED for posix_fadvise(), which Linux supports. That is probably the way to go if you care for an operating system lacking proper POSIX support. ;-)