Segfault in IO.select

[nirvdrum]

I have a Rails application that periodically segfaults. The hs_err log indicates it's stemming from the dnsruby gem as it calls IO.select. I'm seeing this with TruffleRuby 23.0.0 on Oracle GraalVM running in JVM mode. The gem is used by an in-house gem that wraps around ruby-kafka. As the hs_err shows, there's a background thread that does periodic DNS checks.

I haven't yet debugged the code to see what values are flowing through the system when the error occurs. The dnsruby gem uses class variables for various bits of state and I haven't yet verified the thread-safety of that code.

hs_err.log

[eregon]

So it must be from this line:
https://github.com/alexdalitz/dnsruby/blob/053d4743e9a74c2d9c5fe34eebea6bec7984e632/lib/dnsruby/select_thread.rb#L193

ready, _write, _errors = IO.select(sockets, nil, nil, timeout)

It could be an issue in truffleposix_select() (more likely) or in NFI.
Obviously IO.select should never segfault.

I wonder if it's a case of a file descriptor being > FD_SETSIZE (1024).
If so the reason is most likely that we use FD_ZERO() and that doesn't support such big fds.
BTW select(2) does not officially support any FD bigger than FD_SETSIZE but it seems in practice it does:

DESCRIPTION
       WARNING:  select()  can  monitor only file descriptors numbers that are less than FD_SETSIZE (1024)—an unreasonably
       low limit for many modern applications—and this limitation will not change.  All modern applications should instead
       use poll(2) or epoll(7), which do not suffer this limitation.

[eregon]

There is rb_fd_init(), rb_fd_zero() etc to deal with such big fds.
Those headers are not really accessible to libtruffleposix though, and the implementation is in libtruffleruby.
But we don't really want IO.select depend on loading C extensions support.

Another idea would be to use poll(2) to implement IO.select, not sure whether if it's a good idea.

[eregon]

I tried to reproduce this with:
On CRuby:

$ ruby -ve 'ios=2048.times.map { io=File.new("README.md"); p io.fileno; io }; p ios.last.fileno; IO.select([ios.last], nil, nil, 1.0)'
ruby 3.2.2 (2023-03-30 revision e51014f9c0) [x86_64-linux]
...
1022
1023
-e:1:in `initialize': Too many open files @ rb_sysopen - README.md (Errno::EMFILE)

On TruffleRuby:

$ ruby -ve 'ios=4096.times.map { io=File.new("README.md"); p io.fileno; io }; p ios.last.fileno; p IO.select([ios.last], nil, nil, 1.0)'
truffleruby 23.1.0-dev-b6c229a4, like ruby 3.1.3, GraalVM CE Native [x86_64-linux]
...
4095
4096
4097
4098
4099
4100
4101
4101
zsh: segmentation fault (core dumped)  ruby -ve 

So interestingly CRuby raises EMFILE for what would be fd 1024.
And for some unknown reason TruffleRuby goes higher.

At least what we can do is add a check in IO.select to raise if fd > FD_SETSIZE.

[eregon]

open(2) mentions:

EMFILE The per-process limit on the number of open file descriptors has been reached (see the description of RLIMIT_NOFILE in getrlimit(2)).

and getrlimit(2) mentions:

       RLIMIT_NOFILE
              This specifies a value one greater than the maximum file descriptor  number  that  can  be  opened  by  this
              process.  Attempts (open(2), pipe(2), dup(2), etc.)  to exceed this limit yield the error EMFILE.  (Histori‐
              cally, this limit was named RLIMIT_OFILE on BSD.)

              Since Linux 4.5, this limit also defines the maximum number of file descriptors that an unprivileged process
              (one  without  the  CAP_SYS_RESOURCE  capability)  may  have "in flight" to other processes, by being passed
              across UNIX domain sockets.  This limit applies to the sendmsg(2) system call.   For  further  details,  see
              unix(7).

And the default limit for me is $ limit: descriptors 1024.
So it seems the JVM/SVM increases the soft limit to the hard limit:

$ ruby -ve 'p Process.getrlimit Process::RLIMIT_NOFILE'
ruby 3.2.2 (2023-03-30 revision e51014f9c0) [x86_64-linux]
[1024, 524288]

$ ruby -ve 'p Process.getrlimit Process::RLIMIT_NOFILE'
truffleruby 23.1.0-dev-b6c229a4, like ruby 3.1.3, GraalVM CE Native [x86_64-linux]
[524288, 524288]

$ jt -q ruby -ve 'p Process.getrlimit Process::RLIMIT_NOFILE'
truffleruby 23.1.0-dev-82d3a4a5, like ruby 3.1.3, Interpreted JVM [x86_64-linux]
[524288, 524288]

In SVM it's done in https://github.com/oracle/graal/blob/79738daa192f8dd941691a07e4bbd5238fcccd81/substratevm/src/com.oracle.svm.core.posix/src/com/oracle/svm/core/posix/PosixNativeLibraryFeature.java#L81-L90

[eregon]

On CRuby if we set the limit then it works with higher fds:

$ ruby -ve 'Process.setrlimit(Process::RLIMIT_NOFILE, 524288); ios=4096.times.map { io=File.new("README.md"); p io.fileno; io }; p ios.last.fileno; p IO.select([ios.last], nil, nil, 1.0)'
...
4099
4100
4100
[[#<File:README.md>], [], []]

[eregon]

I have a branch reimplementing IO.select with poll() or ppoll() which seems a pretty good fit, removes this limit of fd < 1024 and is likely faster than select(2).