rustc reliably dies partway through a Firefox build on Mac

[bzbarsky]

For at least the last several weeks, whenever I do a clean Firefox build (ccache, bit no sccache) on Mac OSX 10.12, rustc dies partway through the build. If I just restart the build it usually completes, though sometimes I'll need multiple restarts.

The key part of the build log seems to be:

17:48.59 thread 'main' panicked at 'failed printing to stdout: Resource temporarily unavailable (os error 35)', libstd/io/stdio.rs:700:9
17:48.59 stack backtrace:
...
17:48.59    6:        0x100e0e4a0 - std::panicking::begin_panic_fmt::h2bdefd173f570a0b
17:48.59    7:        0x100e03b8b - std::io::stdio::_print::hd811110021038dff
17:48.59    8:        0x1007bb02c - cargo::core::compiler::job_queue::JobQueue::drain_the_queue::h5d61210172b85c62
17:48.59    9:        0x100742f42 - crossbeam_utils::thread::scope::h299c5d8d8ddddf08
17:48.59   10:        0x1007b93b5 - cargo::core::compiler::job_queue::JobQueue::execute::h5aaf65413952cf5b

I expect "os error 35" is EAGAIN. It's possible that rustc is running out of file descriptors or something...

I filed a bug on this on the Firefox build system as well: https://bugzilla.mozilla.org/show_bug.cgi?id=1520159

[jonas-schievink]

Have you tried rising your fd limit? Although printing to stdout shouldn't really need to allocate an fd, and EAGAIN seems like an odd error to get when running out of fds.

[bzbarsky]

Have you tried rising your fd limit?

I tried raising it from 10240 to 40000, but I still get the same failure.

It's possible that fds are not the issue...

[mtak-]

@bzbarsky Is this still happening on the latest nightly? I recently made a PR that fixed issues with thread locals on OSX. This looks potentially related.

[bzbarsky]

@mtak Yes, I get the same issue on nightly. Specifically, "rustc 1.34.0-nightly (da6ab95 2019-01-27)".

[luser]

Note that the Firefox build pipes the output of the actual build tool, so cargo's stdout/stderr will be a pipe, but I still wouldn't expect that to cause this error.

[bsdkurt]

This is caused by stdout/stderr being set to non-blocking by node earlier in the build. cargo writes to stdout, receives EAGAIN/EWOULDBLOCK and fails with the above error. Firefox changed the way it invokes node to use popen which islates node's non-blocking changes to its side of the pipe. See:
https://bugzilla.mozilla.org/show_bug.cgi?id=1500436
https://bugzilla.mozilla.org/show_bug.cgi?id=1520159

This is not a cargo bug (unless cargo is intended to deal with non-blocking stdout/stderr file descriptors).

[Mark-Simulacrum]

Sounds like this has been resolved upstream and indeed does not sound like a rustc/cargo bug (we don't expect non-blocking stdout).

Closing.