Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

'rustc' panicked at 'failed to acquire jobserver token: ...' #46981

Closed
matthiasbeyer opened this issue Dec 24, 2017 · 25 comments · Fixed by #75295
Closed

'rustc' panicked at 'failed to acquire jobserver token: ...' #46981

matthiasbeyer opened this issue Dec 24, 2017 · 25 comments · Fixed by #75295
Labels
C-bug Category: This is a bug. I-ICE Issue: The compiler panicked, giving an Internal Compilation Error (ICE) ❄️ T-cargo Relevant to the cargo team, which will review and decide on the PR/issue.

Comments

@matthiasbeyer
Copy link

This seems to be related to #42867, though not entirely the same.

I am in the process of cargo publishing my crate (multiple crates in a workspace project) imag v0.5.0 (from this commit).

Instead of a normal compile process, I got this:

error: internal compiler error: unexpected panic

note: the compiler unexpectedly panicked. this is a bug.

note: we would appreciate a bug report: https://github.com/rust-lang/rust/blob/master/CONTRIBUTING.md#bug-reports

note: rustc 1.20.0 (f3d6973f4 2017-08-27) running on x86_64-unknown-linux-gnu

note: run with `RUST_BACKTRACE=1` for a backtrace

thread 'rustc' panicked at 'failed to acquire jobserver token: Error { repr: Os { code: 11, message: "Resource temporarily unavailable" } }', /checkout/src/libcore/result.rs:860:4
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
stack backtrace:
   0: std::sys::imp::backtrace::tracing::imp::unwind_backtrace
   1: std::sys_common::backtrace::_print
   2: std::panicking::default_hook::{{closure}}
   3: std::panicking::default_hook
   4: std::panicking::rust_panic_with_hook
   5: std::panicking::begin_panic_new
   6: std::panicking::begin_panic_fmt
   7: rust_begin_unwind
   8: core::panicking::panic_fmt
   9: core::result::unwrap_failed
  10: rustc_trans::back::write::execute_work
  11: rustc_trans::back::write::run_passes
  12: rustc_driver::driver::phase_5_run_llvm_passes
  13: rustc_driver::driver::compile_input
  14: rustc_driver::run_compiler

error: Could not compile `serde_derive_internals`.
warning: build failed, waiting for other jobs to finish...
error: failed to verify package tarball

Caused by:
  build failed

rustc --version --verbose:

rustc 1.20.0 (f3d6973f4 2017-08-27)
binary: rustc
commit-hash: f3d6973f41a7d1fb83029c9c0ceaf0f5d4fd7208
commit-date: 2017-08-27
host: x86_64-unknown-linux-gnu
release: 1.20.0
LLVM version: 4.0
@matthiasbeyer
Copy link
Author

I experienced this error now multiple times. Maybe it is a good idea to include the complete build output here:

    Updating registry `https://github.com/rust-lang/crates.io-index`
warning: manifest has no documentation, homepage or repository.
See http://doc.crates.io/manifest.html#package-metadata for more info.
   Packaging libimagentryannotation v0.5.0 (file:///home/m/archive/development/rust/imag/lib/entry/libimagentryannotation)
   Verifying libimagentryannotation v0.5.0 (file:///home/m/archive/development/rust/imag/lib/entry/libimagentryannotation)
    Updating registry `https://github.com/rust-lang/crates.io-index`
 Downloading libimagentrylink v0.5.0
   Compiling num-traits v0.1.41
   Compiling cfg-if v0.1.2
   Compiling unicode-xid v0.0.4
   Compiling rustc-demangle v0.1.5
   Compiling glob v0.2.11
   Compiling itoa v0.3.4
   Compiling boolinator v2.4.0
   Compiling dtoa v0.4.2
   Compiling semver-parser v0.7.0
   Compiling lazy_static v0.2.11
   Compiling cc v1.0.3
   Compiling is-match v0.1.0
   Compiling libc v0.2.34
   Compiling serde v1.0.25
   Compiling lazy_static v1.0.0
   Compiling same-file v0.1.3
   Compiling percent-encoding v1.0.1
   Compiling gcc v0.3.54
   Compiling quote v0.3.15
   Compiling rustc-serialize v0.3.24
   Compiling either v1.4.0
   Compiling version v2.0.1
   Compiling utf8-ranges v1.0.0
   Compiling log v0.3.8
   Compiling regex-syntax v0.4.1
   Compiling unicode-normalization v0.1.5
   Compiling ansi_term v0.10.2
   Compiling void v1.0.2
   Compiling matches v0.1.6
   Compiling synom v0.11.3
   Compiling walkdir v1.0.7
   Compiling semver v0.8.0
   Compiling memchr v2.0.1
   Compiling time v0.1.38
   Compiling rand v0.3.18
   Compiling itertools v0.7.4
   Compiling unreachable v1.0.0
   Compiling unicode-bidi v0.3.4
   Compiling libimagerror v0.5.0
   Compiling syn v0.11.11
   Compiling backtrace-sys v0.1.16
   Compiling aho-corasick v0.6.4
   Compiling rust-crypto v0.2.36
   Compiling thread_local v0.3.5
   Compiling tempfile v2.2.0
   Compiling idna v0.1.4
   Compiling regex v0.2.3
   Compiling url v1.6.0
   Compiling toml v0.4.5
   Compiling serde_json v1.0.8
   Compiling serde_derive_internals v0.18.1
   Compiling backtrace v0.3.4
   Compiling error-chain v0.11.0
error: internal compiler error: unexpected panic

note: the compiler unexpectedly panicked. this is a bug.

note: we would appreciate a bug report: https://github.com/rust-lang/rust/blob/master/CONTRIBUTING.md#bug-reports

note: rustc 1.20.0 (f3d6973f4 2017-08-27) running on x86_64-unknown-linux-gnu

note: run with `RUST_BACKTRACE=1` for a backtrace

thread 'rustc' panicked at 'failed to acquire jobserver token: Error { repr: Os { code: 11, message: "Resource temporarily unavailable" } }', /checkout/src/libcore/result.rs:860:4
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
stack backtrace:
   0: std::sys::imp::backtrace::tracing::imp::unwind_backtrace
   1: std::sys_common::backtrace::_print
   2: std::panicking::default_hook::{{closure}}
   3: std::panicking::default_hook
   4: std::panicking::rust_panic_with_hook
   5: std::panicking::begin_panic_new
   6: std::panicking::begin_panic_fmt
   7: rust_begin_unwind
   8: core::panicking::panic_fmt
   9: core::result::unwrap_failed
  10: rustc_trans::back::write::execute_work
  11: rustc_trans::back::write::run_passes
  12: rustc_driver::driver::phase_5_run_llvm_passes
  13: rustc_driver::driver::compile_input
  14: rustc_driver::run_compiler

error: Could not compile `serde_derive_internals`.
warning: build failed, waiting for other jobs to finish...
error: internal compiler error: unexpected panic

note: the compiler unexpectedly panicked. this is a bug.

note: we would appreciate a bug report: https://github.com/rust-lang/rust/blob/master/CONTRIBUTING.md#bug-reports

note: rustc 1.20.0 (f3d6973f4 2017-08-27) running on x86_64-unknown-linux-gnu

note: run with `RUST_BACKTRACE=1` for a backtrace

thread 'rustc' panicked at 'failed to acquire jobserver token: Error { repr: Os { code: 11, message: "Resource temporarily unavailable" } }', /checkout/src/libcore/result.rs:860:4
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
stack backtrace:
   0: std::sys::imp::backtrace::tracing::imp::unwind_backtrace
   1: std::sys_common::backtrace::_print
   2: std::panicking::default_hook::{{closure}}
   3: std::panicking::default_hook
   4: std::panicking::rust_panic_with_hook
   5: std::panicking::begin_panic_new
   6: std::panicking::begin_panic_fmt
   7: rust_begin_unwind
   8: core::panicking::panic_fmt
   9: core::result::unwrap_failed
  10: rustc_trans::back::write::execute_work
  11: rustc_trans::back::write::run_passes
  12: rustc_driver::driver::phase_5_run_llvm_passes
  13: rustc_driver::driver::compile_input
  14: rustc_driver::run_compiler

error: Could not compile `toml`.
warning: build failed, waiting for other jobs to finish...
error: failed to verify package tarball

Caused by:
  build failed

@matthiasbeyer
Copy link
Author

With rustc 1.22.1 I get this error in a different form:

thread '<unnamed>' panicked at 'failed to acquire jobserver token: Resource temporarily unavailable (os error 11)', /checkout/src/librustc_trans/back/write.rs:1619:28
note: Run with `RUST_BACKTRACE=1` for a backtrace.
error: failed to acquire jobserver token: Resource temporarily unavailable (os error 11)

error: Could not compile `serde_json`.
warning: build failed, waiting for other jobs to finish...
thread '<unnamed>' panicked at 'failed to acquire jobserver token: Resource temporarily unavailable (os error 11)', /checkout/src/librustc_trans/back/write.rs:1619:28
note: Run with `RUST_BACKTRACE=1` for a backtrace.
error: failed to acquire jobserver token: Resource temporarily unavailable (os error 11)

error: Could not compile `toml`.
warning: build failed, waiting for other jobs to finish...
thread '<unnamed>' panicked at 'failed to acquire jobserver token: Resource temporarily unavailable (os error 11)', /checkout/src/librustc_trans/back/write.rs:1619:28
note: Run with `RUST_BACKTRACE=1` for a backtrace.
error: failed to acquire jobserver token: Resource temporarily unavailable (os error 11)

error: Could not compile `backtrace`.
warning: build failed, waiting for other jobs to finish...
thread '<unnamed>' panicked at 'failed to acquire jobserver token: Resource temporarily unavailable (os error 11)', /checkout/src/librustc_trans/back/write.rs:1619:28
note: Run with `RUST_BACKTRACE=1` for a backtrace.
error: failed to acquire jobserver token: Resource temporarily unavailable (os error 11)

error: Could not compile `serde_derive`.
warning: build failed, waiting for other jobs to finish...
error: failed to verify package tarball

Caused by:
  build failed

@matthiasbeyer
Copy link
Author

For the record: cargo build --release works on all crates where this bug was triggered (though I did not test all of them, just some).

@Ralith
Copy link
Contributor

Ralith commented Jan 19, 2018

For whatever reason, this is making it effectively impossible for me to package projects with lots of dependencies for Nix.

@pietroalbini pietroalbini added I-ICE Issue: The compiler panicked, giving an Internal Compilation Error (ICE) ❄️ T-cargo Relevant to the cargo team, which will review and decide on the PR/issue. C-bug Category: This is a bug. labels Jan 30, 2018
@bennofs
Copy link

bennofs commented Jan 30, 2018

I can reliably hit this bug when compiling my project (https://github.com/bennofs/nix-index) with Nix

@bennofs
Copy link

bennofs commented Jan 31, 2018

I believe this is a bug in our rustc package in nixpkgs, we appear to build it against jobserver 0.1.6 which is bugged: https://nix-cache.s3.amazonaws.com/log/j4n6lk9fl5ngby7cvhxm5jys8zz62xf1-rustc-1.22.1.drv. Maybe it would be a good idea to have a hard dependency on jobserver >= 0.1.8 for rustc?

@bennofs
Copy link

bennofs commented Jan 31, 2018

There is an issue here: rustc version 1.22.1 has jobserver-0.1.6 in its lockfile, and the nix build appears to reliably trigger this bug while compiling 1.23.0. So we have a problem in nixpkgs that we cannot bootstrap 1.23.0 using the binary distribution from rust-lang.org.

@alexcrichton you are the author of jobserver-rs, is there something we could do to workaround this bug for bootstrapping 1.23.0? Or is it necessary to use a patched 1.22.1 for compiling 1.23.0?

@alexcrichton
Copy link
Member

@bennofs the workaround would be to remove any ambient jobservers and let cargo/rustc manage it, if any

@Ralith
Copy link
Contributor

Ralith commented Jan 31, 2018

Can't we bootstrap using 1.23.0 binaries?

@arianvp
Copy link

arianvp commented Feb 15, 2018

I can reliably hit this bug as well in nix

8573 added a commit to 8573/irc-bot.rs that referenced this issue Mar 19, 2018
Bump the Rust toolchain version specified in `shell.nix` from 1.22.1
to 1.24.1 to escape a `jobserver` bug:
<rust-lang/rust#46981>.

Version 1.24.1 comes with rustfmt, so this patch also drops the
dependency on the Nix package for rustfmt. The version of rustfmt
bundled with rustc 1.24.1 does not accept any of the `rustfmt.toml`
configuration file options I was using, so this patch deletes
`rustfmt.toml`.
8573 added a commit to 8573/nixos-config that referenced this issue May 28, 2018
Update my Rust toolchain installation from version 1.22.1 to version
1.24.1 to escape a jobserver bug:
<rust-lang/rust#46981>.

Version 1.24.1 comes with rustfmt, so this patch uninstalls the Nix
package for rustfmt.
@kevincox
Copy link
Contributor

kevincox commented Jan 25, 2019

I can reliably hit this bug using entr -r.

% entr -r cargo test <<</tmp/file
   Compiling proc-macro2 v0.4.26
   Compiling cc v1.0.28
   Compiling libc v0.2.48
   Compiling unicode-xid v0.1.0
   Compiling autocfg v0.1.2
   Compiling serde v1.0.85
   Compiling semver-parser v0.7.0
   Compiling ryu v0.2.7
thread '<unnamed>' panicked at 'failed to acquire jobserver token: early EOF on jobserver pipe', src/librustc_codegen_ssa/back/write.rs:1348:29
note: Run with `RUST_BACKTRACE=1` environment variable to display a backtrace.
error: failed to acquire jobserver token: early EOF on jobserver pipe

error: aborting due to previous error

thread '<unnamed>' panicked at 'failed to acquire jobserver token: early EOF on jobserver pipe', src/librustc_codegen_ssa/back/write.rs:1348:29
note: Run with `RUST_BACKTRACE=1` environment variable to display a backtrace.
error: failed to acquire jobserver token: early EOF on jobserver pipe

error: Could not compile `unicode-xid`.
warning: build failed, waiting for other jobs to finish...
error: aborting due to previous error

thread '<unnamed>' panicked at 'failed to acquire jobserver token: early EOF on jobserver pipe', src/librustc_codegen_ssa/back/write.rs:1348:29
note: Run with `RUST_BACKTRACE=1` environment variable to display a backtrace.
error: Could not compile `libc`.
warning: build failed, waiting for other jobs to finish...
...

As far as I am aware I don't have any jobservers running.

@kevincox
Copy link
Contributor

kevincox commented Mar 4, 2019

I realized that the following issue was only occurring on nightly. However after the latest update this is now occurring on stable as well.

% cargo version
cargo 1.33.0 (f099fe94b 2019-02-12)
% rustc --version
rustc 1.33.0 (2aa4c46cf 2019-02-28)

@davidnorthetal
Copy link

davidnorthetal commented Apr 23, 2019

@kevincox
Copy link
Contributor

@alexcrichton Can you please take another look at this. It appears that your diagnosis is wrong and many people are experiencing this issue.

There is no workaround as far as I can tell and this is seriously workflow disrupting.

@alexcrichton
Copy link
Member

A misconfigured jobserver can still cause issues in rustc, and this error is indicative of a misconfigured jobserver in one way or another. This indicates an environmental problem rather than a problem in rustc AFAIK. If someone else manages to fix rustc we can of course merge, but I don't personally believe there's much we can do in rustc for this.

@kevincox
Copy link
Contributor

There is no other jobserver in my environment. I have checked numerous ways and am highly confident.

@AlvaroBernalG
Copy link

AlvaroBernalG commented Apr 27, 2019

@alexcrichton I am also getting the same error as @kevincox when running entr with the reload flag.

find . -name "*.rs" | entr -rs 'cargo run'

Throws the following error:

error: failed to acquire jobserver token: early EOF on jobserver pipe

According to the entr documentation whats happening is :

  1. immediately start the server
  2. block until any of the listed files change
  3. terminate the background process
  4. wait for the server to exit before restarting

@davidnorthetal
Copy link

I created an upstream issue for entr as well - it surrounds entr closing STDIN, which was added in later entr versions for compatibility with some programs reasons - seems it then breaks with rustc.

https://bitbucket.org/eradman/entr/issues/81/restart_ops-closing-of-stdin-causes

@CleanCut
Copy link
Contributor

It sounds like the underlying issue is #60447

@dbohdan
Copy link

dbohdan commented Jan 4, 2020

A workaround for those using entr:

find Cargo.* src/ static/ ... -type f | entr -r sh -c 'cargo run < /dev/null'

It works for me with rustc and cargo 1.40.0 on Linux.

@kevincox
Copy link
Contributor

kevincox commented Apr 3, 2020

This is no longer happening for me. I don't know what has changed but I don't see this anymore.

% cargo --version
cargo 1.42.0 (86334295e 2020-01-31

@CleanCut
Copy link
Contributor

CleanCut commented Apr 3, 2020

It still happens for me with rustc 1.42.0 (b8cedc004 2020-03-09) and cargo 1.42.0 (86334295e 2020-01-31). 😢 Just tested it with cargo run <&- and got a panic starting with the same failed to acquire jobserver token... bit.

@petrochenkov
Copy link
Contributor

So, I've found an entirely legitimate way to trigger this error (the "early EOF on jobserver pipe" case) on new versions of make, without having stdin closed or anything other exotic setup.

If you call cargo from an outer make-based build system

# Assume we are using `cmake`
add_custom_target(
    ...
    COMMAND cargo build
    ...
)

then cargo will receive file descriptors for the jobserver pipe through an env var called MAKEFLAGS.

Except that new versions of make will mark those descriptors as FD_CLOEXEC so they will be either dead or, worse, replaced with other random files when cargo tries to access them :D

The solution is to prevent make from closing the pipe by using string $(MAKE) anywhere in the command.
(Yep, that's how make works - https://www.gnu.org/software/make/manual/make.html#How-the-MAKE-Variable-Works.)

# Assume we are using `cmake`
add_custom_target(
    ...
    COMMAND SNEAK_DOLLAR_MAKE_INTO_THE_COMMAND=$(MAKE) cargo build
    ...
)

This is silly, but it seems like make doesn't leave any other way to inherit jobserver (at least not readily available from higher level makefile generators).

RalfJung added a commit to RalfJung/rust that referenced this issue Sep 25, 2020
Reopen standard file descriptors when they are missing on Unix

The syscalls returning a new file descriptors generally return lowest-numbered
file descriptor not currently opened, without any exceptions for those
corresponding to stdin, sdout, or stderr.

Previously when any of standard file descriptors has been closed before starting
the application, operations on std::io::{stderr,stdin,stdout} were likely to
either succeed while being performed on unrelated file descriptor, or fail with
EBADF which is silently ignored.

Avoid the issue by using /dev/null as a replacement when the standard file
descriptors are missing.

The implementation is based on the one found in musl. It was selected among a
few others on the basis of the lowest overhead in the case when all descriptors
are already present (measured on GNU/Linux).

Closes rust-lang#57728.
Closes rust-lang#46981.
Closes rust-lang#60447.

Benefits:
* Makes applications robust in the absence of standard file descriptors.
* Upholds IntoRawFd / FromRawFd safety contract (which was broken previously).

Drawbacks:
* Additional syscall during startup.
* The standard descriptors might have been closed intentionally.
* Requires /dev/null.

Alternatives:
* Check if stdin, stdout, stderr are opened and provide no-op substitutes in std::io::{stdin,stdout,stderr} without reopening them directly.
* Leave the status quo, expect robust applications to reopen them manually.
@bors bors closed this as completed in db7ee7c Sep 28, 2020
matthiaskrgr added a commit to matthiaskrgr/rust that referenced this issue Mar 29, 2023
… r=bjorn3

do not panic on failure to acquire jobserver token

Purpose: remove `panic`.

Rust fails to acquire token if an error in build system occurs - environment variable contains incorrect `jobserver-auth`. It isn't ice so compiler shouldn't panic on such error.

Related issue: rust-lang#46981
@ShubhamKamatgekar
Copy link

ShubhamKamatgekar commented Mar 22, 2024

was recently facing similar issue while trying to run cargo build inside a makefile with buildroot
was facing this everytime when i run cargo build
Build fails: error: failed to acquire jobserver token: Bad file descriptor (os error 9)

This worked for me.
MAKEFLAGS= JOBS= cargo build

@kevincox
Copy link
Contributor

If you are actually running cargo inside of make you are hitting a different problem and I recommend opening a different issue.

This issue was about a bug where no job server was running. This issue was fixed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
C-bug Category: This is a bug. I-ICE Issue: The compiler panicked, giving an Internal Compilation Error (ICE) ❄️ T-cargo Relevant to the cargo team, which will review and decide on the PR/issue.
Projects
None yet
Development

Successfully merging a pull request may close this issue.