Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

rustc uses a constantly increasing amount of memory up to OS crash or OOM kill #128365

Open
vigna opened this issue Jul 29, 2024 · 5 comments
Open

rustc uses a constantly increasing amount of memory up to OS crash or OOM kill #128365

vigna opened this issue Jul 29, 2024 · 5 comments
Labels
C-bug Category: This is a bug. fixed-by-next-solver Fixed by the next-generation trait solver, `-Znext-solver`. I-crash Issue: The compiler crashes (SIGSEGV, SIGABRT, etc). Use I-ICE instead when the compiler panics. T-types Relevant to the types team, which will review and decide on the PR/issue.

Comments

@vigna
Copy link

vigna commented Jul 29, 2024 •
edited
Loading

Clone the bug branch of webgraph-rs and run a check:

git clone [email protected]:vigna/webgraph-rs
cd webgraph-rs
git checkout bug
cargo c

On MacOs 14.5 (M1), nixos 23.11, and Fedora 38 the compilation process stops at this message:

    Building [=======================> ] 207/209: webgraph

At this point, rustc uses a constantly increasing amount of memory. The result is an OS crash in MacOS, and OOM-kill on Linux.

The problem appears both in stable and nightly.

rustc --version --verbose:

rustc 1.82.0-nightly (2cbbe8b8b 2024-07-28)
binary: rustc
commit-hash: 2cbbe8b8bb2be672b14cf741a2f0ec24a49f3f0b
commit-date: 2024-07-28
host: aarch64-apple-darwin
release: 1.82.0-nightly
LLVM version: 18.1.7

rustc 1.80.0 (051478957 2024-07-21)
binary: rustc
commit-hash: 051478957371ee0084a7c0913941d2a8c4757bb9
commit-date: 2024-07-21
host: aarch64-apple-darwin
release: 1.80.0
LLVM version: 18.1.7

The same setup works without problems on Arch Linux.

UPDATE: the first post claimed mistakenly that the check was working on Fedora 38.
@vigna vigna added the C-bug Category: This is a bug. label Jul 29, 2024
@rustbot rustbot added the needs-triage This issue may need triage. Remove it if it has been sufficiently triaged. label Jul 29, 2024
@tgross35
Copy link
Contributor

Could you add the backtrace and relevant logs?

@vigna
Copy link
Author

vigna commented Jul 29, 2024

I have no backtrace—I have to pkill -9 rustc to avoid that the OS crashes.

Here is the log of cargo c up to the point where the compiler gets stuck.

   Compiling proc-macro2 v1.0.86
   Compiling unicode-ident v1.0.12
   Compiling libc v0.2.155
   Compiling autocfg v1.3.0
    Checking cfg-if v1.0.0
    Checking memchr v2.7.4
   Compiling pkg-config v0.3.30
   Compiling crossbeam-utils v0.8.20
   Compiling version_check v0.9.5
    Checking regex-syntax v0.8.4
    Checking adler v1.0.2
   Compiling vcpkg v0.2.15
   Compiling proc-macro-error-attr v1.0.4
    Checking aho-corasick v1.1.3
    Checking miniz_oxide v0.7.4
    Checking gimli v0.29.0
   Compiling rayon-core v1.12.1
   Compiling tinyvec_macros v0.1.1
   Compiling tinyvec v1.8.0
   Compiling memoffset v0.7.1
    Checking object v0.36.2
    Checking crossbeam-epoch v0.9.18
    Checking crossbeam-deque v0.8.5
    Checking regex-automata v0.4.7
   Compiling proc-macro-error v1.0.4
   Compiling thiserror v1.0.63
   Compiling quote v1.0.36
   Compiling anyhow v1.0.86
   Compiling jobserver v0.1.32
   Compiling syn v2.0.72
    Checking utf8parse v0.2.2
   Compiling heck v0.4.1
   Compiling cc v1.1.7
    Checking rustc-demangle v0.1.24
    Checking addr2line v0.22.0
    Checking anstyle v1.0.8
    Checking bitflags v2.6.0
    Checking either v1.13.0
    Checking anstyle-parse v0.2.5
   Compiling unicode-normalization v0.1.23
   Compiling num-traits v0.2.19
   Compiling unicode-bidi v0.3.15
    Checking is_terminal_polyfill v1.70.1
    Checking colorchoice v1.0.2
    Checking anstyle-query v1.1.1
   Compiling percent-encoding v2.3.1
    Checking anstream v0.6.15
   Compiling form_urlencoded v1.2.1
   Compiling idna v0.5.0
    Checking regex v1.10.5
    Checking rayon v1.10.0
    Checking getrandom v0.2.15
   Compiling impl-tools v0.10.0
    Checking half v2.4.1
    Checking pin-utils v0.1.0
    Checking byteorder v1.5.0
   Compiling libz-sys v1.1.18
   Compiling backtrace v0.3.73
   Compiling zstd-sys v2.0.12+zstd.1.5.6
   Compiling libgit2-sys v0.17.0+1.8.1
   Compiling fnv v1.0.7
   Compiling strsim v0.11.1
    Checking bitflags v1.3.2
    Checking bytes v1.6.1
   Compiling syn v1.0.109
   Compiling core-foundation-sys v0.8.6
   Compiling ident_case v1.0.1
    Checking combine v4.6.7
   Compiling iana-time-zone v0.1.60
    Checking nix v0.26.4
   Compiling url v2.5.2
   Compiling darling_core v0.20.10
   Compiling impl-tools-lib v0.10.0
    Checking rand_core v0.6.4
    Checking mach2 v0.4.2
   Compiling log v0.4.22
    Checking ppv-lite86 v0.2.17
   Compiling zstd-safe v7.2.0
   Compiling rustix v0.38.34
    Checking lazy_static v1.5.0
   Compiling radium v0.7.0
    Checking rand_chacha v0.3.1
   Compiling chrono v0.4.38
    Checking errno v0.3.9
    Checking clap_lex v0.7.2
   Compiling heck v0.5.0
    Checking arrayvec v0.7.4
    Checking maligned v0.2.1
   Compiling thiserror-impl v1.0.63
   Compiling enum-as-inner v0.6.0
   Compiling mem_dbg-derive v0.1.5
   Compiling darling_macro v0.20.10
    Checking itoa v1.0.11
    Checking tap v1.0.1
   Compiling epserde v0.6.1
    Checking clap_builder v4.5.11
    Checking wyz v0.5.1
   Compiling darling v0.20.10
    Checking num-format v0.4.4
   Compiling itertools v0.10.5
    Checking common_traits v0.11.0
   Compiling clap_derive v4.5.11
   Compiling epserde-derive v0.6.0
   Compiling sealed v0.5.0
   Compiling lender-derive v0.1.2
    Checking sysctl v0.5.5
    Checking rand v0.8.5
    Checking sysinfo v0.29.11
    Checking env_filter v0.1.2
    Checking pluralizer v0.4.0
    Checking common_traits v0.10.2
    Checking num_cpus v1.16.0
    Checking crc32fast v1.4.2
    Checking mmap-rs v0.6.1
    Checking partition v0.1.2
    Checking stable_deref_trait v1.2.0
    Checking arbitrary-chunks v0.4.1
    Checking fastrand v2.1.0
    Checking xxhash-rust v0.8.12
    Checking zerofrom v0.1.4
    Checking humantime v2.1.0
    Checking funty v2.0.0
    Checking tempfile v3.10.1
    Checking yoke v0.7.4
    Checking mem_dbg v0.2.2
    Checking env_logger v0.11.5
    Checking rdst v0.20.14
    Checking flate2 v1.0.30
    Checking core_affinity v0.8.1
    Checking dsi-progress-logger v0.2.4
    Checking lender v0.2.9
   Compiling derive_setters v0.1.6
    Checking float-cmp v0.9.0
    Checking clap v4.5.11
    Checking bitvec v1.0.1
    Checking encoding_rs v0.8.34
    Checking difflib v0.4.0
    Checking predicates-core v1.0.8
    Checking normalize-line-endings v0.3.0
    Checking bytemuck v1.16.1
    Checking xxh3 v0.1.1
    Checking predicates v3.1.2
    Checking dsi-bitstream v0.4.0
    Checking clap_complete v4.5.11
    Checking sysinfo v0.30.13
    Checking itertools v0.12.1
    Checking dary_heap v0.3.6
    Checking java-properties v2.0.0
   Compiling ambassador v0.4.1
   Compiling derivative v2.2.0
    Checking zstd v0.13.2
    Checking sux v0.4.1
   Compiling git2 v0.19.0
   Compiling built v0.7.4
   Compiling webgraph v0.1.0 (/private/tmp/webgraph-rs)
error[E0412]: cannot find type `BR` in this scope
   --> src/labels/bitstream.rs:197:26
    |
197 | impl BitDeserializer<BE, BR> for SwhDeserializer<'_, BR> {
    |                          ^^
    |
   ::: /Users/vigna/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dsi-bitstream-0.4.0/src/traits/endianness.rs:78:1
    |
78  | pub type BE = BigEndian;
    | ----------- similarly named type alias `BE` defined here
    |
help: a type alias with a similar name exists
    |
197 | impl BitDeserializer<BE, BE> for SwhDeserializer<'_, BR> {
    |                          ~~
help: you might be missing a type parameter
    |
197 | impl<BR> BitDeserializer<BE, BR> for SwhDeserializer<'_, BR> {
    |     ++++

error[E0412]: cannot find type `BR` in this scope
   --> src/labels/bitstream.rs:197:54
    |
197 | impl BitDeserializer<BE, BR> for SwhDeserializer<'_, BR> {
    |                                                      ^^
    |
   ::: /Users/vigna/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dsi-bitstream-0.4.0/src/traits/endianness.rs:78:1
    |
78  | pub type BE = BigEndian;
    | ----------- similarly named type alias `BE` defined here
    |
help: a type alias with a similar name exists
    |
197 | impl BitDeserializer<BE, BR> for SwhDeserializer<'_, BE> {
    |                                                      ~~
help: you might be missing a type parameter
    |
197 | impl<BR> BitDeserializer<BE, BR> for SwhDeserializer<'_, BR> {
    |     ++++

error[E0412]: cannot find type `BR` in this scope
   --> src/labels/bitstream.rs:202:25
    |
202 |         bitstream: &mut BR,
    |                         ^^ help: a type alias with a similar name exists: `BE`
    |
   ::: /Users/vigna/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dsi-bitstream-0.4.0/src/traits/endianness.rs:78:1
    |
78  | pub type BE = BigEndian;
    | ----------- similarly named type alias `BE` defined here

error[E0412]: cannot find type `BR` in this scope
   --> src/labels/bitstream.rs:203:48
    |
203 |     ) -> std::result::Result<Self::DeserType, <BR as BitRead>::Error> {
    |                                                ^^
    |
   ::: /Users/vigna/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dsi-bitstream-0.4.0/src/traits/endianness.rs:78:1
    |
78  | pub type BE = BigEndian;
    | ----------- similarly named type alias `BE` defined here
    |
help: a type alias with a similar name exists
    |
203 |     ) -> std::result::Result<Self::DeserType, <BE as BitRead>::Error> {
    |                                                ~~
help: you might be missing a type parameter
    |
197 | impl<BR> BitDeserializer<BE, BR> for SwhDeserializer<'_, BR> {
    |     ++++

error[E0107]: missing generics for trait `dsi_bitstream::traits::BitRead`
   --> src/labels/bitstream.rs:203:54
    |
203 |     ) -> std::result::Result<Self::DeserType, <BR as BitRead>::Error> {
    |                                                      ^^^^^^^ expected 1 generic argument
    |
note: trait defined here, with 1 generic parameter: `E`
   --> /Users/vigna/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dsi-bitstream-0.4.0/src/traits/bits.rs:81:11
    |
81  | pub trait BitRead<E: Endianness> {
    |           ^^^^^^^ -
help: add missing generic argument
    |
203 |     ) -> std::result::Result<Self::DeserType, <BR as BitRead<E>>::Error> {
    |                                                             +++

^C  Building [=======================> ] 207/209: webgraph

@saethlin saethlin added I-crash Issue: The compiler crashes (SIGSEGV, SIGABRT, etc). Use I-ICE instead when the compiler panics. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. T-types Relevant to the types team, which will review and decide on the PR/issue. and removed T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. labels Jul 29, 2024
@SNCPlay42
Copy link
Contributor

Reduced:

pub trait BitRead {}
pub trait BitSeek {}

pub trait ReaderBuilder {
    type Reader<'a>;
}

pub struct MmapReaderBuilder {}

impl ReaderBuilder for MmapReaderBuilder where for<'a> Self::Reader<'a>: BitRead + BitSeek
{}

@shadyfennec
Copy link

Ran a cargo-bisect-rustc on a Windows 10 machine with the reduced code, that platform also reproduces the problem, unbound consumption of memory until OS crash/OOM killed (haven't tried to let it do any of these). I hope I bisected correctly, the regression is going quite far and goes into "edition 2021 isn't stable yet" territory, but as far as I can tell the edition doesn't matter. No per-commit bisection because it's too old to have CI artifacts.

  commit[0] 2021-07-15: Auto merge of #83319 - tmiasko:packed-aligned, r=jackh726
  commit[1] 2021-07-16: Auto merge of #86993 - jackh726:project-gat-binders, r=nikomatsakis
  commit[2] 2021-07-16: Auto merge of #87177 - ehuss:update-cargo, r=ehuss
  commit[3] 2021-07-16: Auto merge of #86662 - mockersf:fix-86620-link-unknown-location, r=jyn514
  commit[4] 2021-07-16: Auto merge of #87182 - GuillaumeGomez:rollup-whwohua, r=GuillaumeGomez
  commit[5] 2021-07-16: Auto merge of #87140 - camsteffen:pat-slice-refs, r=oli-obk
  commit[6] 2021-07-16: Auto merge of #84623 - jackh726:gats-incomplete, r=nikomatsakis
  commit[7] 2021-07-16: Auto merge of #87201 - GuillaumeGomez:rollup-4loi2q9, r=GuillaumeGomez

Two of those commits stand out to me: #84623 is the GAT incomplete feature commit, but as far as I can tell the "this feature is incomplete" message doesn't appear in any of the outputs I have found for the bisected nightlies, before and after. The other is #86993, which also touches on GATs and seem to be a bit more related to the faulty code.

I hope this bisection is correct, I might be totally wrong; but for my bisect toolchain of 2021-07-16, cargo check doesn't hang, and in 2021-07-17, it does hang.

@lqd
Copy link
Member

lqd commented Jul 30, 2024

There are many ways (and open issues) to have the old solver infinite loop like this, and here as well the overflow is detected by the new solver.

@rustbot label +fixed-by-next-solver

@rustbot rustbot added the fixed-by-next-solver Fixed by the next-generation trait solver, `-Znext-solver`. label Jul 30, 2024
@saethlin saethlin removed the needs-triage This issue may need triage. Remove it if it has been sufficiently triaged. label Aug 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
C-bug Category: This is a bug. fixed-by-next-solver Fixed by the next-generation trait solver, `-Znext-solver`. I-crash Issue: The compiler crashes (SIGSEGV, SIGABRT, etc). Use I-ICE instead when the compiler panics. T-types Relevant to the types team, which will review and decide on the PR/issue.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@lqd @vigna @SNCPlay42 @saethlin @tgross35 @rustbot @shadyfennec