Compiling code hangs compiler

[joshlf]

When compiling this code, if this test is uncommented, the compiler hangs indefinitely during compilation.

Details:

• Platform: macOS High Sierra 10.13.6
• cargo version - cargo 1.34.0-nightly (245818076 2019-01-27)
• rustc --version - rustc 1.34.0-nightly (8ae730a44 2019-02-04)
• invocation: cargo test --no-run
• While running, rustc uses a constant 80% CPU (of 200% - 2-core CPU)

[Centril]

@rust-lang/infra Extremely wild speculation... might this have anything to do with #58506?

@joshlf Could you try to reduce the code example that hangs the compiler to be self-contained and minimal?

[joshlf]

Extremely wild speculation... might this have anything to do with #58506?

I will note that this has been happening for more than six months; I just haven't bothered reporting it until now.

Could you try to reduce the code example that hangs the compiler to be self-contained and minimal?

Minimizing the body of the test to this seems to a) cause the problem to happen on my machine and, b) cause the Rust playground's rustc to get SIGKILL.

let mut buf = vec![0u8; 1 << 32];
(&[0u8; (1 << 32) - 8]);

[Mark-Simulacrum]

Highly unlikely to have anything to do with #58506 -- that's likely a hardware, not a hanging issue.

The SIGKILL on playground is likely because we limit memory to 256 MB, at least as far as I can tell, (see docker invocation here).

The hang appears to be in rustc_mir (<rustc_mir::interpret::memory::Memory<'a, 'mir, 'tcx, M>>::copy_repeatedly); cc @oli-obk @RalfJung @rust-lang/compiler @nnethercote -- but it might be in some higher function. Here's the backtrace I hit:

#0  core::iter::range::<impl core::iter::traits::iterator::Iterator for core::ops::range::Range<A>>::next (self=<optimized out>)
    at /home/mark/Build/rust/src/libcore/iter/range.rs:212
#1  <rustc_mir::interpret::memory::Memory<'a, 'mir, 'tcx, M>>::copy_undef_mask (self=<optimized out>, src=..., dest=..., size=...,
    repeat=<optimized out>) at src/librustc_mir/interpret/memory.rs:790
#2  <rustc_mir::interpret::memory::Memory<'a, 'mir, 'tcx, M>>::copy_repeatedly (self=0x7f045e41c1f8, src=<optimized out>, src_align=...,
    dest=<optimized out>, dest_align=..., size=..., length=1, nonoverlapping=<optimized out>) at src/librustc_mir/interpret/memory.rs:766
#3  0x00007f0462835528 in <rustc_mir::interpret::memory::Memory<'a, 'mir, 'tcx, M>>::copy (self=0x7f045e41c1f8, src_align=..., dest_align=...,
    size=..., nonoverlapping=255, src=<optimized out>, dest=<optimized out>) at src/librustc_mir/interpret/memory.rs:673
#4  rustc_mir::interpret::place::<impl rustc_mir::interpret::eval_context::EvalContext<'a, 'mir, 'tcx, M>>::copy_op_no_validate (
    self=<optimized out>, src=..., dest=...) at src/librustc_mir/interpret/place.rs:838
#5  0x00007f0462846287 in rustc_mir::interpret::place::<impl rustc_mir::interpret::eval_context::EvalContext<'a, 'mir, 'tcx, M>>::copy_op (
    self=<optimized out>, src=..., dest=...) at src/librustc_mir/interpret/place.rs:796
#6  rustc_mir::interpret::step::<impl rustc_mir::interpret::eval_context::EvalContext<'a, 'mir, 'tcx, M>>::eval_rvalue_into_place (
    self=0x7f045e41c190, rvalue=<optimized out>, place=<optimized out>) at src/librustc_mir/interpret/step.rs:147
#7  0x00007f0462844e39 in rustc_mir::interpret::step::<impl rustc_mir::interpret::eval_context::EvalContext<'a, 'mir, 'tcx, M>>::statement (
    self=<optimized out>, stmt=<optimized out>) at src/librustc_mir/interpret/step.rs:85
#8  rustc_mir::interpret::step::<impl rustc_mir::interpret::eval_context::EvalContext<'a, 'mir, 'tcx, M>>::step (self=0x7f045e41c190)
    at src/librustc_mir/interpret/step.rs:61
#9  rustc_mir::interpret::step::<impl rustc_mir::interpret::eval_context::EvalContext<'a, 'mir, 'tcx, M>>::run (self=0x7f045e41c190)
    at src/librustc_mir/interpret/step.rs:40
#10 0x00007f0462a5b50e in rustc_mir::const_eval::eval_body_using_ecx (ecx=0x7f045e41c190, cid=..., mir=<optimized out>, param_env=...)
    at src/librustc_mir/const_eval.rs:157
#11 0x00007f0462a5ea59 in rustc_mir::const_eval::eval_body_and_ecx (tcx=..., cid=...,
    mir=<unknown type in /home/mark/.rustup/toolchains/stage1/lib/librustc_mir-cd23d8cc09622ccc.so, CU 0x17247a7, DIE 0x1901b01>, param_env=...)
    at src/librustc_mir/const_eval.rs:120
#12 rustc_mir::const_eval::const_eval_raw_provider (tcx=..., key=...) at src/librustc_mir/const_eval.rs:636
#13 0x00007f04627392cf in rustc::ty::query::<impl rustc::ty::query::config::QueryAccessors<'tcx> for rustc::ty::query::queries::const_eval_raw<'tcx>>::compute::{{closure}} () at /home/mark/Build/rust/src/librustc/ty/query/plumbing.rs:963
#14 rustc::ty::query::__query_compute::const_eval_raw (f=...) at /home/mark/Build/rust/src/librustc/ty/query/plumbing.rs:922
#15 0x00007f0462413cfc in rustc::ty::query::<impl rustc::ty::query::config::QueryAccessors<'tcx> for rustc::ty::query::queries::const_eval_raw<'tcx>>::compute (tcx=..., key=<error reading variable: Cannot access memory at address 0x32d7007>)
    at /home/mark/Build/rust/src/librustc/ty/query/plumbing.rs:955
#16 0x00007f046277817e in rustc::dep_graph::graph::DepGraph::with_task_impl (self=<optimized out>, key=..., cx=..., arg=..., no_tcx=false,
    task=<optimized out>, create_task=0x7f046272a400 <core::ops::function::FnOnce::call_once>,
    finish_task_and_alloc_depnode=0x7f046272a200 <core::ops::function::FnOnce::call_once>, hash_result=<optimized out>)
    at /home/mark/Build/rust/src/librustc/dep_graph/graph.rs:334
#17 0x00007f04625b994c in rustc::dep_graph::graph::DepGraph::with_task (self=0x1, cx=..., task=0x7f03314df010, key=..., arg=...,
    hash_result=<optimized out>) at /home/mark/Build/rust/src/librustc/dep_graph/graph.rs:202
#18 rustc::ty::query::plumbing::<impl rustc::ty::context::TyCtxt<'a, 'gcx, 'tcx>>::force_query_with_job::{{closure}}::{{closure}} (tcx=...)
    at /home/mark/Build/rust/src/librustc/ty/query/plumbing.rs:567
#19 rustc::ty::query::plumbing::<impl rustc::ty::context::TyCtxt<'a, 'gcx, 'tcx>>::start_query::{{closure}}::{{closure}} ()
    at /home/mark/Build/rust/src/librustc/ty/query/plumbing.rs:280
#20 rustc::ty::context::tls::enter_context::{{closure}} () at /home/mark/Build/rust/src/librustc/ty/context.rs:1955
#21 rustc::ty::context::tls::set_tlv (f=..., value=<optimized out>) at /home/mark/Build/rust/src/librustc/ty/context.rs:1888
#22 rustc::ty::context::tls::enter_context (f=..., context=<optimized out>) at /home/mark/Build/rust/src/librustc/ty/context.rs:1954
#23 rustc::ty::query::plumbing::<impl rustc::ty::context::TyCtxt<'a, 'gcx, 'tcx>>::start_query::{{closure}} (current_icx=<optimized out>)
    at /home/mark/Build/rust/src/librustc/ty/query/plumbing.rs:279

[Mark-Simulacrum]

Oh, forgot to note -- I am able to reproduce on Linux as well and with this code (same as the previous minimal, just with fn main added.

pub fn main() {
    let mut _buf = vec![0u8; 1 << 32];
    (&[0u8; (1 << 32) - 8]);
}

[RalfJung]

Is this more than "just" CTFE going OOM?

[Mark-Simulacrum]

Well, maybe - I haven't actually waited, but even if it does eventually OOM, it certainly is also quite slow so this could be a good test case for optimizing what I assume is a relatively tight loop.

[oli-obk]

This is not even ctfe going OOM (except maybe on the playground), this is just ctfe taking forever. We are running a loop for 2^32 copies of an undef mask (which keeps setting single bits in bytes)

I have started setting up some optimizations in #58556

[joshlf]

On version rustc 1.35.0-nightly (94fd04589 2019-03-21), this unfortunately still eventually runs LLVM out of memory.

[oli-obk]

running out of memory meaning this consumes more than 8GB of RAM for a 4GB array? One duplication is pretty much guaranteed the way we work with constants and I don't see many possibilites around (though I think we could try to tell llvm to just use our memory and not duplicate it. Not sure how well that works with LLVM optimizations).

Some Playground tests showed that anything beyond 60MB gets killed (though 60MB gets killed randomly every now and then). So I guess if 256MB is the kill limit, we have a duplication of 4. Meaning in order to comiple a 4GB array, you need 16GB of RAM plus anything you need for compiling a 1MB array.

[RalfJung]

Might be caused by or at least related to superlinear compile times for array literals: #52868, #37155

[Enselic]

Triage: Looks a lot like a duplicate, so let's close as duplicate of #37155.