Crater shows many crates as "regressed" but there is no error

[RalfJung]

Most of the recent crater runs I looked at have a ton of crates that "regressed", but there's no build failure to be seen in the logfile. For instance, in https://crater-reports.s3.amazonaws.com/pr-129199/index.html, all the items I looked at in "build failed (unknown)" fall in that category. Concrete examples:

• https://crater-reports.s3.amazonaws.com/pr-129199/try%232c70eb47850052381670935f807d41647a7dc992/gh/pba-pba-pba.blockchain-from-scratch/log.txt
• https://crater-reports.s3.amazonaws.com/pr-129199/try%232c70eb47850052381670935f807d41647a7dc992/reg/cryo_python-0.3.2/log.txt
• https://crater-reports.s3.amazonaws.com/pr-129199/try%232c70eb47850052381670935f807d41647a7dc992/reg/earendil_packet-0.5.8/log.txt
• https://crater-reports.s3.amazonaws.com/pr-129199/try%232c70eb47850052381670935f807d41647a7dc992/reg/waynest-0.0.9/log.txt

Also see here for some more examples of this.

[Skgland]

3 out of the four listed examples appear to have an error in crater/rustwide itself

• The second example has this error:

[INFO] running Command { std: "docker" "start" "-a" "4587da511f2e05fc5af9052e2a4c27f9b9c9a90262f127468c228df05bbf3564", kill_on_drop: false }
[INFO] [stderr] time="2024-08-22T06:05:15Z" level=error msg="error waiting for container: "
[INFO] [stderr] Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: unable to apply cgroup configuration: unable to start unit "docker-4587da511f2e05fc5af9052e2a4c27f9b9c9a90262f127468c228df05bbf3564.scope" (properties [{Name:Description Value:"libcontainer container 4587da511f2e05fc5af9052e2a4c27f9b9c9a90262f127468c228df05bbf3564"} {Name:Slice Value:"system.slice"} {Name:Delegate Value:true} {Name:PIDs Value:@au [763051]} {Name:MemoryAccounting Value:true} {Name:CPUAccounting Value:true} {Name:IOAccounting Value:true} {Name:TasksAccounting Value:true} {Name:DefaultDependencies Value:false}]): Message recipient disconnected from message bus without replying: unknown
[INFO] running Command { std: "docker" "inspect" "4587da511f2e05fc5af9052e2a4c27f9b9c9a90262f127468c228df05bbf3564", kill_on_drop: false }

• The third example has a basically identical error.
• The fourth example ends with the error

[INFO] [stderr] Error response from daemon: write /var/lib/docker/overlay2/aedbbd379c1a6eb2ba4545c5b28a51da14744c09bf720798329d19a1e8fffa42-init/.tmp-lower3624005366: no space left on device

[Skgland]

The fourth one might be sorted properly with 397f3f0 from #729

[RalfJung]

Some more examples:

• https://crater-reports.s3.amazonaws.com/pr-127907/try%236223d840318ac1f9c8b6bf3aef24c3ff67683479/gh/sleeyax.advent-of-code-2023/log.txt
• https://crater-reports.s3.amazonaws.com/pr-127907/try%236223d840318ac1f9c8b6bf3aef24c3ff67683479/gh/tljubej.tljl/log.txt

These have

[INFO] [stderr] error: could not compile `advent_of_code` (bin "04" test); 1 warning emitted

which is very strange -- it seems like rustc says "could not compile" but there has been no error emitted.

That's the same as what the first of the above has.

[RalfJung]

These errors all seem to be spurious, for the errors mentioned in the OP none of them reproduced in a rerun.

[Mark-Simulacrum]

Interestingly, "could not compile" messaging actually comes from Cargo, not rustc (https://github.com/rust-lang/cargo/blob/3d2cf56d40041b4974bae74e7ad6b293f4f5daad/src/cargo/core/compiler/mod.rs#L426). That same code has a debug assert trying to cover this case. My suspicion is that we're seeing something in exec fail in an unexpected way -- but I'm not sure what it could be. It seems like this is probably a Cargo bug though.