Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

std: Switch from libbacktrace to gimli #73441

Merged
merged 1 commit into from
Jul 18, 2020

Conversation

alexcrichton
Copy link
Member

This commit is a proof-of-concept for switching the standard library's
backtrace symbolication mechanism on most platforms from libbacktrace to
gimli. The standard library's support for RUST_BACKTRACE=1 requires
in-process parsing of object files and DWARF debug information to
interpret it and print the filename/line number of stack frames as part
of a backtrace.

Historically this support in the standard library has come from a
library called "libbacktrace". The libbacktrace library seems to have
been extracted from gcc at some point and is written in C. We've had a
lot of issues with libbacktrace over time, unfortunately, though. The
library does not appear to be actively maintained since we've had
patches sit for months-to-years without comments. We have discovered a
good number of soundness issues with the library itself, both when
parsing valid DWARF as well as invalid DWARF. This is enough of an issue
that the libs team has previously decided that we cannot feed untrusted
inputs to libbacktrace. This also doesn't take into account the
portability of libbacktrace which has been difficult to manage and
maintain over time. While possible there are lots of exceptions and it's
the main C dependency of the standard library right now.

For years it's been the desire to switch over to a Rust-based solution
for symbolicating backtraces. It's been assumed that we'll be using the
Gimli family of crates for this purpose, which are targeted at safely
and efficiently parsing DWARF debug information. I've been working
recently to shore up the Gimli support in the backtrace crate. As of a
few weeks ago the backtrace crate, by default, uses Gimli when loaded
from crates.io. This transition has gone well enough that I figured it
was time to start talking seriously about this change to the standard
library.

This commit is a preview of what's probably the best way to integrate
the backtrace crate into the standard library with the Gimli feature
turned on. While today it's used as a crates.io dependency, this commit
switches the backtrace crate to a submodule of this repository which
will need to be updated manually. This is not done lightly, but is
thought to be the best solution. The primary reason for this is that the
backtrace crate needs to do some pretty nontrivial filesystem
interactions to locate debug information. Working without std::fs is
not an option, and while it might be possible to do some sort of
trait-based solution when prototyped it was found to be too unergonomic.
Using a submodule allows the backtrace crate to build as a submodule
of the std crate itself, enabling it to use std::fs and such.

Otherwise this adds new dependencies to the standard library. This step
requires extra attention because this means that these crates are now
going to be included with all Rust programs by default. It's important
to note, however, that we're already shipping libbacktrace with all Rust
programs by default and it has a bunch of C code implementing all of
this internally anyway, so we're basically already switching
already-shipping functionality to Rust from C.

  • object - this crate is used to parse object file headers and
    contents. Very low-level support is used from this crate and almost
    all of it is disabled. Largely we're just using struct definitions as
    well as convenience methods internally to read bytes and such.

  • addr2line - this is the main meat of the implementation for
    symbolication. This crate depends on gimli for DWARF parsing and
    then provides interfaces needed by the backtrace crate to turn an
    address into a filename / line number. This crate is actually pretty
    small (fits in a single file almost!) and mirrors most of what
    dwarf.c does for libbacktrace.

  • miniz_oxide - the libbacktrace crate transparently handles
    compressed debug information which is compressed with zlib. This crate
    is used to decompress compressed debug sections.

  • gimli - not actually used directly, but a dependency of addr2line.

  • adler32- not used directly either, but a dependency of
    miniz_oxide.

The goal of this change is to improve the safety of backtrace
symbolication in the standard library, especially in the face of
possibly malformed DWARF debug information. Even to this day we're still
seeing segfaults in libbacktrace which could possibly become security
vulnerabilities. This change should almost entirely eliminate this
possibility whilc also paving the way forward to adding more features
like split debug information.

Some references for those interested are:

Switching to Rust will not make us immune to all of these issues. The
crashes are expected to go away, but correctness and performance may
still have bugs arise. The gimli and backtrace crates, however, are
actively maintained unlike libbacktrace, so this should enable us to at
least efficiently apply fixes as situations come up.


I want to note that my purpose for creating a PR here is to start a conversation about this. I think that all the various pieces are in place that this is compelling enough that I think this transition should be talked about seriously. There are a number of items which still need to be addressed before actually merging this PR, however:

Most of the merging/publishing I'm not actively pushing on right now. It's a bit wonky for crates to support libstd so I'm holding off on pulling the trigger everywhere until there's a bit more discussion about how to go through with this. Namely rust-lang/backtrace-rs#349 I'm going to hold off merging until we decide to go through with the submodule strategy.

In any case this is a pretty major change, so I suspect that the compiler team is likely going to be interested in this. I don't mean to force changes by dumping a bunch of code by any means. Integration of external crates into the standard library is so difficult I wanted to have a proof-of-concept to review while talking about whether to do this at all (hence the PR), but I'm more than happy to follow any processes needed to merge this. I must admit though that I'm not entirely sure myself at this time what the process would be to decide to merge this, so I'm hoping others can help me figure that out!

@rust-highfive
Copy link
Collaborator

r? @Mark-Simulacrum

(rust_highfive has picked a reviewer for you, use r? to override)

@rust-highfive
Copy link
Collaborator

⚠️ Warning ⚠️

  • These commits modify submodules.

@rust-highfive rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Jun 17, 2020
@Mark-Simulacrum
Copy link
Member

I think the process to follow here is to file a major change proposal (https://github.com/rust-lang/compiler-team/blob/master/.github/ISSUE_TEMPLATE/major_change.md) on the compiler-team repository, and then we can go from there.

I suspect that the body of that proposal can basically just link to this PR's description.


I would personally be on board with this change, I think I agree with the proposal for how to integrate into std via the submodule -- I don't like it, but I don't think we have a better approach today -- and, more broadly, that we should be looking to migrate std away from libbacktrace.

One thing that I don't see addressed in the PR description is the licensing of these libraries we're adding -- are they all MIT or Apache 2.0, same as the Rust repository, or do we need to do some legwork in that respect?

Also, it would be good to get some sense of what kind of review you think would be appropriate here -- e.g., should we be trying to find some person or persons to try and review at least the unsafe code we're bringing in (if any) or perhaps more broadly the various libraries we're adding in?

I presume that backtrace isn't going to be relying on any unstable code, right, just using stabilized APIs in std? Otherwise a submodule may be a bit painful with bootstrap bumping and such (though nothing we can't deal with if necessary).

@alexcrichton
Copy link
Member Author

Excellent point on licenses! The license are:

  • gimli - MIT/Apache-2.0
  • addr2line - MIT/Apache-2.0
  • object - MIT/Apache-2.0
  • miniz_oxide - MIT
  • adler32 - Zlib

The adler32 crate may need some extra attention.

For review, that's also a very good question. Here's the various big "chunks"

  • miniz_oxide / adler32 - these have been heavily used by flate2 which is a heavily used dependency (and already used by rustc itself). I don't think these need too much extra review
  • object / gimli / addr2line - these have been independently reviewed and developed as part of the gimli ecosystem. They do have some unsafe code but have also seen lots of usage outside of the backtrace crate. Additionally they've gotten a lot of usage on crates.io with the backtrace crate using them by default. I think it's ok to largely skip this review.
  • backtrace with gimli-symbolize feature - this has only been pretty lightly reviewed. I would appreciate a review of the various files here (src/symbolize/gimli*).
  • Integration points of the backtrace crate - this is the main area to review here. This would encompass the changes listed here in this PR as well as Prepare for this crate to go into libstd backtrace-rs#349.

Also, to make sure that everything's within context, we added libbacktrace to the standard library with practically zero review. The backtrace-sys crate never really correctly reported its license and the code in libbacktrace wasn't really reviewed other than over time we found bugs in it. I, for example, was surprised just last week that it embedded a zlib inflate implementation. I don't say this to dissuade review of this PR, but I wanted to at least provide the context of how we got here in the first place.

@alexcrichton alexcrichton marked this pull request as draft June 17, 2020 16:29
@rust-highfive
Copy link
Collaborator

The job x86_64-gnu-llvm-8 of your PR failed (pretty log, raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.
##[section]Starting: Linux x86_64-gnu-llvm-8
##[section]Starting: Initialize job
Agent name: 'Azure Pipelines 3'
Agent machine name: 'fv-az578'
Current agent version: '2.170.1'
##[group]Operating System
16.04.6
LTS
LTS
##[endgroup]
##[group]Virtual Environment
Environment: ubuntu-16.04
Version: 20200604.1
Included Software: https://github.com/actions/virtual-environments/blob/ubuntu16/20200604.1/images/linux/Ubuntu1604-README.md
##[endgroup]
Agent running as: 'vsts'
Prepare build directory.
Set build variables.
Download all required tasks.
Download all required tasks.
Downloading task: Bash (3.163.3)
Checking job knob settings.
   Knob: AgentToolsDirectory = /opt/hostedtoolcache Source: ${AGENT_TOOLSDIRECTORY} 
   Knob: AgentPerflog = /home/vsts/perflog Source: ${VSTS_AGENT_PERFLOG} 
Start tracking orphan processes.
##[section]Finishing: Initialize job
##[section]Starting: Configure Job Name
==============================================================================
---
========================== Starting Command Output ===========================
[command]/bin/bash --noprofile --norc /home/vsts/work/_temp/34864ab6-6287-46fa-8434-2ee29a237d4f.sh

##[section]Finishing: Disable git automatic line ending conversion
##[section]Starting: Checkout rust-lang/rust@refs/pull/73441/merge to s
Task         : Get sources
Description  : Get sources from a repository. Supports Git, TfsVC, and SVN repositories.
Version      : 1.0.0
Author       : Microsoft
---
##[command]git remote add origin https://github.com/rust-lang/rust
##[command]git config gc.auto 0
##[command]git config --get-all http.https://github.com/rust-lang/rust.extraheader
##[command]git config --get-all http.proxy
##[command]git -c http.extraheader="AUTHORIZATION: basic ***" fetch --force --tags --prune --progress --no-recurse-submodules --depth=2 origin +refs/heads/*:refs/remotes/origin/* +refs/pull/73441/merge:refs/remotes/pull/73441/merge
---
 ---> 38c164126245
Step 5/8 : ENV RUST_CONFIGURE_ARGS       --build=x86_64-unknown-linux-gnu       --llvm-root=/usr/lib/llvm-8       --enable-llvm-link-shared       --set rust.thin-lto-import-instr-limit=10
 ---> Using cache
 ---> afe0ff39f7ee
Step 6/8 : ENV SCRIPT python2.7 ../x.py test --exclude src/tools/tidy &&            python2.7 ../x.py test src/test/mir-opt --pass=build                                   --target=armv5te-unknown-linux-gnueabi &&            python2.7 ../x.py test src/tools/tidy
 ---> 45ef90c713f0
Step 7/8 : ENV NO_DEBUG_ASSERTIONS=1
 ---> Using cache
 ---> d9e5bf90d143
---
####################################################                      72.6%
###################################################################       94.1%
######################################################################## 100.0%
extracting /checkout/obj/build/cache/2020-04-22/rustfmt-nightly-x86_64-unknown-linux-gnu.tar.gz
    Updating git repository `https://github.com/alexcrichton/addr2line`
    Updating git repository `https://github.com/gimli-rs/gimli`
    Updating git repository `https://github.com/Frommi/miniz_oxide`
---
   Compiling rustc-demangle v0.1.16
   Compiling panic_abort v0.0.0 (/checkout/src/libpanic_abort)
   Compiling rustc-std-workspace-alloc v1.99.0 (/checkout/src/tools/rustc-std-workspace-alloc)
   Compiling panic_unwind v0.0.0 (/checkout/src/libpanic_unwind)
   Compiling gimli v0.21.0 (https://github.com/gimli-rs/gimli#ea707d78)
   Compiling miniz_oxide v0.3.7 (https://github.com/Frommi/miniz_oxide#7f5aedd7)
   Compiling object v0.20.0
   Compiling addr2line v0.12.0 (https://github.com/alexcrichton/addr2line?branch=dep-of-std#94e604b2)
   Compiling term v0.0.0 (/checkout/src/libterm)
   Compiling proc_macro v0.0.0 (/checkout/src/libproc_macro)
   Compiling unicode-width v0.1.6
   Compiling getopts v0.2.21
---
   Compiling block-padding v0.1.5
   Compiling crossbeam-utils v0.7.2
   Compiling getopts v0.2.21
   Compiling rustc_lexer v0.1.0 (/checkout/src/librustc_lexer)
   Compiling miniz_oxide v0.3.7 (https://github.com/Frommi/miniz_oxide#7f5aedd7)
   Compiling petgraph v0.4.13
   Compiling rustc_version v0.2.3
   Compiling crossbeam-queue v0.1.2
   Compiling unicode-security v0.0.3
---
   Compiling rustc_parse_format v0.0.0 (/checkout/src/librustc_parse_format)
   Compiling chalk-rust-ir v0.10.0
   Compiling rustc_ast_pretty v0.0.0 (/checkout/src/librustc_ast_pretty)
   Compiling rustc_hir v0.0.0 (/checkout/src/librustc_hir)
   Compiling rustc_query_system v0.0.0 (/checkout/src/librustc_query_system)
   Compiling chalk-solve v0.10.0
   Compiling rustc_hir_pretty v0.0.0 (/checkout/src/librustc_hir_pretty)
   Compiling rustc_parse v0.0.0 (/checkout/src/librustc_parse)
   Compiling rustc_ast_lowering v0.0.0 (/checkout/src/librustc_ast_lowering)
---
   Compiling rustc-demangle v0.1.16
   Compiling panic_abort v0.0.0 (/checkout/src/libpanic_abort)
   Compiling rustc-std-workspace-alloc v1.99.0 (/checkout/src/tools/rustc-std-workspace-alloc)
   Compiling panic_unwind v0.0.0 (/checkout/src/libpanic_unwind)
   Compiling gimli v0.21.0 (https://github.com/gimli-rs/gimli#ea707d78)
   Compiling miniz_oxide v0.3.7 (https://github.com/Frommi/miniz_oxide#7f5aedd7)
   Compiling object v0.20.0
   Compiling addr2line v0.12.0 (https://github.com/alexcrichton/addr2line?branch=dep-of-std#94e604b2)
   Compiling proc_macro v0.0.0 (/checkout/src/libproc_macro)
   Compiling term v0.0.0 (/checkout/src/libterm)
   Compiling unicode-width v0.1.6
   Compiling getopts v0.2.21
---
   Compiling block-padding v0.1.5
   Compiling crossbeam-utils v0.7.2
   Compiling getopts v0.2.21
   Compiling rustc_lexer v0.1.0 (/checkout/src/librustc_lexer)
   Compiling miniz_oxide v0.3.7 (https://github.com/Frommi/miniz_oxide#7f5aedd7)
   Compiling petgraph v0.4.13
   Compiling rustc_version v0.2.3
   Compiling crossbeam-queue v0.1.2
   Compiling unicode-security v0.0.3
---
   Compiling rustc_parse_format v0.0.0 (/checkout/src/librustc_parse_format)
   Compiling chalk-rust-ir v0.10.0
   Compiling rustc_ast_pretty v0.0.0 (/checkout/src/librustc_ast_pretty)
   Compiling rustc_hir v0.0.0 (/checkout/src/librustc_hir)
   Compiling rustc_query_system v0.0.0 (/checkout/src/librustc_query_system)
   Compiling chalk-solve v0.10.0
   Compiling rustc_hir_pretty v0.0.0 (/checkout/src/librustc_hir_pretty)
   Compiling rustc_parse v0.0.0 (/checkout/src/librustc_parse)
   Compiling rustc_ast_lowering v0.0.0 (/checkout/src/librustc_ast_lowering)
---
.................................................................................................... 1900/10319
.................................................................................................... 2000/10319
...............i..i................................................................................. 2100/10319
.................................................................................................... 2200/10319
.....iiiii.......................................................................................... 2300/10319
.................................................................................................... 2500/10319
.................................................................................................... 2600/10319
.................................................................................................... 2700/10319
.................................................................................................... 2800/10319
---
.................................................................................................... 6000/10319
.......ii.....................................i..................................................... 6100/10319
.................................................................................................... 6200/10319
.................................................................................................... 6300/10319
......................................................................ii...i..ii...........i........ 6400/10319
.................................................................................................... 6600/10319
.................................................................................................... 6700/10319
.................................................................................................... 6800/10319
.................................................................................................... 6800/10319
....i..ii........................................................................................... 6900/10319
.................................................................................................... 7100/10319
...........................................................i........................................ 7200/10319
.................................................................................................... 7300/10319
.................................................................................................... 7400/10319
---
.................................................................................................... 8200/10319
.................................................................................................... 8300/10319
.................................................................................................... 8400/10319
.i.................................................................................................. 8500/10319
.......................................................iiiiii.iiiiii.i.............................. 8600/10319
............i....................................................................................... 8800/10319
.................................................................................................... 8900/10319
.................................................................................................... 9000/10319
.................................................................................................... 9100/10319
---
Suite("src/test/codegen") not skipped for "bootstrap::test::Codegen" -- not in ["src/tools/tidy"]
Check compiletest suite=codegen mode=codegen (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)

running 193 tests
iiii......i..............ii.i..........i......................i...........i..i................i....i 100/193
.............i.i.i...iii..iiiiiiiiiiiiiiiii.......................iii................ii......

 finished in 6.998
Suite("src/test/codegen-units") not skipped for "bootstrap::test::CodegenUnits" -- not in ["src/tools/tidy"]
Check compiletest suite=codegen-units mode=codegen-units (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
---
Suite("src/test/assembly") not skipped for "bootstrap::test::Assembly" -- not in ["src/tools/tidy"]
Check compiletest suite=assembly mode=assembly (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)

running 21 tests
iiiiiiiiiiiiiiiiiiiii

 finished in 0.165
Suite("src/test/incremental") not skipped for "bootstrap::test::Incremental" -- not in ["src/tools/tidy"]
Check compiletest suite=incremental mode=incremental (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
---
Suite("src/test/debuginfo") not skipped for "bootstrap::test::Debuginfo" -- not in ["src/tools/tidy"]
Check compiletest suite=debuginfo mode=debuginfo (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)

running 116 tests
iiiii..i.....i..i...i..i.i.i..i..i..ii....i.i....ii..........iiii.........i.....i...i.......ii.i.ii. 100/116
....iiii.....ii.

 finished in 15.879
Suite("src/test/ui-fulldeps") not skipped for "bootstrap::test::UiFullDeps" -- not in ["src/tools/tidy"]
Uplifting stage1 rustc (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
---

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

 finished in 0.931
Set({"/checkout/src/librustc_query_system"}) not skipped for "bootstrap::test::CrateLibrustc" -- not in ["src/tools/tidy"]
Testing rustc_query_system stage1 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
   Compiling rustc_query_system v0.0.0 (/checkout/src/librustc_query_system)
     Running build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/rustc_query_system-101a02b96cd72ebe

running 0 tests

---
   Compiling itoa v0.4.4
   Compiling getrandom v0.1.14
   Compiling new_debug_unreachable v1.0.3
   Compiling smallvec v1.4.0
   Compiling gimli v0.21.0 (https://github.com/gimli-rs/gimli#ea707d78)
   Compiling ucd-trie v0.1.1
   Compiling fnv v1.0.6
   Compiling adler32 v1.1.0
   Compiling matches v0.1.8
---
   Compiling libnghttp2-sys v0.1.2
   Compiling curl-sys v0.4.25
   Compiling unicode-normalization v0.1.12
   Compiling pest v2.1.0
   Compiling miniz_oxide v0.3.7 (https://github.com/Frommi/miniz_oxide#7f5aedd7)
   Compiling futf v0.1.4
   Compiling textwrap v0.11.0
   Compiling getopts v0.2.21
   Compiling getopts v0.2.21
   Compiling addr2line v0.12.0 (https://github.com/alexcrichton/addr2line?branch=dep-of-std#94e604b2)
   Compiling heck v0.3.0
   Compiling humantime v1.3.0
   Compiling foreign-types v0.3.2
   Compiling itertools v0.8.0
---
    Checking rustc-demangle v0.1.16
    Checking panic_abort v0.0.0 (/checkout/src/libpanic_abort)
    Checking rustc-std-workspace-alloc v1.99.0 (/checkout/src/tools/rustc-std-workspace-alloc)
    Checking panic_unwind v0.0.0 (/checkout/src/libpanic_unwind)
    Checking gimli v0.21.0 (https://github.com/gimli-rs/gimli#ea707d78)
    Checking object v0.20.0
    Checking miniz_oxide v0.3.7 (https://github.com/Frommi/miniz_oxide#7f5aedd7)
    Checking addr2line v0.12.0 (https://github.com/alexcrichton/addr2line?branch=dep-of-std#94e604b2)
    Finished release [optimized] target(s) in 19.60s
   Compiling std v0.0.0 (/checkout/src/libstd)
 Documenting proc_macro v0.0.0 (/checkout/src/libproc_macro)
    Finished release [optimized] target(s) in 7.46s
---
Set({"/checkout/src/librustc_parse_format"}) not skipped for "bootstrap::doc::Rustc" -- not in ["src/tools/tidy"]
Set({"/checkout/src/librustc_passes"}) not skipped for "bootstrap::doc::Rustc" -- not in ["src/tools/tidy"]
Set({"/checkout/src/librustc_plugin_impl"}) not skipped for "bootstrap::doc::Rustc" -- not in ["src/tools/tidy"]
Set({"/checkout/src/librustc_privacy"}) not skipped for "bootstrap::doc::Rustc" -- not in ["src/tools/tidy"]
Set({"/checkout/src/librustc_query_system"}) not skipped for "bootstrap::doc::Rustc" -- not in ["src/tools/tidy"]
Set({"/checkout/src/librustc_save_analysis"}) not skipped for "bootstrap::doc::Rustc" -- not in ["src/tools/tidy"]
Set({"/checkout/src/librustc_serialize"}) not skipped for "bootstrap::doc::Rustc" -- not in ["src/tools/tidy"]
Set({"/checkout/src/librustc_session"}) not skipped for "bootstrap::doc::Rustc" -- not in ["src/tools/tidy"]
Set({"/checkout/src/librustc_span"}) not skipped for "bootstrap::doc::Rustc" -- not in ["src/tools/tidy"]
---
Rustbook (x86_64-unknown-linux-gnu) - edition-guide
Building stage0 tool linkchecker (x86_64-unknown-linux-gnu)
   Compiling linkchecker v0.1.0 (/checkout/src/tools/linkchecker)
    Finished release [optimized] target(s) in 1.64s
std/primitive.u64.html:928: broken link fragment `#tymethod.from_u64` pointing to `std/primitive.u64.html`
std/primitive.u32.html:943: broken link fragment `#tymethod.from_u64` pointing to `std/primitive.u32.html`
std/primitive.usize.html:941: broken link fragment `#tymethod.from_u64` pointing to `std/primitive.usize.html`
thread 'main' panicked at 'found some broken links', src/tools/linkchecker/main.rs:43:9


command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-tools-bin/linkchecker" "/checkout/obj/build/x86_64-unknown-linux-gnu/doc"
expected success, got: exit code: 101
---
  local time: Wed Jun 17 16:35:48 UTC 2020
  network time: Wed, 17 Jun 2020 16:35:48 GMT
== end clock drift check ==

##[error]Bash exited with code '1'.
##[section]Finishing: Run build
##[section]Starting: Checkout rust-lang/rust@refs/pull/73441/merge to s
Task         : Get sources
Description  : Get sources from a repository. Supports Git, TfsVC, and SVN repositories.
Version      : 1.0.0
Author       : Microsoft
Author       : Microsoft
Help         : [More Information](https://go.microsoft.com/fwlink/?LinkId=798199)
==============================================================================
Cleaning any cached credential from repository: rust-lang/rust (GitHub)
##[section]Finishing: Checkout rust-lang/rust@refs/pull/73441/merge to s
Cleaning up task key
Start cleaning up orphan processes.
Terminate orphan process: pid (3605) (python)
##[section]Finishing: Finalize Job

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @rust-lang/infra. (Feature Requests)

@cuviper
Copy link
Member

cuviper commented Jun 17, 2020

std/primitive.u64.html:928: broken link fragment `#tymethod.from_u64` pointing to `std/primitive.u64.html`
std/primitive.u32.html:943: broken link fragment `#tymethod.from_u64` pointing to `std/primitive.u32.html`
std/primitive.usize.html:941: broken link fragment `#tymethod.from_u64` pointing to `std/primitive.usize.html`

I suppose that's gimli's ReaderOffset leaking?

benesch added a commit to benesch/materialize that referenced this pull request Jun 17, 2020
Switch over to the new Gimli-based backtrace symbolicator. Gimli is a
pure-Rust reimplementation of libbacktrace that will soon be the default
backtrace implementation in the Rust standard library [0]. Switching to
it now means we can help shake out problems, like [1], before it ships
in the standard library, where fixing problems will require a full six
week release cycle.

This switch has the side effect of fixing backtraces on macOS, which
seem to be broken at the moment because the backtrace crate is picking
libbacktrace over coresymbolication. There isn't a good way of fixing
that because Cargo doesn't support per-platform features [2]; using
neither libbacktrace or coresymbolicator is a kind of stupid but
effective way of sidestepping the problem.

[0]: rust-lang/rust#73441
[1]: rust-lang/backtrace-rs#342
[2]: rust-lang/cargo#1197
@alexcrichton
Copy link
Member Author

@cuviper hm yes indeed, you wouldn't happen to know off-hand how to fix that would you?

addr2line = { version = "0.12.0", optional = true, default-features = false }
rustc-demangle = { version = "0.1.4", optional = true }
miniz_oxide = { version = "0.3.7", optional = true }
[dependencies.object]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
[dependencies.object]
[dependencies.object]

src/tools/tidy/src/deps.rs Outdated Show resolved Hide resolved
@cuviper
Copy link
Member

cuviper commented Jun 17, 2020

you wouldn't happen to know off-hand how to fix that would you?

Sorry, no -- I guess you should consult some rustdoc folks?

@alexcrichton
Copy link
Member Author

Ok I made some rustc/rustdoc changes to fix the issue about suggestions and broken links pointing to internal crates libstd depends on.

// Don't issue suggestions for unstable traits since they're
// unlikely to be implementable anyway
.filter(|info| match self.tcx.lookup_stability(info.def_id) {
Some(attr) => attr.level.is_stable(),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could this keep suggesting unstable traits on nightly?

@Mark-Simulacrum
Copy link
Member

I guess we also want some legal attention on miniz_oxide as well? We'll want both MIT and Apache 2.0 I think for standard library deps -- I don't think it matters as much for the compiler?

I've posted a topic on the licensing stream on Zulip as well about this: https://rust-lang.zulipchat.com/#narrow/stream/231349-t-core.2Flicensing/topic/backtrace.20crate.20inclusion.20in.20std

Also, to be up front with some of the expectations on my end as to when I can get around to reviewing, I can't be sure, but I'll try to allocate some time to do so for the things you noted as needing review. I'll make a note on the MCP as well.

  • backtrace with gimli-symbolize feature - this has only been pretty lightly reviewed. I would appreciate a review of the various files here (src/symbolize/gimli*).
  • Integration points of the backtrace crate - this is the main area to review here. This would encompass the changes listed here in this PR as well as Prepare for this crate to go into libstd backtrace-rs#349.

@Mark-Simulacrum
Copy link
Member

I'm thinking that it would be good to perhaps land parts of this separately as well -- say, the unstable trait changes etc I don't know a lot about so probably not the best reviewer -- and then we can follow up with a PR that mostly just adds in the submodule and doesn't have ~any compiler changes.

@alexcrichton
Copy link
Member Author

Thanks for following up on the license bits! I'm not really sure how to go about that and am happy to follow whatever recommendation is necessary.

For splitting off changes, I don't mind doing so. I'm not sure if it's too beneficial though. This is a large-ish PR since it changes how libstd uses the backtrace crate (submodule rather than external crate) so it won't really get all that much smaller. The small diffs in src/librustc_typeck/check/method/suggest.rs and src/librustdoc/clean/inline.rs are the only bits relating to suggestions.

@Mark-Simulacrum
Copy link
Member

For splitting off changes, I don't mind doing so. I'm not sure if it's too beneficial though. This is a large-ish PR since it changes how libstd uses the backtrace crate (submodule rather than external crate) so it won't really get all that much smaller. The small diffs in src/librustc_typeck/check/method/suggest.rs and src/librustdoc/clean/inline.rs are the only bits relating to suggestions.

To be clear this is because I feel pretty comfortable reviewing the std/backtrace bits but pretty much not at all so for the typeck and rustdoc, and splitting them out makes that review easier to make happen naturally I think (rather than trying to cc folks into this PR and identify what they need to review).

@alexcrichton
Copy link
Member Author

Ok no worries! I'll look to splitting that out tomorrow.

@bors
Copy link
Contributor

bors commented Jun 26, 2020

☔ The latest upstream changes (presumably #73746) made this pull request unmergeable. Please resolve the merge conflicts.

alexcrichton added a commit to alexcrichton/rust that referenced this pull request Jun 26, 2020
Currently dependency crates of the standard library can sometimes leak
into error messages such as when traits to import are suggested.
Additionally they can leak into documentation such as in the list of
"all traits implemented by `u32`". The dependencies of the standard
library, however, are intended to be private.

The dependencies of the standard library can't actually be stabl-y
imported nor is the documentation that relevant since you can't import
them on stable either. This commit updates both the compiler and rustdoc
to ignore unstable traits in these two scenarios.

Specifically the suggestion for traits to import ignore unstable traits,
and similarly the list of traits implemented by a type excludes unstable
traits.

This commit is extracted from rust-lang#73441 where the addition of some new
dependencies to the standard library was showed to leak into various
error messages and documentation. The intention here is to go ahead and
land these changes ahead of that since it will likely take some time to
land.
@alexcrichton
Copy link
Member Author

I'll leave the changes here for now to ensure CI stays green, but I've also extracted them out for independent review to #73771

@rust-highfive
Copy link
Collaborator

The job x86_64-gnu-llvm-8 of your PR failed (pretty log, raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.
##[section]Starting: Linux x86_64-gnu-llvm-8
##[section]Starting: Initialize job
Agent name: 'Azure Pipelines 6'
Agent machine name: 'fv-az619'
Current agent version: '2.171.1'
##[group]Operating System
16.04.6
LTS
LTS
##[endgroup]
##[group]Virtual Environment
Environment: ubuntu-16.04
Version: 20200621.1
Included Software: https://github.com/actions/virtual-environments/blob/ubuntu16/20200621.1/images/linux/Ubuntu1604-README.md
##[endgroup]
Agent running as: 'vsts'
Prepare build directory.
Set build variables.
Download all required tasks.
Download all required tasks.
Downloading task: Bash (3.171.1)
Checking job knob settings.
   Knob: AgentToolsDirectory = /opt/hostedtoolcache Source: ${AGENT_TOOLSDIRECTORY} 
   Knob: AgentPerflog = /home/vsts/perflog Source: ${VSTS_AGENT_PERFLOG} 
Start tracking orphan processes.
##[section]Finishing: Initialize job
##[section]Starting: Configure Job Name
==============================================================================
---
========================== Starting Command Output ===========================
[command]/bin/bash --noprofile --norc /home/vsts/work/_temp/c1ca4711-4bd9-4f60-9f3a-57325df27e75.sh

##[section]Finishing: Disable git automatic line ending conversion
##[section]Starting: Checkout rust-lang/rust@refs/pull/73441/merge to s
Task         : Get sources
Description  : Get sources from a repository. Supports Git, TfsVC, and SVN repositories.
Version      : 1.0.0
Author       : Microsoft
---
##[command]git remote add origin https://github.com/rust-lang/rust
##[command]git config gc.auto 0
##[command]git config --get-all http.https://github.com/rust-lang/rust.extraheader
##[command]git config --get-all http.proxy
##[command]git -c http.extraheader="AUTHORIZATION: basic ***" fetch --force --tags --prune --progress --no-recurse-submodules --depth=2 origin +refs/heads/*:refs/remotes/origin/* +refs/pull/73441/merge:refs/remotes/pull/73441/merge
---
 ---> 31fea614d2f3
Step 5/8 : ENV RUST_CONFIGURE_ARGS       --build=x86_64-unknown-linux-gnu       --llvm-root=/usr/lib/llvm-8       --enable-llvm-link-shared       --set rust.thin-lto-import-instr-limit=10
 ---> Using cache
 ---> 4195cadf126d
Step 6/8 : ENV SCRIPT python2.7 ../x.py test --exclude src/tools/tidy &&            python2.7 ../x.py test src/test/mir-opt --pass=build                                   --target=armv5te-unknown-linux-gnueabi &&            python2.7 ../x.py test src/tools/tidy
 ---> 4e90f6b48f05
Step 7/8 : ENV NO_DEBUG_ASSERTIONS=1
 ---> Using cache
 ---> dfa0a356d899
---

############################################################              84.7%
######################################################################## 100.0%
extracting /checkout/obj/build/cache/2020-04-22/rustfmt-nightly-x86_64-unknown-linux-gnu.tar.gz
    Updating git repository `https://github.com/alexcrichton/addr2line`
    Updating git repository `https://github.com/gimli-rs/gimli`
    Updating git repository `https://github.com/Frommi/miniz_oxide`
---
Set({"src/librustc_parse_format"}) not skipped for "bootstrap::test::CrateLibrustc" -- not in ["src/tools/tidy"]
Set({"src/librustc_passes"}) not skipped for "bootstrap::test::CrateLibrustc" -- not in ["src/tools/tidy"]
Set({"src/librustc_plugin_impl"}) not skipped for "bootstrap::test::CrateLibrustc" -- not in ["src/tools/tidy"]
Set({"src/librustc_privacy"}) not skipped for "bootstrap::test::CrateLibrustc" -- not in ["src/tools/tidy"]
Set({"src/librustc_query_system"}) not skipped for "bootstrap::test::CrateLibrustc" -- not in ["src/tools/tidy"]
Set({"src/librustc_save_analysis"}) not skipped for "bootstrap::test::CrateLibrustc" -- not in ["src/tools/tidy"]
Set({"src/librustc_serialize"}) not skipped for "bootstrap::test::CrateLibrustc" -- not in ["src/tools/tidy"]
Set({"src/librustc_session"}) not skipped for "bootstrap::test::CrateLibrustc" -- not in ["src/tools/tidy"]
Set({"src/librustc_span"}) not skipped for "bootstrap::test::CrateLibrustc" -- not in ["src/tools/tidy"]
---
Set({"src/librustc_parse_format"}) not skipped for "bootstrap::doc::Rustc" -- not in ["src/tools/tidy"]
Set({"src/librustc_passes"}) not skipped for "bootstrap::doc::Rustc" -- not in ["src/tools/tidy"]
Set({"src/librustc_plugin_impl"}) not skipped for "bootstrap::doc::Rustc" -- not in ["src/tools/tidy"]
Set({"src/librustc_privacy"}) not skipped for "bootstrap::doc::Rustc" -- not in ["src/tools/tidy"]
Set({"src/librustc_query_system"}) not skipped for "bootstrap::doc::Rustc" -- not in ["src/tools/tidy"]
Set({"src/librustc_save_analysis"}) not skipped for "bootstrap::doc::Rustc" -- not in ["src/tools/tidy"]
Set({"src/librustc_serialize"}) not skipped for "bootstrap::doc::Rustc" -- not in ["src/tools/tidy"]
Set({"src/librustc_session"}) not skipped for "bootstrap::doc::Rustc" -- not in ["src/tools/tidy"]
Set({"src/librustc_span"}) not skipped for "bootstrap::doc::Rustc" -- not in ["src/tools/tidy"]
---
   Compiling rustc-demangle v0.1.16
   Compiling panic_abort v0.0.0 (/checkout/src/libpanic_abort)
   Compiling rustc-std-workspace-alloc v1.99.0 (/checkout/src/tools/rustc-std-workspace-alloc)
   Compiling panic_unwind v0.0.0 (/checkout/src/libpanic_unwind)
   Compiling gimli v0.21.0 (https://github.com/gimli-rs/gimli#ea707d78)
   Compiling miniz_oxide v0.3.7 (https://github.com/Frommi/miniz_oxide#7f5aedd7)
   Compiling object v0.20.0
   Compiling addr2line v0.12.0 (https://github.com/alexcrichton/addr2line?branch=dep-of-std#94e604b2)
   Compiling term v0.0.0 (/checkout/src/libterm)
   Compiling proc_macro v0.0.0 (/checkout/src/libproc_macro)
   Compiling unicode-width v0.1.6
   Compiling getopts v0.2.21
---
   Compiling block-padding v0.1.5
   Compiling getopts v0.2.21
   Compiling rustc_lexer v0.1.0 (/checkout/src/librustc_lexer)
   Compiling crossbeam-utils v0.7.2
   Compiling miniz_oxide v0.3.7 (https://github.com/Frommi/miniz_oxide#7f5aedd7)
   Compiling petgraph v0.5.1
   Compiling rustc_version v0.2.3
   Compiling crossbeam-queue v0.1.2
   Compiling unicode-security v0.0.5
---
   Compiling rustc_parse_format v0.0.0 (/checkout/src/librustc_parse_format)
   Compiling tracing v0.1.15
   Compiling rustc_ast_pretty v0.0.0 (/checkout/src/librustc_ast_pretty)
   Compiling rustc_hir v0.0.0 (/checkout/src/librustc_hir)
   Compiling rustc_query_system v0.0.0 (/checkout/src/librustc_query_system)
   Compiling chalk-engine v0.14.0
   Compiling rustc_hir_pretty v0.0.0 (/checkout/src/librustc_hir_pretty)
   Compiling rustc_parse v0.0.0 (/checkout/src/librustc_parse)
   Compiling rustc_ast_lowering v0.0.0 (/checkout/src/librustc_ast_lowering)
---
   Compiling rustc-demangle v0.1.16
   Compiling panic_abort v0.0.0 (/checkout/src/libpanic_abort)
   Compiling rustc-std-workspace-alloc v1.99.0 (/checkout/src/tools/rustc-std-workspace-alloc)
   Compiling panic_unwind v0.0.0 (/checkout/src/libpanic_unwind)
   Compiling gimli v0.21.0 (https://github.com/gimli-rs/gimli#ea707d78)
   Compiling object v0.20.0
   Compiling miniz_oxide v0.3.7 (https://github.com/Frommi/miniz_oxide#7f5aedd7)
   Compiling addr2line v0.12.0 (https://github.com/alexcrichton/addr2line?branch=dep-of-std#94e604b2)
   Compiling proc_macro v0.0.0 (/checkout/src/libproc_macro)
   Compiling term v0.0.0 (/checkout/src/libterm)
   Compiling unicode-width v0.1.6
   Compiling getopts v0.2.21
---
   Compiling crossbeam-utils v0.7.2
   Compiling block-padding v0.1.5
   Compiling getopts v0.2.21
   Compiling rustc_lexer v0.1.0 (/checkout/src/librustc_lexer)
   Compiling miniz_oxide v0.3.7 (https://github.com/Frommi/miniz_oxide#7f5aedd7)
   Compiling petgraph v0.5.1
   Compiling rustc_version v0.2.3
   Compiling crossbeam-queue v0.1.2
   Compiling unicode-security v0.0.5
---
   Compiling rustc_parse_format v0.0.0 (/checkout/src/librustc_parse_format)
   Compiling tracing v0.1.15
   Compiling rustc_ast_pretty v0.0.0 (/checkout/src/librustc_ast_pretty)
   Compiling rustc_hir v0.0.0 (/checkout/src/librustc_hir)
   Compiling rustc_query_system v0.0.0 (/checkout/src/librustc_query_system)
   Compiling chalk-engine v0.14.0
   Compiling rustc_hir_pretty v0.0.0 (/checkout/src/librustc_hir_pretty)
   Compiling rustc_parse v0.0.0 (/checkout/src/librustc_parse)
   Compiling rustc_ast_lowering v0.0.0 (/checkout/src/librustc_ast_lowering)
---
...........................i........................................................................ 1900/10407
.................................................................................................... 2000/10407
......................................................i..i.......................................... 2100/10407
.................................................................................................... 2200/10407
............................................iiiii................................................... 2300/10407
.................................................................................................... 2500/10407
.................................................................................................... 2600/10407
.................................................................................................... 2700/10407
.................................................................................................... 2800/10407
---
...i................................................................................................ 5300/10407
.................................................................................................... 5400/10407
...................................i................................................................ 5500/10407
.............................i...................................................................... 5600/10407
.................................................ii.ii........i...i................................. 5700/10407
..................i................................................................................. 5900/10407
...............i.................................................................................... 6000/10407
.........................................................................ii......................... 6100/10407
............i....................................................................................... 6200/10407
............i....................................................................................... 6200/10407
.................................................................................................... 6300/10407
.................................................................................................... 6400/10407
....................................ii...i..ii...........i.......................................... 6500/10407
.................................................................................................... 6700/10407
.................................................................................................... 6800/10407
.................................................................................................... 6800/10407
.......................................................................i..ii........................ 6900/10407
.................................................................................................... 7100/10407
.................................................................................................... 7200/10407
...........................i........................................................................ 7300/10407
.................................................................................................... 7400/10407
---
.................................................................................................... 8300/10407
.................................................................................................... 8400/10407
............................................................................i....................... 8500/10407
.................................................................................................... 8600/10407
..............................iiiiii..iiiiii.i...................................................... 8700/10407
.................................................................................................... 8900/10407
.................................................................................................... 9000/10407
.................................................................................................... 9100/10407
.................................................................................................... 9200/10407
---
Suite("src/test/codegen") not skipped for "bootstrap::test::Codegen" -- not in ["src/tools/tidy"]
Check compiletest suite=codegen mode=codegen (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)

running 202 tests
iiii......i..i...............ii..i..........i...........i............i...........i..i........i...... 100/202
..i....i.............i.i.i...iii..iiii....................................iii.................ii.... 200/202
test result: ok. 170 passed; 0 failed; 32 ignored; 0 measured; 0 filtered out

 finished in 9.526
Suite("src/test/codegen-units") not skipped for "bootstrap::test::CodegenUnits" -- not in ["src/tools/tidy"]
---
Suite("src/test/assembly") not skipped for "bootstrap::test::Assembly" -- not in ["src/tools/tidy"]
Check compiletest suite=assembly mode=assembly (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)

running 22 tests
iiiiiiiiiiiiiiiiiiiiii

 finished in 0.173
Suite("src/test/incremental") not skipped for "bootstrap::test::Incremental" -- not in ["src/tools/tidy"]
Check compiletest suite=incremental mode=incremental (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
---
Suite("src/test/debuginfo") not skipped for "bootstrap::test::Debuginfo" -- not in ["src/tools/tidy"]
Check compiletest suite=debuginfo mode=debuginfo (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)

running 116 tests
iiiii..i.....i..i...i..i.i.i..i..i..ii....i.i....ii..........iiii.........i.....i...i.......ii.i.ii. 100/116
....iiii.....ii.

 finished in 17.456
Suite("src/test/ui-fulldeps") not skipped for "bootstrap::test::UiFullDeps" -- not in ["src/tools/tidy"]
Uplifting stage1 rustc (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
---

   Doc-tests core

running 2586 tests
......iiiii......................................................................................... 100/2586
.................................................................................................ii. 200/2586
.......................................i............................................................ 400/2586
...............................................................................................i..i. 500/2586
...............................................................................................i..i. 500/2586
.................iiii............................................................................... 600/2586
.................................................................................................... 800/2586
.................................................................................................... 900/2586
.................................................................................................... 1000/2586
.................................................................................................... 1100/2586
---
............
test result: ok. 12 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

   Doc-tests std
error[E0659]: `std` is ambiguous (name vs any other name during import resolution)
   --> /checkout/src/libstd/../backtrace/src/symbolize/gimli.rs:344:13
    |
344 |         use std::ffi::{OsStr, CStr};
    |
    = note: `std` could refer to a crate passed with `--extern`
    = help: use `::std` to refer to this crate unambiguously
note: `std` could also refer to the module defined here
note: `std` could also refer to the module defined here
   --> /checkout/src/libstd/../backtrace/src/symbolize/gimli.rs:25:1
25  | / mod std {
26  | |     pub use crate::*;
27  | | }
    | |_^
    | |_^
    = help: use `self::std` to refer to this module unambiguously

error[E0659]: `std` is ambiguous (name vs any other name during import resolution)
  --> /checkout/src/libstd/../backtrace/src/symbolize/gimli.rs:19:5
19 | use std::ffi::OsString;
   |     ^^^ ambiguous name
   |
   = note: `std` could refer to a crate passed with `--extern`
   = note: `std` could refer to a crate passed with `--extern`
   = help: use `::std` to refer to this crate unambiguously
note: `std` could also refer to the module defined here
  --> /checkout/src/libstd/../backtrace/src/symbolize/gimli.rs:25:1
25 | / mod std {
26 | |     pub use crate::*;
27 | | }
   | |_^
   | |_^
   = help: use `self::std` to refer to this module unambiguously

error[E0659]: `std` is ambiguous (name vs any other name during import resolution)
  --> /checkout/src/libstd/../backtrace/src/symbolize/gimli.rs:20:5
20 | use std::fs::File;
   |     ^^^ ambiguous name
   |
   = note: `std` could refer to a crate passed with `--extern`
   = note: `std` could refer to a crate passed with `--extern`
   = help: use `::std` to refer to this crate unambiguously
note: `std` could also refer to the module defined here
  --> /checkout/src/libstd/../backtrace/src/symbolize/gimli.rs:25:1
25 | / mod std {
26 | |     pub use crate::*;
27 | | }
   | |_^
   | |_^
   = help: use `self::std` to refer to this module unambiguously

error[E0659]: `std` is ambiguous (name vs any other name during import resolution)
  --> /checkout/src/libstd/../backtrace/src/symbolize/gimli.rs:21:5
21 | use std::path::Path;
   |     ^^^ ambiguous name
   |
   = note: `std` could refer to a crate passed with `--extern`
   = note: `std` could refer to a crate passed with `--extern`
   = help: use `::std` to refer to this crate unambiguously
note: `std` could also refer to the module defined here
  --> /checkout/src/libstd/../backtrace/src/symbolize/gimli.rs:25:1
25 | / mod std {
26 | |     pub use crate::*;
27 | | }
   | |_^
   | |_^
   = help: use `self::std` to refer to this module unambiguously

error[E0659]: `std` is ambiguous (name vs any other name during import resolution)
  --> /checkout/src/libstd/../backtrace/src/symbolize/gimli.rs:22:5
22 | use std::prelude::v1::*;
   |     ^^^ ambiguous name
   |
   = note: `std` could refer to a crate passed with `--extern`
   = note: `std` could refer to a crate passed with `--extern`
   = help: use `::std` to refer to this crate unambiguously
note: `std` could also refer to the module defined here
  --> /checkout/src/libstd/../backtrace/src/symbolize/gimli.rs:25:1
25 | / mod std {
26 | |     pub use crate::*;
27 | | }
   | |_^
   | |_^
   = help: use `self::std` to refer to this module unambiguously

error[E0659]: `std` is ambiguous (name vs any other name during import resolution)
   --> /checkout/src/libstd/../backtrace/src/symbolize/gimli.rs:343:13
343 |         use std::os::unix::prelude::*;
    |             ^^^ ambiguous name
    |
    = note: `std` could refer to a crate passed with `--extern`
    = note: `std` could refer to a crate passed with `--extern`
    = help: use `::std` to refer to this crate unambiguously
note: `std` could also refer to the module defined here
   --> /checkout/src/libstd/../backtrace/src/symbolize/gimli.rs:25:1
25  | / mod std {
26  | |     pub use crate::*;
27  | | }
    | |_^
---
  local time: Fri Jun 26 18:10:59 UTC 2020
  network time: Fri, 26 Jun 2020 18:11:00 GMT
== end clock drift check ==

##[error]Bash exited with code '1'.
##[section]Finishing: Run build
##[section]Starting: Checkout rust-lang/rust@refs/pull/73441/merge to s
Task         : Get sources
Description  : Get sources from a repository. Supports Git, TfsVC, and SVN repositories.
Version      : 1.0.0
Author       : Microsoft
Author       : Microsoft
Help         : [More Information](https://go.microsoft.com/fwlink/?LinkId=798199)
==============================================================================
Cleaning any cached credential from repository: rust-lang/rust (GitHub)
##[section]Finishing: Checkout rust-lang/rust@refs/pull/73441/merge to s
Cleaning up task key
Start cleaning up orphan processes.
Terminate orphan process: pid (4042) (python)
##[section]Finishing: Finalize Job

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @rust-lang/infra. (Feature Requests)

@bors
Copy link
Contributor

bors commented Jul 4, 2020

☔ The latest upstream changes (presumably #72978) made this pull request unmergeable. Please resolve the merge conflicts.

@alexcrichton
Copy link
Member Author

Thanks for the heads up, it's definitely not intended that this regresses anything. As mentioned a bit on the Zulip thread I think perf data has filled in and the precise regression range for just this PR is at https://perf.rust-lang.org/compare.html?start=7d31ffc1ac9e9ea356e896e63307168a64501b9d&end=1fa54ad9680cc82e7301f8ed4e9b7402dfd6ce0e&stat=instructions%3Au. Still bad, but wanted to make sure it didn't accidentally include other PRs.

@nnethercote
Copy link
Contributor

@alexcrichton: Unfortunately a regression this large is unacceptable. Can you file a PR to revert the change? Thank you.

@nnethercote
Copy link
Contributor

This was also a max-rss regression, with lots of benchmarks in the 5-10% range, including plenty of real-world crates.

@alexcrichton
Copy link
Member Author

Ok I'm looking at the perf data this morning trying to figure out what's best to do next. I want to be clear that I'm very wary of simply reverting this change based solely on the numbers. I don't think that makes sense to do unless there's actually a feasible route to re-landing this, but that does imply further understanding of the numbers as well.

From what I can tell, the regressions fall into these categories:

  • Instruction regressions, from 1% to 45%. I checked every >10% instruction regression and the wall-clock time regresses at most 40ms.
  • Wall-clock regressions, from 1% to 50%. Like above though the regressions are all in the <50ms range.
  • Memory usage regressions up to 15%

Looking into the details of regressions the lion's share (e.g. almost all of it) of time regressions (especially those >5ms) are entirely within the run_linker step, which means the linker is running more slowly. My guess is that this is where most of the instruction regressions are coming from as well. Queries such as metadata_decode_entry are executed more often (as well as metadata_register_crate for the delta of 3 more crates to load), and this is happening because the standard library, which all perf benchmarks depend on, pulls in more dependencies (the gimli deps here).

My understanding of this is that the linker is slower because it has more work to do. #74560 shows that removing debuginfo from these crates can reduce the amount of work the linker has to do, but it's still got to resolve all the symbols and probably gc a good deal of object code out of the final object. Additionally the compiler simply fundamentally has more work to do because there's more crates in play that it needs to load and inspect. There are presumably a number of loops in the compiler that query all crates loaded, and those loops are all a little bit longer now. And finally memory regressions are basically guaranteed because we're mmaping rlibs and metadata into the process, so since we're mmap'ing more things then memory usage is basically guaranteed to go up.

With my understanding of all this I don't really see a bug here. These regressions are all defined by how rustc works with loading external crates and what it means to add more dependencies to a crate. I don't think there's any way to remove these regressions unless we simply never land this feature. Put that way, I would personally prefer to not revert this. There's definitely a lot of noise in the measurements, and if we'd like to get more precise measurements I think it makes sense to post a PR that's a revert and run perf on that (but not for the intention of landing). Perf gains would translate to perf losses if we were to revert all this and re-land it.

In general though if the attitude is "one of those numbers is too negative, back out full-stop" then that, I believe, provides very little means to land this. I do not have the time to investigate how we can rearchitect rustc to lessen the impact of adding dependencies to a crate, and I'm not sure whether that's even possible. It's extremely hard to justify spending time on that when the practical impact of this PR is that crate compilations linking to libstd are about 5ms slower and if you run the linker it's about 30-40ms slower.

This PR is also not simply a random PR out of the weeds, but rather one that's been intended since basically 1.0 to fix a number of longstanding bugs:

  • We don't trust libbacktrace with arbitrary data. We go to great lengths to try to not feed it untrusted data but even still we see segfaults in libbacktrace (even recently) which will inevitably become CVEs in time.
  • Development on libbacktrace has all but halted. We have a good number of bug fixes we've applied over time that never made their way upstream. Moving to a maintained and Rust based solution provides an actual means to fix issues in a timely and non-stressful manner. One example is that MinGW has backtraces with dynamic libraries now.
  • Development is not only intended for bug fixes but also for new features. Perhaps what'll be one of the biggest improvements to compile time in a long time, split debuginfo, is gated on this PR because we're not going to implement split-debuginfo-hunting in libbacktrace.

I understand how the regression numbers in this PR seem alarming. They're very surprising to me because I wasn't expecting them as well. I would like to, however, push back against automatic reactions that we should back this out because of these numbers. I think it's very worthwhile to analyze and understand what happened, but I think it's harmful to require that rustc's perf monotonically improves for the rest of time at the cost of feature development.

@oli-obk
Copy link
Contributor

oli-obk commented Jul 21, 2020

I think It's fine to regress this as long as we track progress on its root cause instead of reverting a symptom like this PR. It seems like a problem to me if merging crates into one crate manually will lead to speed ups and we should look into addressing this where possible. This kind of work will benefit all crates, and with growing numbers of dependencies it seems like the effects could really be felt at some point.

@nnethercote
Copy link
Contributor

nnethercote commented Jul 21, 2020

The comment above lays most of the blame on linking. check builds didn't regress as much as debug and opt builds, but they still regressed a lot. So I wonder if too much blame is being placed on linking.

In general though if the attitude is "one of those numbers is too negative, back out full-stop" then that I believe, provides very little means to land this.

Here's my view.

  • This caused a lot of regressions, and any of the regressions were very large in percentage terms.
  • The regressions were entirely unexpected.
  • The PR landed in a rollup, which means that if any other PRs in that rollup affected performance, those effects will have been masked by this.
  • There are ideas for follow-up changes to reduce the cost, but it's been an unlucky week, perf-wise, with lots of other regressions confusing things, which means teasing out the true effects of follow-ups is even harder than it normally would be.

The comments above suggest that a backout is a draconian response. I disagree. Perhaps this is because I come from Firefox where the landing process is different and backouts are frequent and no big deal. To me, it just says "hey, this didn't work out like we expected, and we should undo things, take a minute, and try again".

Finally, I'm not saying something extreme like "no perf regressions are ever allowed", and I don't think anyone else is saying that. But the above circumstances indicate there is a great deal of surprise and uncertainty surrounding the effects of this PR. I recommend that the PR be backed out, and that the follow-ups be added to this PR so clear perf measurements can be made. If there are still lots of double-digit regressions, then perhaps it's worth a discussion at a compiler meeting.

@mati865
Copy link
Contributor

mati865 commented Jul 21, 2020

The PR landed in a rollup

This was not the case.

@nnethercote
Copy link
Contributor

You are right. Apologies, there were numerous regressions this week and I mixed this one up with some of the others.

Nonetheless, my basic argument still holds, which is that a backout is a reasonable thing to do here.

@nnethercote
Copy link
Contributor

#74613 is the reversion.

@mati865
Copy link
Contributor

mati865 commented Jul 22, 2020

Could we benchmark gimili with LLD to see if linker regression could be related to some inefficiency in BFD?

@alexcrichton
Copy link
Member Author

My point I was trying to explain is that if this is backed out because of performance it has no recourse for landing again without seemingly major architectural changes in rustc. The performance will not change the next time around it's landed. If the purpose is to get reliable performance numbers backing out is not required, as I explained we can benchmark the reversion.

@Mark-Simulacrum
Copy link
Member

I think to some extent we'll probably want to just land it regardless of the performance, but it's easier to have that conversation (amongst compiler team or so) when we're talking about the regression without needing to invert numbers and such.

@nnethercote
Copy link
Contributor

if this is backed out because of performance it has no recourse for landing again without seemingly major architectural changes in rustc

Intentionality is key.

To me, there is a large difference between "whoops this regressed, let's try to improve it afterwards if we can" and "we know in advance that this will regress by x%, we have taken measures to minimize this, and we think it's worth it because of $Y, do we have agreement?". The latter gets agreement up front on a known performance regression amount.

bors added a commit to rust-lang-ci/rust that referenced this pull request Jul 30, 2020
… r=Mark-Simulacrum

std: Switch from libbacktrace to gimli (take 2)

This is the second attempt to land rust-lang#73441 after being reverted in rust-lang#74613. Will be gathering precise perf numbers here in this take.

Closes rust-lang#71060
@cuviper cuviper added this to the 1.47.0 milestone May 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
merged-by-bors This PR was explicitly merged by bors. S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion.
Projects
None yet
Development

Successfully merging this pull request may close these issues.