Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Serialize all foreign SourceFiles into proc-macro crate metadata #73706

Merged
merged 3 commits into from
Jul 1, 2020

Conversation

Aaron1011
Copy link
Member

Normally, we encode a Span that references a foreign SourceFile by
encoding information about the foreign crate. When we decode this
Span, we lookup the foreign crate in order to decode the SourceFile.

However, this approach does not work for proc-macro crates. When we load
a proc-macro crate, we do not deserialzie any of its dependencies (since
a proc-macro crate can only export proc-macros). This means that we
cannot serialize a reference to an upstream crate, since the associated
metadata will not be available when we try to deserialize it.

This commit modifies foreign span handling so that we treat all foreign
SourceFiles as local SourceFiles when serializing a proc-macro.
All SourceFiles will be stored into the metadata of a proc-macro
crate, allowing us to cotinue to deserialize a proc-macro crate without
needing to load any of its dependencies.

Since the number of foreign SourceFiles that we load during a
compilation session may be very large, we only serialize a SourceFile
if we have also serialized a Span which requires it.

@rust-highfive
Copy link
Collaborator

r? @matthewjasper

(rust_highfive has picked a reviewer for you, use r? to override)

@rust-highfive rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Jun 24, 2020
@Aaron1011
Copy link
Member Author

I think this code doesn't need to be changed, but I'm not 100% certain:

// We hash the final, remapped names of all local source files so we
// don't have to include the path prefix remapping commandline args.
// If we included the full mapping in the SVH, we could only have
// reproducible builds by compiling from the same directory. So we just
// hash the result of the mapping instead of the mapping itself.
let mut source_file_names: Vec<_> = self
.source_map
.files()
.iter()
.filter(|source_file| source_file.cnum == LOCAL_CRATE)
.map(|source_file| source_file.name_hash)
.collect();

When we encode a proc-macro crate, we'll end up serializing some foreign SourceFiles, which we won't hash. However, we will still include our dependency hashes in the crate hash, which will cause those SourceFiles to influence the final hash.

@petrochenkov
Copy link
Contributor

r? @petrochenkov

@petrochenkov
Copy link
Contributor

I think this code doesn't need to be changed, but I'm not 100% certain

I'm also not sure, but the logic seems reasonable.

@petrochenkov
Copy link
Contributor

Something like this is definitely preferable to #69976 as and end state.

r=me with the nit addressed

@petrochenkov petrochenkov added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jun 27, 2020
@Aaron1011 Aaron1011 force-pushed the fix/proc-macro-foreign-span branch from 5115132 to 3c1743c Compare June 27, 2020 15:36
@Aaron1011
Copy link
Member Author

I've removed the call to ensure.

@bors r=petrochenkov

@bors
Copy link
Contributor

bors commented Jun 27, 2020

📌 Commit 3c1743cb40da105a120133fb3d041c66b8e092e7 has been approved by petrochenkov

@bors bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Jun 27, 2020
@rust-highfive

This comment has been minimized.

@Aaron1011 Aaron1011 force-pushed the fix/proc-macro-foreign-span branch from 3c1743c to 2765149 Compare June 27, 2020 17:16
@Aaron1011
Copy link
Member Author

@bors r=petrochenkov

@bors
Copy link
Contributor

bors commented Jun 27, 2020

📌 Commit 2765149 has been approved by petrochenkov

Manishearth added a commit to Manishearth/rust that referenced this pull request Jun 28, 2020
…an, r=petrochenkov

Serialize all foreign `SourceFile`s into proc-macro crate metadata

Normally, we encode a `Span` that references a foreign `SourceFile` by
encoding information about the foreign crate. When we decode this
`Span`, we lookup the foreign crate in order to decode the `SourceFile`.

However, this approach does not work for proc-macro crates. When we load
a proc-macro crate, we do not deserialzie any of its dependencies (since
a proc-macro crate can only export proc-macros). This means that we
cannot serialize a reference to an upstream crate, since the associated
metadata will not be available when we try to deserialize it.

This commit modifies foreign span handling so that we treat all foreign
`SourceFile`s as local `SourceFile`s when serializing a proc-macro.
All `SourceFile`s will be stored into the metadata of a proc-macro
crate, allowing us to cotinue to deserialize a proc-macro crate without
needing to load any of its dependencies.

Since the number of foreign `SourceFile`s that we load during a
compilation session may be very large, we only serialize a `SourceFile`
if we have also serialized a `Span` which requires it.
@Manishearth
Copy link
Member

@bors r-

failed the rollup, likely

#73824 (comment)

https://dev.azure.com/rust-lang/rust/_build/results?buildId=33134&view=logs&j=1c64c3ff-4c37-56fe-a133-b407354bbfbf&t=4d85e1cd-e19c-562d-4a04-e8daca7cfc12&l=17311

failures:

---- [ui] ui/proc-macro/meta-macro-hygiene.rs stdout ----
diff of stdout:
#1: parent: #0, outer_mark: (ExpnId(1), Opaque)
#2: parent: #0, outer_mark: (ExpnId(1), Transparent)
#3: parent: #0, outer_mark: (ExpnId(2), Opaque)
#4: parent: #0, outer_mark: (ExpnId(2), Transparent)
#5: parent: #0, outer_mark: (ExpnId(2), SemiTransparent)
*/

------------------------------------------
stderr:
------------------------------------------

------------------------------------------



failures:
    [ui] ui/proc-macro/meta-macro-hygiene.rs

test result: FAILED. 10350 passed; 1 failed; 62 ignored; 0 measured; 0 filtered out

@bors bors added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Jun 28, 2020
@Aaron1011
Copy link
Member Author

@bors r+ rollup=never

It looks like one of the other PRs in the rollup affected a symbol ID in src/test/ui/proc-macro/meta-macro-hygiene.rs

@bors
Copy link
Contributor

bors commented Jun 28, 2020

📌 Commit 2765149 has been approved by Aaron1011

@bors bors added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Jun 30, 2020
@Aaron1011
Copy link
Member Author

@bors r=petrochenkov

@bors
Copy link
Contributor

bors commented Jun 30, 2020

📌 Commit 5e0396555fc0b3721c90a97e030800714c7d3d51 has been approved by petrochenkov

@bors bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jun 30, 2020
@rust-highfive
Copy link
Collaborator

The job x86_64-gnu-llvm-8 of your PR failed (pretty log, raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.
##[section]Starting: Linux x86_64-gnu-llvm-8
##[section]Starting: Initialize job
Agent name: 'Azure Pipelines 14'
Agent machine name: 'fv-az578'
Current agent version: '2.171.1'
##[group]Operating System
16.04.6
LTS
LTS
##[endgroup]
##[group]Virtual Environment
Environment: ubuntu-16.04
Version: 20200621.1
Included Software: https://github.com/actions/virtual-environments/blob/ubuntu16/20200621.1/images/linux/Ubuntu1604-README.md
##[endgroup]
Agent running as: 'vsts'
Prepare build directory.
Set build variables.
Download all required tasks.
Download all required tasks.
Downloading task: Bash (3.171.1)
Checking job knob settings.
   Knob: AgentToolsDirectory = /opt/hostedtoolcache Source: ${AGENT_TOOLSDIRECTORY} 
   Knob: AgentPerflog = /home/vsts/perflog Source: ${VSTS_AGENT_PERFLOG} 
Start tracking orphan processes.
##[section]Finishing: Initialize job
##[section]Starting: Configure Job Name
==============================================================================
---
========================== Starting Command Output ===========================
[command]/bin/bash --noprofile --norc /home/vsts/work/_temp/56c03cb0-3770-48db-8c1d-b21148b48c91.sh

##[section]Finishing: Disable git automatic line ending conversion
##[section]Starting: Checkout rust-lang/rust@refs/pull/73706/merge to s
Task         : Get sources
Description  : Get sources from a repository. Supports Git, TfsVC, and SVN repositories.
Version      : 1.0.0
Author       : Microsoft
---
##[command]git remote add origin https://github.com/rust-lang/rust
##[command]git config gc.auto 0
##[command]git config --get-all http.https://github.com/rust-lang/rust.extraheader
##[command]git config --get-all http.proxy
##[command]git -c http.extraheader="AUTHORIZATION: basic ***" fetch --force --tags --prune --progress --no-recurse-submodules --depth=2 origin +refs/heads/*:refs/remotes/origin/* +refs/pull/73706/merge:refs/remotes/pull/73706/merge
---
 ---> 31fea614d2f3
Step 5/8 : ENV RUST_CONFIGURE_ARGS       --build=x86_64-unknown-linux-gnu       --llvm-root=/usr/lib/llvm-8       --enable-llvm-link-shared       --set rust.thin-lto-import-instr-limit=10
 ---> Using cache
 ---> 4195cadf126d
Step 6/8 : ENV SCRIPT python2.7 ../x.py test --exclude src/tools/tidy &&            python2.7 ../x.py test src/test/mir-opt --pass=build                                   --target=armv5te-unknown-linux-gnueabi &&            python2.7 ../x.py test src/tools/tidy
 ---> 4e90f6b48f05
Step 7/8 : ENV NO_DEBUG_ASSERTIONS=1
 ---> Using cache
 ---> dfa0a356d899
---
Set({"src/librustc_parse_format"}) not skipped for "bootstrap::test::CrateLibrustc" -- not in ["src/tools/tidy"]
Set({"src/librustc_passes"}) not skipped for "bootstrap::test::CrateLibrustc" -- not in ["src/tools/tidy"]
Set({"src/librustc_plugin_impl"}) not skipped for "bootstrap::test::CrateLibrustc" -- not in ["src/tools/tidy"]
Set({"src/librustc_privacy"}) not skipped for "bootstrap::test::CrateLibrustc" -- not in ["src/tools/tidy"]
Set({"src/librustc_query_system"}) not skipped for "bootstrap::test::CrateLibrustc" -- not in ["src/tools/tidy"]
Set({"src/librustc_save_analysis"}) not skipped for "bootstrap::test::CrateLibrustc" -- not in ["src/tools/tidy"]
Set({"src/librustc_serialize"}) not skipped for "bootstrap::test::CrateLibrustc" -- not in ["src/tools/tidy"]
Set({"src/librustc_session"}) not skipped for "bootstrap::test::CrateLibrustc" -- not in ["src/tools/tidy"]
Set({"src/librustc_span"}) not skipped for "bootstrap::test::CrateLibrustc" -- not in ["src/tools/tidy"]
---
Set({"src/librustc_parse_format"}) not skipped for "bootstrap::doc::Rustc" -- not in ["src/tools/tidy"]
Set({"src/librustc_passes"}) not skipped for "bootstrap::doc::Rustc" -- not in ["src/tools/tidy"]
Set({"src/librustc_plugin_impl"}) not skipped for "bootstrap::doc::Rustc" -- not in ["src/tools/tidy"]
Set({"src/librustc_privacy"}) not skipped for "bootstrap::doc::Rustc" -- not in ["src/tools/tidy"]
Set({"src/librustc_query_system"}) not skipped for "bootstrap::doc::Rustc" -- not in ["src/tools/tidy"]
Set({"src/librustc_save_analysis"}) not skipped for "bootstrap::doc::Rustc" -- not in ["src/tools/tidy"]
Set({"src/librustc_serialize"}) not skipped for "bootstrap::doc::Rustc" -- not in ["src/tools/tidy"]
Set({"src/librustc_session"}) not skipped for "bootstrap::doc::Rustc" -- not in ["src/tools/tidy"]
Set({"src/librustc_span"}) not skipped for "bootstrap::doc::Rustc" -- not in ["src/tools/tidy"]
---
   Compiling rustc_parse_format v0.0.0 (/checkout/src/librustc_parse_format)
   Compiling tracing v0.1.15
   Compiling rustc_ast_pretty v0.0.0 (/checkout/src/librustc_ast_pretty)
   Compiling rustc_hir v0.0.0 (/checkout/src/librustc_hir)
   Compiling rustc_query_system v0.0.0 (/checkout/src/librustc_query_system)
   Compiling chalk-engine v0.14.0
   Compiling rustc_hir_pretty v0.0.0 (/checkout/src/librustc_hir_pretty)
   Compiling rustc_parse v0.0.0 (/checkout/src/librustc_parse)
   Compiling rustc_ast_lowering v0.0.0 (/checkout/src/librustc_ast_lowering)
---
   Compiling rustc_parse_format v0.0.0 (/checkout/src/librustc_parse_format)
   Compiling tracing v0.1.15
   Compiling rustc_ast_pretty v0.0.0 (/checkout/src/librustc_ast_pretty)
   Compiling rustc_hir v0.0.0 (/checkout/src/librustc_hir)
   Compiling rustc_query_system v0.0.0 (/checkout/src/librustc_query_system)
   Compiling chalk-engine v0.14.0
   Compiling rustc_hir_pretty v0.0.0 (/checkout/src/librustc_hir_pretty)
   Compiling rustc_parse v0.0.0 (/checkout/src/librustc_parse)
   Compiling rustc_ast_lowering v0.0.0 (/checkout/src/librustc_ast_lowering)
---
..............................i..................................................................... 1900/10408
.................................................................................................... 2000/10408
...............................................i..i................................................. 2100/10408
.................................................................................................... 2200/10408
.....................................iiiii.......................................................... 2300/10408
.................................................................................................... 2500/10408
.................................................................................................... 2600/10408
.................................................................................................... 2700/10408
.................................................................................................... 2800/10408
---
.................................................................................................... 5300/10408
.................................................................................................... 5400/10408
.............................i...................................................................... 5500/10408
.......................i............................................................................ 5600/10408
...........................................ii.ii........i...i....................................... 5700/10408
............i....................................................................................... 5900/10408
.........i.......................................................................................... 6000/10408
...................................................................ii............................... 6100/10408
......i............................................................................................. 6200/10408
......i............................................................................................. 6200/10408
.................................................................................................... 6300/10408
.................................................................................................... 6400/10408
...............................ii...i..ii...........i............................................... 6500/10408
.................................................................................................... 6700/10408
.................................................................................................... 6800/10408
.................................................................................................... 6800/10408
...................................................................i..ii............................ 6900/10408
.................................................................................................... 7100/10408
.................................................................................................... 7200/10408
.......................i............................................................................ 7300/10408
.................................................................................................... 7400/10408
---
.................................................................................................... 8300/10408
.................................................................................................... 8400/10408
............................................................................i....................... 8500/10408
.................................................................................................... 8600/10408
..............................iiiiii..iiiiii.i...................................................... 8700/10408
.................................................................................................... 8900/10408
.................................................................................................... 9000/10408
.................................................................................................... 9100/10408
.................................................................................................... 9200/10408
---

---- [ui] ui/proc-macro/meta-macro.rs stdout ----
diff of stdout:

- Def site: $DIR/auxiliary/make-macro.rs:5:9: 8:10 (#3)
+ Def site: $DIR/auxiliary/make-macro.rs:7:9: 10:10 (#3)


The actual stdout differed from the expected stdout.
Actual stdout saved to /checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/proc-macro/meta-macro/meta-macro.stdout
Actual stdout saved to /checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/proc-macro/meta-macro/meta-macro.stdout
To update references, rerun the tests and pass the `--bless` flag
To only update this specific test, also pass `--test-args proc-macro/meta-macro.rs`
error: 1 errors occurred comparing output.
status: exit code: 0
status: exit code: 0
command: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" "/checkout/src/test/ui/proc-macro/meta-macro.rs" "-Zthreads=1" "--target=x86_64-unknown-linux-gnu" "--error-format" "json" "-Zui-testing" "-Zdeduplicate-diagnostics=no" "-C" "prefer-dynamic" "-o" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/proc-macro/meta-macro/a" "-Crpath" "-O" "-Cdebuginfo=0" "-Zunstable-options" "-Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "--edition=2018" "-Z" "span-debug" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/proc-macro/meta-macro/auxiliary"
------------------------------------------
------------------------------------------
Def site: /checkout/src/test/ui/proc-macro/auxiliary/make-macro.rs:7:9: 10:10 (#3)
------------------------------------------
stderr:
------------------------------------------

---
thread 'main' panicked at 'Some tests failed', src/tools/compiletest/src/main.rs:344:22
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace


command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-tools-bin/compiletest" "--compile-lib-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib" "--run-lib-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/lib" "--rustc-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" "--src-base" "/checkout/src/test/ui" "--build-base" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui" "--stage-id" "stage2-x86_64-unknown-linux-gnu" "--mode" "ui" "--target" "x86_64-unknown-linux-gnu" "--host" "x86_64-unknown-linux-gnu" "--llvm-filecheck" "/usr/lib/llvm-8/bin/FileCheck" "--nodejs" "/usr/bin/node" "--host-rustcflags" "-Crpath -O -Cdebuginfo=0 -Zunstable-options  -Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "--target-rustcflags" "-Crpath -O -Cdebuginfo=0 -Zunstable-options  -Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "--docck-python" "/usr/bin/python2.7" "--lldb-python" "/usr/bin/python2.7" "--gdb" "/usr/bin/gdb" "--quiet" "--llvm-version" "8.0.0" "--system-llvm" "--cc" "" "--cxx" "" "--cflags" "" "--llvm-components" "" "--adb-path" "adb" "--adb-test-dir" "/data/tmp/work" "--android-cross-path" "" "--color" "always"


failed to run: /checkout/obj/build/bootstrap/debug/bootstrap test --exclude src/tools/tidy
Build completed unsuccessfully in 1:11:48
Build completed unsuccessfully in 1:11:48
== clock drift check ==
  local time: Tue Jun 30 19:59:06 UTC 2020
  network time: Tue, 30 Jun 2020 19:59:06 GMT
== end clock drift check ==

##[error]Bash exited with code '1'.
##[section]Finishing: Run build
##[section]Starting: Checkout rust-lang/rust@refs/pull/73706/merge to s
Task         : Get sources
Description  : Get sources from a repository. Supports Git, TfsVC, and SVN repositories.
Version      : 1.0.0
Author       : Microsoft
Author       : Microsoft
Help         : [More Information](https://go.microsoft.com/fwlink/?LinkId=798199)
==============================================================================
Cleaning any cached credential from repository: rust-lang/rust (GitHub)
##[section]Finishing: Checkout rust-lang/rust@refs/pull/73706/merge to s
Cleaning up task key
Start cleaning up orphan processes.
Terminate orphan process: pid (3793) (python)
##[section]Finishing: Finalize Job

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @rust-lang/infra. (Feature Requests)

@Aaron1011 Aaron1011 force-pushed the fix/proc-macro-foreign-span branch from 5e03965 to 37a48fa Compare June 30, 2020 20:12
@Aaron1011
Copy link
Member Author

Forgot to bless a stderr file

@bors r=petrochenkov

@bors
Copy link
Contributor

bors commented Jun 30, 2020

📌 Commit 37a48fa has been approved by petrochenkov

@bors
Copy link
Contributor

bors commented Jul 1, 2020

⌛ Testing commit 37a48fa with merge 83c62c0537052bce30041002fc57bc73c98b96cf...

@Manishearth
Copy link
Member

@bors r- yield retry

sorry, just made a rollup

@bors bors added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Jul 1, 2020
@Manishearth
Copy link
Member

@bors r=petrochenkov retry

@bors
Copy link
Contributor

bors commented Jul 1, 2020

📌 Commit 37a48fa has been approved by petrochenkov

@bors bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Jul 1, 2020
@bors
Copy link
Contributor

bors commented Jul 1, 2020

⌛ Testing commit 37a48fa with merge d462551...

@bors
Copy link
Contributor

bors commented Jul 1, 2020

☀️ Test successful - checks-actions, checks-azure
Approved by: petrochenkov
Pushing d462551 to master...

@bors bors added the merged-by-bors This PR was explicitly merged by bors. label Jul 1, 2020
@bors bors merged commit d462551 into rust-lang:master Jul 1, 2020
@mati865
Copy link
Contributor

mati865 commented Jul 4, 2020

Nice performance improvement 🎉: https://perf.rust-lang.org/compare.html?start=16957bd4d3a5377263f76ed74c572aad8e4b7e59&end=d462551a8600e57d8b6f87e71ea56868bc5da6cf&stat=instructions:u

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
merged-by-bors This PR was explicitly merged by bors. S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

9 participants