Rustbuild cleanups/fixes and improvements #43630

Mark-Simulacrum · 2017-08-03T17:11:37Z

Each commit is a standalone change, and can/should be reviewed separately.

This adds two new functionalities:

--target and --host can be passed without changing config.toml, and we'll respect the users' wishes, instead of requiring that all possible targets are passed.
- Note that this means that ./x.py clean won't be quite as wide-spread as before, since it limits itself to the configured hosts, not all hosts. This could be considered a feature as well.
ignore-git field in config.toml which tells Rustbuild to not attempt to load git hashes from .git.

This is a precursor to eventual further simplification of the configuration system, but I want to get this merged first so that later work can be made in individual PRs.

r? @alexcrichton

alexcrichton · 2017-08-05T03:39:41Z

@bors: r+

Nice!

bors · 2017-08-05T03:39:42Z

📌 Commit 1121c75 has been approved by alexcrichton

bors · 2017-08-05T04:21:12Z

⌛ Testing commit 1121c75c32c0d840bd0ef73d026628863e4bc50f with merge 1df9f9c7316c2942fa87cdf72a7293262be8f696...

bors · 2017-08-05T05:08:09Z

💔 Test failed - status-travis

kennytm · 2017-08-05T08:13:58Z

armhf-gnu failed, the UI run-pass tests cannot be executed. Cross-testing issue.

[00:42:05] ---- [ui] ui/deprecated-macro_escape-inner.rs stdout ----
[00:42:05] 	ui: /checkout/src/test/ui/deprecated-macro_escape-inner.rs
[00:42:05] thread '[ui] ui/deprecated-macro_escape-inner.rs' panicked at 'failed to exec `/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-tools/x86_64-unknown-linux-gnu/release/remote-test-client`: Error { repr: Os { code: 2, message: "No such file or directory" } }', /checkout/src/libcore/result.rs:860:4
[00:42:05] note: Run with `RUST_BACKTRACE=1` for a backtrace.
[00:42:05] 
...
[00:42:05] 
[00:42:05] failures:
[00:42:05]     [ui] ui/deprecated-macro_escape-inner.rs
[00:42:05]     [ui] ui/deprecated-macro_escape.rs
[00:42:05]     [ui] ui/deriving-meta-empty-trait-list.rs
[00:42:05]     [ui] ui/enum-size-variance.rs
[00:42:05]     [ui] ui/issue-19100.rs
[00:42:05]     [ui] ui/path-lookahead.rs
[00:42:05]     [ui] ui/test-should-panic-attr.rs
[00:42:05] 
[00:42:05] test result: FAILED. 348 passed; 7 failed; 1 ignored; 0 measured; 0 filtered out

Mark-Simulacrum · 2017-08-05T10:58:03Z

Hm, odd. I can't tell why it wouldn't have been available, since it was built as far as I can tell, and for the right target and stage. @alexcrichton Do we copy the RemoteTestClient to the qemu emulator usually? In the meantime, as the queue is mostly empty, I'm going to retry this just in case it was a spurious failure, since I don't believe I changed anything in relevant code...

@bors retry

bors · 2017-08-05T12:15:47Z

⌛ Testing commit 1121c75c32c0d840bd0ef73d026628863e4bc50f with merge 6ca50f7533e08bcf5ea39eaddb9f0f446ba55393...

bors · 2017-08-05T13:02:11Z

💔 Test failed - status-travis

kennytm · 2017-08-05T13:51:36Z

Same error @Mark-Simulacrum.

alexcrichton · 2017-08-07T06:39:57Z

Hm yeah I dunno what's going on :(

Files like remote-test-client indeed never go to the emulator, although this error looks like it's coming from the host, not the emulator? The remote-test-client binary was compiled, though, so I'm not sure what's up?

Mark-Simulacrum · 2017-08-08T00:53:49Z

I'm wondering if maybe some change I added makes it so that we clear out the stage0 tools directory at some point? I'll try to investigate soon, though not sure when I'll get the chance.

bors · 2017-08-08T12:16:40Z

☔ The latest upstream changes (presumably #43694) made this pull request unmergeable. Please resolve the merge conflicts.

Mark-Simulacrum · 2017-08-10T05:25:23Z

@bors r=alexcrichton

The latest commit (7d2e1a9) has the fix for the problem encountered in CI. I'm not sure why the problem doesn't occur on master, and the fix contained isn't perfect, but I hope it'll be sufficient. See the commit description for more details.

bors · 2017-08-10T05:25:23Z

📌 Commit 7d2e1a9 has been approved by alexcrichton

bors · 2017-08-10T06:28:22Z

⌛ Testing commit 7d2e1a9e848a1a303ae9be5f73f7418d9d57f83d with merge d20881a5d77c454a61b2b739b8251c0abdbe692c...

bors · 2017-08-10T06:32:16Z

💔 Test failed - status-appveyor

kennytm · 2017-08-10T06:57:57Z

src/bootstrap/compile.rs

@@ -198,6 +198,12 @@ impl Step for StdLink {
            // for reason why the sanitizers are not built in stage0.
            copy_apple_sanitizer_dylibs(&build.native_dir(target), "osx", &libdir);
        }
+
+        builder.ensure(tool::CleanTools {


Failed to bootstrap. Missing use tool;?

Compiling bootstrap v0.0.0 (file:///C:/projects/rust/src/bootstrap) error[E0433]: failed to resolve. Use of undeclared type or module `tool` --> src\bootstrap\compile.rs:202:24 | 202 | builder.ensure(tool::CleanTools { | ^^^^^^^^^^^^^^^^ Use of undeclared type or module `tool` error[E0433]: failed to resolve. Use of undeclared type or module `tool` --> src\bootstrap\compile.rs:398:24 | 398 | builder.ensure(tool::CleanTools { | ^^^^^^^^^^^^^^^^ Use of undeclared type or module `tool` error[E0433]: failed to resolve. Use of undeclared type or module `tool` --> src\bootstrap\compile.rs:581:24 | 581 | builder.ensure(tool::CleanTools { | ^^^^^^^^^^^^^^^^ Use of undeclared type or module `tool`

This introduces a slight change in behavior, where we unilaterally respect the --host and --target parameters passed for all sanity checking and runtime configuration.

We no longer care about the source of this information, so there is no reason to restrict users.

We first check the configuration, then passed parameters (--build), then fall back to the auto-detection that bootstrap.py does. Fixes rust-lang#39673.

Some users of the build system change the git sha on every build due to utilizing git to push changes to a remote server. This allows them to simply configure that away instead of depending on custom patches to rustbuild.

This permits proc macro crates to correctly work with rustdoc.

This fixes the bug we previously had where we'd build a libtest tool after building a libstd tool and clear out the libstd tool. Since we clear out all tools for a given stage on invocations of CleanTools after lib{std, test, rustc} change, we need to make sure that all tools built with that stage will be built after the clearing is done. The fix contained here technically isn't perfect; there is still an edge case of compiling a libstd tool, then compiling libtest, which will clear out the libstd tool and it won't ever get rebuilt within that session of rustbuild. This is where the caching system used today shows it's problems -- in effect, all tools depend on a global counter of the stage being cleared out. We can implement such a counter in a future patch to ensure that tools are rebuilt as needed, but it is deemed unlikely that it will be required in practice, since most if not all tools are built after the relevant stage's std/test/rustc are built, though this is only an opinion and hasn't been verified.

Mark-Simulacrum · 2017-08-13T00:20:23Z

Thanks for investigating! I've pushed a fix that just renames the "tool" rustdoc to rustdoc-tool-binary which should work if I understand correctly, and rebased.

@bors r+

bors · 2017-08-13T00:20:24Z

📌 Commit 82cdf10 has been approved by Mark-Simulacrum

Mark-Simulacrum · 2017-08-13T00:21:25Z

Ah,

@bors r=alexcrichton

bors · 2017-08-13T00:21:25Z

💡 This pull request was already approved, no need to approve it again.

There's another pull request that is currently being tested, blocking this pull request: rustdoc: Don't add external impls to implementors js #43736

bors · 2017-08-13T00:21:26Z

📌 Commit 82cdf10 has been approved by alexcrichton

bors · 2017-08-13T00:43:31Z

⌛ Testing commit 82cdf10 with merge 7eb0bd28d5a88e2971159c8237c95c41c5e1b456...

bors · 2017-08-13T02:18:10Z

💔 Test failed - status-travis

Mark-Simulacrum · 2017-08-13T03:59:58Z

Compiler hosts aren't always capable of running on the current platform...

@bors r=alexcrichton

bors · 2017-08-13T03:59:59Z

📌 Commit 01641c7 has been approved by alexcrichton

bors · 2017-08-13T05:24:54Z

⌛ Testing commit 01641c7 with merge adbce60...

@alexcrichton

…hton Rustbuild cleanups/fixes and improvements Each commit is a standalone change, and can/should be reviewed separately. This adds two new functionalities: - `--target` and `--host` can be passed without changing config.toml, and we'll respect the users' wishes, instead of requiring that all possible targets are passed. - Note that this means that `./x.py clean` won't be quite as wide-spread as before, since it limits itself to the configured hosts, not all hosts. This could be considered a feature as well. - `ignore-git` field in `config.toml` which tells Rustbuild to not attempt to load git hashes from `.git`. This is a precursor to eventual further simplification of the configuration system, but I want to get this merged first so that later work can be made in individual PRs. r? @alexcrichton

bors · 2017-08-13T07:50:28Z

☀️ Test successful - status-appveyor, status-travis
Approved by: alexcrichton
Pushing adbce60 to master...

rust-highfive assigned alexcrichton Aug 3, 2017

aidanhs added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Aug 3, 2017

This was referenced Aug 4, 2017

Add warnings when rustdoc html rendering differs #41991

Merged

Build rustdoc only at the topmost stage. #43528

Closed

Mark-Simulacrum added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Aug 10, 2017

Mark-Simulacrum force-pushed the rustbuild-cleanups branch from 3d78ff9 to 7d2e1a9 Compare August 10, 2017 05:24

Mark-Simulacrum added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Aug 10, 2017

kennytm reviewed Aug 10, 2017

View reviewed changes

Mark-Simulacrum force-pushed the rustbuild-cleanups branch from 7d2e1a9 to 5fd5c7a Compare August 10, 2017 07:12

Mark-Simulacrum added 10 commits August 13, 2017 05:15

Unify flags into config.

44ffb61

This introduces a slight change in behavior, where we unilaterally respect the --host and --target parameters passed for all sanity checking and runtime configuration.

Allow specifiying targets and hosts not in the config file.

84d9a6e

We no longer care about the source of this information, so there is no reason to restrict users.

Allow overriding build triple via flag.

5290c6c

We first check the configuration, then passed parameters (--build), then fall back to the auto-detection that bootstrap.py does. Fixes rust-lang#39673.

Add ability to ignore git when building rust.

40dea65

Some users of the build system change the git sha on every build due to utilizing git to push changes to a remote server. This allows them to simply configure that away instead of depending on custom patches to rustbuild.

Make the message for building rustdoc slightly nicer

83b6812

Build rustdoc with the stageN compiler in N >= 2.

ad4acba

This permits proc macro crates to correctly work with rustdoc.

Build rustdoc only at the top stage

facf5a9

Correct code to not run host-only tests.

6571968

Cargotest needs only one rustdoc.exe to exist on Windows

82cdf10

Mark-Simulacrum force-pushed the rustbuild-cleanups branch from 13cebf6 to 82cdf10 Compare August 13, 2017 00:16

Build rustdoc with the native build triple

01641c7

bors merged commit 01641c7 into rust-lang:master Aug 13, 2017

kennytm mentioned this pull request Aug 15, 2017

AppVeyor's 32-bit MSVC testing become extremely slow since August 13th #43881

Closed

o01eg mentioned this pull request Nov 30, 2017

Custom libdir breaks std crate #45345

Closed

Mark-Simulacrum deleted the rustbuild-cleanups branch November 30, 2017 18:15

mati865 mentioned this pull request Mar 21, 2019

error: unknown debugging option: dual-proc-macros when running cross tests #59110

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Rustbuild cleanups/fixes and improvements #43630

Rustbuild cleanups/fixes and improvements #43630

Mark-Simulacrum commented Aug 3, 2017

alexcrichton commented Aug 5, 2017

bors commented Aug 5, 2017

bors commented Aug 5, 2017

bors commented Aug 5, 2017

kennytm commented Aug 5, 2017 •

edited

Loading

Mark-Simulacrum commented Aug 5, 2017

bors commented Aug 5, 2017

bors commented Aug 5, 2017

kennytm commented Aug 5, 2017

alexcrichton commented Aug 7, 2017

Mark-Simulacrum commented Aug 8, 2017

bors commented Aug 8, 2017

Mark-Simulacrum commented Aug 10, 2017

bors commented Aug 10, 2017

bors commented Aug 10, 2017

bors commented Aug 10, 2017

kennytm Aug 10, 2017

Mark-Simulacrum commented Aug 13, 2017

bors commented Aug 13, 2017

Mark-Simulacrum commented Aug 13, 2017

bors commented Aug 13, 2017

bors commented Aug 13, 2017

bors commented Aug 13, 2017

bors commented Aug 13, 2017

Mark-Simulacrum commented Aug 13, 2017

bors commented Aug 13, 2017

bors commented Aug 13, 2017

bors commented Aug 13, 2017

Rustbuild cleanups/fixes and improvements #43630

Rustbuild cleanups/fixes and improvements #43630

Conversation

Mark-Simulacrum commented Aug 3, 2017

alexcrichton commented Aug 5, 2017

bors commented Aug 5, 2017

bors commented Aug 5, 2017

bors commented Aug 5, 2017

kennytm commented Aug 5, 2017 • edited Loading

Mark-Simulacrum commented Aug 5, 2017

bors commented Aug 5, 2017

bors commented Aug 5, 2017

kennytm commented Aug 5, 2017

alexcrichton commented Aug 7, 2017

Mark-Simulacrum commented Aug 8, 2017

bors commented Aug 8, 2017

Mark-Simulacrum commented Aug 10, 2017

bors commented Aug 10, 2017

bors commented Aug 10, 2017

bors commented Aug 10, 2017

kennytm Aug 10, 2017

Choose a reason for hiding this comment

Mark-Simulacrum commented Aug 13, 2017

bors commented Aug 13, 2017

Mark-Simulacrum commented Aug 13, 2017

bors commented Aug 13, 2017

bors commented Aug 13, 2017

bors commented Aug 13, 2017

bors commented Aug 13, 2017

Mark-Simulacrum commented Aug 13, 2017

bors commented Aug 13, 2017

bors commented Aug 13, 2017

bors commented Aug 13, 2017

kennytm commented Aug 5, 2017 •

edited

Loading