Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

If local-rust is the same as the current version, then force a local-rebuild #34779

Merged
merged 3 commits into from
Jul 16, 2016

Conversation

infinity0
Copy link
Contributor

In Debian, we would like the option to build/rebuild the current release from
either the current or previous stable release. So we use enable-local-rust
instead of enable-local-rebuild, and read the bootstrap key dynamically from
whatever is installed locally.

In general, it does not make much sense to allow enable-local-rust without also
setting the bootstrap key, since the build would fail otherwise.

(The way I detect "the bootstrap key of [the local] rustc installation" is a bit hacky, suggestions welcome.)

@rust-highfive
Copy link
Collaborator

Thanks for the pull request, and welcome! The Rust team is excited to review your changes, and you should hear from @brson (or someone else) soon.

If any changes to this PR are deemed necessary, please add them as extra commits. This ensures that the reviewer can see what has changed since they last reviewed the code. Due to the way GitHub handles out-of-date commits, this should also make it reasonably obvious what issues have or haven't been addressed. Large or tricky changes may require several passes of review and changes.

Please see the contribution instructions for more information.

@TimNN
Copy link
Contributor

TimNN commented Jul 12, 2016

I use export RUSTC_BOOTSTRAP_KEY=rustc --version | cut -f2 -d' ' | tr -d "\n" | md5sum | cut -c-8`` to get the bootstrap key for a specific version of rustc.

@eddyb
Copy link
Member

eddyb commented Jul 12, 2016

In Debian, we would like the option to build/rebuild the current release from either the current or previous stable release.

How is that supported in any way? AFAIK the only supported versions are "previous release" and "exact same as the source being compiled".

cc @rust-lang/tools

@cuviper
Copy link
Member

cuviper commented Jul 12, 2016

If you're setting the --release-channel correctly, so you don't get "-dev" in your version, then your debian builds should get the exact same bootstrap key as upstream rust, whether for the previous or current local rust. So I'm not sure why you need any guesswork/hack detection.

The local-rebuild option also affects how unstable features are used internally. If you don't use this, then the first pass will get --cfg stage0 which toggles code to use only the previous release's features, not anything new. With rebuild it invokes even the first pass with --cfg stage1, as if saying "everything is new!" You might be lucky if this doesn't matter right now, but any breaking transition in unstable features needs to get this right.

find ${PREFIX}/${LIB_DIR} -type f -name ${LIB_PREFIX}std-*${LIB_SUF} \
| sed -ne 's,^.*/'${LIB_PREFIX}'std-\(.*\)'${LIB_SUF}'$,\1,gp' \
| head -n1
;;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can't tell what this is doing. Can you add more comments?

@brson
Copy link
Contributor

brson commented Jul 12, 2016

Bootstrapping from the current release has been a known requirement (#29556), but that issue was closed by this PR (by @cuviper). Why did that solution end up not working?

@alexcrichton
Copy link
Member

Hm yeah I believe the intention of these two flags was:

  • --enable-local-rust just means you arranged a stage0 download yourself, but the build system assumes that it's the exact same stage0 that it would have downloaded.
  • --enable-local-rebuild means that the stage0 in use is the exact same as the compiler that's going to be produced, in which case the bootstrap key the same.

Along those lines the two situations you've described in the description I think are covered by these two flags? When you're bootstrapping from a previous release --enable-local-rust should be all you need, and when you bootstrap from the current release you'll also pass --enable-local-rebuild.

@infinity0
Copy link
Contributor Author

@TimNN unfortunately that doesn't work for stable builds.

@alexcrichton Thanks, yes --enable-local-rust does work with the previous stable release. I just thought it would be simpler to enable this functionality under one option, rather than two. IMO it's a little bit pointless for every distributor to have to pass an extra --enable-local-rebuild plus some custom version detection code for the current local rust. However, I do understand the desire to restrict the support to exactly "the current or previous stable release".

Going forward I could:

a) revert this whole PR and instead improve the documentation to be a bit clearer about what distributors are supposed to do with those flags
b) build on top of this PR to make local-stage0.sh error in case the output is not exactly equal to one of the two supported values for the bootstrap key. Then distributors would need to write less code, but still have the version of the stage0 be restricted to two possible options.

@TimNN
Copy link
Contributor

TimNN commented Jul 13, 2016

@infinity0 Then I probably misunderstood something, this script behaves as expected for me (tested with stable 1.9.0 & 1.10.0): the first rustc code.rs invocation fails, the second one succeeds.

@cuviper
Copy link
Member

cuviper commented Jul 13, 2016

Again, the bootstrap key isn't the whole story. Using --cfg stage0 is a problem if the stage0 compiler is actually the current release, instead of the previous release. That's the other part "rebuild" fixes.

I think just improving the documentation is the better choice, if it's not clear what these are doing.

@cuviper
Copy link
Member

cuviper commented Jul 13, 2016

FWIW, so far I'm not trying to get fancy detecting the local version for the Fedora spec. I just have a rebuild flag to manually toggle in the spec, and then BuildRequires: rust with appropriate version constraints.

@alexcrichton
Copy link
Member

@infinity0 actually both of your suggestions sound great to me! I think we could definitely use some bolstered docs here, and I'd love to see the local_stage0.sh script beefed up.

I think the script should:

  • If only --enable-local-rust is passed, verify the bootstrap key is the same as the stage0 bootstrap key. In other words, md5 the version and ensure the first 8 characters are the same.
  • If --enable-local-rebulid is also passed, verify the bootstrap key is the same as the bootstrap key for the compiler that will be created.

@infinity0
Copy link
Contributor Author

@cuviper Could you explain in more detail this comment: "Using --cfg stage0 is a problem if the stage0 compiler is actually the current release, instead of the previous release"? I can see how it might be more efficient to "assume stage1 features" if doing a rebuild with the current version, but not why not-assuming stage1 would cause a problem.

@cuviper
Copy link
Member

cuviper commented Jul 14, 2016

@infinity0 Sometimes there are breaking changes in syntax that hasn't stabilized yet, and --cfg stage0 is used to alternate between using the old and new way in the compiler itself. Consider #32202 for example, particularly commit f0174fc. That "HACK" would do the wrong thing with stage0 if the local rust is really the same current version. So --enable-local-rebuild tricks it to act like stage1 instead, here.

@infinity0
Copy link
Contributor Author

@TimNN whoops my bad, your snippet works fine. I was confused and expected rustc --version to output the hash as well, and "skimmed" the snippet execution in my head.

@cuviper OK thanks for the explanation. I wasn't expecting those sorts of behaviour changes, but that makes sense.

Based on everyone's comments, my new idea is to set enable-local-rebuild implicitly, if the local-rust version matches CFG_RELEASE. Then I can also revert most of what I wrote in local-stage0.sh. I'll also force-push to this PR because this new idea basically obsoletes most of the original commit, so there's no point leaving it in the history. Please let me know if you prefer otherwise.

Technical details: I have to do this in the makefiles because ./configure does not have access to CFG_RELEASE. But then, config.mk will contain a value for CFG_ENABLE_LOCAL_REBUILD which gets effectively overridden later on, which might be a bit counter-intuitive. To fix this I can refactor the build scripts so that CFG_RELEASE is defined in ./configure instead, but this is more invasive so I'll leave it out for the time being. But if anyone prefers that, please let me know and I'll do that as well.

Naming suggestion: I'd also suggest to rename enable-local-rebuild to enable-stage1-rebuild or something like that, to make it a bit more explicit. The documentation will then read: opt stage1-rebuild 0 "assume the stage0 compiler has stage1 features; implies enable-local-rust. furthermore, this option is implicitly forced on if the local-rust version is the same as the current version being built."

@alexcrichton
Copy link
Member

I actually like the idea of inferring enable-local-rebuild, we could basically remove that option from the ./configure script!

Also yeah doing this in the makefiles is fine, but could you also be sure to add support in rustbuild (src/bootstrap) as well?

@infinity0 infinity0 force-pushed the master branch 3 times, most recently from a6f5030 to d98a256 Compare July 14, 2016 17:40
@infinity0
Copy link
Contributor Author

infinity0 commented Jul 14, 2016

@alexcrichton @brson Updated commits, rewriting the original ones. Now I'm comparing local-rustc --version with CFG_RELEASE, and using this to set local-rebuild even if not explicitly set by the user. This should handle the issues raised by @cuviper but also result in less extra code for distributors to write.

These new commits make it harder for distributors to say "I really only want to use the {previous,current} version, fail if local-rust is the {current,previous} version", but I assume that this won't be a common use-case. The build still fails for any other versions of local-rust, and one could always check the build logs for the presence of the "auto-detected local-rebuild" message to see what was actually done.

I've left the actual config options in for now, but could also remove those.

@infinity0 infinity0 changed the title When enable-local-rust is given, use this to set the stage0 bootstrap key If local-rust is the same as the current version, then force a local-rebuild Jul 14, 2016
@brson
Copy link
Contributor

brson commented Jul 14, 2016

Patch is looking better. Nice work.

let local_version = String::from_utf8(
Command::new(&self.rustc)
.arg("--version").output()
.expect("failed to execute rustc --version").stdout).unwrap();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe there's a build_helper::output function which should help with running a command and getting stdout as a string

@alexcrichton
Copy link
Member

Nice! Looking great to me as well

# If local-rust is the same as the current version, then force a local-rebuild
ifdef CFG_ENABLE_LOCAL_RUST
ifeq (rustc $(CFG_RELEASE),\
$(shell $(Q)$(S)src/etc/local_stage0.sh --print-rustc-version $(CFG_LOCAL_RUST_ROOT)))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, adding on to my note about --version, I see you are actually matching the "rustc" prefix at least. The optional git info would throw it off. I still think we should be a little more precise, because we might want vendor info to show in the basic --version string too, the same way gcc calls itself "gcc version 6.1.1 20160621 (Red Hat 6.1.1-3) (GCC)".

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Understood, it does look like --verbose is more specifically designed to be machine-parsed. I will go ahead and make that change.

@infinity0
Copy link
Contributor Author

OK, I've made some fixes. I only forced-pushed some of the later commits, but it looks like Github thinks the whole set is "outdated" - they aren't, the 3 initial commits with all your comments are still part of the PR.

Anyway, let me know if you want me to do any commit squashing.

case "$TARG_DIR" in
--print-rustc-release)
# not actually copying to TARG_DIR, just print the local rustc version and exit
${PREFIX}/bin/rustc${BIN_SUF} --version --verbose | sed -ne 's/^release: \(.*\)/\1/gp'
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not a big deal, but the wildcard is unnecessary. Just s/^release: //p will work.

@cuviper
Copy link
Member

cuviper commented Jul 14, 2016

Apart from my sed-golfing, it looks good to me, thanks!

@infinity0
Copy link
Contributor Author

Ack, have fixed the sed expressions as well.

@alexcrichton
Copy link
Member

Looks good to me, thanks! Want to squash the commits down as well?

@infinity0
Copy link
Contributor Author

Sure, done:

old (reviewed) commit: bf4cfb413b5ec08344f738a42ef5e7324899148d
new (squashed) commit: bbff336
shared tree (git cat-file -p $commit): 6288ecb3546e8be7841ae67f4e54eae501d2328f

@alexcrichton
Copy link
Member

@bors: r+ bbff336

Thanks!

@bors
Copy link
Contributor

bors commented Jul 15, 2016

⌛ Testing commit bbff336 with merge 9f9b415...

@infinity0
Copy link
Contributor Author

Argh, I forgot to update the first post of this PR, and now it's included in the merge commit's commit message.

@alexcrichton
Copy link
Member

@bors: r+ bbff336 force

@bors
Copy link
Contributor

bors commented Jul 16, 2016

💡 This pull request was already approved, no need to approve it again.

  • This pull request is currently being tested. If there's no response from the continuous integration service, you may use retry to trigger a build again.

@bors
Copy link
Contributor

bors commented Jul 16, 2016

⛄ The build was interrupted to prioritize another pull request.

@alexcrichton
Copy link
Member

Oh hm, I may have just gotten confused...

@bors
Copy link
Contributor

bors commented Jul 16, 2016

⌛ Testing commit bbff336 with merge dc8212f...

bors added a commit that referenced this pull request Jul 16, 2016
If local-rust is the same as the current version, then force a local-rebuild

In Debian, we would like the option to build/rebuild the current release from
*either* the current or previous stable release. So we use enable-local-rust
instead of enable-local-rebuild, and read the bootstrap key dynamically from
whatever is installed locally.

In general, it does not make much sense to allow enable-local-rust without also
setting the bootstrap key, since the build would fail otherwise.

(The way I detect "the bootstrap key of [the local] rustc installation" is a bit hacky, suggestions welcome.)
@bors bors merged commit bbff336 into rust-lang:master Jul 16, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

8 participants