Download pre-compiled bootstrap from CI #112598

dvtkrlbs · 2023-06-13T22:12:37Z

This PR builds on the work done by #98483 and downloads the bootstrap from the artifacts server if there is not any changes on the bootstrap source code. If the download fails it fallbacks to local build.

fixes #99989
helps with #94829

rustbot · 2023-06-13T22:12:45Z

r? @ozkanonur

(rustbot has picked a reviewer for you, use r? to override)

dvtkrlbs · 2023-06-13T22:13:41Z

r? @jyn514

dvtkrlbs · 2023-06-13T22:14:44Z

i could not find the button to turn the pr into draft it right now works but it is not a clean solution. I am not sure how to clean this it has a few hacks that i am not happy with

dvtkrlbs · 2023-06-14T00:21:51Z

RIght now the wip code still tries to download even if there is a bootstrap locally. I am not sure what is the best way to handle this case. I first thought about chekcing if the cache folder has a bootstrap binary and then skipping download but this does not consider the case when user updates their old copy of the source code and in this case x.py will still use old bootstrap binary. What do you think is the best way to handle this case ?

jyn514 · 2023-06-15T12:41:50Z

i don't have time for reviews, sorry. onur is working on the shell script side, he's a good reviewer.

r? @ozkanonur

onur-ozkan · 2023-06-15T14:54:25Z

I can do the review by the coming weekend. Until then, you can squash your commits and undraft the PR once it's ready.

@rustbot author

dvtkrlbs · 2023-06-17T16:05:38Z

I think this is ready for initial review the redownload logic is not fully clean but I am open to suggestions for best way to detect changes. I also screwed up my first rebase and the bot added a label.

onur-ozkan

Maybe it's time to update this log

https://github.com/rust-lang/rust/blob/1673e54f679ebf67fc7c889b4bcba6d93add0456/src/bootstrap/bootstrap.py#L1124

Something like Checking if bootstrap needs to be downloaded or built before processing. should fit better

src/bootstrap/bootstrap.py

dvtkrlbs · 2023-06-17T17:49:06Z

The one problem that I did not find a clean solution for still is even if there is a built bootstrap binary in the build folder and there is no changes it still tries to download so I think the bootstrap_modified also needs to detect if there is an up to date bootstrap binary. I am not sure how to best solve this.

onur-ozkan · 2023-06-17T18:05:03Z

I think the bootstrap_modified also needs to detect if there is an up to date bootstrap binary. I am not sure how to best solve this.

Yes, this should be checked too. You can create a stamp file on the same path with bootstrap binary and check that for this particular check.

dvtkrlbs · 2023-06-18T13:05:37Z

I added the stamp check. I am not sure about the best way to check the logic without having a CI artifact for the latest commit

onur-ozkan · 2023-06-19T05:11:52Z

src/bootstrap/bootstrap.py

+        return True
+
+
+
+    def download_bootstrap(self, commit_hash):


Too many spaces

onur-ozkan · 2023-06-19T05:22:18Z

src/bootstrap/bootstrap.py

+    def last_bootstrap_commit(self):
+        top_level = output(["git", "rev-parse", "--show-toplevel"])
+
+        cmd = ["git", "rev-list", "--author={}".format(self.git_merge_commit_mail),
+               "-n1", "--first-parent", "HEAD"]
+        merge_base = output(cmd)
+
+        path = "{}/src/bootstrap".format(top_level)
+        cmd = ["git", "diff-index", "--quiet", "HEAD", "--", path]
+
+        ret = subprocess.Popen(cmd, cwd=self.rust_root)
+        code = ret.wait()
+        if code != 0:
+            return None
+        else:
+            return merge_base


This command alone git log -1 --pretty=format:%H src/bootstrap does what you need

if you need short version of the commit hash, simply lowercase the H in --pretty

onur-ozkan · 2023-06-19T05:29:09Z

src/bootstrap/bootstrap.py

+    def bootstrap_out_of_date(self, commit):
+        stamp_path = pathlib.Path(self.bin_root()).joinpath("bootstrap/.bootstrap-stamp")
+        if not stamp_path.exists():
+            return True
+
+        return stamp_path.read_text != commit


This seems wrong as it will always return True because we don't have the stamp file on fresh repository.

onur-ozkan · 2023-06-19T05:39:59Z

I haven't looked into this in detail, but I think the following steps should get the job done easily:

Check if anything has been changed in the src/bootstrap folder. If there are any modifications, compile the bootstrap.
If there haven't been any changes, first check if we have a stamp file:
- If we do, compare its value with the latest bootstrap commit. If they don't match, download the latest version from the CI system.
- If we don't have a stamp file, download it from the CI system.

Regardless of whether we need to compile or download, we need to make sure the stamp file has been generated.

dvtkrlbs · 2023-06-19T11:05:49Z

Wait do we distribute the stamp-file with the CI as well. I thought it gets created after a successful download to mark the artifact.

onur-ozkan · 2023-06-19T11:07:00Z

Wait do we distribute the stamp-file with the CI as well.

No, this is not needed.

bors · 2023-06-25T21:51:33Z

☔ The latest upstream changes (presumably #113037) made this pull request unmergeable. Please resolve the merge conflicts.

dvtkrlbs · 2023-06-29T12:58:47Z

fmt check
tidy check
dependency exception `dunce` license has changed
    previously `CC0-1.0 OR MIT-0 OR Apache-2.0` now `CC0-1.0 OR MIT-0`
    update EXCEPTIONS for the new license
some tidy checks failed
Build completed unsuccessfully in 0:00:37
error: failed to push some refs to 'github.com:dvtkrlbs/rust'

trying to rebasae my branch and getting this error. and it is not helpful at all it first does not tell which crate has this dependency and second it does not tell where this exceptions file or sections is

dvtkrlbs · 2023-06-29T13:01:42Z

oh found it it was in the tools tidy deps

Kobzol · 2023-06-29T13:02:01Z

rust/src/tools/tidy/src/deps.rs

Line 57 in 214a0d1

("dunce", "CC0-1.0 OR MIT-0 OR Apache-2.0"),

rustbot · 2023-06-29T13:02:17Z

Some changes occurred in src/tools/cargo

cc @ehuss

bors · 2023-07-01T15:17:48Z

☔ The latest upstream changes (presumably #113229) made this pull request unmergeable. Please resolve the merge conflicts.

Dylan-DPC · 2023-08-16T17:24:41Z

@dvtkrlbs any updates on resolving the conflicts?

dvtkrlbs · 2023-08-16T17:41:24Z

I will update my branch in the following days. Was there an easy way to resolve submodule conflicts there was a way to force reset them to HEAD without going through the conflict resolving but I don't remember how.

dvtkrlbs · 2023-08-21T18:47:00Z

i really despise git sometimes I thought the rebase was resolved did the push and forgot about it. somehow it reset my branch. Looking at the reflog now to restore it.

rustbot assigned onur-ozkan Jun 13, 2023

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) labels Jun 13, 2023

rustbot assigned jyn514 and unassigned onur-ozkan Jun 13, 2023

This comment has been minimized.

Sign in to view

dvtkrlbs marked this pull request as draft June 13, 2023 22:40

dvtkrlbs changed the title ~~Download precompiled bootstrap from CI~~ Download pre-compiled bootstrap from CI Jun 13, 2023

rustbot assigned onur-ozkan and unassigned jyn514 Jun 15, 2023

rustbot added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jun 15, 2023

dvtkrlbs force-pushed the master branch from ebee149 to 215624a Compare June 17, 2023 16:02

rustbot added the WG-trait-system-refactor The Rustc Trait System Refactor Initiative (-Znext-solver) label Jun 17, 2023

dvtkrlbs force-pushed the master branch from 215624a to 1673e54 Compare June 17, 2023 16:04

dvtkrlbs marked this pull request as ready for review June 17, 2023 16:05

jyn514 removed the WG-trait-system-refactor The Rustc Trait System Refactor Initiative (-Znext-solver) label Jun 17, 2023

onur-ozkan requested changes Jun 17, 2023

View reviewed changes

src/bootstrap/bootstrap.py Outdated Show resolved Hide resolved

src/bootstrap/bootstrap.py Outdated Show resolved Hide resolved

This comment has been minimized.

Sign in to view

dvtkrlbs force-pushed the master branch from fca7020 to eae95d3 Compare June 17, 2023 17:59

dvtkrlbs requested a review from onur-ozkan June 18, 2023 13:05

onur-ozkan reviewed Jun 19, 2023

View reviewed changes

dvtkrlbs force-pushed the master branch from 6807e43 to 22b2ba3 Compare June 29, 2023 13:02

rustbot added the A-testsuite Area: The testsuite used to check the correctness of rustc label Jun 29, 2023

dvtkrlbs force-pushed the master branch from efe8bd8 to 0f0113f Compare June 29, 2023 13:17

This comment has been minimized.

Sign in to view

dvtkrlbs closed this Aug 21, 2023

dvtkrlbs force-pushed the master branch from 09b70a0 to c40cfcf Compare August 21, 2023 07:32

dvtkrlbs mentioned this pull request Aug 21, 2023

Download pre-compiled bootstrap from CI #115076

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Download pre-compiled bootstrap from CI #112598

Download pre-compiled bootstrap from CI #112598

dvtkrlbs commented Jun 13, 2023 •

edited

Loading

rustbot commented Jun 13, 2023

dvtkrlbs commented Jun 13, 2023

dvtkrlbs commented Jun 13, 2023 •

edited

Loading

This comment has been minimized.

dvtkrlbs commented Jun 14, 2023

jyn514 commented Jun 15, 2023

onur-ozkan commented Jun 15, 2023

dvtkrlbs commented Jun 17, 2023 •

edited

Loading

onur-ozkan left a comment

dvtkrlbs commented Jun 17, 2023

This comment has been minimized.

onur-ozkan commented Jun 17, 2023

dvtkrlbs commented Jun 18, 2023 •

edited

Loading

onur-ozkan Jun 19, 2023

onur-ozkan Jun 19, 2023

onur-ozkan Jun 19, 2023

onur-ozkan Jun 19, 2023

onur-ozkan commented Jun 19, 2023

dvtkrlbs commented Jun 19, 2023

onur-ozkan commented Jun 19, 2023

bors commented Jun 25, 2023

dvtkrlbs commented Jun 29, 2023

dvtkrlbs commented Jun 29, 2023

Kobzol commented Jun 29, 2023

rustbot commented Jun 29, 2023

This comment has been minimized.

This comment has been minimized.

bors commented Jul 1, 2023

Dylan-DPC commented Aug 16, 2023

dvtkrlbs commented Aug 16, 2023

dvtkrlbs commented Aug 21, 2023 •

edited

Loading

Download pre-compiled bootstrap from CI #112598

Download pre-compiled bootstrap from CI #112598

Conversation

dvtkrlbs commented Jun 13, 2023 • edited Loading

rustbot commented Jun 13, 2023

dvtkrlbs commented Jun 13, 2023

dvtkrlbs commented Jun 13, 2023 • edited Loading

This comment has been minimized.

dvtkrlbs commented Jun 14, 2023

jyn514 commented Jun 15, 2023

onur-ozkan commented Jun 15, 2023

dvtkrlbs commented Jun 17, 2023 • edited Loading

onur-ozkan left a comment

Choose a reason for hiding this comment

dvtkrlbs commented Jun 17, 2023

This comment has been minimized.

onur-ozkan commented Jun 17, 2023

dvtkrlbs commented Jun 18, 2023 • edited Loading

onur-ozkan Jun 19, 2023

Choose a reason for hiding this comment

onur-ozkan Jun 19, 2023

Choose a reason for hiding this comment

onur-ozkan Jun 19, 2023

Choose a reason for hiding this comment

onur-ozkan Jun 19, 2023

Choose a reason for hiding this comment

onur-ozkan commented Jun 19, 2023

dvtkrlbs commented Jun 19, 2023

onur-ozkan commented Jun 19, 2023

bors commented Jun 25, 2023

dvtkrlbs commented Jun 29, 2023

dvtkrlbs commented Jun 29, 2023

Kobzol commented Jun 29, 2023

rustbot commented Jun 29, 2023

This comment has been minimized.

This comment has been minimized.

bors commented Jul 1, 2023

Dylan-DPC commented Aug 16, 2023

dvtkrlbs commented Aug 16, 2023

dvtkrlbs commented Aug 21, 2023 • edited Loading

dvtkrlbs commented Jun 13, 2023 •

edited

Loading

dvtkrlbs commented Jun 13, 2023 •

edited

Loading

dvtkrlbs commented Jun 17, 2023 •

edited

Loading

dvtkrlbs commented Jun 18, 2023 •

edited

Loading

dvtkrlbs commented Aug 21, 2023 •

edited

Loading