Add unstable support for outputting file checksums for use in cargo #126930

Xaeroxe · 2024-06-25T05:20:42Z

Adds an unstable option that appends file checksums and expected lengths to the end of the dep-info file such that cargo can read and use these values as an alternative to file mtimes.

This PR powers the changes made in this cargo PR rust-lang/cargo#14137

Here's the tracking issue for the cargo feature rust-lang/cargo#14136.

rustbot · 2024-06-25T05:20:51Z

r? @fmease

rustbot has assigned @fmease.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

Urgau · 2024-06-25T05:59:13Z

Given that this PR adds a new concept and a new flag it requires a minimum of discussion within the team.
Could you start a T-compiler Major Change Proposol following the guide.

rustbot · 2024-06-26T05:09:49Z

These commits modify the Cargo.lock file. Unintentional changes to Cargo.lock can be introduced when switching branches and rebasing PRs.

If this was unintentional then you should revert the changes before this PR is merged.
Otherwise, you can ignore this comment.

Xaeroxe · 2024-07-01T03:31:26Z

This now has an MCP rust-lang/compiler-team#765

fmease · 2024-07-07T15:57:29Z

Waiting for rust-lang/compiler-team#765 FCP to complete. That is 2024-07-11 07:23h UTC.
@rustbot blocked

Xaeroxe · 2024-07-13T13:15:09Z

The MCP has been accepted

@rustbot ready

rustbot · 2024-07-13T14:19:47Z

The list of allowed third-party dependencies may have been modified! You must ensure that any new dependencies have compatible licenses before merging.

cc @davidtwco, @wesleywiser

Xaeroxe · 2024-07-13T14:21:18Z

Correct, blake3 has been added as it was mentioned during the MCP review as being desirable.

Urgau · 2024-07-17T11:38:40Z

We had some LLVM BOLT issues with the blake3 crate in #127754, a couple of days ago, on a simple try build.
Let's try this PR, to see if is also has those compilation issues.

~~@bors try~~ (@Xaeroxe could you fix the conflicts please)

Xaeroxe · 2024-07-17T14:53:59Z

@Urgau conflicts resolved.

Urgau · 2024-07-17T15:23:38Z

@bors try @rust-timer queue

Xaeroxe · 2024-10-02T05:45:37Z

Follow-up at #131139

workingjubilee · 2024-10-02T06:15:25Z

The rollup hasn't actually started, so it's actually quite cost-free to just

@bors r-

Xaeroxe · 2024-10-02T06:39:12Z

Okay, in that case here is the commit that @weihanglo wanted b48c5f1

Xaeroxe · 2024-10-02T06:40:20Z

The Cargo PR has already been adjusted to this new behavior.

weihanglo

Checked that 58c5ac4 works with the latest commit rust-lang/cargo@c755949 in rust-lang/cargo#14137.

I would like to r+ this, but I am not in t-compiler so probably shouldn't do that.

It would be appreciated if @chenyukang or someone on t-compiler could take another look. It should be pretty much the same as before except writing # checksum comments to separate lines (b48c5f1)

chenyukang · 2024-10-03T02:43:50Z

@bors r+ rollup

bors · 2024-10-03T02:43:52Z

📌 Commit 58c5ac4 has been approved by chenyukang

It is now in the queue for this repository.

…kingjubilee Rollup of 3 pull requests Successful merges: - rust-lang#126930 (Add unstable support for outputting file checksums for use in cargo) - rust-lang#130725 (Parser: better error messages for ``@`` in struct patterns) - rust-lang#131166 (Fix `target_vendor` for `aarch64-nintendo-switch-freestanding`) r? `@ghost` `@rustbot` modify labels: rollup

Rollup merge of rust-lang#126930 - Xaeroxe:file-checksum-hint, r=chenyukang Add unstable support for outputting file checksums for use in cargo Adds an unstable option that appends file checksums and expected lengths to the end of the dep-info file such that `cargo` can read and use these values as an alternative to file mtimes. This PR powers the changes made in this cargo PR rust-lang/cargo#14137 Here's the tracking issue for the cargo feature rust-lang/cargo#14136.

klensy · 2024-10-03T12:09:43Z

Can PR description be updated with actual changes in that PR? Currently is slightly misleading.

klensy · 2024-10-03T12:11:55Z

compiler/rustc_session/src/options.rs

+    checksum_hash_algorithm: Option<SourceFileHashAlgorithm> = (None, parse_cargo_src_file_hash, [TRACKED],
+        "hash algorithm of source files used to check freshness in cargo (`blake3` or `sha256`)"),


What will happened if md5 passed here? SourceFileHashAlgorithm allows it.

Then md5 checksums will be emitted. This doesn’t make any guarantees about whether the tooling surrounding rustc can actually ingest those. It’s up to the tooling to understand what it can ingest. An earlier version of this PR did forbid things like md5 but I was instructed to not build in that kind of restriction by prior reviewers.

Well, at least help message mentions only blake3 and sha256, while actually accepting something else too.

Hmm… still seeing "cargo" mentioned here and elsewhere. Not really critical but would love to see we make the option more general than a Cargo-specific compiler flag (though in fact it was made for Cargo 😆)

Xaeroxe · 2024-10-03T12:39:18Z

Can PR description be updated with actual changes in that PR? Currently is slightly misleading.

How so? I just went back and read it and it seems correct to me. What’s misleading about it?

klensy · 2024-10-03T14:08:53Z

PR also adds option for using blake3 source file hashing (or not? I'm not clearly understand that, the same enum used for source file hash for debug and for new cargo feature)

Xaeroxe · 2024-10-03T16:24:38Z

blake3 can’t be used in debug data as no debug format supports it, but it can be used in dep-info files. If you try to use blake3 here then you simply won’t get a hash in your debug files.

initial version of checksum based freshness Implementation for #14136 and resolves #6529 This PR implements the use of checksums in cargo fingerprints as an alternative to using mtimes. This is most useful on systems with poor mtime implementations. This has a dependency on rust-lang/rust#126930. It's expected this will increase the time it takes to declare a build to be fresh. Still this loss in performance may be preferable to the issues the ecosystem has had with the use of mtimes for determining freshness.

This also adds three license exceptions to Cargo. * arrayref — BSD-2-Clause * blake3 — CC0-1.0 OR Apache-2.0 OR Apache-2.0 WITH LLVM-exception * constant_time_eq — CC0-1.0 OR MIT-0 OR Apache-2.0 These exceptions were added to rustc in rust-lang#126930, so should be fine for Cargo as well.

Update cargo 8 commits in ad074abe3a18ce8444c06f962ceecfd056acfc73..15fbd2f607d4defc87053b8b76bf5038f2483cf4 2024-10-04 18:18:15 +0000 to 2024-10-08 21:08:11 +0000 - initial version of checksum based freshness (rust-lang/cargo#14137) - feat: Add custom completer for completing registry name (rust-lang/cargo#14656) - Document build-plan as being deprecated (rust-lang/cargo#14657) - fix(complete): Don't complete files for any value (rust-lang/cargo#14653) - Add more SAT resolver tests (rust-lang/cargo#14614) - fix: avoid inserting duplicate `dylib_path_envvar` when calling `cargo run` recursively (rust-lang/cargo#14464) - chore(deps): bump gix-path from 0.10.9 to 0.10.11 (rust-lang/cargo#14489) - improve error reporting when feature not found in `activated_features` (rust-lang/cargo#14647) --- This also adds three license exceptions to Cargo. * arrayref — BSD-2-Clause * blake3 — CC0-1.0 OR Apache-2.0 OR Apache-2.0 WITH LLVM-exception * constant_time_eq — CC0-1.0 OR MIT-0 OR Apache-2.0 These exceptions were added to rustc in rust-lang#126930, so should be fine for Cargo as well.

This also adds three license exceptions to Cargo. * arrayref — BSD-2-Clause * blake3 — CC0-1.0 OR Apache-2.0 OR Apache-2.0 WITH LLVM-exception * constant_time_eq — CC0-1.0 OR MIT-0 OR Apache-2.0 These exceptions were added to rustc in rust-lang#126930, so should be fine for Cargo as well.

rustbot assigned fmease Jun 25, 2024

This was referenced Jun 25, 2024

initial version of checksum based freshness rust-lang/cargo#14137

Merged

Tracking Issue for checksum freshness rust-lang/cargo#14136

Open

Xaeroxe mentioned this pull request Jul 1, 2024

MCP: Alternate cargo freshness algorithm, unstable flag to annotate depinfo file with checksums and file sizes rust-lang/compiler-team#765

Closed

3 tasks

rustbot added S-blocked Status: Blocked on something else such as an RFC or other implementation work. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jul 7, 2024

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-blocked Status: Blocked on something else such as an RFC or other implementation work. labels Jul 13, 2024

Xaeroxe force-pushed the file-checksum-hint branch from ee53f79 to 21c2e87 Compare July 13, 2024 14:19

rustbot added A-testsuite Area: The testsuite used to check the correctness of rustc T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) labels Jul 13, 2024

This comment was marked as outdated.

Sign in to view

Xaeroxe force-pushed the file-checksum-hint branch from 21c2e87 to d80d73e Compare July 17, 2024 14:53

This comment has been minimized.

Sign in to view

rustbot added the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Jul 17, 2024

bors added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Oct 2, 2024

restore prior more readable suggestion

58c5ac4

weihanglo approved these changes Oct 2, 2024

View reviewed changes

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Oct 3, 2024

workingjubilee mentioned this pull request Oct 3, 2024

Rollup of 3 pull requests #131185

Merged

bors merged commit 44f6275 into rust-lang:master Oct 3, 2024
6 checks passed

rustbot added this to the 1.83.0 milestone Oct 3, 2024

klensy reviewed Oct 3, 2024

View reviewed changes

Xaeroxe deleted the file-checksum-hint branch October 3, 2024 12:41

saethlin mentioned this pull request Oct 4, 2024

Add linkage attributes to extern "C" blocks BLAKE3-team/BLAKE3#429

Open

weihanglo mentioned this pull request Oct 9, 2024

Update cargo #131421

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add unstable support for outputting file checksums for use in cargo #126930

Add unstable support for outputting file checksums for use in cargo #126930

Xaeroxe commented Jun 25, 2024

rustbot commented Jun 25, 2024

Urgau commented Jun 25, 2024

rustbot commented Jun 26, 2024

Xaeroxe commented Jul 1, 2024

fmease commented Jul 7, 2024 •

edited

Loading

Xaeroxe commented Jul 13, 2024 •

edited

Loading

rustbot commented Jul 13, 2024

Xaeroxe commented Jul 13, 2024

This comment was marked as outdated.

Urgau commented Jul 17, 2024 •

edited

Loading

This comment was marked as outdated.

Xaeroxe commented Jul 17, 2024

Urgau commented Jul 17, 2024

This comment has been minimized.

Xaeroxe commented Oct 2, 2024

workingjubilee commented Oct 2, 2024

Xaeroxe commented Oct 2, 2024

Xaeroxe commented Oct 2, 2024

weihanglo left a comment

chenyukang commented Oct 3, 2024

bors commented Oct 3, 2024

klensy commented Oct 3, 2024

klensy Oct 3, 2024 •

edited

Loading

Xaeroxe Oct 3, 2024

klensy Oct 3, 2024

weihanglo Oct 4, 2024

Xaeroxe commented Oct 3, 2024

klensy commented Oct 3, 2024

Xaeroxe commented Oct 3, 2024

		checksum_hash_algorithm: Option<SourceFileHashAlgorithm> = (None, parse_cargo_src_file_hash, [TRACKED],
		"hash algorithm of source files used to check freshness in cargo (`blake3` or `sha256`)"),

Add unstable support for outputting file checksums for use in cargo #126930

Add unstable support for outputting file checksums for use in cargo #126930

Conversation

Xaeroxe commented Jun 25, 2024

rustbot commented Jun 25, 2024

Urgau commented Jun 25, 2024

rustbot commented Jun 26, 2024

Xaeroxe commented Jul 1, 2024

fmease commented Jul 7, 2024 • edited Loading

Xaeroxe commented Jul 13, 2024 • edited Loading

rustbot commented Jul 13, 2024

Xaeroxe commented Jul 13, 2024

This comment was marked as outdated.

Urgau commented Jul 17, 2024 • edited Loading

This comment was marked as outdated.

Xaeroxe commented Jul 17, 2024

Urgau commented Jul 17, 2024

This comment has been minimized.

Xaeroxe commented Oct 2, 2024

workingjubilee commented Oct 2, 2024

Xaeroxe commented Oct 2, 2024

Xaeroxe commented Oct 2, 2024

weihanglo left a comment

Choose a reason for hiding this comment

chenyukang commented Oct 3, 2024

bors commented Oct 3, 2024

klensy commented Oct 3, 2024

klensy Oct 3, 2024 • edited Loading

Choose a reason for hiding this comment

Xaeroxe Oct 3, 2024

Choose a reason for hiding this comment

klensy Oct 3, 2024

Choose a reason for hiding this comment

weihanglo Oct 4, 2024

Choose a reason for hiding this comment

Xaeroxe commented Oct 3, 2024

klensy commented Oct 3, 2024

Xaeroxe commented Oct 3, 2024

fmease commented Jul 7, 2024 •

edited

Loading

Xaeroxe commented Jul 13, 2024 •

edited

Loading

Urgau commented Jul 17, 2024 •

edited

Loading

klensy Oct 3, 2024 •

edited

Loading