Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[7.3.0] Back-port vendor mode for 7.3.0 #22794

Merged

Conversation

meteorcloudy
Copy link
Member

@meteorcloudy meteorcloudy commented Jun 18, 2024

Commits back-ported:

RELNOTES: Bazel now supports vendoring external dependencies with Bzlmod, see https://bazel.build/external/vendor for details.

SalmaSamy and others added 7 commits June 18, 2024 16:55
Use the TargetFetcher used for fetch to fetch targets, while internally runs a build and creates a dependency graph. Then traverse this graph for each target to collect the repos and vendor them.

PiperOrigin-RevId: 627047679
Change-Id: I78a33c4258099a45d4d3a345bd075badc8985209
A Starlark config file to allow controlling the vendored repos via:
- ignore('repo_name','repo_name', ....) function -->
Bazel will never vendor the repo or consider this directory while building in vendor mode.
- pin('repo_name','repo_name', ....) function -->
Bazel will pin the contents of this repo, will not update it while vendoring, and will always use it as is when building in vendor mode

Co-authored-by: Yun Peng <[email protected]>
PiperOrigin-RevId: 639000162
Change-Id: I278359f6e92eab290dac9cc4ef3136aa140cf8ba
RELNOTES: None
PiperOrigin-RevId: 639001377
Change-Id: Ib914fa8c62999b334ebea552481be5b7ebf52a2e
PiperOrigin-RevId: 641013801
Change-Id: Ia74b7b79a870d74b9055f32e4a4a77418fd3f555
- Vendor registry files needed for Bazel module resolution to achieve
offline build with vendor mode.
- Also refactored bazel_vendor_test to avoid vendoring dependencies of
  bazel_tools, which speeds up the test significantly.

Fixes bazelbuild#22554

Closes bazelbuild#22566.

PiperOrigin-RevId: 641246903
Change-Id: I01a78612ad7ca454df2c70dc5dcc38ec2fbfb71c
This drastically improves the speed of vendoring external repositories.

Related: bazelbuild#19563

Closes bazelbuild#22668.

PiperOrigin-RevId: 642338030
Change-Id: Idcba16c491711cf8fa6637d1e9c42cfc65e87599
To make sure symlinks work correctly, Bazel uses the following strategy to rewrite symlinks in the vendored source:

  - Create a symlink `<vendor_dir>/bazel-external` that points to `$(bazel info output_base)/external`. It is refreshed by every Bazel command automatically.
  - For the vendored source, rewrite all symlinks that originally point to a path under `$(bazel info output_base)/external` to a relative path under `<vendor_dir>/bazel-external`.

Fixes bazelbuild#22303

Closes bazelbuild#22723.

PiperOrigin-RevId: 644349481
Change-Id: I853ac0ea5405f0cf58431e988d727e690cbbb013
@meteorcloudy meteorcloudy requested a review from a team as a code owner June 18, 2024 15:00
@github-actions github-actions bot added awaiting-review PR is awaiting review from an assigned reviewer team-Configurability platforms, toolchains, cquery, select(), config transitions team-Documentation Documentation improvements that cannot be directly linked to other team labels team-ExternalDeps External dependency handling, remote repositiories, WORKSPACE file. team-Performance Issues for Performance teams labels Jun 18, 2024
@meteorcloudy
Copy link
Member Author

meteorcloudy commented Jun 18, 2024

@fmeum @Wyverald I had to apply 91708d2 to fix the build.

Looks like SymbolGenerator has been heavily refactored. In 7.2.0, it was moved from src/main/java/com/google/devtools/build/lib/packages/SymbolGenerator.java to src/main/java/com/google/devtools/build/lib/cmdline/SymbolGenerator.java for some reason. And at Bazel HEAD, it's now under net.starlark.java.eval.SymbolGenerator.

Also the StarlarkThread initialiser in 7.3.0 doesn't take SymbolGenerator at all, so I removed it. Can you please take a look if it's fine?

Currently `pin` and `ignore` show up in the page for MODULE.bazel globals (https://bazel.build/rules/lib/globals/module#pin). They should be in their own page instead.

PiperOrigin-RevId: 641816839
Change-Id: I923ec2e800870dd565ea3e2ba36886b5e92ba947
@meteorcloudy meteorcloudy force-pushed the vendor_mode_for_7_3_0 branch from 91708d2 to bf189e6 Compare June 18, 2024 15:37
@meteorcloudy meteorcloudy requested review from Wyverald and fmeum June 18, 2024 15:49
PiperOrigin-RevId: 644405623
Change-Id: Ic89e4ba7056fe4927470715babe87a8a617da197
@meteorcloudy meteorcloudy force-pushed the vendor_mode_for_7_3_0 branch from bf189e6 to 4046ec9 Compare June 18, 2024 16:22
@iancha1992 iancha1992 added this to the 7.3.0 release blockers milestone Jun 18, 2024
@meteorcloudy meteorcloudy added this pull request to the merge queue Jun 20, 2024
@iancha1992 iancha1992 added this pull request to the merge queue Jun 20, 2024
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Jun 20, 2024
@meteorcloudy meteorcloudy added this pull request to the merge queue Jun 21, 2024
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Jun 21, 2024
@meteorcloudy meteorcloudy enabled auto-merge June 21, 2024 08:47
@meteorcloudy meteorcloudy added this pull request to the merge queue Jun 21, 2024
Merged via the queue into bazelbuild:release-7.3.0 with commit 31fc2ed Jun 21, 2024
32 checks passed
@github-actions github-actions bot removed the awaiting-review PR is awaiting review from an assigned reviewer label Jun 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
team-Configurability platforms, toolchains, cquery, select(), config transitions team-Documentation Documentation improvements that cannot be directly linked to other team labels team-ExternalDeps External dependency handling, remote repositiories, WORKSPACE file. team-Performance Issues for Performance teams
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants