Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[6.2.0] Construct TreeArtifactValues on multiple threads. #18194

Merged
merged 2 commits into from
Apr 25, 2023
Merged

[6.2.0] Construct TreeArtifactValues on multiple threads. #18194

merged 2 commits into from
Apr 25, 2023

Conversation

BalestraPatrick
Copy link
Member

This makes it possible to use every core available for checksumming, which makes a huge difference for large tree artifacts.

Fixes #17009.

RELNOTES: None.
PiperOrigin-RevId: 525085502
Change-Id: I2a995d3445940333c21eeb89b4ba60887f99e51b (cherry picked from commit 368bf11)

Context

This is an attempt at cherry-picking this change to 6.2.0 as described in this comment: 368bf11#commitcomment-110335567

Not totally sure if this is the right approach or there are more parts missing, but at least it built locally for me.

@lberki @BalestraPatrick
Construct TreeArtifactValues on multiple threads.
3fe74c1 
This makes it possible to use every core available for checksumming, which
makes a huge difference for large tree artifacts.

Fixes bazelbuild#17009.

RELNOTES: None.
PiperOrigin-RevId: 525085502
Change-Id: I2a995d3445940333c21eeb89b4ba60887f99e51b
(cherry picked from commit 368bf11)

# Conflicts:
#	src/main/java/com/google/devtools/build/lib/skyframe/TreeArtifactValue.java
BalestraPatrick referenced this pull request Apr 24, 2023
@lberki @copybara-github
Construct TreeArtifactValues on multiple threads.
368bf11 
This makes it possible to use every core available for checksumming, which
makes a huge difference for large tree artifacts.

Fixes #17009.

RELNOTES: None.
PiperOrigin-RevId: 525085502
Change-Id: I2a995d3445940333c21eeb89b4ba60887f99e51b
@BalestraPatrick BalestraPatrick marked this pull request as ready for review April 25, 2023 09:34
BalestraPatrick
BalestraPatrick commented Apr 25, 2023
View reviewed changes
src/main/java/com/google/devtools/build/lib/skyframe/TreeArtifactValue.java
Visitor visitor = new Visitor(parentDir, treeArtifactVisitor);
visitor.run();
} catch (InterruptedException e) {
throw new IOException(e);
Copy link
Member Author

@BalestraPatrick BalestraPatrick Apr 25, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The key part that is different is really only this one compared to 368bf11

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, looks reasonable. Let's merge this!

@lberki
Copy link
Contributor

lberki commented Apr 25, 2023

@keertk : mind merging this into 6.2.0? It's a backport of the fix for #17009 which needed to be backported and tweaked because it doesn't apply cleanly here

@keertk keertk enabled auto-merge (squash) April 25, 2023 14:05
@keertk keertk mentioned this pull request Apr 25, 2023
Closed
@keertk keertk added the team-Remote-Exec Issues and PRs for the Execution (Remote) team label Apr 25, 2023
@keertk keertk merged commit d94dee2 into bazelbuild:release-6.2.0 Apr 25, 2023
@BalestraPatrick BalestraPatrick deleted the cherry-pick-tree-artifact-value-multiple-threads branch April 27, 2023 08:25
@chiragramani
Copy link
Contributor

Branch: release-6.2.0rc2
Commit: 96eb991

We are seeing a regression on 6.2.0rc2 and looks like it might be related to this change.

Bazel 6.1.0
Screenshot 2023-05-05 at 4 41 51 PM

  • actuallyCompleteAction: 1.3seconds

Bazel 6.2.0
Screenshot 2023-05-05 at 4 42 51 PM

  • actuallyCompleteAction: 6.7seconds

@keertk
Copy link
Member

keertk commented May 6, 2023

@chiragramani would you mind opening an issue for this so we can track it separately?

@meteorcloudy
Copy link
Member

@chiragramani Can you include some steps to reproduce the potential performance regression in the issue?

/cc @lberki

@BalestraPatrick
Copy link
Member Author

I did 3 tries before this change and after, and here are the results of how long the actuallyCompleteAction in our bundling step of our app took (which is a tree artifact).

Before: 9.3s, 6.4s, 9.8s
After: 4.2s, 5.5s, 4.5s

So in our case we definitely see an improvement after this change. All measurements were performed by applying the same incremental modification to a single module, causing it to link and bundle the app again I’ve noticed a big variance in these measurements from our developers as well after being on this change for a few days. Sometimes for the same exact change it can take 20s to compute the metadata, and other times it can take 4s.

copybara-service bot pushed a commit that referenced this pull request May 9, 2023
@keertk
Release 6.2.0 (2023-05-09)
76ef9e9 
Baseline: d60ce2c

Cherry picks:

   + 490f8ba:
     kshyanashree is the release manager of 6.0.0
   + 4e66d93:
     Automated rollback of commit
     2a28909.
   + 48a8d01:
     Allow C/C++ coverage collection for external targets
   + ec7be34:
     Temporarily set parent directory of the input to writable if it
     is not.
   + b098434:
     Infer CPUs for tvOS and watchOS when on Apple Silicon
   + 5cea7dd:
     Improve the documentation for ctx.actions.symlink.
   + a82d26f:
     Add quotes to "Tip"
   + 75b0f3a:
     Write a repo mapping manifest in the runfiles directory (#16555)
   + 07c5c1a:
     Ensure repository names don't start with `~`
   + 30f6c82:
     Escape tilde more gracefully
   + cf3f48c:
     Relax `Label` repo visibility validation
   + 4477823:
     Use "apparent name" instead of "local name" in Bzlmod docs
   + 55f4f48:
     Bazel DevSite: Add required css file.
   + a706994:
     Remove logic that increases delay between progress updates over
     time
   + 1d514ab:
     Remove references to io_bazel repository
   + b0357bd:
     Relnotes python script
   + 8f95651:
     Allow Java coverage collection for external targets
   + bddb191:
     Fix race condition in package-bazel.sh
   + a757852:
     Decanonicalize labels emitted by {a,c,}query if possible
   + 9d250ed:
     Add uniquify parameter to TemplateDict.add_joined
   + f02bcf8:
     Fix identical gcov json file name problem
   + a159330:
     Add `$(rlocationpath(s) ...)` expansion
   + 56f54da:
     Rollup of SBOM correctness fixes (#16655)
   + 72e6e94:
     [cc] Add cc toolchain to starlark cc_proto_library (#16661)
   + 8f28513:
     Make C++ runfiles library repo mapping aware
   + 95f9adc:
     Always collect FileProvider's filesToBuild as data runfiles
   + 4959d34:
     fix: export constraints.bzl file from @local_config_platform so
     it can be used in downstream bzl_library targets
   + 4919d4a:
     Add --host_per_file_copt (#16695)
   + 0a23d46:
     Moving some tests for `RemoteActionFileSystem` of BwoB to a base
     class.
   + 9296068:
     [remote/downloader] Don't include headers in `FetchBlobRequest`
   + 128d833:
     [remote] Respect whether the server supports action cache updates
   + 38c5019:
     [remote/downloader] Migrate `Downloader` to take `Credentials`
     (#16732)
   + 5929cb7:
     Stage repository mapping manifest as a root symlink (#16733)
   + 455454a:
     Expose current repository name to Java with @AutoBazelRepository
   + 97dea59:
     Implement getDirectoryEntries and readdir for
     RemoteActionFileSystem.
   + aa45f5f:
     Move integration tests for BwoB to a base class and add more
     tests there.
   + 1b073ac:
     Make Java runfiles library repo mapping aware
   + 148bbb1:
     Use `_repo_mapping` in C++ runfiles library (#16752)
   + 25558ad:
     Add --experimental_action_cache_store_output_metadata to the
     expansio…
   + 6e945e8:
     Treat `DEBUG` events as progress-like.
   + 1940c5d:
     redact 'token' strings from logging
   + 0b64525:
     Make Bash runfiles library repo mapping aware (#16753)
   + 4caae75:
     Automated rollback of commit
     8f95651.
   + 13ff6d9:
     Fix Bash rlocation failure with stricter Bash options
   + bd88c7e:
     Make bazel Android remote deps compatible with bzlmod (#16772)
   + 6f6d4cc:
     Flip `--incompatible_always_include_files_in_data`
   + 94355b1:
     Add required `--add-opens` server JVM args also with
     non-embedded JDK
   + 8349c95:
     Also collect clang resource directory with
     `-no-canonical-prefixes`
   + dce6ed7:
     Make `bazel run` works with minimal mode
   + ca8674c:
     Include source files with `cquery --output=files` (#16826)
   + 9cb5e0a:
     Fsync before rename after copy in DiskCacheClient
   + 23ffce5:
     Update GetActionResult for disk cache to check referenced files
     when …
   + 0bc0ff5:
     Make Java runfiles library compilable with JDK 8
   + fe16965:
     [6.0.0] Default --incompatible_strict_conflict_checks to true.
     (#16864)
   + 99ca86f:
     Revert "[6.0.0] Default --incompatible_strict_conflict_checks to
     true. (#16864)" (#16872)
   + 312238f:
     Make C++ runfiles library build with `-Werror=shadow`
   + 2baa5a4:
     Keep credentials cached across build commands.
   + 8818a57:
     [6.0.0] Only inject metadata for outputs that cannot be
     reconstructed by skyf… (#16879)
   + 31e4bf4:
     Update java_tools v11.9
   + cd40666:
     replace _get_canonical_form(label) with str(label)
   + e2bc237:
     Avoid exceptions from hermetic sandbox for unsupported artifact
     subclasses
   + b564d14:
     Remove warnings about ignored starlark options
   + 7ccc661:
     [6.0.0] Emit Tree objects in topological order (#16904)
   + 86dee6d:
     Correctly match regex with tree artifact
   + 14925b5:
     Always use target's attributes to set Python version (#16959)
   + a247502:
     Add 'toolchain' parameter to actions.{run,run_shell}
   + 497b7e4:
     Fix Bash `runfiles_current_repository` with RBE
   + 0a2c4ed:
     Fix corner cases in Bash runfiles library
   + 33b514b:
     Fix runfiles creation with MANIFEST when building without the
     bytes
   + 3a13af4:
     Remove LCOV merger dependency of `cc_test` without coverage
     (#17004)
   + 25ba76c:
     Include full tree artifact in inputs when prefetcher doesn't
     support partial tree artifacts.
   + 06deebf:
     Release 6.0.0 (2022-12-19)
   + c2bfb2e:
     Enable git_submodules tests and merge git_repository and
     starlark_git_repository (#17247)
   + e8182ce:
     have 'bazel test' non-test targets depend on
     --remote_download_output
   + c3245cd:
     Add `SpellChecker` suggestions for common Bzlmod errors
   + 8ec8743:
     Use xcrun to invoke install_name_tool
   + 2b2bea4:
     Extra resources
   + 5b4de12:
     Do not clear `--platforms` on no-op change to `--cpu`
   + db3c013:
     Report `AspectCreationException` to the user
   + 53e9fea:
     Use long executable path instead of `argv[0]` in all launchers
   + abae5ca:
     Add sanitizer features to unix_cc_toolchain_config
   + db68419:
     Allow `map_each` to return `None` in `TemplateDict#add_joined`
   + 66b58ee:
     Always emit trailing zero in human-readable download progress
   + 17b8e44:
     Upload all logs in BEP even with minimal upload
   + 28d2daa:
     Set `--experimental_action_listeners` to default in `exec` config
   + 16427c9:
     Do not count tests as failed that have not started
   + 094a0aa:
     Accept tildes in --override_module
   + 5f2866f:
     Do the AC integrity check for disk part of the combined cache.
   + cd10d50:
     Fix `--nozip_undeclared_test_outputs` on Windows
   + 073f54b:
     Allow pyd in extensions of dynamic libraries
   + ac504cb:
     Register JavaCompileActionContext in Bazel.
   + f090433:
     Rollback #14510 because it causes remote test execution to fail
   + 942ddda:
     Prepare backwards compatible usage of optional C++ toolchain
   + 88b51f5:
     Emit LLVM coverage for source file paths with a `tmp` segment
   + bb0b886:
     Enable archive_param_file feature by default for Windows
   + baf97c0:
     Allow `TemplateDict#map_each` callback to return a list of
     strings (#17306)
   + fcb0077:
     Only try to create groups of test actions in the ui.
   + 3c47b47:
     Find `libtool` when using `BAZEL_USE_CPP_ONLY_TOOLCHAIN`.
   + 1da8a82:
     Add -dead_strip in default opt link flags for darwin
   + b0c5eb3:
     Dont query remote cache but always use bytestream protocol
   + 00e9af1:
     Allow Java coverage collection for external targets.
   + dd24a00:
     Test and fix root symlink edge case in runfiles library
   + af97d98:
     [6.1.0] Fix broken CI (#17422)
   + b859571:
     Add `native.package_relative_label` function
   + 82168d4:
     Make Bazel more responsive and use less memory when --jobs is
     high (#17398)
   + 84c1ed4:
     Multiplex worker fixes and tests for Android busybox tools
     (#17371)
   + 0759081:
     Improve error message for concurrent modifications (#17396)
   + 27bc896:
     [6.1.0] Make bazel coverage work with minimal mode (#17397)
   + 544b816:
     [6.1.0] Fix cc_binary bug related to cc_shared_library on
     Windows and prepare for future removal of
     --experimental_cc_shared_library flag (#17445)
   + 9a7aa21:
     Fix Bash `runfiles_current_repository` for tools
   + 911eedc:
     Fix label unambiguous canonical form to correctly report
     non-visible repo names
   + e132653:
     Remove O1 from sanitizer feature flag defaults
   + ba9e2f8:
     Remove usage of gRPC Context cancellation in the remote
     execution client.
   + de03a23:
     Allow -v to libtool
   + 4e35c02:
     Prettify labels in action progress messages with Bzlmod (#17278)
   + 1be0ac3:
     Expand tree outputs before eagerly prefetching them for local
     actions. (#17494)
   + c266651:
     Fix Java coverage collection with Java 8 runtime
   + 1a6ffe6:
     Add a flag to disable execution log sorting.
   + fbec8e2:
     Reduce flakiness on Windows for BwoB tests
   + 420659a:
     Make AutoBazelRepositoryProcessor compatible with Java 8
   + dde6d20:
     Do not recommend `shallow_since` for `git_repository`
   + f76fc61:
     Allow the timeout value for execute calls on a mac to be set via
     an environment variable
   + 773d232:
     Fix RPATHs for cc toolchain solib when sibling layout is used
   + 5932b3b:
     [6.1.0] Add --host_features (#17528)
   + a0fa77c:
     Exit with code 39 if remote cache evicted blobs that Bazel need
     during an invocation (#17496)
   + 1a438b4:
     Only fetch @remote_coverage_tools when collecting coverage
     (#17512)
   + ee1daaf:
     [6.1.0]Only fetch @remote_coverage_tools when collecting
     coverage (#17287)
   + 350e329:
     [6.1.0]Fix symlink file creation overhead (#17488)
   + 5406c95:
     [6.1.0] Cleanup stale state when remote cache evicted (#17538)
   + 2d1b570:
     Bump version number in runfiles.bash init code
   + 3ab8a0a:
     [6.1.0]Let `aquery` print effective environment for all
     `CommandAction`s (#17274)
   + 51e6c38:
     Properly report repo fetch progress during main repo mapping
     computation
   + 744108e:
     [docs] Update migrating to Xcode tutorial (#17563)
   + 9dc7a7e:
     Update //third_party:jsr305 (#17557)
   + 43dadb2:
     Bump minimum supported macOS versions to 10.13
   + 7d9d23c:
     Correctly set default subprocess factory when loading class
     `SubprocessBuilder`.
   + 77f82f4:
     Add an --incompatible_strict_conflict_checks alias for
     --experimental_strict_conflict_checks.
   + e05345d:
     Add support for wrapping system streams in WorkRequestHandler
   + 68e1924:
     Also handle remote cache eviction for tree artifacts. (#17601)
   + 4a6d056:
     Add host transition to lcov_merger in Java version of py_test
   + 546e9e2:
     Fix Bazel 6.0 crash regression (#17613)
   + 7e328bb:
     Include Bazel version information in profile metadata. (#17616)
   + aafe123:
     [6.1.0] Handle remote cache eviction when uploading inputs for
     remote actions. (#17605)
   + 938e348:
     [6.1.0] Rerun the artifact conflict check when
     --incompatible_strict_conflict_checks changes. (#17592)
   + 034a281:
     Report background download for BwoB (#17619)
   + 5afbce5:
     [6.1.0] Flag for writable outputs (experimental) (#17617)
   + d4105e6:
     [6.1.0] Allow .wasm cc executable and dynamic library extension
     (#17440)
   + a306f4f:
     make desugar dependencies deterministic
   + 37953c5:
     Apply exec transition to lcov_merger in sh_test and cc_test
   + 1d73d72:
     [6.1.0]Fix --host_features with multiple transitions (#17641)
   + 755cd4b:
     Release 6.1.0 (2023-03-06)
   + 885ae7e:
     Revert "Add -dead_strip in default opt link flags for darwin
     (#17312)" (#17696)
   + 94c519b:
     Skip empty directories instead of throwing in prefetcher.
     (#17718)
   + 60749d5:
     [6.2.0] Remove actionId from RemoteFileArtifactValue. (#17724)
   + 52deefe:
     Correctly handle templated tree artifacts in the prefetcher.
   + 10587b1:
     Silence setlocale warnings in Java stub
   + 3ad3927:
     Force the Bazel server Java runtime to use the root locale
   + 7c235ff:
     [credentialhelper] Ignore failing to provide request
   + daa3dbe:
     [remote] upload: treat `ALREADY_EXISTS` as success
   + e0cdace:
     Fix data race in prefetcher.
   + c9e3eeb:
     [6.2.0] Update java_tools v11.12 (#17761)
   + 6c89303:
     python: Remove temporary module space created for zip-based
     binaries (#17764)
   + b874e5f:
     [6.2.0]Add test coverage support to android_local_test (#17467)
   + 6fc9bed:
     Fix wasm dynamic library extension crash (#17765)
   + 38ece3c:
     Handle `SIGWINCH` under `bazel run //some:test` (i.e. test
     rules) correctly.
   + 49a9502:
     Clear all remote metadata if any of them are evicted from remote
     cache
   + 8fab22e:
     Include build-tools/X.Y.Z/{lib,lib64}/** in the minimal set of
     SDK files used by the Android integration tests.
   + 3a7236b:
     Allow WORKSPACE and WORKSPACE-loaded .bzl files to see Bzlmod
     root module's mappings (#17818)
   + a87b8e0:
     [6.2.0]Add support for alias targets to cquery's `providers`
     (#17786)
   + ee32eff:
     [6.2.0] Allows --query_file to be used for cquery and aquery
     too. (#17823)
   + cfef67d:
     Fix seeking of empty chunkers.
   + 50ec6bb:
     Rework ByteStreamUploader early return logic.
   + 745ca28:
     Suggest alternatives for typoed rule attribute names
   + 8548e20:
     Relax JSON parser, ensure duplicate keys are overridden
   + c04f0d4:
     Use less subshells and `tee`s in running tests with `bazel run`.
   + 8c6d02e:
     Upgrade Azul JDK 17 archives. (#17852)
   + 3c1c0c0:
     Add suggestions to invalid tag class errors
   + 99b3f38:
     [6.2.0] Add native.module_{name,version} (#17893)
   + f7627e0:
     Support (workspace) relative paths in --override_module closes
     #17551
   + 109b290:
     Fail early if `use_extension` has a bad label
   + f63ce79:
     Avoid unnecessary copying when building Merkle trees.
   + 3ea18cc:
     [6.2.0]Add `module_ctx.is_dev_dependency` (#17934)
   + 2d04c91:
     [6.2.0] Add version to JavaRuntimeInfo (#17913)
   + 2c1a63b:
     Fix CI breakages for release-6.2.0 (#17969)
   + 788801a:
     Enable C++ deps pruning on Windows when PARSE_SHOWINCLUDES is
     available.
   + fb4a0c2:
     [6.2.0] TargetPattern parsing fixes (#17945)
   + 70ce837:
     Add attribute 'provides = [CcInfo]' to '_cc_proto_aspect'
   + 6e18829:
     [6.2.0]Expose cc_proto_aspect as a toplevel symbol. (#17980)
   + 0f55d12:
     Cache Merkle trees for tree artifacts.
   + 6d6fa81:
     Deduplicate concurrent computations of the same Merkle tree.
   + 1f2b3ed:
     Patch zlib to fix compatibility with latest Xcode
   + 27e4c62:
     Add exception message to 'failed to create output directory'
   + 4050120:
     Prevent failures creating output directories
   + 1474b5b:
     Allow multiple matching select branches if they resolve to the
     same value (#18066)
   + 5ddef47:
     Turn applicable_licenses on platform() into a no-op. (#18089)
   + e00509b:
     Use ctime in file digest cache key (#18105)
   + ca30372:
     Gracefully handle output symlinks with BwoB
   + a0cb57f:
     [6.2.0] profile: add profile_finish_ts (#18129)
   + 1a60fad:
     fix(python): Set envvar for runfiles manifest, not runfiles dir,
     when using a manifest (#18133)
   + 5e9fa39:
     Add attribute validation to IncompatibleTargetChecker. (#18135)
   + 97312f3:
     [6.2.0] Update CODEOWNERS (#18149)
   + 76ad4a9:
     [6.2.0]Fix worker and multiplex workers for DexBuilder and
     Desugar actions (#17965)
   + 5afb8b6:
     Lockfile cherry picks (#18143)
   + 1940dfb:
     Automatically retry the build if encountered remote cache
     eviction error (#18171)
   + 755cf95:
     [6.2.0]Allow remote retry max delay to be user configurable
     (#18061)
   + 6c61110:
     Add `module_ctx.extension_metadata` (#18174)
   + c1fea13:
     Introduce max_compatibility_level for bazel_dep (#18178)
   + f95b80d:
     Include cause when reporting `ActionExecutionException`
   + 8a41be9:
     Check for the nullness of AspectValue.
   + 2715120:
     Rename `--experimental_remote_grpc_log` to `--remote_grpc_log`
   + e4682f6:
     [6.2.0] Update java_tools v12.1 (#18197)
   + d94dee2:
     Construct TreeArtifactValues on multiple threads. (#18194)
   + 519eef4:
     Fix crash on multiple output symlinks with BwoB
   + 2442a2e:
     [6.2.0] default_java_toolchain.bzl cherry-picks to fix
     regression (#18225)
   + e4f3d4d:
     Fix message generation of `ActionExecutionException`
   + f39454f:
     Add `dev_dependency` to
     `register_{execution_platforms,toolchains}`
   + bf188c1:
     Fine tune the number of test jobs running in parallel to avoid
     timeout on Intel macOS platform
   + e024247:
     Update java_tools v12.2
   + 2dca982:
     Update java_tools v12.3

Incompatible changes:

  - Bazel no longer increases the delay between progress updates when
    there is no cursor control.
  - `--incompatible_always_include_files_in_data` is flipped
    to true. See #16654 for
    details.
  - `cquery --output=files` also outputs source files.

    Closes #16602.
  - `--incompatible_strict_conflict_checks` is flipped to true. See
    #16729 for details.
  - This changes the behavior of Python version in exec/host
    configuration. Mitigation is to set Python version on the targets.
  - `--features` only applies to targets built in the target
    configuration, and `--host_features` is used for the host / exec
    configuration (gated behind `--incompatible_use_host_features`)

    Fixes #13839

    Closes #16626.

New features:

  - The aquery and cquery commands now respect the --query_file flag
    just like the query command.

Important changes:

  - The new path variable `$(rlocationpath ...)` and its plural form
    `$(rlocationpaths ...)` can be used to expand labels to the paths
    accepted by the `Rlocation` function of runfiles libraries. This
    is the preferred way to access data dependencies at runtime and
    works on all platforms, even when runfiles are not enabled (e.g.,
    on Windows by default).

    Work towards #16124
    Fixes #10923

    Closes #16428.
  - Starlark `print()` statements are now emitted iff the line of
    code is executed. They are no longer replayed on subsequent
    invocations unless the Starlark code is re-executed.
    Additionally, multiple identical `print()` statements (same
    string from the same line of code, e.g. from a loop) are all
    emitted and no longer deduplicated.
  - Fixes a bug where some compilation flags would not be applied to
    a cc_test
  - Added a `native.package_relative_label()` function, which
    converts a label string to a Label object in the context of the
    calling package, in contrast to `Label()`, which does so in the
    context of the current .bzl file. Both functions now also accept
    relative labels such as `:foo`, and are idempotent.
  - Adds coverage metric support to android_local_test
  - Fixed an issue where WORKSPACE and WORKSPACE-loaded .bzl files
    couldn't see the Bzlmod root module's mappings when Bzlmod is
    enabled.
  - Added `native.module_name()` and `native.module_version()` to
    allow BUILD macro authors to acquire information about which
    Bazel module the current repo is associated with.
  - `@foo` labels can now be used on the command line as the
    top-level target (that is, `bazel build @foo` now works).
    Double-dot syntax is now forbidden (`bazel build ../foo` will no
    longer work).
  - Added preliminary support for a lockfile for Bzlmod. It's
    disabled by default; use `--lockfile_mode=update` to enable it.
    This lockfile contains only Bazel module information; it does not
    involve module extensions.
  - Added a new `max_compatibility_level` attribute to the
    `bazel_dep` directive, which allows version selection to upgrade
    a dependency up to the specified compatibility level.

    Co-authored-by: Brentley Jones <[email protected]>
  - `--experimental_remote_grpc_log` has been renamed to
    `--remote_grpc_log`

    Closes #18180.

This release contains contributions from many people at Google, as well as Ben Lee, Brentley Jones, Ezekiel Warren, Fabian Meumertzheim, keertk, Keith Smiley, Krzysztof Naglik, kshyanashree, lunch-glide-pepper, oquenchil, Patrick Balestra, Son Luong Ngoc, Ted Kaplan, Ted Kaplan, UebelAndre, Xùdōng Yáng, Yannic, yuzhy8701.
@lberki
Copy link
Contributor

lberki commented May 10, 2023

@chiragramani do you have large tree artifacts on the inputs or outputs of that action?

My guess is that this is caused by the naive way parallel checksumming just starts as many threads as there are cores, without much respect for anything else. Thus, if there is already high load (which is totally possible, given that it's the middle of a build), badness happens.

The band-aid here is to add a flag to control whether parallel checksumming should happen, but the principled solution would be to take a good hard look at how Bazel managed CPU-bound work. We should do that in any case because it would help in other scenarios, too, but maybe this is the bug that would provide the impetus to do so?

@meisterT WDYT?

@meisterT
Copy link
Member

I would hope that once we have migrated to JDK20, and start using loom things like this become a lot easier to manage. cc @coeuvre

sluongng pushed a commit to sluongng/bazel that referenced this pull request May 10, 2023
@keertk
Release 6.2.0 (2023-05-09)
758b44d 
Baseline: d60ce2c

Cherry picks:

   + 490f8ba:
     kshyanashree is the release manager of 6.0.0
   + 4e66d93:
     Automated rollback of commit
     2a28909.
   + 48a8d01:
     Allow C/C++ coverage collection for external targets
   + ec7be34:
     Temporarily set parent directory of the input to writable if it
     is not.
   + b098434:
     Infer CPUs for tvOS and watchOS when on Apple Silicon
   + 5cea7dd:
     Improve the documentation for ctx.actions.symlink.
   + a82d26f:
     Add quotes to "Tip"
   + 75b0f3a:
     Write a repo mapping manifest in the runfiles directory (bazelbuild#16555)
   + 07c5c1a:
     Ensure repository names don't start with `~`
   + 30f6c82:
     Escape tilde more gracefully
   + cf3f48c:
     Relax `Label` repo visibility validation
   + 4477823:
     Use "apparent name" instead of "local name" in Bzlmod docs
   + 55f4f48:
     Bazel DevSite: Add required css file.
   + a706994:
     Remove logic that increases delay between progress updates over
     time
   + 1d514ab:
     Remove references to io_bazel repository
   + b0357bd:
     Relnotes python script
   + 8f95651:
     Allow Java coverage collection for external targets
   + bddb191:
     Fix race condition in package-bazel.sh
   + a757852:
     Decanonicalize labels emitted by {a,c,}query if possible
   + 9d250ed:
     Add uniquify parameter to TemplateDict.add_joined
   + f02bcf8:
     Fix identical gcov json file name problem
   + a159330:
     Add `$(rlocationpath(s) ...)` expansion
   + 56f54da:
     Rollup of SBOM correctness fixes (bazelbuild#16655)
   + 72e6e94:
     [cc] Add cc toolchain to starlark cc_proto_library (bazelbuild#16661)
   + 8f28513:
     Make C++ runfiles library repo mapping aware
   + 95f9adc:
     Always collect FileProvider's filesToBuild as data runfiles
   + 4959d34:
     fix: export constraints.bzl file from @local_config_platform so
     it can be used in downstream bzl_library targets
   + 4919d4a:
     Add --host_per_file_copt (bazelbuild#16695)
   + 0a23d46:
     Moving some tests for `RemoteActionFileSystem` of BwoB to a base
     class.
   + 9296068:
     [remote/downloader] Don't include headers in `FetchBlobRequest`
   + 128d833:
     [remote] Respect whether the server supports action cache updates
   + 38c5019:
     [remote/downloader] Migrate `Downloader` to take `Credentials`
     (bazelbuild#16732)
   + 5929cb7:
     Stage repository mapping manifest as a root symlink (bazelbuild#16733)
   + 455454a:
     Expose current repository name to Java with @AutoBazelRepository
   + 97dea59:
     Implement getDirectoryEntries and readdir for
     RemoteActionFileSystem.
   + aa45f5f:
     Move integration tests for BwoB to a base class and add more
     tests there.
   + 1b073ac:
     Make Java runfiles library repo mapping aware
   + 148bbb1:
     Use `_repo_mapping` in C++ runfiles library (bazelbuild#16752)
   + 25558ad:
     Add --experimental_action_cache_store_output_metadata to the
     expansio…
   + 6e945e8:
     Treat `DEBUG` events as progress-like.
   + 1940c5d:
     redact 'token' strings from logging
   + 0b64525:
     Make Bash runfiles library repo mapping aware (bazelbuild#16753)
   + 4caae75:
     Automated rollback of commit
     8f95651.
   + 13ff6d9:
     Fix Bash rlocation failure with stricter Bash options
   + bd88c7e:
     Make bazel Android remote deps compatible with bzlmod (bazelbuild#16772)
   + 6f6d4cc:
     Flip `--incompatible_always_include_files_in_data`
   + 94355b1:
     Add required `--add-opens` server JVM args also with
     non-embedded JDK
   + 8349c95:
     Also collect clang resource directory with
     `-no-canonical-prefixes`
   + dce6ed7:
     Make `bazel run` works with minimal mode
   + ca8674c:
     Include source files with `cquery --output=files` (bazelbuild#16826)
   + 9cb5e0a:
     Fsync before rename after copy in DiskCacheClient
   + 23ffce5:
     Update GetActionResult for disk cache to check referenced files
     when …
   + 0bc0ff5:
     Make Java runfiles library compilable with JDK 8
   + fe16965:
     [6.0.0] Default --incompatible_strict_conflict_checks to true.
     (bazelbuild#16864)
   + 99ca86f:
     Revert "[6.0.0] Default --incompatible_strict_conflict_checks to
     true. (bazelbuild#16864)" (bazelbuild#16872)
   + 312238f:
     Make C++ runfiles library build with `-Werror=shadow`
   + 2baa5a4:
     Keep credentials cached across build commands.
   + 8818a57:
     [6.0.0] Only inject metadata for outputs that cannot be
     reconstructed by skyf… (bazelbuild#16879)
   + 31e4bf4:
     Update java_tools v11.9
   + cd40666:
     replace _get_canonical_form(label) with str(label)
   + e2bc237:
     Avoid exceptions from hermetic sandbox for unsupported artifact
     subclasses
   + b564d14:
     Remove warnings about ignored starlark options
   + 7ccc661:
     [6.0.0] Emit Tree objects in topological order (bazelbuild#16904)
   + 86dee6d:
     Correctly match regex with tree artifact
   + 14925b5:
     Always use target's attributes to set Python version (bazelbuild#16959)
   + a247502:
     Add 'toolchain' parameter to actions.{run,run_shell}
   + 497b7e4:
     Fix Bash `runfiles_current_repository` with RBE
   + 0a2c4ed:
     Fix corner cases in Bash runfiles library
   + 33b514b:
     Fix runfiles creation with MANIFEST when building without the
     bytes
   + 3a13af4:
     Remove LCOV merger dependency of `cc_test` without coverage
     (bazelbuild#17004)
   + 25ba76c:
     Include full tree artifact in inputs when prefetcher doesn't
     support partial tree artifacts.
   + 06deebf:
     Release 6.0.0 (2022-12-19)
   + c2bfb2e:
     Enable git_submodules tests and merge git_repository and
     starlark_git_repository (bazelbuild#17247)
   + e8182ce:
     have 'bazel test' non-test targets depend on
     --remote_download_output
   + c3245cd:
     Add `SpellChecker` suggestions for common Bzlmod errors
   + 8ec8743:
     Use xcrun to invoke install_name_tool
   + 2b2bea4:
     Extra resources
   + 5b4de12:
     Do not clear `--platforms` on no-op change to `--cpu`
   + db3c013:
     Report `AspectCreationException` to the user
   + 53e9fea:
     Use long executable path instead of `argv[0]` in all launchers
   + abae5ca:
     Add sanitizer features to unix_cc_toolchain_config
   + db68419:
     Allow `map_each` to return `None` in `TemplateDict#add_joined`
   + 66b58ee:
     Always emit trailing zero in human-readable download progress
   + 17b8e44:
     Upload all logs in BEP even with minimal upload
   + 28d2daa:
     Set `--experimental_action_listeners` to default in `exec` config
   + 16427c9:
     Do not count tests as failed that have not started
   + 094a0aa:
     Accept tildes in --override_module
   + 5f2866f:
     Do the AC integrity check for disk part of the combined cache.
   + cd10d50:
     Fix `--nozip_undeclared_test_outputs` on Windows
   + 073f54b:
     Allow pyd in extensions of dynamic libraries
   + ac504cb:
     Register JavaCompileActionContext in Bazel.
   + f090433:
     Rollback bazelbuild#14510 because it causes remote test execution to fail
   + 942ddda:
     Prepare backwards compatible usage of optional C++ toolchain
   + 88b51f5:
     Emit LLVM coverage for source file paths with a `tmp` segment
   + bb0b886:
     Enable archive_param_file feature by default for Windows
   + baf97c0:
     Allow `TemplateDict#map_each` callback to return a list of
     strings (bazelbuild#17306)
   + fcb0077:
     Only try to create groups of test actions in the ui.
   + 3c47b47:
     Find `libtool` when using `BAZEL_USE_CPP_ONLY_TOOLCHAIN`.
   + 1da8a82:
     Add -dead_strip in default opt link flags for darwin
   + b0c5eb3:
     Dont query remote cache but always use bytestream protocol
   + 00e9af1:
     Allow Java coverage collection for external targets.
   + dd24a00:
     Test and fix root symlink edge case in runfiles library
   + af97d98:
     [6.1.0] Fix broken CI (bazelbuild#17422)
   + b859571:
     Add `native.package_relative_label` function
   + 82168d4:
     Make Bazel more responsive and use less memory when --jobs is
     high (bazelbuild#17398)
   + 84c1ed4:
     Multiplex worker fixes and tests for Android busybox tools
     (bazelbuild#17371)
   + 0759081:
     Improve error message for concurrent modifications (bazelbuild#17396)
   + 27bc896:
     [6.1.0] Make bazel coverage work with minimal mode (bazelbuild#17397)
   + 544b816:
     [6.1.0] Fix cc_binary bug related to cc_shared_library on
     Windows and prepare for future removal of
     --experimental_cc_shared_library flag (bazelbuild#17445)
   + 9a7aa21:
     Fix Bash `runfiles_current_repository` for tools
   + 911eedc:
     Fix label unambiguous canonical form to correctly report
     non-visible repo names
   + e132653:
     Remove O1 from sanitizer feature flag defaults
   + ba9e2f8:
     Remove usage of gRPC Context cancellation in the remote
     execution client.
   + de03a23:
     Allow -v to libtool
   + 4e35c02:
     Prettify labels in action progress messages with Bzlmod (bazelbuild#17278)
   + 1be0ac3:
     Expand tree outputs before eagerly prefetching them for local
     actions. (bazelbuild#17494)
   + c266651:
     Fix Java coverage collection with Java 8 runtime
   + 1a6ffe6:
     Add a flag to disable execution log sorting.
   + fbec8e2:
     Reduce flakiness on Windows for BwoB tests
   + 420659a:
     Make AutoBazelRepositoryProcessor compatible with Java 8
   + dde6d20:
     Do not recommend `shallow_since` for `git_repository`
   + f76fc61:
     Allow the timeout value for execute calls on a mac to be set via
     an environment variable
   + 773d232:
     Fix RPATHs for cc toolchain solib when sibling layout is used
   + 5932b3b:
     [6.1.0] Add --host_features (bazelbuild#17528)
   + a0fa77c:
     Exit with code 39 if remote cache evicted blobs that Bazel need
     during an invocation (bazelbuild#17496)
   + 1a438b4:
     Only fetch @remote_coverage_tools when collecting coverage
     (bazelbuild#17512)
   + ee1daaf:
     [6.1.0]Only fetch @remote_coverage_tools when collecting
     coverage (bazelbuild#17287)
   + 350e329:
     [6.1.0]Fix symlink file creation overhead (bazelbuild#17488)
   + 5406c95:
     [6.1.0] Cleanup stale state when remote cache evicted (bazelbuild#17538)
   + 2d1b570:
     Bump version number in runfiles.bash init code
   + 3ab8a0a:
     [6.1.0]Let `aquery` print effective environment for all
     `CommandAction`s (bazelbuild#17274)
   + 51e6c38:
     Properly report repo fetch progress during main repo mapping
     computation
   + 744108e:
     [docs] Update migrating to Xcode tutorial (bazelbuild#17563)
   + 9dc7a7e:
     Update //third_party:jsr305 (bazelbuild#17557)
   + 43dadb2:
     Bump minimum supported macOS versions to 10.13
   + 7d9d23c:
     Correctly set default subprocess factory when loading class
     `SubprocessBuilder`.
   + 77f82f4:
     Add an --incompatible_strict_conflict_checks alias for
     --experimental_strict_conflict_checks.
   + e05345d:
     Add support for wrapping system streams in WorkRequestHandler
   + 68e1924:
     Also handle remote cache eviction for tree artifacts. (bazelbuild#17601)
   + 4a6d056:
     Add host transition to lcov_merger in Java version of py_test
   + 546e9e2:
     Fix Bazel 6.0 crash regression (bazelbuild#17613)
   + 7e328bb:
     Include Bazel version information in profile metadata. (bazelbuild#17616)
   + aafe123:
     [6.1.0] Handle remote cache eviction when uploading inputs for
     remote actions. (bazelbuild#17605)
   + 938e348:
     [6.1.0] Rerun the artifact conflict check when
     --incompatible_strict_conflict_checks changes. (bazelbuild#17592)
   + 034a281:
     Report background download for BwoB (bazelbuild#17619)
   + 5afbce5:
     [6.1.0] Flag for writable outputs (experimental) (bazelbuild#17617)
   + d4105e6:
     [6.1.0] Allow .wasm cc executable and dynamic library extension
     (bazelbuild#17440)
   + a306f4f:
     make desugar dependencies deterministic
   + 37953c5:
     Apply exec transition to lcov_merger in sh_test and cc_test
   + 1d73d72:
     [6.1.0]Fix --host_features with multiple transitions (bazelbuild#17641)
   + 755cd4b:
     Release 6.1.0 (2023-03-06)
   + 885ae7e:
     Revert "Add -dead_strip in default opt link flags for darwin
     (bazelbuild#17312)" (bazelbuild#17696)
   + 94c519b:
     Skip empty directories instead of throwing in prefetcher.
     (bazelbuild#17718)
   + 60749d5:
     [6.2.0] Remove actionId from RemoteFileArtifactValue. (bazelbuild#17724)
   + 52deefe:
     Correctly handle templated tree artifacts in the prefetcher.
   + 10587b1:
     Silence setlocale warnings in Java stub
   + 3ad3927:
     Force the Bazel server Java runtime to use the root locale
   + 7c235ff:
     [credentialhelper] Ignore failing to provide request
   + daa3dbe:
     [remote] upload: treat `ALREADY_EXISTS` as success
   + e0cdace:
     Fix data race in prefetcher.
   + c9e3eeb:
     [6.2.0] Update java_tools v11.12 (bazelbuild#17761)
   + 6c89303:
     python: Remove temporary module space created for zip-based
     binaries (bazelbuild#17764)
   + b874e5f:
     [6.2.0]Add test coverage support to android_local_test (bazelbuild#17467)
   + 6fc9bed:
     Fix wasm dynamic library extension crash (bazelbuild#17765)
   + 38ece3c:
     Handle `SIGWINCH` under `bazel run //some:test` (i.e. test
     rules) correctly.
   + 49a9502:
     Clear all remote metadata if any of them are evicted from remote
     cache
   + 8fab22e:
     Include build-tools/X.Y.Z/{lib,lib64}/** in the minimal set of
     SDK files used by the Android integration tests.
   + 3a7236b:
     Allow WORKSPACE and WORKSPACE-loaded .bzl files to see Bzlmod
     root module's mappings (bazelbuild#17818)
   + a87b8e0:
     [6.2.0]Add support for alias targets to cquery's `providers`
     (bazelbuild#17786)
   + ee32eff:
     [6.2.0] Allows --query_file to be used for cquery and aquery
     too. (bazelbuild#17823)
   + cfef67d:
     Fix seeking of empty chunkers.
   + 50ec6bb:
     Rework ByteStreamUploader early return logic.
   + 745ca28:
     Suggest alternatives for typoed rule attribute names
   + 8548e20:
     Relax JSON parser, ensure duplicate keys are overridden
   + c04f0d4:
     Use less subshells and `tee`s in running tests with `bazel run`.
   + 8c6d02e:
     Upgrade Azul JDK 17 archives. (bazelbuild#17852)
   + 3c1c0c0:
     Add suggestions to invalid tag class errors
   + 99b3f38:
     [6.2.0] Add native.module_{name,version} (bazelbuild#17893)
   + f7627e0:
     Support (workspace) relative paths in --override_module closes
     bazelbuild#17551
   + 109b290:
     Fail early if `use_extension` has a bad label
   + f63ce79:
     Avoid unnecessary copying when building Merkle trees.
   + 3ea18cc:
     [6.2.0]Add `module_ctx.is_dev_dependency` (bazelbuild#17934)
   + 2d04c91:
     [6.2.0] Add version to JavaRuntimeInfo (bazelbuild#17913)
   + 2c1a63b:
     Fix CI breakages for release-6.2.0 (bazelbuild#17969)
   + 788801a:
     Enable C++ deps pruning on Windows when PARSE_SHOWINCLUDES is
     available.
   + fb4a0c2:
     [6.2.0] TargetPattern parsing fixes (bazelbuild#17945)
   + 70ce837:
     Add attribute 'provides = [CcInfo]' to '_cc_proto_aspect'
   + 6e18829:
     [6.2.0]Expose cc_proto_aspect as a toplevel symbol. (bazelbuild#17980)
   + 0f55d12:
     Cache Merkle trees for tree artifacts.
   + 6d6fa81:
     Deduplicate concurrent computations of the same Merkle tree.
   + 1f2b3ed:
     Patch zlib to fix compatibility with latest Xcode
   + 27e4c62:
     Add exception message to 'failed to create output directory'
   + 4050120:
     Prevent failures creating output directories
   + 1474b5b:
     Allow multiple matching select branches if they resolve to the
     same value (bazelbuild#18066)
   + 5ddef47:
     Turn applicable_licenses on platform() into a no-op. (bazelbuild#18089)
   + e00509b:
     Use ctime in file digest cache key (bazelbuild#18105)
   + ca30372:
     Gracefully handle output symlinks with BwoB
   + a0cb57f:
     [6.2.0] profile: add profile_finish_ts (bazelbuild#18129)
   + 1a60fad:
     fix(python): Set envvar for runfiles manifest, not runfiles dir,
     when using a manifest (bazelbuild#18133)
   + 5e9fa39:
     Add attribute validation to IncompatibleTargetChecker. (bazelbuild#18135)
   + 97312f3:
     [6.2.0] Update CODEOWNERS (bazelbuild#18149)
   + 76ad4a9:
     [6.2.0]Fix worker and multiplex workers for DexBuilder and
     Desugar actions (bazelbuild#17965)
   + 5afb8b6:
     Lockfile cherry picks (bazelbuild#18143)
   + 1940dfb:
     Automatically retry the build if encountered remote cache
     eviction error (bazelbuild#18171)
   + 755cf95:
     [6.2.0]Allow remote retry max delay to be user configurable
     (bazelbuild#18061)
   + 6c61110:
     Add `module_ctx.extension_metadata` (bazelbuild#18174)
   + c1fea13:
     Introduce max_compatibility_level for bazel_dep (bazelbuild#18178)
   + f95b80d:
     Include cause when reporting `ActionExecutionException`
   + 8a41be9:
     Check for the nullness of AspectValue.
   + 2715120:
     Rename `--experimental_remote_grpc_log` to `--remote_grpc_log`
   + e4682f6:
     [6.2.0] Update java_tools v12.1 (bazelbuild#18197)
   + d94dee2:
     Construct TreeArtifactValues on multiple threads. (bazelbuild#18194)
   + 519eef4:
     Fix crash on multiple output symlinks with BwoB
   + 2442a2e:
     [6.2.0] default_java_toolchain.bzl cherry-picks to fix
     regression (bazelbuild#18225)
   + e4f3d4d:
     Fix message generation of `ActionExecutionException`
   + f39454f:
     Add `dev_dependency` to
     `register_{execution_platforms,toolchains}`
   + bf188c1:
     Fine tune the number of test jobs running in parallel to avoid
     timeout on Intel macOS platform
   + e024247:
     Update java_tools v12.2
   + 2dca982:
     Update java_tools v12.3

Incompatible changes:

  - Bazel no longer increases the delay between progress updates when
    there is no cursor control.
  - `--incompatible_always_include_files_in_data` is flipped
    to true. See bazelbuild#16654 for
    details.
  - `cquery --output=files` also outputs source files.

    Closes bazelbuild#16602.
  - `--incompatible_strict_conflict_checks` is flipped to true. See
    bazelbuild#16729 for details.
  - This changes the behavior of Python version in exec/host
    configuration. Mitigation is to set Python version on the targets.
  - `--features` only applies to targets built in the target
    configuration, and `--host_features` is used for the host / exec
    configuration (gated behind `--incompatible_use_host_features`)

    Fixes bazelbuild#13839

    Closes bazelbuild#16626.

New features:

  - The aquery and cquery commands now respect the --query_file flag
    just like the query command.

Important changes:

  - The new path variable `$(rlocationpath ...)` and its plural form
    `$(rlocationpaths ...)` can be used to expand labels to the paths
    accepted by the `Rlocation` function of runfiles libraries. This
    is the preferred way to access data dependencies at runtime and
    works on all platforms, even when runfiles are not enabled (e.g.,
    on Windows by default).

    Work towards bazelbuild#16124
    Fixes bazelbuild#10923

    Closes bazelbuild#16428.
  - Starlark `print()` statements are now emitted iff the line of
    code is executed. They are no longer replayed on subsequent
    invocations unless the Starlark code is re-executed.
    Additionally, multiple identical `print()` statements (same
    string from the same line of code, e.g. from a loop) are all
    emitted and no longer deduplicated.
  - Fixes a bug where some compilation flags would not be applied to
    a cc_test
  - Added a `native.package_relative_label()` function, which
    converts a label string to a Label object in the context of the
    calling package, in contrast to `Label()`, which does so in the
    context of the current .bzl file. Both functions now also accept
    relative labels such as `:foo`, and are idempotent.
  - Adds coverage metric support to android_local_test
  - Fixed an issue where WORKSPACE and WORKSPACE-loaded .bzl files
    couldn't see the Bzlmod root module's mappings when Bzlmod is
    enabled.
  - Added `native.module_name()` and `native.module_version()` to
    allow BUILD macro authors to acquire information about which
    Bazel module the current repo is associated with.
  - `@foo` labels can now be used on the command line as the
    top-level target (that is, `bazel build @foo` now works).
    Double-dot syntax is now forbidden (`bazel build ../foo` will no
    longer work).
  - Added preliminary support for a lockfile for Bzlmod. It's
    disabled by default; use `--lockfile_mode=update` to enable it.
    This lockfile contains only Bazel module information; it does not
    involve module extensions.
  - Added a new `max_compatibility_level` attribute to the
    `bazel_dep` directive, which allows version selection to upgrade
    a dependency up to the specified compatibility level.

    Co-authored-by: Brentley Jones <[email protected]>
  - `--experimental_remote_grpc_log` has been renamed to
    `--remote_grpc_log`

    Closes bazelbuild#18180.

This release contains contributions from many people at Google, as well as Ben Lee, Brentley Jones, Ezekiel Warren, Fabian Meumertzheim, keertk, Keith Smiley, Krzysztof Naglik, kshyanashree, lunch-glide-pepper, oquenchil, Patrick Balestra, Son Luong Ngoc, Ted Kaplan, Ted Kaplan, UebelAndre, Xùdōng Yáng, Yannic, yuzhy8701.
fweikert pushed a commit to fweikert/bazel that referenced this pull request May 25, 2023
@fweikert
Release 6.2.0 (2023-05-09)
1a333eb 
Baseline: d60ce2c

Cherry picks:

   + 490f8ba:
     kshyanashree is the release manager of 6.0.0
   + 4e66d93:
     Automated rollback of commit
     2a28909.
   + 48a8d01:
     Allow C/C++ coverage collection for external targets
   + ec7be34:
     Temporarily set parent directory of the input to writable if it
     is not.
   + b098434:
     Infer CPUs for tvOS and watchOS when on Apple Silicon
   + 5cea7dd:
     Improve the documentation for ctx.actions.symlink.
   + a82d26f:
     Add quotes to "Tip"
   + 75b0f3a:
     Write a repo mapping manifest in the runfiles directory (bazelbuild#16555)
   + 07c5c1a:
     Ensure repository names don't start with `~`
   + 30f6c82:
     Escape tilde more gracefully
   + cf3f48c:
     Relax `Label` repo visibility validation
   + 4477823:
     Use "apparent name" instead of "local name" in Bzlmod docs
   + 55f4f48:
     Bazel DevSite: Add required css file.
   + a706994:
     Remove logic that increases delay between progress updates over
     time
   + 1d514ab:
     Remove references to io_bazel repository
   + b0357bd:
     Relnotes python script
   + 8f95651:
     Allow Java coverage collection for external targets
   + bddb191:
     Fix race condition in package-bazel.sh
   + a757852:
     Decanonicalize labels emitted by {a,c,}query if possible
   + 9d250ed:
     Add uniquify parameter to TemplateDict.add_joined
   + f02bcf8:
     Fix identical gcov json file name problem
   + a159330:
     Add `$(rlocationpath(s) ...)` expansion
   + 56f54da:
     Rollup of SBOM correctness fixes (bazelbuild#16655)
   + 72e6e94:
     [cc] Add cc toolchain to starlark cc_proto_library (bazelbuild#16661)
   + 8f28513:
     Make C++ runfiles library repo mapping aware
   + 95f9adc:
     Always collect FileProvider's filesToBuild as data runfiles
   + 4959d34:
     fix: export constraints.bzl file from @local_config_platform so
     it can be used in downstream bzl_library targets
   + 4919d4a:
     Add --host_per_file_copt (bazelbuild#16695)
   + 0a23d46:
     Moving some tests for `RemoteActionFileSystem` of BwoB to a base
     class.
   + 9296068:
     [remote/downloader] Don't include headers in `FetchBlobRequest`
   + 128d833:
     [remote] Respect whether the server supports action cache updates
   + 38c5019:
     [remote/downloader] Migrate `Downloader` to take `Credentials`
     (bazelbuild#16732)
   + 5929cb7:
     Stage repository mapping manifest as a root symlink (bazelbuild#16733)
   + 455454a:
     Expose current repository name to Java with @AutoBazelRepository
   + 97dea59:
     Implement getDirectoryEntries and readdir for
     RemoteActionFileSystem.
   + aa45f5f:
     Move integration tests for BwoB to a base class and add more
     tests there.
   + 1b073ac:
     Make Java runfiles library repo mapping aware
   + 148bbb1:
     Use `_repo_mapping` in C++ runfiles library (bazelbuild#16752)
   + 25558ad:
     Add --experimental_action_cache_store_output_metadata to the
     expansio…
   + 6e945e8:
     Treat `DEBUG` events as progress-like.
   + 1940c5d:
     redact 'token' strings from logging
   + 0b64525:
     Make Bash runfiles library repo mapping aware (bazelbuild#16753)
   + 4caae75:
     Automated rollback of commit
     8f95651.
   + 13ff6d9:
     Fix Bash rlocation failure with stricter Bash options
   + bd88c7e:
     Make bazel Android remote deps compatible with bzlmod (bazelbuild#16772)
   + 6f6d4cc:
     Flip `--incompatible_always_include_files_in_data`
   + 94355b1:
     Add required `--add-opens` server JVM args also with
     non-embedded JDK
   + 8349c95:
     Also collect clang resource directory with
     `-no-canonical-prefixes`
   + dce6ed7:
     Make `bazel run` works with minimal mode
   + ca8674c:
     Include source files with `cquery --output=files` (bazelbuild#16826)
   + 9cb5e0a:
     Fsync before rename after copy in DiskCacheClient
   + 23ffce5:
     Update GetActionResult for disk cache to check referenced files
     when …
   + 0bc0ff5:
     Make Java runfiles library compilable with JDK 8
   + fe16965:
     [6.0.0] Default --incompatible_strict_conflict_checks to true.
     (bazelbuild#16864)
   + 99ca86f:
     Revert "[6.0.0] Default --incompatible_strict_conflict_checks to
     true. (bazelbuild#16864)" (bazelbuild#16872)
   + 312238f:
     Make C++ runfiles library build with `-Werror=shadow`
   + 2baa5a4:
     Keep credentials cached across build commands.
   + 8818a57:
     [6.0.0] Only inject metadata for outputs that cannot be
     reconstructed by skyf… (bazelbuild#16879)
   + 31e4bf4:
     Update java_tools v11.9
   + cd40666:
     replace _get_canonical_form(label) with str(label)
   + e2bc237:
     Avoid exceptions from hermetic sandbox for unsupported artifact
     subclasses
   + b564d14:
     Remove warnings about ignored starlark options
   + 7ccc661:
     [6.0.0] Emit Tree objects in topological order (bazelbuild#16904)
   + 86dee6d:
     Correctly match regex with tree artifact
   + 14925b5:
     Always use target's attributes to set Python version (bazelbuild#16959)
   + a247502:
     Add 'toolchain' parameter to actions.{run,run_shell}
   + 497b7e4:
     Fix Bash `runfiles_current_repository` with RBE
   + 0a2c4ed:
     Fix corner cases in Bash runfiles library
   + 33b514b:
     Fix runfiles creation with MANIFEST when building without the
     bytes
   + 3a13af4:
     Remove LCOV merger dependency of `cc_test` without coverage
     (bazelbuild#17004)
   + 25ba76c:
     Include full tree artifact in inputs when prefetcher doesn't
     support partial tree artifacts.
   + 06deebf:
     Release 6.0.0 (2022-12-19)
   + c2bfb2e:
     Enable git_submodules tests and merge git_repository and
     starlark_git_repository (bazelbuild#17247)
   + e8182ce:
     have 'bazel test' non-test targets depend on
     --remote_download_output
   + c3245cd:
     Add `SpellChecker` suggestions for common Bzlmod errors
   + 8ec8743:
     Use xcrun to invoke install_name_tool
   + 2b2bea4:
     Extra resources
   + 5b4de12:
     Do not clear `--platforms` on no-op change to `--cpu`
   + db3c013:
     Report `AspectCreationException` to the user
   + 53e9fea:
     Use long executable path instead of `argv[0]` in all launchers
   + abae5ca:
     Add sanitizer features to unix_cc_toolchain_config
   + db68419:
     Allow `map_each` to return `None` in `TemplateDict#add_joined`
   + 66b58ee:
     Always emit trailing zero in human-readable download progress
   + 17b8e44:
     Upload all logs in BEP even with minimal upload
   + 28d2daa:
     Set `--experimental_action_listeners` to default in `exec` config
   + 16427c9:
     Do not count tests as failed that have not started
   + 094a0aa:
     Accept tildes in --override_module
   + 5f2866f:
     Do the AC integrity check for disk part of the combined cache.
   + cd10d50:
     Fix `--nozip_undeclared_test_outputs` on Windows
   + 073f54b:
     Allow pyd in extensions of dynamic libraries
   + ac504cb:
     Register JavaCompileActionContext in Bazel.
   + f090433:
     Rollback bazelbuild#14510 because it causes remote test execution to fail
   + 942ddda:
     Prepare backwards compatible usage of optional C++ toolchain
   + 88b51f5:
     Emit LLVM coverage for source file paths with a `tmp` segment
   + bb0b886:
     Enable archive_param_file feature by default for Windows
   + baf97c0:
     Allow `TemplateDict#map_each` callback to return a list of
     strings (bazelbuild#17306)
   + fcb0077:
     Only try to create groups of test actions in the ui.
   + 3c47b47:
     Find `libtool` when using `BAZEL_USE_CPP_ONLY_TOOLCHAIN`.
   + 1da8a82:
     Add -dead_strip in default opt link flags for darwin
   + b0c5eb3:
     Dont query remote cache but always use bytestream protocol
   + 00e9af1:
     Allow Java coverage collection for external targets.
   + dd24a00:
     Test and fix root symlink edge case in runfiles library
   + af97d98:
     [6.1.0] Fix broken CI (bazelbuild#17422)
   + b859571:
     Add `native.package_relative_label` function
   + 82168d4:
     Make Bazel more responsive and use less memory when --jobs is
     high (bazelbuild#17398)
   + 84c1ed4:
     Multiplex worker fixes and tests for Android busybox tools
     (bazelbuild#17371)
   + 0759081:
     Improve error message for concurrent modifications (bazelbuild#17396)
   + 27bc896:
     [6.1.0] Make bazel coverage work with minimal mode (bazelbuild#17397)
   + 544b816:
     [6.1.0] Fix cc_binary bug related to cc_shared_library on
     Windows and prepare for future removal of
     --experimental_cc_shared_library flag (bazelbuild#17445)
   + 9a7aa21:
     Fix Bash `runfiles_current_repository` for tools
   + 911eedc:
     Fix label unambiguous canonical form to correctly report
     non-visible repo names
   + e132653:
     Remove O1 from sanitizer feature flag defaults
   + ba9e2f8:
     Remove usage of gRPC Context cancellation in the remote
     execution client.
   + de03a23:
     Allow -v to libtool
   + 4e35c02:
     Prettify labels in action progress messages with Bzlmod (bazelbuild#17278)
   + 1be0ac3:
     Expand tree outputs before eagerly prefetching them for local
     actions. (bazelbuild#17494)
   + c266651:
     Fix Java coverage collection with Java 8 runtime
   + 1a6ffe6:
     Add a flag to disable execution log sorting.
   + fbec8e2:
     Reduce flakiness on Windows for BwoB tests
   + 420659a:
     Make AutoBazelRepositoryProcessor compatible with Java 8
   + dde6d20:
     Do not recommend `shallow_since` for `git_repository`
   + f76fc61:
     Allow the timeout value for execute calls on a mac to be set via
     an environment variable
   + 773d232:
     Fix RPATHs for cc toolchain solib when sibling layout is used
   + 5932b3b:
     [6.1.0] Add --host_features (bazelbuild#17528)
   + a0fa77c:
     Exit with code 39 if remote cache evicted blobs that Bazel need
     during an invocation (bazelbuild#17496)
   + 1a438b4:
     Only fetch @remote_coverage_tools when collecting coverage
     (bazelbuild#17512)
   + ee1daaf:
     [6.1.0]Only fetch @remote_coverage_tools when collecting
     coverage (bazelbuild#17287)
   + 350e329:
     [6.1.0]Fix symlink file creation overhead (bazelbuild#17488)
   + 5406c95:
     [6.1.0] Cleanup stale state when remote cache evicted (bazelbuild#17538)
   + 2d1b570:
     Bump version number in runfiles.bash init code
   + 3ab8a0a:
     [6.1.0]Let `aquery` print effective environment for all
     `CommandAction`s (bazelbuild#17274)
   + 51e6c38:
     Properly report repo fetch progress during main repo mapping
     computation
   + 744108e:
     [docs] Update migrating to Xcode tutorial (bazelbuild#17563)
   + 9dc7a7e:
     Update //third_party:jsr305 (bazelbuild#17557)
   + 43dadb2:
     Bump minimum supported macOS versions to 10.13
   + 7d9d23c:
     Correctly set default subprocess factory when loading class
     `SubprocessBuilder`.
   + 77f82f4:
     Add an --incompatible_strict_conflict_checks alias for
     --experimental_strict_conflict_checks.
   + e05345d:
     Add support for wrapping system streams in WorkRequestHandler
   + 68e1924:
     Also handle remote cache eviction for tree artifacts. (bazelbuild#17601)
   + 4a6d056:
     Add host transition to lcov_merger in Java version of py_test
   + 546e9e2:
     Fix Bazel 6.0 crash regression (bazelbuild#17613)
   + 7e328bb:
     Include Bazel version information in profile metadata. (bazelbuild#17616)
   + aafe123:
     [6.1.0] Handle remote cache eviction when uploading inputs for
     remote actions. (bazelbuild#17605)
   + 938e348:
     [6.1.0] Rerun the artifact conflict check when
     --incompatible_strict_conflict_checks changes. (bazelbuild#17592)
   + 034a281:
     Report background download for BwoB (bazelbuild#17619)
   + 5afbce5:
     [6.1.0] Flag for writable outputs (experimental) (bazelbuild#17617)
   + d4105e6:
     [6.1.0] Allow .wasm cc executable and dynamic library extension
     (bazelbuild#17440)
   + a306f4f:
     make desugar dependencies deterministic
   + 37953c5:
     Apply exec transition to lcov_merger in sh_test and cc_test
   + 1d73d72:
     [6.1.0]Fix --host_features with multiple transitions (bazelbuild#17641)
   + 755cd4b:
     Release 6.1.0 (2023-03-06)
   + 885ae7e:
     Revert "Add -dead_strip in default opt link flags for darwin
     (bazelbuild#17312)" (bazelbuild#17696)
   + 94c519b:
     Skip empty directories instead of throwing in prefetcher.
     (bazelbuild#17718)
   + 60749d5:
     [6.2.0] Remove actionId from RemoteFileArtifactValue. (bazelbuild#17724)
   + 52deefe:
     Correctly handle templated tree artifacts in the prefetcher.
   + 10587b1:
     Silence setlocale warnings in Java stub
   + 3ad3927:
     Force the Bazel server Java runtime to use the root locale
   + 7c235ff:
     [credentialhelper] Ignore failing to provide request
   + daa3dbe:
     [remote] upload: treat `ALREADY_EXISTS` as success
   + e0cdace:
     Fix data race in prefetcher.
   + c9e3eeb:
     [6.2.0] Update java_tools v11.12 (bazelbuild#17761)
   + 6c89303:
     python: Remove temporary module space created for zip-based
     binaries (bazelbuild#17764)
   + b874e5f:
     [6.2.0]Add test coverage support to android_local_test (bazelbuild#17467)
   + 6fc9bed:
     Fix wasm dynamic library extension crash (bazelbuild#17765)
   + 38ece3c:
     Handle `SIGWINCH` under `bazel run //some:test` (i.e. test
     rules) correctly.
   + 49a9502:
     Clear all remote metadata if any of them are evicted from remote
     cache
   + 8fab22e:
     Include build-tools/X.Y.Z/{lib,lib64}/** in the minimal set of
     SDK files used by the Android integration tests.
   + 3a7236b:
     Allow WORKSPACE and WORKSPACE-loaded .bzl files to see Bzlmod
     root module's mappings (bazelbuild#17818)
   + a87b8e0:
     [6.2.0]Add support for alias targets to cquery's `providers`
     (bazelbuild#17786)
   + ee32eff:
     [6.2.0] Allows --query_file to be used for cquery and aquery
     too. (bazelbuild#17823)
   + cfef67d:
     Fix seeking of empty chunkers.
   + 50ec6bb:
     Rework ByteStreamUploader early return logic.
   + 745ca28:
     Suggest alternatives for typoed rule attribute names
   + 8548e20:
     Relax JSON parser, ensure duplicate keys are overridden
   + c04f0d4:
     Use less subshells and `tee`s in running tests with `bazel run`.
   + 8c6d02e:
     Upgrade Azul JDK 17 archives. (bazelbuild#17852)
   + 3c1c0c0:
     Add suggestions to invalid tag class errors
   + 99b3f38:
     [6.2.0] Add native.module_{name,version} (bazelbuild#17893)
   + f7627e0:
     Support (workspace) relative paths in --override_module closes
     bazelbuild#17551
   + 109b290:
     Fail early if `use_extension` has a bad label
   + f63ce79:
     Avoid unnecessary copying when building Merkle trees.
   + 3ea18cc:
     [6.2.0]Add `module_ctx.is_dev_dependency` (bazelbuild#17934)
   + 2d04c91:
     [6.2.0] Add version to JavaRuntimeInfo (bazelbuild#17913)
   + 2c1a63b:
     Fix CI breakages for release-6.2.0 (bazelbuild#17969)
   + 788801a:
     Enable C++ deps pruning on Windows when PARSE_SHOWINCLUDES is
     available.
   + fb4a0c2:
     [6.2.0] TargetPattern parsing fixes (bazelbuild#17945)
   + 70ce837:
     Add attribute 'provides = [CcInfo]' to '_cc_proto_aspect'
   + 6e18829:
     [6.2.0]Expose cc_proto_aspect as a toplevel symbol. (bazelbuild#17980)
   + 0f55d12:
     Cache Merkle trees for tree artifacts.
   + 6d6fa81:
     Deduplicate concurrent computations of the same Merkle tree.
   + 1f2b3ed:
     Patch zlib to fix compatibility with latest Xcode
   + 27e4c62:
     Add exception message to 'failed to create output directory'
   + 4050120:
     Prevent failures creating output directories
   + 1474b5b:
     Allow multiple matching select branches if they resolve to the
     same value (bazelbuild#18066)
   + 5ddef47:
     Turn applicable_licenses on platform() into a no-op. (bazelbuild#18089)
   + e00509b:
     Use ctime in file digest cache key (bazelbuild#18105)
   + ca30372:
     Gracefully handle output symlinks with BwoB
   + a0cb57f:
     [6.2.0] profile: add profile_finish_ts (bazelbuild#18129)
   + 1a60fad:
     fix(python): Set envvar for runfiles manifest, not runfiles dir,
     when using a manifest (bazelbuild#18133)
   + 5e9fa39:
     Add attribute validation to IncompatibleTargetChecker. (bazelbuild#18135)
   + 97312f3:
     [6.2.0] Update CODEOWNERS (bazelbuild#18149)
   + 76ad4a9:
     [6.2.0]Fix worker and multiplex workers for DexBuilder and
     Desugar actions (bazelbuild#17965)
   + 5afb8b6:
     Lockfile cherry picks (bazelbuild#18143)
   + 1940dfb:
     Automatically retry the build if encountered remote cache
     eviction error (bazelbuild#18171)
   + 755cf95:
     [6.2.0]Allow remote retry max delay to be user configurable
     (bazelbuild#18061)
   + 6c61110:
     Add `module_ctx.extension_metadata` (bazelbuild#18174)
   + c1fea13:
     Introduce max_compatibility_level for bazel_dep (bazelbuild#18178)
   + f95b80d:
     Include cause when reporting `ActionExecutionException`
   + 8a41be9:
     Check for the nullness of AspectValue.
   + 2715120:
     Rename `--experimental_remote_grpc_log` to `--remote_grpc_log`
   + e4682f6:
     [6.2.0] Update java_tools v12.1 (bazelbuild#18197)
   + d94dee2:
     Construct TreeArtifactValues on multiple threads. (bazelbuild#18194)
   + 519eef4:
     Fix crash on multiple output symlinks with BwoB
   + 2442a2e:
     [6.2.0] default_java_toolchain.bzl cherry-picks to fix
     regression (bazelbuild#18225)
   + e4f3d4d:
     Fix message generation of `ActionExecutionException`
   + f39454f:
     Add `dev_dependency` to
     `register_{execution_platforms,toolchains}`
   + bf188c1:
     Fine tune the number of test jobs running in parallel to avoid
     timeout on Intel macOS platform
   + e024247:
     Update java_tools v12.2
   + 2dca982:
     Update java_tools v12.3

Incompatible changes:

  - Bazel no longer increases the delay between progress updates when
    there is no cursor control.
  - `--incompatible_always_include_files_in_data` is flipped
    to true. See bazelbuild#16654 for
    details.
  - `cquery --output=files` also outputs source files.

    Closes bazelbuild#16602.
  - `--incompatible_strict_conflict_checks` is flipped to true. See
    bazelbuild#16729 for details.
  - This changes the behavior of Python version in exec/host
    configuration. Mitigation is to set Python version on the targets.
  - `--features` only applies to targets built in the target
    configuration, and `--host_features` is used for the host / exec
    configuration (gated behind `--incompatible_use_host_features`)

    Fixes bazelbuild#13839

    Closes bazelbuild#16626.

New features:

  - The aquery and cquery commands now respect the --query_file flag
    just like the query command.

Important changes:

  - The new path variable `$(rlocationpath ...)` and its plural form
    `$(rlocationpaths ...)` can be used to expand labels to the paths
    accepted by the `Rlocation` function of runfiles libraries. This
    is the preferred way to access data dependencies at runtime and
    works on all platforms, even when runfiles are not enabled (e.g.,
    on Windows by default).

    Work towards bazelbuild#16124
    Fixes bazelbuild#10923

    Closes bazelbuild#16428.
  - Starlark `print()` statements are now emitted iff the line of
    code is executed. They are no longer replayed on subsequent
    invocations unless the Starlark code is re-executed.
    Additionally, multiple identical `print()` statements (same
    string from the same line of code, e.g. from a loop) are all
    emitted and no longer deduplicated.
  - Fixes a bug where some compilation flags would not be applied to
    a cc_test
  - Added a `native.package_relative_label()` function, which
    converts a label string to a Label object in the context of the
    calling package, in contrast to `Label()`, which does so in the
    context of the current .bzl file. Both functions now also accept
    relative labels such as `:foo`, and are idempotent.
  - Adds coverage metric support to android_local_test
  - Fixed an issue where WORKSPACE and WORKSPACE-loaded .bzl files
    couldn't see the Bzlmod root module's mappings when Bzlmod is
    enabled.
  - Added `native.module_name()` and `native.module_version()` to
    allow BUILD macro authors to acquire information about which
    Bazel module the current repo is associated with.
  - `@foo` labels can now be used on the command line as the
    top-level target (that is, `bazel build @foo` now works).
    Double-dot syntax is now forbidden (`bazel build ../foo` will no
    longer work).
  - Added preliminary support for a lockfile for Bzlmod. It's
    disabled by default; use `--lockfile_mode=update` to enable it.
    This lockfile contains only Bazel module information; it does not
    involve module extensions.
  - Added a new `max_compatibility_level` attribute to the
    `bazel_dep` directive, which allows version selection to upgrade
    a dependency up to the specified compatibility level.

    Co-authored-by: Brentley Jones <[email protected]>
  - `--experimental_remote_grpc_log` has been renamed to
    `--remote_grpc_log`

    Closes bazelbuild#18180.

This release contains contributions from many people at Google, as well as Ben Lee, Brentley Jones, Ezekiel Warren, Fabian Meumertzheim, keertk, Keith Smiley, Krzysztof Naglik, kshyanashree, lunch-glide-pepper, oquenchil, Patrick Balestra, Son Luong Ngoc, Ted Kaplan, Ted Kaplan, UebelAndre, Xùdōng Yáng, Yannic, yuzhy8701.
This was referenced May 30, 2023
Merged
Merged
@chiragramani chiragramani mentioned this pull request Jun 8, 2023
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@keertk keertk keertk approved these changes

@lberki lberki lberki approved these changes

Assignees
No one assigned
Labels
team-Remote-Exec Issues and PRs for the Execution (Remote) team
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@BalestraPatrick @lberki @chiragramani @keertk @meteorcloudy @meisterT