Bazel 6.0.0rc4 breaks Bazel RBE tests #16933
Comments
|
This issue also happens with Bazel@HEAD: https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/2766#0184e535-8df3-4114-9f65-8777f5bc7fe4, but the Bazel Auto Sheriff points to a commit that's not in Bazel 6.0.0rc4: https://buildkite.com/bazel/bazel-auto-sheriff-face-with-cowboy-hat/builds/1118 Update: culprit finder failed due to not being able to download the Bazel binary. We should fix culprit finder |
|
Bisecting the rc4 branch at https://buildkite.com/bazel/culprit-finder/builds/3876 and it points to 0b64525 |
… commit Related bazelbuild/bazel#16933 (comment) - The "Publish Bazel binaries" pipeline sometimes fails to publish Bazel binaries for a commit, in this case, the Culprit Finder should immediately fail instead consider the commit as a bad Bazel commit. - Also removed some unused parameters in the function.
|
Bisecting again on master branch after fixing the CI issue: https://buildkite.com/bazel/culprit-finder/builds/3877 |
|
While waiting for the bisect result, I can also reproduce at 839ce7f (original commit of the 6.0.0 cherrypick 0b64525) But not at it's parent commit. |
|
@meteorcloudy Could you run these tests with |
|
@fmeum Here is the test log: |
|
@coeuvre I suspected a different issue, so I would like to confirm that isn't a problem either: The Bash runfiles library currently relies on absolute (!) paths of generated files containing an |
|
The logic for absolute paths passed to |
My gut feeling is that |
|
@coeuvre Thanks, I will remove that assumption in a second PR. @meteorcloudy The Bash runfiles test didn't cover the exit codes of |
|
I think I know how to fix the RBE test failures, but now that I studied this more closely, there is some underlying inconsistent behavior that makes the library difficult to test and potentially also hard to use: @laszlocsomor Sorry for the sideline mention, but can you recall how you intended the exit code of Bash |
Did you mean |
|
@Wyverald Yes, sorry, fixed. |
|
Fixing this inconsistency isn't required to get the tests to pass, but if we want to do it, 6.0.0 is probably the right time. Since @Wyverald @meteorcloudy What do you think? |
Due to tests not actually verifying the exit code of rlocation calls, a few special cases were not handled correctly. In particular, absolute paths still went through regular rlocation lookup and could fail there even after the absolute path had been printed to stdout. Also adds a missing newline in the (very rare) case that a manifest entry is found, but doesn't point to an existing file (e.g. if it is an unresolved symlink), and `rlocation` is not used with command substitution, but some other mechanism that doesn't strip trailing newlines. The tests now assert the expected exit code (== 0 or != 0). Fixes bazelbuild#16933
Due to tests not actually verifying the exit code of rlocation calls, a few special cases were not handled correctly. In particular, absolute paths still went through regular rlocation lookup and could fail there even after the absolute path had been printed to stdout. Also adds a missing newline in the (very rare) case that a manifest entry is found, but doesn't point to an existing file (e.g. if it is an unresolved symlink), and `rlocation` is not used with command substitution, but some other mechanism that doesn't strip trailing newlines. The tests now assert the expected exit code (== 0 or != 0). Fixes bazelbuild#16933
During remote execution, absolute paths may not contain `execroot` segments. Work towards #16933 Closes #16946. PiperOrigin-RevId: 493876135 Change-Id: I5ceb8c6edbe0e58aae2066efe1230e622eabf9bf Co-authored-by: Fabian Meumertzheim <[email protected]>
… commit (#1507) Related bazelbuild/bazel#16933 (comment) BuildKiteException is always caught at https://github.com/bazelbuild/continuous-integration/blob/4592271b2fdcf26f865d9c0082e1e53add7ea9a2/buildkite/bazelci.py#L3592-L3594 - The "Publish Bazel binaries" pipeline sometimes fails to publish Bazel binaries for a commit, in this case, the Culprit Finder should immediately fail instead consider the commit as a bad Bazel commit due to failing build. - Also removed some unused parameters in the function.
Due to tests not actually verifying the exit code of rlocation calls, a few special cases were not handled correctly. In particular, absolute paths still went through regular rlocation lookup and could fail there even after the absolute path had been printed to stdout. Also adds a missing newline in the (very rare) case that a manifest entry is found, but doesn't point to an existing file (e.g. if it is an unresolved symlink), and `rlocation` is not used with command substitution, but some other mechanism that doesn't strip trailing newlines. The tests now assert the expected exit code (== 0 or != 0). Fixes bazelbuild#16933 Closes bazelbuild#16945. PiperOrigin-RevId: 494609104 Change-Id: I30333219176a61bda51f08c2c6bc927ce653d681
Due to tests not actually verifying the exit code of rlocation calls, a few special cases were not handled correctly. In particular, absolute paths still went through regular rlocation lookup and could fail there even after the absolute path had been printed to stdout. Also adds a missing newline in the (very rare) case that a manifest entry is found, but doesn't point to an existing file (e.g. if it is an unresolved symlink), and `rlocation` is not used with command substitution, but some other mechanism that doesn't strip trailing newlines. The tests now assert the expected exit code (== 0 or != 0). Fixes #16933 Closes #16945. PiperOrigin-RevId: 494609104 Change-Id: I30333219176a61bda51f08c2c6bc927ce653d681
Due to tests not actually verifying the exit code of rlocation calls, a few special cases were not handled correctly. In particular, absolute paths still went through regular rlocation lookup and could fail there even after the absolute path had been printed to stdout. Also adds a missing newline in the (very rare) case that a manifest entry is found, but doesn't point to an existing file (e.g. if it is an unresolved symlink), and `rlocation` is not used with command substitution, but some other mechanism that doesn't strip trailing newlines. The tests now assert the expected exit code (== 0 or != 0). Fixes bazelbuild#16933
Due to tests not actually verifying the exit code of rlocation calls, a few special cases were not handled correctly. In particular, absolute paths still went through regular rlocation lookup and could fail there even after the absolute path had been printed to stdout. Also adds a missing newline in the (very rare) case that a manifest entry is found, but doesn't point to an existing file (e.g. if it is an unresolved symlink), and `rlocation` is not used with command substitution, but some other mechanism that doesn't strip trailing newlines. The tests now assert the expected exit code (== 0 or != 0). Fixes bazelbuild#16933
… commit (bazelbuild#1507) Related bazelbuild/bazel#16933 (comment) BuildKiteException is always caught at https://github.com/bazelbuild/continuous-integration/blob/4592271b2fdcf26f865d9c0082e1e53add7ea9a2/buildkite/bazelci.py#L3592-L3594 - The "Publish Bazel binaries" pipeline sometimes fails to publish Bazel binaries for a commit, in this case, the Culprit Finder should immediately fail instead consider the commit as a bad Bazel commit due to failing build. - Also removed some unused parameters in the function.
This was first detected in the downstream test for Bazel 6.0.0rc4: https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/2761#0184d3b5-3cf0-410a-984d-2fdfdf78a49b
And verified in https://buildkite.com/bazel/bazel-bazel/builds/21471#0184e876-6406-4909-9a81-e8dc957b471f
Integration tests with test args are failing remotely with test logs look like:
The text was updated successfully, but these errors were encountered: