bazel: high Bazel CI concurrency affects timing-sensitive tests #65407
Labels
A-testing
Testing tools and infrastructure
C-test-failure
Broken test (automatically or manually discovered).
skipped-test
Comments
erikgrinaker
added
C-test-failure
rickystewart
changed the title
craig bot
pushed a commit
that referenced
this issue
May 24, 2021
65582: bazel: mark `kvserver` test as `exclusive` r=jlinder a=rickystewart This tag tells Bazel the test shouldn't be run at the same time as any other build or test tasks, which solves a problem where timing-sensitive tests will sometimes spuriously time out. Ref: https://docs.bazel.build/versions/master/be/common-definitions.html Closes #65407. Release note: None 65616: bazel: add additional resolve hints for `gazelle` r=jlinder a=rickystewart I'm not really sure why gazelle wants these, but this quiets some error messages. Fixes #65615. Release note: None Co-authored-by: Ricky Stewart <[email protected]>
rickystewart
added a commit
to rickystewart/cockroach
that referenced
this issue
Nov 23, 2021
The `exclusive` tag here prevents the shards from running concurrently. See cockroachdb#65407, cockroachdb#65582. Release note: None
craig bot
pushed a commit
that referenced
this issue
Nov 23, 2021
71770: bazel: run acceptance tests under Bazel r=rickystewart a=rail This patch makes the acceptance test work under Bazel. * Add `AbsCertsDir()` in order to keep track of certificate path for cases when tests change the working directory. * docker-compose tests to use interpolation and environment variables in order to override `CERTS_DIR` and `COCKROACH_BINARY`. * Add `copyRunfiles()` in order to copy Bazel-generated symlinked runfiles as regular files to make them available in docker mounted volumes. Related: #71932, #71930 Fixes: #59446 Release note: None 72574: ci: add bazel roachtest gce teamcity job r=rail a=rickystewart Release note: None 73055: ui: prevent undefined axis label on custom chart r=zachlite a=zachlite Release note (bug fix): Y-axis labels on custom charts no longer display 'undefined'. @thtruo, this fix addresses the issue as described in #72115. Now, when the user selects a new unit from the dropdown, the user will experience up to a 10 second delay before the axis label refreshes. @nathanstilwell and I investigated this last week, and we'd need more time to think of a solution that doesn't create long term maintenance headaches. The benefit of this PR as it stands is that the Y-axis label is noticeably less broken. FYI. 73080: bazel: don't shard `kvserver` test r=rail a=rickystewart The `exclusive` tag here prevents the shards from running concurrently. See #65407, #65582. Release note: None Co-authored-by: Rail Aliiev <[email protected]> Co-authored-by: Ricky Stewart <[email protected]> Co-authored-by: Zach Lite <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
As seen in e.g. #65380, timing-sensitive tests consistently fail with Bazel CI as they take much longer to run than with regular CI (i.e.
make test). Assuming this is caused by higher test concurrency, we should either tune Bazel CI concurrency such that each test gets e.g. 1 CPU core's worth of reserved CPU, or allow tagging timing-sensitive tests such that they run with appropriate isolation from other tests.The text was updated successfully, but these errors were encountered: