Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

multiple test failiures #98905

Closed
smg260 opened this issue Mar 17, 2023 · 7 comments
Closed

multiple test failiures #98905

smg260 opened this issue Mar 17, 2023 · 7 comments
Assignees
@smg260
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). GA-blocker skipped-test

Comments

@smg260
Copy link
Contributor

smg260 commented Mar 17, 2023 •
edited by cockroach-jira-scripts
Loading

On 2023-03-17 BORS has been failing with multiple issues. This is an umbrella issue to link to, and can be updated as required.

changefeedccl/TestChangefeedRandomExpressions/kafka - 96532

changefeedccl/TestChangefeedSchemaChangeAllowBackfill 5 fails https://teamcity.cockroachdb.com/test/-626867411426624385?currentProjectId=Cockroach_Ci_TestsGcpLinuxX8664BigVm&expandTestHistoryChartSection=true

changefeedccl.TestChangefeedNemeses 5 fails https://teamcity.cockroachdb.com/test/4626064293122723814?currentProjectId=Cockroach_Ci_TestsGcpLinuxX8664BigVm&expandTestHistoryChartSection=true

kvserver/TestDecommission - > 20 fails https://teamcity.cockroachdb.com/test/-6750397557676147176?currentProjectId=Cockroach_Ci_TestsGcpLinuxX8664BigVm&expandTestHistoryChartSection=true

kvserver/TestSystemZoneConfigs - 3 fails staging - https://teamcity.cockroachdb.com/test/1437690617008383303?currentProjectId=Cockroach_Ci_TestsGcpLinuxX8664BigVm&expandTestHistoryChartSection=true

kvserver/TestReplicaTombstone/(4) (4.1) raft messages to newer replicaID
multiregionccl/TestMultiRegionDataDriven/regional_by_table
logictest/distsql_join - 2 fails https://teamcity.cockroachdb.com/test/7978722684846482798?currentProjectId=Cockroach_Ci_TestsGcpLinuxX8664BigVm&branch=&expandTestHistoryChartSection=true

logictest/sequences_distsql

Jira issue: CRDB-25581
@smg260 smg260 added the C-test-failure Broken test (automatically or manually discovered). label Mar 17, 2023
@smg260 smg260 self-assigned this Mar 17, 2023
@smg260 smg260 mentioned this issue Mar 17, 2023
Closed
@rafiss rafiss added branch-master Failures and bugs on the master branch. GA-blocker skipped-test labels Mar 18, 2023
@aadityasondhi
Copy link
Collaborator

aadityasondhi commented Mar 20, 2023 •
edited
Loading

multiregionccl/TestMultiRegionDataDriven/regional_by_table failed again on: #98077

https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_BazelEssentialCi/9156237?showRootCauses=true&expandBuildChangesSection=true&expandBuildProblemsSection=true&expandBuildTestsSection=true

@adityamaru adityamaru mentioned this issue Mar 21, 2023
Merged
craig bot pushed a commit that referenced this issue Mar 21, 2023
@adityamaru
Merge #99121
5c48aa2 
99121: *: skip flaky tests r=adityamaru a=adityamaru

Informs: #98987
Informs: #98020
Informs: #98905
Informs: #98564
Informs: #98991

Release note: None
Epic: none

Co-authored-by: adityamaru <[email protected]>
@mgartner mgartner self-assigned this Mar 22, 2023
@blathers-crl blathers-crl bot added the T-sql-queries SQL Queries Team label Mar 22, 2023
@erikgrinaker
Copy link
Contributor

Fix for TestReplicaTombstone being merged in #99099.

@mgartner
Copy link
Collaborator

@smg260 Do you have more context about the logictest/sequences_distsql flakes? I'm unable to reproduce any flakes with that test.

@mgartner
Copy link
Collaborator

I'm also unable to reproduce a failure with logictest/distsql_join.

@rafiss
Copy link
Collaborator

rafiss commented Mar 22, 2023

I believe all logictests running under the fakedist-vec-off config are currently affected by the "slow quiesce" flakes as seen in #99242. Here's an example of it affecting sequences_distsql: https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_BazelEssentialCi/9172335?showRootCauses=false&expandBuildProblemsSection=true&expandBuildTestsSection=true

@smg260
Copy link
Contributor Author

smg260 commented Mar 22, 2023

@mgartner sorry for the lack of detail; it was getting late on Friday. @rafiss is correct, the flakes I saw were "slow quiesce". This was in the staging branch.

This was referenced Mar 23, 2023
Closed
Closed
Closed
@andrewbaptist andrewbaptist mentioned this issue Mar 23, 2023
Closed
@mgartner mgartner removed their assignment Mar 27, 2023
@mgartner mgartner removed the T-sql-queries SQL Queries Team label Mar 29, 2023
@renatolabs
Copy link
Contributor

Closing this:

  • it's inactive and out of date;
  • many of the failures here were related to the "slow quiesce" issue;
  • we would benefit from issues that are specific to a test failure for any future flakes we see.

@srosenberg srosenberg mentioned this issue Jun 27, 2023
Closed
srosenberg added a commit to srosenberg/cockroach that referenced this issue Jun 27, 2023
@srosenberg
kv: unskip TestSystemZoneConfigs
78a92cb 
TestSystemZoneConfigs was part of a batch of skipped tests [1].
It appears several tests in that batch flaked due to a
"slow quiesce" issue. The root cause for TestSystemZoneConfigs
flaking during that time remains unknown, yet the other tests
in the same batch have since been unskipped.

After stress-testing via,

```
./dev test pkg/kv/kvserver -f=TestSystemZoneConfigs --stress --stress-args '-maxruns 10000 -p 23' --timeout=10m
```
we surmise this test might have been indirectly deflaked.
Thus, we unskip it.

[1] cockroachdb#99121

Epic: none
Fixes: cockroachdb#98905

Release note: None
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@smg260 smg260

Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). GA-blocker skipped-test
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

kv: unskip TestSystemZoneConfigs srosenberg/cockroach
6 participants
@erikgrinaker @mgartner @rafiss @aadityasondhi @smg260 @renatolabs