Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

teamcity: failed test: TestReplicaRefreshPendingCommandsTicks #38525

Closed
cockroach-teamcity opened this issue Jun 27, 2019 · 2 comments · Fixed by #38532
Closed

teamcity: failed test: TestReplicaRefreshPendingCommandsTicks #38525

cockroach-teamcity opened this issue Jun 27, 2019 · 2 comments · Fixed by #38532
Assignees
@nvanbenschoten
Labels
C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.
Milestone
19.2

Comments

@cockroach-teamcity
Copy link
Member

The following tests appear to have failed on master (testrace): TestReplicaRefreshPendingCommandsTicks

You may want to check for open issues.

#1364461:

TestReplicaRefreshPendingCommandsTicks
--- FAIL: testrace/TestReplicaRefreshPendingCommandsTicks (0.100s)

------- Stdout: -------
I190627 21:56:24.319148 27933 util/stop/stopper.go:542  quiescing; tasks left:
1      [async] intent_resolver_ir_batcher
1      [async] intent_resolver_gc_batcher
    replica_test.go:7645: 0: expected no reproposed commands, but found [0xc00123c510]

Please assign, take a look and update the issue accordingly.
@cockroach-teamcity cockroach-teamcity added this to the 19.2 milestone Jun 27, 2019
@cockroach-teamcity cockroach-teamcity added C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. labels Jun 27, 2019
@nvanbenschoten nvanbenschoten self-assigned this Jun 27, 2019
@nvanbenschoten
Copy link
Member

I'm seeing proposals being refreshed after the first proposal in the test due to a reasonNewLeaderOrConfigChange. This is likely due to #38343, although I'm not sure how.

@nvanbenschoten
Copy link
Member

I confirmed that I can only reproduce this after 1ff3556.

nvanbenschoten added a commit to nvanbenschoten/cockroach that referenced this issue Jun 28, 2019
@nvanbenschoten
storage: deflake TestReplicaRefreshPendingCommandsTicks
7d3583d 
Fixes cockroachdb#38525.

The test was occasionally failing because the proposals map was being refreshed
due to a `reasonNewLeaderOrConfigChange`. This was caused by 1ff3556, which
replaced `submitProposalFn` with `proposalBuf.testing.submitProposalFilter`. The
new approach to mocking out this function doesn't prevent a Raft ready iteration
from being scheduled, so `handleRaftReady` was being called and noticing the
leadership change, resulting in the refresh.

I've stressed this for 200,000 iterations without a failure.

Release note: None
@nvanbenschoten nvanbenschoten mentioned this issue Jun 28, 2019
Merged
craig bot pushed a commit that referenced this issue Jun 28, 2019
@nvanbenschoten
Merge #38532
77e4611 
38532: storage: deflake TestReplicaRefreshPendingCommandsTicks r=tbg a=nvanbenschoten

Fixes #38525.

The test was occasionally failing because the proposals map was being refreshed
due to a `reasonNewLeaderOrConfigChange`. This was caused by 1ff3556, which
replaced `submitProposalFn` with `proposalBuf.testing.submitProposalFilter`. The
new approach to mocking out this function doesn't prevent a Raft ready iteration
from being scheduled, so `handleRaftReady` was being called and noticing the
leadership change, resulting in the refresh.

I've stressed this for 200,000 iterations without a failure.

Co-authored-by: Nathan VanBenschoten <[email protected]>
@craig craig bot closed this as completed in #38532 Jun 28, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@nvanbenschoten nvanbenschoten

Labels
C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.
Projects
None yet
Milestone
19.2
Development

Successfully merging a pull request may close this issue.

storage: deflake TestReplicaRefreshPendingCommandsTicks nvanbenschoten/cockroach
2 participants
@nvanbenschoten @cockroach-teamcity