Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

teamcity: failed test: TestCleanupTxnIntentsOnGCAsync #35340

Closed
cockroach-teamcity opened this issue Mar 3, 2019 · 2 comments · Fixed by #35667
Closed

teamcity: failed test: TestCleanupTxnIntentsOnGCAsync #35340

cockroach-teamcity opened this issue Mar 3, 2019 · 2 comments · Fixed by #35667
Assignees
@ajwerner
Labels
C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.

Comments

@cockroach-teamcity
Copy link
Member

The following tests appear to have failed on master (test): TestCleanupTxnIntentsOnGCAsync, TestCleanupTxnIntentsOnGCAsync/#1, TestCleanupTxnIntentsOnGCAsync/#00, TestCleanupTxnIntentsOnGCAsync/#3, TestCleanupTxnIntentsOnGCAsync/#2

You may want to check for open issues.

#1160310:

TestCleanupTxnIntentsOnGCAsync/#02
--- FAIL: test/TestCleanupTxnIntentsOnGCAsync/#02 (0.000s)
Test ended in panic.

------- Stdout: -------
W190303 18:44:35.806826 49 storage/intentresolver/intent_resolver.go:729  failed to cleanup transaction intents: failed to resolve intents: boom



TestCleanupTxnIntentsOnGCAsync
--- FAIL: testrace/TestCleanupTxnIntentsOnGCAsync (0.040s)
Test ended in panic.




TestCleanupTxnIntentsOnGCAsync/#00
--- FAIL: test/TestCleanupTxnIntentsOnGCAsync/#00 (0.000s)
Test ended in panic.




TestCleanupTxnIntentsOnGCAsync
--- FAIL: test/TestCleanupTxnIntentsOnGCAsync (0.000s)
Test ended in panic.




TestCleanupTxnIntentsOnGCAsync/#03
...
	/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:190 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000504120, 0x11034c0, 0xc0000ba010, 0xc000343050, 0x22, 0x0, 0x0, 0xc00052a0f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:325 +0xe6
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:320 +0x134

goroutine 47 [select]:
github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).run(0xc00052ea90, 0x11034c0, 0xc0000ba010)
	/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:373 +0x2fa
github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).run-fm(0x11034c0, 0xc0000ba010)
	/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:190 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000504120, 0x11034c0, 0xc0000ba010, 0xc000343110, 0x22, 0x0, 0x0, 0xc00052a170)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:325 +0xe6
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:320 +0x134

goroutine 48 [select]:
github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).run(0xc00052eb60, 0x11034c0, 0xc0000ba010)
	/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:373 +0x2fa
github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).run-fm(0x11034c0, 0xc0000ba010)
	/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:190 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000504120, 0x11034c0, 0xc0000ba010, 0xc0003431a0, 0x22, 0x0, 0x0, 0xc00052a180)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:325 +0xe6
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:320 +0x134

goroutine 59 [select]:
github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).run(0xc0000b48f0, 0x11034c0, 0xc0000ba010)
	/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:373 +0x2fa
github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).run-fm(0x11034c0, 0xc0000ba010)
	/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:190 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000504120, 0x11034c0, 0xc0000ba010, 0xc0005a4030, 0x22, 0x0, 0x0, 0xc00049c0c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:325 +0xe6
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:320 +0x134

goroutine 60 [select]:
github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).run(0xc0000b4d00, 0x11034c0, 0xc0000ba010)
	/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:373 +0x2fa
github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).run-fm(0x11034c0, 0xc0000ba010)
	/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:190 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000504120, 0x11034c0, 0xc0000ba010, 0xc0005a4060, 0x22, 0x0, 0x0, 0xc00049c0d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:325 +0xe6
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:320 +0x134



TestCleanupTxnIntentsOnGCAsync/#01
--- FAIL: test/TestCleanupTxnIntentsOnGCAsync/#01 (0.000s)
Test ended in panic.

Please assign, take a look and update the issue accordingly.
@cockroach-teamcity cockroach-teamcity added C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. labels Mar 3, 2019
@bdarnell
Copy link
Contributor

I've hit a flake on this test several times:

https://teamcity.cockroachdb.com/viewLog.html?buildId=1174203&buildTypeId=Cockroach_UnitTests_Test&tab=buildLog&_focus=349993

panic: interface conversion: roachpb.Response is *roachpb.GCResponse, not *roachpb.ResolveIntentRangeResponse [recovered]
	panic: interface conversion: roachpb.Response is *roachpb.GCResponse, not *roachpb.ResolveIntentRangeResponse

goroutine 63 [running]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).Recover(0xc0003aa1b0, 0x1105fc0, 0xc000034108)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:185 +0x11f
panic(0xe79480, 0xc00019f1d0)
	/usr/local/go/src/runtime/panic.go:513 +0x1b9
github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).ResolveIntents(0xc00051c680, 0x1105f80, 0xc000040e00, 0xc000360540, 0x3, 0x3, 0x410001, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:945 +0xfa1
github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).cleanupFinishedTxnIntents(0xc00051c680, 0x1105fc0, 0xc000034108, 0x1, 0xc000313940, 0xc000360540, 0x3, 0x3, 0x158b4bf7b338dbe5, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:808 +0xed
github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).CleanupTxnIntentsOnGCAsync.func1(0x1105fc0, 0xc000034108)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:726 +0x205
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunLimitedAsyncTask.func1(0xc0003aa1b0, 0x1105fc0, 0xc000034108, 0xf95af4, 0x16, 0xc00032f0e0, 0x0, 0x0, 0xc000286280)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:385 +0x110
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunLimitedAsyncTask
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:379 +0x23a

ajwerner added a commit to ajwerner/cockroach that referenced this issue Mar 12, 2019
@ajwerner
intentresolver: fix test flake due to timing of batches
601b37c 
Before this change we'd assume that two requests addressed to the same range
would end up in the same batch. This patch adopts the more flexible logic
introduced in cockroachdb#35273 to resolve similar test flakes in TestCleanupIntents.
This new infrastructure behaves correctly when batches are sent due to time
with less batching than the test assumed.

Fixes cockroachdb#35340.

Release note: None
@ajwerner ajwerner mentioned this issue Mar 12, 2019
Merged
@ajwerner
Copy link
Contributor

Sorry you ran in to this. Typed up a patch.

craig bot pushed a commit that referenced this issue Mar 13, 2019
@ajwerner
Merge #35667
41e67d0 
35667: intentresolver: fix test flake due to timing of batches r=ajwerner a=ajwerner

Before this change we'd assume that two requests addressed to the same range
would end up in the same batch. This patch adopts the more flexible logic
introduced in #35273 to resolve similar test flakes in TestCleanupIntents.
This new infrastructure behaves correctly when batches are sent due to time
with less batching than the test assumed.

Fixes #35340.

Release note: None

Co-authored-by: Andrew Werner <[email protected]>
@craig craig bot closed this as completed in #35667 Mar 13, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ajwerner ajwerner

Labels
C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

intentresolver: fix test flake due to timing of batches ajwerner/cockroach
3 participants
@bdarnell @ajwerner @cockroach-teamcity