Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

kv/kvnemesis: TestKVNemesisMultiNode failed #108241

Closed
cockroach-teamcity opened this issue Aug 5, 2023 · 5 comments · Fixed by #108775
Closed

kv/kvnemesis: TestKVNemesisMultiNode failed #108241

cockroach-teamcity opened this issue Aug 5, 2023 · 5 comments · Fixed by #108775
Assignees
@pav-kv
Labels
A-kv-replication Relating to Raft, consensus, and coordination. branch-master Failures and bugs on the master branch. C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. X-duplicate Closed as a duplicate of another issue.
Milestone
23.2

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Aug 5, 2023 •
edited by cockroach-jira-scripts
Loading

kv/kvnemesis.TestKVNemesisMultiNode failed with artifacts on master @ 695d4ec85157a4da0c97237e6f16bf919eebf6c5:

Fatal error:

panic: use of Span after Finish. Span: /cockroach.roachpb.Internal/Batch. Finish previously called at: <stack not captured. Set debugUseAfterFinish> [recovered]
	panic: use of Span after Finish. Span: /cockroach.roachpb.Internal/Batch. Finish previously called at: <stack not captured. Set debugUseAfterFinish>

Stack:

goroutine 6569871 [running]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).recover(0x1f81c6a?, {0x6976bc8, 0xc014ecf170})
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:230 +0x6a
panic({0x4b2d840, 0xc01b174160})
	GOROOT/src/runtime/panic.go:884 +0x212
github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).detectUseAfterFinish(0x6976bc8?)
	github.com/cockroachdb/cockroach/pkg/util/tracing/span.go:182 +0x12e
github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).Tracer(...)
	github.com/cockroachdb/cockroach/pkg/util/tracing/span.go:225
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaDecoder).createTracingSpans(0xc00a3270d8, {0x6976bc8, 0xc0204de0f0})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_application_decoder.go:154 +0x4f0
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaDecoder).DecodeAndBind(0xc00a326c80?, {0x6976bc8, 0xc0204de0f0}, {0xc02851e210?, 0x3?, 0x10?})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_application_decoder.go:63 +0x66
github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).Decode(...)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/task.go:142
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked(_, {_, _}, {{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:878 +0x73a
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady(_, {_, _}, {{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:728 +0x1b0
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady(0xc0165eea80, 0x2?)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:689 +0x14b
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker(0xc0085cbd50, {0x6976bc8, 0xc014ecf170}, {0x695c980, 0xc0165eea80}, 0xc0165ee000)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:416 +0x274
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2({0x6976bc8?, 0xc014ecf170?})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:321 +0x45
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:484 +0x146
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:475 +0x43b
Log preceding fatal error 

=== RUN   TestKVNemesisMultiNode
    test_log_scope.go:167: test logs captured to: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/logTestKVNemesisMultiNode1784560352
    test_log_scope.go:81: use -show-logs to present logs inline
    kvnemesis_test.go:283: seed: 2381536854787156731
    test_server_shim.go:98: 
        Test server was configured to route SQL queries to a secondary tenant (virtual cluster).
        If you are only seeing a test failure when this message appears, there may be a problem
        specific to cluster virtualization or multi-tenancy.
        
        To investigate, consider using "COCKROACH_TEST_TENANT=true" to force-enable just
        the secondary tenant in all runs (or, alternatively, "false" to force-disable), or use
        "COCKROACH_INTERNAL_DISABLE_METAMORPHIC_TESTING=false" to disable all random test variables altogether.
*
* WARNING: test tenant requested by configuration, but code organization prevents start!
* OSS binaries do not include enterprise features
*
*
* WARNING: test tenant requested by configuration, but code organization prevents start!
* OSS binaries do not include enterprise features
*
*
* WARNING: test tenant requested by configuration, but code organization prevents start!
* OSS binaries do not include enterprise features
*
*
* WARNING: test tenant requested by configuration, but code organization prevents start!
* OSS binaries do not include enterprise features
*
    kvnemesis_test.go:197: kvnemesis logging to /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis1734753383

Parameters: TAGS=bazel,gss,deadlock , stress=true

Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/kv

This test on roachdash | Improve this report!

Jira issue: CRDB-30377
@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. T-kv KV Team labels Aug 5, 2023
@cockroach-teamcity cockroach-teamcity added this to the 23.2 milestone Aug 5, 2023
@nvanbenschoten
Copy link
Member

This is an instance of the span use after finish failure described in #107521.

@nvanbenschoten nvanbenschoten added A-kv-replication Relating to Raft, consensus, and coordination. and removed T-kv KV Team labels Aug 6, 2023
@blathers-crl
Copy link

blathers-crl bot commented Aug 6, 2023

cc @cockroachdb/replication

@nvanbenschoten
Copy link
Member

@pavelkalinnikov I'm assigning you for now because you're already assigned to #107521, but feel free to re-assign.

@nvanbenschoten nvanbenschoten added the C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. label Aug 6, 2023
@cockroach-teamcity
Copy link
Member Author

kv/kvnemesis.TestKVNemesisMultiNode failed with artifacts on master @ 3e7991d1b2bb23ce06d36cf2adc9905134e22d8e:

Fatal error:

panic: use of Span after Finish. Span: /cockroach.roachpb.Internal/Batch. Finish previously called at: <stack not captured. Set debugUseAfterFinish> [recovered]
	panic: use of Span after Finish. Span: /cockroach.roachpb.Internal/Batch. Finish previously called at: <stack not captured. Set debugUseAfterFinish>

Stack:

goroutine 8855242 [running]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).recover(0x1f81d4a?, {0x6977888, 0xc00bd14960})
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:230 +0x6a
panic({0x4b2f360, 0xc0288f0330})
	GOROOT/src/runtime/panic.go:884 +0x212
github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).detectUseAfterFinish(0x6977888?)
	github.com/cockroachdb/cockroach/pkg/util/tracing/span.go:182 +0x12e
github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).Tracer(...)
	github.com/cockroachdb/cockroach/pkg/util/tracing/span.go:225
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaDecoder).createTracingSpans(0xc025dea458, {0x6977888, 0xc025433bc0})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_application_decoder.go:154 +0x4f0
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaDecoder).DecodeAndBind(0xc025dea000?, {0x6977888, 0xc025433bc0}, {0xc0009dca80?, 0x2?, 0x10?})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_application_decoder.go:63 +0x66
github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).Decode(...)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/task.go:142
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked(_, {_, _}, {{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:878 +0x73a
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady(_, {_, _}, {{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:728 +0x1b0
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady(0xc004025500, 0x2?)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:689 +0x14b
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker(0xc00e0185b0, {0x6977888, 0xc00bd14960}, {0x695d640, 0xc004025500}, 0xc008ac6000)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:416 +0x274
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2({0x6977888?, 0xc00bd14960?})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:321 +0x45
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:484 +0x146
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:475 +0x43b
Log preceding fatal error 

=== RUN   TestKVNemesisMultiNode
    test_log_scope.go:167: test logs captured to: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/logTestKVNemesisMultiNode3974267412
    test_log_scope.go:81: use -show-logs to present logs inline
    kvnemesis_test.go:283: seed: 5132744873277486582
    kvnemesis_test.go:197: kvnemesis logging to /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis1130432553

Parameters: TAGS=bazel,gss,deadlock , stress=true

Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

@pav-kv pav-kv mentioned this issue Aug 7, 2023
Closed
@cockroach-teamcity
Copy link
Member Author

kv/kvnemesis.TestKVNemesisMultiNode failed with artifacts on master @ c13bf7633cbb416d9e43f8c57b1e309fab1110ce:

Fatal error:

panic: use of Span after Finish. Span: /cockroach.roachpb.Internal/Batch. Finish previously called at: <stack not captured. Set debugUseAfterFinish> [recovered]
	panic: use of Span after Finish. Span: /cockroach.roachpb.Internal/Batch. Finish previously called at: <stack not captured. Set debugUseAfterFinish>

Stack:

goroutine 8388484 [running]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).recover(0x1f8976a?, {0x69b21a8, 0xc008aa3ce0})
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:230 +0x6a
panic({0x4b58900, 0xc01654c060})
	GOROOT/src/runtime/panic.go:884 +0x212
github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).detectUseAfterFinish(0x9ac8c70?)
	github.com/cockroachdb/cockroach/pkg/util/tracing/span.go:182 +0x12e
github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).RecordingType(0xc0028ca400)
	github.com/cockroachdb/cockroach/pkg/util/tracing/span.go:442 +0x1e
github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).IsVerbose(...)
	github.com/cockroachdb/cockroach/pkg/util/tracing/span.go:450
github.com/cockroachdb/cockroach/pkg/util/log.getSpanOrEventLog({0x69b21a8, 0xc009411bc0})
	github.com/cockroachdb/cockroach/pkg/util/log/trace.go:92 +0x3a
github.com/cockroachdb/cockroach/pkg/util/log.vEventf({0x69b21a8, 0xc009411bc0}, 0x0, 0x1, 0x0?, 0x0?, {0x502c335, 0x19}, {0x0, 0x0, ...})
	github.com/cockroachdb/cockroach/pkg/util/log/trace.go:212 +0x11f
github.com/cockroachdb/cockroach/pkg/util/log.VEvent(...)
	github.com/cockroachdb/cockroach/pkg/util/log/trace.go:231
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*propBuf).FlushLockedWithRaftGroup(0xc0103354a0, {0x69b21a8, 0xc02912cd20}, {0x69d0c18, 0xc019afa230})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_proposal_buf.go:601 +0x12c8
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked.func2(0x4893b00?)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:775 +0xd4
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).withRaftGroupLocked(0xc010334c80, 0x4afa240?)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:1998 +0x4e
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked(_, {_, _}, {{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:772 +0x317
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady(_, {_, _}, {{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:728 +0x1b0
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady(0xc004b6ca80, 0x69b21a8?)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:689 +0x14b
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker(0xc00bb7a850, {0x69b21a8, 0xc008aa3ce0}, {0x6997e70, 0xc004b6ca80}, 0xc004d98000)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:418 +0x195
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2({0x69b21a8?, 0xc008aa3ce0?})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:321 +0x45
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:484 +0x146
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:475 +0x43b
Log preceding fatal error 

=== RUN   TestKVNemesisMultiNode
    test_log_scope.go:167: test logs captured to: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/logTestKVNemesisMultiNode628115778
    test_log_scope.go:81: use -show-logs to present logs inline
    kvnemesis_test.go:283: seed: 4653790105750483978
    test_server_shim.go:115: 
        Test server was configured to route SQL queries to a secondary tenant (virtual cluster).
        If you are only seeing a test failure when this message appears, there may be a problem
        specific to cluster virtualization or multi-tenancy.
        
        To investigate, consider using "COCKROACH_TEST_TENANT=true" to force-enable just
        the secondary tenant in all runs (or, alternatively, "false" to force-disable), or use
        "COCKROACH_INTERNAL_DISABLE_METAMORPHIC_TESTING=false" to disable all random test variables altogether.
*
* WARNING: test tenant requested by configuration, but code organization prevents start!
* OSS binaries do not include enterprise features
*
*
* WARNING: test tenant requested by configuration, but code organization prevents start!
* OSS binaries do not include enterprise features
*
*
* WARNING: test tenant requested by configuration, but code organization prevents start!
* OSS binaries do not include enterprise features
*
*
* WARNING: test tenant requested by configuration, but code organization prevents start!
* OSS binaries do not include enterprise features
*
    kvnemesis_test.go:197: kvnemesis logging to /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis2755337404

Parameters: TAGS=bazel,gss,deadlock , stress=true

Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

@pav-kv pav-kv mentioned this issue Aug 15, 2023
Merged
@pav-kv pav-kv added the X-duplicate Closed as a duplicate of another issue. label Aug 16, 2023
@craig craig bot closed this as completed in 2ac44e6 Aug 21, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@pav-kv pav-kv

Labels
A-kv-replication Relating to Raft, consensus, and coordination. branch-master Failures and bugs on the master branch. C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. X-duplicate Closed as a duplicate of another issue.
Projects
None yet
Milestone
23.2
Development

Successfully merging a pull request may close this issue.

kvserver: fix double span Finish on reproposals pav-kv/cockroach
3 participants
@pav-kv @nvanbenschoten @cockroach-teamcity