Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

pkg/ccl/logictestccl/tests/3node-tenant/3node-tenant_test: TestTenantLogic_strict_ddl_atomicity failed #102921

Closed
cockroach-teamcity opened this issue May 8, 2023 · 2 comments
Closed

pkg/ccl/logictestccl/tests/3node-tenant/3node-tenant_test: TestTenantLogic_strict_ddl_atomicity failed #102921

cockroach-teamcity opened this issue May 8, 2023 · 2 comments
Labels
branch-release-23.1 Used to mark GA and release blockers, technical advisories, and bugs for 23.1 C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. T-sql-queries SQL Queries Team
Milestone
23.1

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented May 8, 2023 •
edited by cockroach-jira-scripts
Loading

pkg/ccl/logictestccl/tests/3node-tenant/3node-tenant_test.TestTenantLogic_strict_ddl_atomicity failed with artifacts on release-23.1 @ 5376e479204d6e8243f67c30aea3d031df529afd:

F230508 22:25:32.301296 410729 3@pebble/db.go:811  [n3,s3,pebble] 1 !	github.com/cockroachdb/cockroach/pkg/storage/pebble.go:687 +0x54
F230508 22:25:32.301296 410729 3@pebble/db.go:811  [n3,s3,pebble] 1 !github.com/cockroachdb/pebble.(*DB).applyInternal(0x4009de1400, 0x4002fccdc0, 0x4003570438?, 0x40?)
F230508 22:25:32.301296 410729 3@pebble/db.go:811  [n3,s3,pebble] 1 !	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/db.go:811 +0x240
F230508 22:25:32.301296 410729 3@pebble/db.go:811  [n3,s3,pebble] 1 !github.com/cockroachdb/pebble.(*DB).ApplyNoSyncWait(0x4003570458?, 0x320c7e4?, 0x4003570458?)
F230508 22:25:32.301296 410729 3@pebble/db.go:811  [n3,s3,pebble] 1 !	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/db.go:765 +0x28
F230508 22:25:32.301296 410729 3@pebble/db.go:811  [n3,s3,pebble] 1 !github.com/cockroachdb/cockroach/pkg/storage.(*pebbleBatch).CommitNoSyncWait(0x68ddc78?)
F230508 22:25:32.301296 410729 3@pebble/db.go:811  [n3,s3,pebble] 1 !	github.com/cockroachdb/cockroach/pkg/storage/pebble_batch.go:568 +0x34
F230508 22:25:32.301296 410729 3@pebble/db.go:811  [n3,s3,pebble] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvserver/logstore.(*LogStore).storeEntriesAndCommitBatch(0x4003570d08, {0x68ddc78, 0x400ea40090}, {0x0?, 0x0?, 0x0?}, {0x13, 0xffffffffffffffff, 0x2, 0x0, ...}, ...)
F230508 22:25:32.301296 410729 3@pebble/db.go:811  [n3,s3,pebble] 1 !	github.com/cockroachdb/cockroach/pkg/kv/kvserver/logstore/logstore.go:247 +0x594
F230508 22:25:32.301296 410729 3@pebble/db.go:811  [n3,s3,pebble] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvserver/logstore.(*LogStore).StoreEntries(0x4003570d08, {0x68ddc78, 0x400ea40090}, {0xffff59d5d890?, 0xffff8839c878?, 0xa0?}, {0x13, 0xffffffffffffffff, 0x2, 0x0, ...}, ...)
F230508 22:25:32.301296 410729 3@pebble/db.go:811  [n3,s3,pebble] 1 !	github.com/cockroachdb/cockroach/pkg/kv/kvserver/logstore/logstore.go:139 +0xa4
F230508 22:25:32.301296 410729 3@pebble/db.go:811  [n3,s3,pebble] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked(_, {_, _}, {{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...})
F230508 22:25:32.301296 410729 3@pebble/db.go:811  [n3,s3,pebble] 1 !	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:972 +0xc04
F230508 22:25:32.301296 410729 3@pebble/db.go:811  [n3,s3,pebble] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady(_, {_, _}, {{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...})
F230508 22:25:32.301296 410729 3@pebble/db.go:811  [n3,s3,pebble] 1 !	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:718 +0x164
F230508 22:25:32.301296 410729 3@pebble/db.go:811  [n3,s3,pebble] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady(0x4007d0e000, 0x68ddc78?)
F230508 22:25:32.301296 410729 3@pebble/db.go:811  [n3,s3,pebble] 1 !	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:646 +0xd8
F230508 22:25:32.301296 410729 3@pebble/db.go:811  [n3,s3,pebble] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker(0x400d588090, {0x68ddc78, 0x40142fbe30}, {0x68c2320, 0x4007d0e000}, 0x400f5b8400)
F230508 22:25:32.301296 410729 3@pebble/db.go:811  [n3,s3,pebble] 1 !	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:392 +0x20c
F230508 22:25:32.301296 410729 3@pebble/db.go:811  [n3,s3,pebble] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2({0x68ddc78?, 0x40142fbe30?})
F230508 22:25:32.301296 410729 3@pebble/db.go:811  [n3,s3,pebble] 1 !	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:299 +0x48
F230508 22:25:32.301296 410729 3@pebble/db.go:811  [n3,s3,pebble] 1 !github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
F230508 22:25:32.301296 410729 3@pebble/db.go:811  [n3,s3,pebble] 1 !	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:470 +0x128
F230508 22:25:32.301296 410729 3@pebble/db.go:811  [n3,s3,pebble] 1 !created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx
F230508 22:25:32.301296 410729 3@pebble/db.go:811  [n3,s3,pebble] 1 !	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:461 +0x358
F230508 22:25:32.301296 410729 3@pebble/db.go:811  [n3,s3,pebble] 1 !
F230508 22:25:32.301296 410729 3@pebble/db.go:811  [n3,s3,pebble] 1 !For more context, check log files in: /artifacts/tmp/_tmp/29095b981cb432b73f8cdd111429b1bc/logTestTenantLogic_strict_ddl_atomicity2916123937
F230508 22:25:32.301296 410729 3@pebble/db.go:811  [n3,s3,pebble] 1 !
F230508 22:25:32.301296 410729 3@pebble/db.go:811  [n3,s3,pebble] 1 !For more context, check log files in: /artifacts/tmp/_tmp/29095b981cb432b73f8cdd111429b1bc/logTestTenantLogic_strict_ddl_atomicity2916123937
F230508 22:25:32.301296 410729 3@pebble/db.go:811  [n3,s3,pebble] 1 !
F230508 22:25:32.301296 410729 3@pebble/db.go:811  [n3,s3,pebble] 1 !
F230508 22:25:32.301296 410729 3@pebble/db.go:811  [n3,s3,pebble] 1 !****************************************************************************
F230508 22:25:32.301296 410729 3@pebble/db.go:811  [n3,s3,pebble] 1 !
F230508 22:25:32.301296 410729 3@pebble/db.go:811  [n3,s3,pebble] 1 !This node experienced a fatal error (printed above), and as a result the
F230508 22:25:32.301296 410729 3@pebble/db.go:811  [n3,s3,pebble] 1 !process is terminating.
F230508 22:25:32.301296 410729 3@pebble/db.go:811  [n3,s3,pebble] 1 !
F230508 22:25:32.301296 410729 3@pebble/db.go:811  [n3,s3,pebble] 1 !Fatal errors can occur due to faulty hardware (disks, memory, clocks) or a
F230508 22:25:32.301296 410729 3@pebble/db.go:811  [n3,s3,pebble] 1 !problem in CockroachDB. With your help, the support team at Cockroach Labs
F230508 22:25:32.301296 410729 3@pebble/db.go:811  [n3,s3,pebble] 1 !will try to determine the root cause, recommend next steps, and we can
F230508 22:25:32.301296 410729 3@pebble/db.go:811  [n3,s3,pebble] 1 !improve CockroachDB based on your report.
F230508 22:25:32.301296 410729 3@pebble/db.go:811  [n3,s3,pebble] 1 !
F230508 22:25:32.301296 410729 3@pebble/db.go:811  [n3,s3,pebble] 1 !Please submit a crash report by following the instructions here:
F230508 22:25:32.301296 410729 3@pebble/db.go:811  [n3,s3,pebble] 1 !
F230508 22:25:32.301296 410729 3@pebble/db.go:811  [n3,s3,pebble] 1 !    https://github.com/cockroachdb/cockroach/issues/new/choose
F230508 22:25:32.301296 410729 3@pebble/db.go:811  [n3,s3,pebble] 1 !
F230508 22:25:32.301296 410729 3@pebble/db.go:811  [n3,s3,pebble] 1 !If you would rather not post publicly, please contact us directly at:
F230508 22:25:32.301296 410729 3@pebble/db.go:811  [n3,s3,pebble] 1 !
F230508 22:25:32.301296 410729 3@pebble/db.go:811  [n3,s3,pebble] 1 !    [email protected]
F230508 22:25:32.301296 410729 3@pebble/db.go:811  [n3,s3,pebble] 1 !
F230508 22:25:32.301296 410729 3@pebble/db.go:811  [n3,s3,pebble] 1 !The Cockroach Labs team appreciates your feedback.
Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/sql-queries

This test on roachdash | Improve this report!

Jira issue: CRDB-27730
@cockroach-teamcity cockroach-teamcity added branch-release-23.1 Used to mark GA and release blockers, technical advisories, and bugs for 23.1 C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. labels May 8, 2023
@cockroach-teamcity cockroach-teamcity added this to the 23.1 milestone May 8, 2023
@blathers-crl blathers-crl bot added the T-sql-queries SQL Queries Team label May 8, 2023
@mgartner
Copy link
Collaborator

mgartner commented May 9, 2023

The failure:

=== RUN   TestTenantLogic_strict_ddl_atomicity
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/29095b981cb432b73f8cdd111429b1bc/logTestTenantLogic_strict_ddl_atomicity2916123937
    test_log_scope.go:79: use -show-logs to present logs inline
F230508 22:25:32.301296 410729 3@pebble/db.go:811  [n3,s3,pebble] 1  pebble: fatal commit error: allocation failed because arena is full
F230508 22:25:32.301296 410729 3@pebble/db.go:811  [n3,s3,pebble] 1 !goroutine 410729 [running]:
F230508 22:25:32.301296 410729 3@pebble/db.go:811  [n3,s3,pebble] 1 !github.com/cockroachdb/cockroach/pkg/util/log.getStacks(0x0)
F230508 22:25:32.301296 410729 3@pebble/db.go:811  [n3,s3,pebble] 1 !	github.com/cockroachdb/cockroach/pkg/util/log/get_stacks.go:25 +0x8c
F230508 22:25:32.301296 410729 3@pebble/db.go:811  [n3,s3,pebble] 1 !github.com/cockroachdb/cockroach/pkg/util/log.(*loggerT).outputLogEntry(0x401c2aa880, {{{0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}}, 0x175d4b64de1701c8, ...})
F230508 22:25:32.301296 410729 3@pebble/db.go:811  [n3,s3,pebble] 1 !	github.com/cockroachdb/cockroach/pkg/util/log/clog.go:260 +0x9c
F230508 22:25:32.301296 410729 3@pebble/db.go:811  [n3,s3,pebble] 1 !github.com/cockroachdb/cockroach/pkg/util/log.logfDepthInternal({0x68ddc78, 0x40142fb0e0}, 0x3, 0x4, 0x3, 0x0?, {0x53b7248, 0x1e}, {0x400bb8f480, 0x1, ...})
F230508 22:25:32.301296 410729 3@pebble/db.go:811  [n3,s3,pebble] 1 !	github.com/cockroachdb/cockroach/pkg/util/log/channels.go:106 +0x520
F230508 22:25:32.301296 410729 3@pebble/db.go:811  [n3,s3,pebble] 1 !github.com/cockroachdb/cockroach/pkg/util/log.logfDepth(...)
F230508 22:25:32.301296 410729 3@pebble/db.go:811  [n3,s3,pebble] 1 !	github.com/cockroachdb/cockroach/pkg/util/log/channels.go:39
F230508 22:25:32.301296 410729 3@pebble/db.go:811  [n3,s3,pebble] 1 !github.com/cockroachdb/cockroach/pkg/util/log.loggerStorage.FatalfDepth(...)
F230508 22:25:32.301296 410729 3@pebble/db.go:811  [n3,s3,pebble] 1 !	github.com/cockroachdb/cockroach/bazel-out/aarch64-fastbuild/bin/pkg/util/log/log_channels_generated.go:2223
F230508 22:25:32.301296 410729 3@pebble/db.go:811  [n3,s3,pebble] 1 !github.com/cockroachdb/cockroach/pkg/storage.pebbleLogger.Fatalf({{0x68ddc78?, 0x40142fb0e0?}, 0x4003570358?}, {0x53b7248?, 0x10?}, {0x400bb8f480?, 0x1?, 0x0?})
F230508 22:25:32.301296 410729 3@pebble/db.go:811  [n3,s3,pebble] 1 !	github.com/cockroachdb/cockroach/pkg/storage/pebble.go:687 +0x54
F230508 22:25:32.301296 410729 3@pebble/db.go:811  [n3,s3,pebble] 1 !github.com/cockroachdb/pebble.(*DB).applyInternal(0x4009de1400, 0x4002fccdc0, 0x4003570438?, 0x40?)
F230508 22:25:32.301296 410729 3@pebble/db.go:811  [n3,s3,pebble] 1 !	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/db.go:811 +0x240
F230508 22:25:32.301296 410729 3@pebble/db.go:811  [n3,s3,pebble] 1 !github.com/cockroachdb/pebble.(*DB).ApplyNoSyncWait(0x4003570458?, 0x320c7e4?, 0x4003570458?)
F230508 22:25:32.301296 410729 3@pebble/db.go:811  [n3,s3,pebble] 1 !	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/db.go:765 +0x28
F230508 22:25:32.301296 410729 3@pebble/db.go:811  [n3,s3,pebble] 1 !github.com/cockroachdb/cockroach/pkg/storage.(*pebbleBatch).CommitNoSyncWait(0x68ddc78?)
F230508 22:25:32.301296 410729 3@pebble/db.go:811  [n3,s3,pebble] 1 !	github.com/cockroachdb/cockroach/pkg/storage/pebble_batch.go:568 +0x34
F230508 22:25:32.301296 410729 3@pebble/db.go:811  [n3,s3,pebble] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvserver/logstore.(*LogStore).storeEntriesAndCommitBatch(0x4003570d08, {0x68ddc78, 0x400ea40090}, {0x0?, 0x0?, 0x0?}, {0x13, 0xffffffffffffffff, 0x2, 0x0, ...}, ...)
F230508 22:25:32.301296 410729 3@pebble/db.go:811  [n3,s3,pebble] 1 !	github.com/cockroachdb/cockroach/pkg/kv/kvserver/logstore/logstore.go:247 +0x594
F230508 22:25:32.301296 410729 3@pebble/db.go:811  [n3,s3,pebble] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvserver/logstore.(*LogStore).StoreEntries(0x4003570d08, {0x68ddc78, 0x400ea40090}, {0xffff59d5d890?, 0xffff8839c878?, 0xa0?}, {0x13, 0xffffffffffffffff, 0x2, 0x0, ...}, ...)
F230508 22:25:32.301296 410729 3@pebble/db.go:811  [n3,s3,pebble] 1 !	github.com/cockroachdb/cockroach/pkg/kv/kvserver/logstore/logstore.go:139 +0xa4
F230508 22:25:32.301296 410729 3@pebble/db.go:811  [n3,s3,pebble] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked(_, {_, _}, {{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...})
F230508 22:25:32.301296 410729 3@pebble/db.go:811  [n3,s3,pebble] 1 !	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:972 +0xc04
F230508 22:25:32.301296 410729 3@pebble/db.go:811  [n3,s3,pebble] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady(_, {_, _}, {{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...})
F230508 22:25:32.301296 410729 3@pebble/db.go:811  [n3,s3,pebble] 1 !	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:718 +0x164
F230508 22:25:32.301296 410729 3@pebble/db.go:811  [n3,s3,pebble] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady(0x4007d0e000, 0x68ddc78?)
F230508 22:25:32.301296 410729 3@pebble/db.go:811  [n3,s3,pebble] 1 !	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:646 +0xd8
F230508 22:25:32.301296 410729 3@pebble/db.go:811  [n3,s3,pebble] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker(0x400d588090, {0x68ddc78, 0x40142fbe30}, {0x68c2320, 0x4007d0e000}, 0x400f5b8400)
F230508 22:25:32.301296 410729 3@pebble/db.go:811  [n3,s3,pebble] 1 !	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:392 +0x20c
F230508 22:25:32.301296 410729 3@pebble/db.go:811  [n3,s3,pebble] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2({0x68ddc78?, 0x40142fbe30?})
F230508 22:25:32.301296 410729 3@pebble/db.go:811  [n3,s3,pebble] 1 !	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:299 +0x48
F230508 22:25:32.301296 410729 3@pebble/db.go:811  [n3,s3,pebble] 1 !github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
F230508 22:25:32.301296 410729 3@pebble/db.go:811  [n3,s3,pebble] 1 !	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:470 +0x128
F230508 22:25:32.301296 410729 3@pebble/db.go:811  [n3,s3,pebble] 1 !created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx
F230508 22:25:32.301296 410729 3@pebble/db.go:811  [n3,s3,pebble] 1 !	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:461 +0x358
F230508 22:25:32.301296 410729 3@pebble/db.go:811  [n3,s3,pebble] 1 !
F230508 22:25:32.301296 410729 3@pebble/db.go:811  [n3,s3,pebble] 1 !For more context, check log files in: /artifacts/tmp/_tmp/29095b981cb432b73f8cdd111429b1bc/logTestTenantLogic_strict_ddl_atomicity2916123937
F230508 22:25:32.301296 410729 3@pebble/db.go:811  [n3,s3,pebble] 1 !
F230508 22:25:32.301296 410729 3@pebble/db.go:811  [n3,s3,pebble] 1 !For more context, check log files in: /artifacts/tmp/_tmp/29095b981cb432b73f8cdd111429b1bc/logTestTenantLogic_strict_ddl_atomicity2916123937

@mgartner
Copy link
Collaborator

mgartner commented May 9, 2023

Duplicate of #102945.

@mgartner mgartner closed this as completed May 9, 2023
@mgartner mgartner moved this to Done in SQL Queries Jul 24, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
branch-release-23.1 Used to mark GA and release blockers, technical advisories, and bugs for 23.1 C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. T-sql-queries SQL Queries Team
Projects
SQL Queries
Archived in project
Milestone
23.1
Development

No branches or pull requests
2 participants
@mgartner @cockroach-teamcity