Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ccl/kvccl/kvtenantccl: TestTenantUpgradeInterlock failed #99836

Closed
cockroach-teamcity opened this issue Mar 28, 2023 · 32 comments
Closed

ccl/kvccl/kvtenantccl: TestTenantUpgradeInterlock failed #99836

cockroach-teamcity opened this issue Mar 28, 2023 · 32 comments
Labels
branch-release-23.1 Used to mark GA and release blockers, technical advisories, and bugs for 23.1 C-test-failure Broken test (automatically or manually discovered). GA-blocker O-robot Originated from a bot. skipped-test T-multitenant Issues owned by the multi-tenant virtual team
Milestone
23.1

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Mar 28, 2023 •
edited by exalate-issue-sync bot
Loading

ccl/kvccl/kvtenantccl.TestTenantUpgradeInterlock failed with artifacts on release-23.1 @ 14c45b16d482e069d3f92680572a16a85694664d:

Fatal error:

panic: test timed out after 14m55s

Stack:

goroutine 462054 [running]:
testing.(*M).startAlarm.func1()
	GOROOT/src/testing/testing.go:2036 +0x88
created by time.goFunc
	GOROOT/src/time/sleep.go:176 +0x38
Log preceding fatal error 

=== RUN   TestTenantUpgradeInterlock
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/ab29d1b80c7389017099436f61091ca9/logTestTenantUpgradeInterlock3458533120
    test_log_scope.go:79: use -show-logs to present logs inline
=== RUN   TestTenantUpgradeInterlock/current_binary_version_pause_after_first_check_for_instances
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "current binary version", configuration: "pause after first check for instances"
=== RUN   TestTenantUpgradeInterlock/current_binary_version_pause_after_fence_RPC
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "current binary version", configuration: "pause after fence RPC"

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

/cc @cockroachdb/multi-tenant

This test on roachdash | Improve this report!

Jira issue: CRDB-26107

Epic CRDB-27931
@cockroach-teamcity cockroach-teamcity added branch-release-23.1 Used to mark GA and release blockers, technical advisories, and bugs for 23.1 C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. labels Mar 28, 2023
@cockroach-teamcity cockroach-teamcity added this to the 23.1 milestone Mar 28, 2023
@rafiss rafiss added the T-multitenant Issues owned by the multi-tenant virtual team label Mar 28, 2023
@rafiss rafiss mentioned this issue Mar 28, 2023
Closed
@cockroach-teamcity
Copy link
Member Author

ccl/kvccl/kvtenantccl.TestTenantUpgradeInterlock failed with artifacts on release-23.1 @ a34af06d8e7230e4052db80d55da21a06013f81b:

Fatal error:

panic: test timed out after 14m55s

Stack:

goroutine 773513 [running]:
testing.(*M).startAlarm.func1()
	GOROOT/src/testing/testing.go:2036 +0x88
created by time.goFunc
	GOROOT/src/time/sleep.go:176 +0x38
Log preceding fatal error 

=== RUN   TestTenantUpgradeInterlock
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/ab29d1b80c7389017099436f61091ca9/logTestTenantUpgradeInterlock1129449798
    test_log_scope.go:79: use -show-logs to present logs inline
=== RUN   TestTenantUpgradeInterlock/current_binary_version_pause_after_write_to_settings_table
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "current binary version", configuration: "pause after write to settings table"
=== RUN   TestTenantUpgradeInterlock/lagging_binary_version_pause_after_first_check_for_instances
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "lagging binary version", configuration: "pause after first check for instances"
=== RUN   TestTenantUpgradeInterlock/current_binary_version_pause_after_fence_RPC
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "current binary version", configuration: "pause after fence RPC"
=== RUN   TestTenantUpgradeInterlock/current_binary_version_pause_after_second_check_of_instances
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "current binary version", configuration: "pause after second check of instances"
=== RUN   TestTenantUpgradeInterlock/current_binary_version_pause_after_first_check_for_instances
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "current binary version", configuration: "pause after first check for instances"
=== RUN   TestTenantUpgradeInterlock/current_binary_version_pause_after_migration
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "current binary version", configuration: "pause after migration"
=== RUN   TestTenantUpgradeInterlock/current_binary_version_pause_after_version_bump_RPC
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "current binary version", configuration: "pause after version bump RPC"
=== RUN   TestTenantUpgradeInterlock/lagging_binary_version_pause_after_fence_RPC
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "lagging binary version", configuration: "pause after fence RPC"

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@knz knz added the GA-blocker label Mar 29, 2023
@ajstorm
Copy link
Collaborator

ajstorm commented Mar 29, 2023

This one looks to be new. We're getting stuck trying to perform an upgrade of the host cluster (before we get into the interesting parts of tenant upgrade):

goroutine 81345 [select]:
github.com/cockroachdb/cockroach/pkg/util/retry.(*Retry).Next(0x40051d5d48)
	github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:128 +0xe8
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).waitForJobsToBeTerminalOrPaused(0x4005706900, {0x67de968, 0x4004908e70}, {0x400436e898, 0x1, 0x1}, 0x4002acef60)
	github.com/cockroachdb/cockroach/pkg/jobs/wait.go:103 +0x148
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).waitForJobs(0x4005706900?, {0x67de968, 0x4004908e70}, {0x400436e898, 0x1, 0x1}, 0x0?)
	github.com/cockroachdb/cockroach/pkg/jobs/wait.go:146 +0x10c
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Run(0x67de968?, {0x67de968, 0x4004908e70}, {0x400436e898, 0x1, 0x1})
	github.com/cockroachdb/cockroach/pkg/jobs/wait.go:202 +0xc0
github.com/cockroachdb/cockroach/pkg/upgrade/upgrademanager.(*Manager).runMigration(0x40011b9450, {0x67de968, 0x4004908e70}, {0x67fc488?, 0x40011c0820?}, {{0x4009ffd031?, 0x13?}}, {0xf588610?, 0x40?, 0x1?, ...}, ...)
	github.com/cockroachdb/cockroach/pkg/upgrade/upgrademanager/manager.go:709 +0x234
github.com/cockroachdb/cockroach/pkg/upgrade/upgrademanager.(*Manager).Migrate(0x40011b9450, {0x67de968?, 0x4007b35980?}, {{0x4009ffd031?, 0x400c81c6d8?}}, {{0x32a4598?, 0x0?, 0x7e992d0?, 0x40?}}, {{0x7e992e0?, ...}}, ...)
	github.com/cockroachdb/cockroach/pkg/upgrade/upgrademanager/manager.go:559 +0x65c
github.com/cockroachdb/cockroach/pkg/sql.runMigrationsAndUpgradeVersion({0x67de968, 0x4007b35980}, 0x400a8f25a0, {{0x4009ffd031?, 0x4007e99290?}}, {0x6845a30?, 0x400bdb8e80}, {0x6845a30?, 0x400bdb89c0}, 0x10?)
	github.com/cockroachdb/cockroach/pkg/sql/set_cluster_setting.go:662 +0x118
github.com/cockroachdb/cockroach/pkg/sql.setVersionSetting({0x67de968, 0x4007b35980}, 0x0?, 0x40001ac320, {0x400d44e072, 0x7}, {0x67ca940, 0x400c6e58c0}, {{0x4009ffd031?, 0x0?}}, ...)
	github.com/cockroachdb/cockroach/pkg/sql/set_cluster_setting.go:598 +0x474
github.com/cockroachdb/cockroach/pkg/sql.writeNonDefaultSettingValue({0x67de968, 0x4007b35980}, 0x0?, {0x67ca940, 0x400c6e58c0?}, {0x68313a0?, 0x40001ac320?}, {0x400d44e072, 0x7}, {{0x4009ffd031?, ...}}, ...)
	github.com/cockroachdb/cockroach/pkg/sql/set_cluster_setting.go:436 +0x158
github.com/cockroachdb/cockroach/pkg/sql.writeSettingInternal.func1({0x6813540?, 0x400bdb89c0?}, {0x67de968, 0x4007b35980}, {0x67ca940, 0x400c6e58c0}, {0x68313a0, 0x40001ac320}, {0x400d44e072, 0x7}, ...)
	github.com/cockroachdb/cockroach/pkg/sql/set_cluster_setting.go:372 +0xdc
github.com/cockroachdb/cockroach/pkg/sql.writeSettingInternal({0x67de968?, 0x4007b35980?}, 0x0?, {0x67ca940?, 0x400c6e58c0?}, {0x68313a0?, 0x40001ac320?}, {0x400d44e072?, 0x0?}, {{0x4009ffd031?, ...}}, ...)
	github.com/cockroachdb/cockroach/pkg/sql/set_cluster_setting.go:387 +0xc8
github.com/cockroachdb/cockroach/pkg/sql.(*setClusterSettingNode).startExec(0x40091f4200, {{0x67de968?, 0x4007b35980?}, 0x400beb2000?, 0x400740cde8?})
	github.com/cockroachdb/cockroach/pkg/sql/set_cluster_setting.go:259 +0x31c
github.com/cockroachdb/cockroach/pkg/sql.startExec.func2({0x400c81cdb8?, 0x33094e8?}, {0x67e1838, 0x40091f4200})
	github.com/cockroachdb/cockroach/pkg/sql/plan.go:518 +0xf4
github.com/cockroachdb/cockroach/pkg/sql.(*planVisitor).visitInternal.func1()
	github.com/cockroachdb/cockroach/pkg/sql/walk.go:112 +0x44
github.com/cockroachdb/cockroach/pkg/sql.(*planVisitor).visitInternal(0x400c81ced0, {0x67e1838?, 0x40091f4200?}, {0x529dc0f?, 0x13?})
	github.com/cockroachdb/cockroach/pkg/sql/walk.go:299 +0x1700
github.com/cockroachdb/cockroach/pkg/sql.(*planVisitor).visit(0x400c81ced0, {0x67e1838, 0x40091f4200})
	github.com/cockroachdb/cockroach/pkg/sql/walk.go:79 +0xe8
github.com/cockroachdb/cockroach/pkg/sql.walkPlan(...)
	github.com/cockroachdb/cockroach/pkg/sql/walk.go:43
github.com/cockroachdb/cockroach/pkg/sql.startExec({{0x67de968?, 0x4007b35980?}, 0x400beb2000?, 0x400740cde8?}, {0x67e1838, 0x40091f4200})
	github.com/cockroachdb/cockroach/pkg/sql/plan.go:521 +0xf0
github.com/cockroachdb/cockroach/pkg/sql.(*planNodeToRowSource).Start(0x4005c7c840, {0x67de968, 0x4007b35950})
	github.com/cockroachdb/cockroach/pkg/sql/plan_node_to_row_source.go:172 +0xa4
github.com/cockroachdb/cockroach/pkg/sql/colflow.(*FlowCoordinator).Start.func1()
	github.com/cockroachdb/cockroach/pkg/sql/colflow/flow_coordinator.go:121 +0x34
github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError(0x4006b93320?)
	github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:92 +0x58
github.com/cockroachdb/cockroach/pkg/sql/colflow.(*FlowCoordinator).Start(0x4006b93320, {0x67de8c0?, 0x40091f4300?})
	github.com/cockroachdb/cockroach/pkg/sql/colflow/flow_coordinator.go:120 +0x64
github.com/cockroachdb/cockroach/pkg/sql/execinfra.(*ProcessorBaseNoHelper).Run(0x4006b93320, {0x67de8c0?, 0x40091f4300?}, {0x67b2a88?, 0x40043e8a80})
	github.com/cockroachdb/cockroach/pkg/sql/execinfra/processorsbase.go:736 +0x4c
github.com/cockroachdb/cockroach/pkg/sql/flowinfra.(*FlowBase).Run(0x400c4c1b80, {0x67de8c0?, 0x40091f4300}, 0x74?)
	github.com/cockroachdb/cockroach/pkg/sql/flowinfra/flow.go:552 +0x178
github.com/cockroachdb/cockroach/pkg/sql/colflow.(*vectorizedFlow).Run(0x6847960?, {0x67de8c0?, 0x40091f4300?}, 0x50?)
	github.com/cockroachdb/cockroach/pkg/sql/colflow/vectorized_flow.go:311 +0x1dc
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).Run(0x4000a39b80, {0x67de968, 0x4007b357a0}, 0x4009e95600, 0x4009e95550, 0x4008fa6900, 0x40043e8a80, 0x400740d210, 0x400bdb8a30)
	github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:904 +0x884
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRun(0x400740cde8?, {0x67de968, 0x4007b357a0}, 0x0?, 0x4009e95600, 0x960490?, {{0x67e1838?, 0x40091f4200?}, 0x0?}, 0x40043e8a80, ...)
	github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:1900 +0x164
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRunAll.func2(0x400740cde8, 0xffff80165950?, {0x67de968, 0x4007b357a0}, 0x400c54c000?, 0x400bdb8a20?, 0x0?)
	github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:1636 +0x84
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRunAll(0x4000a39b80?, {0x67de968, 0x4007b357a0}, 0x400740d210, 0x400740cde8?, 0x400740cde8, 0x40043e8a80, 0x0)
	github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:1639 +0x148
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execWithDistSQLEngine(0x400740c800, {0x67de968?, 0x4007b357a0}, 0x400740cde8, 0x0?, {0xffff8311ac20?, 0x400527db20}, 0x400c81e038?, 0x400a055dd8)
	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1778 +0x4ac
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).dispatchToExecutionEngine(0x400740c800, {0x67de8c0, 0x40091f41c0}, 0x400740cde8, {0xffff8311ac20, 0x400527db20})
	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1353 +0x8a8
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmtInOpenState(0x400740c800, {0x67de968?, 0x4007b35560?}, {{0x68094c0, 0x4009e8bfa0}, {0x0, 0x0, 0x0}, {0x400d44e05e, 0x20}, ...}, ...)
	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:772 +0x1f94
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt.func1({0x67de968?, 0x4007b35560?})
	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:137 +0x8c
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execWithProfiling(0x67de968?, {0x67de968?, 0x4007b35560?}, {0x68094c0?, 0x4009e8bfa0?}, 0x30ddf80?, 0x400c81f258?)
	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:2667 +0x280
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt(0x400740c800, {0x67de968, 0x4007b35560}, {{0x68094c0, 0x4009e8bfa0}, {0x0, 0x0, 0x0}, {0x400d44e05e, 0x20}, ...}, ...)
	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:136 +0x36c
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execPortal(0x400740c800, {0x67de968, 0x4007b35560}, {{0x400d44e083, 0x0}, 0x400d5ecc60, {0x400bdb89b0, 0x1, 0x1}, {0x9d365e0, ...}, ...}, ...)
	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:226 +0x118
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd.func2({{0x400d44e09b, 0x0}, 0x0, {0xc100e685a723fbfb, 0x114f0158ee, 0x0}, 0x1}, 0x400740c800, 0x40051d97a8, 0x40051d9798, ...)
	github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:2072 +0x454
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd(0x400740c800)
	github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:2074 +0x528
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).run(0x400740c800, {0x67de8c0, 0x400c4da4c0}, 0x10deee8?, 0x400699bdd8?, 0x4007b6cd60?)
	github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1913 +0x18c
github.com/cockroachdb/cockroach/pkg/sql.(*Server).ServeConn(0x400c557f40?, {0x67de8c0?, 0x400c4da4c0?}, {0x400c4da4c0?}, 0x682f370?, 0x400368ce10?)
	github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:868 +0xb4
github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).processCommandsAsync.func1()
	github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:733 +0x298
created by github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).processCommandsAsync
	github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:646 +0x1e8
upgrade dump

and

goroutine 79568 [IO wait, 13 minutes]:
internal/poll.runtime_pollWait(0xffff86d381a8, 0x72)
	GOROOT/src/runtime/netpoll.go:305 +0xa0
internal/poll.(*pollDesc).wait(0x4000e8a600?, 0x40084b0000?, 0x0)
	GOROOT/src/internal/poll/fd_poll_runtime.go:84 +0x28
internal/poll.(*pollDesc).waitRead(...)
	GOROOT/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0x4000e8a600, {0x40084b0000, 0x700, 0x700})
	GOROOT/src/internal/poll/fd_unix.go:167 +0x1e8
net.(*netFD).Read(0x4000e8a600, {0x40084b0000?, 0xb06e35602c1781f8?, 0x1000000f78e6566?})
	GOROOT/src/net/fd_posix.go:55 +0x28
net.(*conn).Read(0x400b18a878, {0x40084b0000?, 0x0?, 0x6fb?})
	GOROOT/src/net/net.go:183 +0x34
crypto/tls.(*atLeastReader).Read(0x40068b0258, {0x40084b0000?, 0x0?, 0x81adb8?})
	GOROOT/src/crypto/tls/conn.go:787 +0x40
bytes.(*Buffer).ReadFrom(0x400711faf8, {0x6796380, 0x40068b0258})
	GOROOT/src/bytes/buffer.go:202 +0x90
crypto/tls.(*Conn).readFromUntil(0x400711f880, {0x67a46e0?, 0x400b18a878}, 0x818058?)
	GOROOT/src/crypto/tls/conn.go:809 +0xd4
crypto/tls.(*Conn).readRecordOrCCS(0x400711f880, 0x0)
	GOROOT/src/crypto/tls/conn.go:616 +0xd8
crypto/tls.(*Conn).readRecord(...)
	GOROOT/src/crypto/tls/conn.go:582
crypto/tls.(*Conn).Read(0x400711f880, {0x40088e4000, 0x1000, 0x200?})
	GOROOT/src/crypto/tls/conn.go:1287 +0x178
bufio.(*Reader).Read(0x400a772660, {0x4006eea920, 0x5, 0x81ca2c?})
	GOROOT/src/bufio/bufio.go:237 +0x1e8
io.ReadAtLeast({0x6794640, 0x400a772660}, {0x4006eea920, 0x5, 0x200}, 0x5)
	GOROOT/src/io/io.go:332 +0xa8
io.ReadFull(...)
	GOROOT/src/io/io.go:351
github.com/lib/pq.(*conn).recvMessage(0x4006eea900, 0x40039f87c0)
	github.com/lib/pq/external/com_github_lib_pq/conn.go:1004 +0xcc
github.com/lib/pq.(*conn).recv1Buf(0x4006eea900, 0x400711fba8?)
	github.com/lib/pq/external/com_github_lib_pq/conn.go:1059 +0x2c
github.com/lib/pq.(*conn).recv1(...)
	github.com/lib/pq/external/com_github_lib_pq/conn.go:1086
github.com/lib/pq.(*conn).readBindResponse(0x4006eea900)
	github.com/lib/pq/external/com_github_lib_pq/conn.go:1856 +0x38
github.com/lib/pq.(*stmt).exec(0x400adcba40, {0x400bdb8960, 0x1, 0x1})
	github.com/lib/pq/external/com_github_lib_pq/conn.go:1448 +0x568
github.com/lib/pq.(*stmt).Exec(0x400adcba40, {0x400bdb8960, 0x1, 0x1})
	github.com/lib/pq/external/com_github_lib_pq/conn.go:1406 +0xc4
github.com/lib/pq.(*conn).Exec(0x4006eea900, {0x52f3676, 0x20}, {0x400bdb8960, 0x1, 0x1})
	github.com/lib/pq/external/com_github_lib_pq/conn.go:939 +0x148
github.com/lib/pq.(*conn).ExecContext(0x4007e98eb0?, {0x67de8f8, 0x40001b2000}, {0x52f3676, 0x20}, {0x4007b34d80, 0x1, 0x0?})
	github.com/lib/pq/external/com_github_lib_pq/conn_go18.go:46 +0xb4
database/sql.ctxDriverExec({0x67de8f8?, 0x40001b2000?}, {0xffff8311ab60?, 0x4006eea900?}, {0x0?, 0x0?}, {0x52f3676?, 0x900900?}, {0x4007b34d80?, 0x47a2f80?, ...})
	GOROOT/src/database/sql/ctxutil.go:31 +0xb0
database/sql.(*DB).execDC.func2()
	GOROOT/src/database/sql/sql.go:1679 +0x134
database/sql.withLock({0x67b1480, 0x400b523710}, 0x4004b1edd8)
	GOROOT/src/database/sql/sql.go:3439 +0x7c
database/sql.(*DB).execDC(0x4005acb6c0?, {0x67de8f8, 0x40001b2000}, 0x400b523710, 0x4004b1eed8?, {0x52f3676, 0x20}, {0x400bdb8940, 0x1, 0x1})
	GOROOT/src/database/sql/sql.go:1674 +0x19c
database/sql.(*DB).exec(0x4004b1ef58?, {0x67de8f8, 0x40001b2000}, {0x52f3676, 0x20}, {0x400bdb8940, 0x1, 0x1}, 0xe4?)
	GOROOT/src/database/sql/sql.go:1659 +0xa0
database/sql.(*DB).ExecContext(0x2?, {0x67de8f8, 0x40001b2000}, {0x52f3676, 0x20}, {0x400bdb8940, 0x1, 0x1})
	GOROOT/src/database/sql/sql.go:1633 +0xa8
github.com/cockroachdb/cockroach/pkg/testutils/sqlutils.(*SQLRunner).Exec(0x4004b1f788, {0x67a4e60?, 0x40099f8000}, {0x52f3676, 0x20}, {0x400bdb8940, 0x1, 0x1})
	github.com/cockroachdb/cockroach/pkg/testutils/sqlutils/sql_runner.go:79 +0xe4
github.com/cockroachdb/cockroach/pkg/ccl/kvccl/kvtenantccl_test.TestTenantUpgradeInterlock.func1(0x40099f8000, 0x0, {{0x52a9cd1, 0x15}, {{0x4000d211e0, 0x1, 0x1}, {0x4000d21200, 0x1, 0x1}}, ...})
	github.com/cockroachdb/cockroach/pkg/ccl/kvccl/kvtenantccl_test/pkg/ccl/kvccl/kvtenantccl/tenant_upgrade_test.go:669 +0xab0
github.com/cockroachdb/cockroach/pkg/ccl/kvccl/kvtenantccl_test.TestTenantUpgradeInterlock.func2(0x67de8f8?)
	github.com/cockroachdb/cockroach/pkg/ccl/kvccl/kvtenantccl_test/pkg/ccl/kvccl/kvtenantccl/tenant_upgrade_test.go:815 +0x58
testing.tRunner(0x40099f8000, 0x40069f8000)
	GOROOT/src/testing/testing.go:1446 +0x10c
created by testing.(*T).Run
	GOROOT/src/testing/testing.go:1493 +0x328
test goroutine

I can't seem to find the cockroach logs to see which migration we're stuck in, but I'm almost certainly just missing where those logs are stored in TC.

@stevendanna stevendanna added the release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. label Mar 29, 2023
@cockroach-teamcity
Copy link
Member Author

ccl/kvccl/kvtenantccl.TestTenantUpgradeInterlock failed with artifacts on release-23.1 @ 26c4bf986eec832f4a68435876b655e268e1359b:

Fatal error:

panic: test timed out after 14m55s

Stack:

goroutine 777655 [running]:
testing.(*M).startAlarm.func1()
	GOROOT/src/testing/testing.go:2036 +0x88
created by time.goFunc
	GOROOT/src/time/sleep.go:176 +0x38
Log preceding fatal error 

=== RUN   TestTenantUpgradeInterlock
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/ab29d1b80c7389017099436f61091ca9/logTestTenantUpgradeInterlock3799328003
    test_log_scope.go:79: use -show-logs to present logs inline
=== RUN   TestTenantUpgradeInterlock/current_binary_version_pause_after_version_bump_RPC
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "current binary version", configuration: "pause after version bump RPC"
=== RUN   TestTenantUpgradeInterlock/current_binary_version_pause_after_write_to_settings_table
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "current binary version", configuration: "pause after write to settings table"
=== RUN   TestTenantUpgradeInterlock/current_binary_version_pause_after_first_check_for_instances
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "current binary version", configuration: "pause after first check for instances"
=== RUN   TestTenantUpgradeInterlock/current_binary_version_pause_after_migration
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "current binary version", configuration: "pause after migration"
=== RUN   TestTenantUpgradeInterlock/current_binary_version_pause_after_second_check_of_instances
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "current binary version", configuration: "pause after second check of instances"
=== RUN   TestTenantUpgradeInterlock/lagging_binary_version_pause_after_first_check_for_instances
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "lagging binary version", configuration: "pause after first check for instances"

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/kvccl/kvtenantccl.TestTenantUpgradeInterlock failed with artifacts on release-23.1 @ 03e2dd23abedb56a045f0d9e3f7313df69296150:

Fatal error:

panic: test timed out after 14m55s

Stack:

goroutine 490179 [running]:
testing.(*M).startAlarm.func1()
	GOROOT/src/testing/testing.go:2036 +0x88
created by time.goFunc
	GOROOT/src/time/sleep.go:176 +0x38
Log preceding fatal error 

=== RUN   TestTenantUpgradeInterlock
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/ab29d1b80c7389017099436f61091ca9/logTestTenantUpgradeInterlock102840197
    test_log_scope.go:79: use -show-logs to present logs inline
=== RUN   TestTenantUpgradeInterlock/current_binary_version_pause_after_fence_RPC
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "current binary version", configuration: "pause after fence RPC"

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/kvccl/kvtenantccl.TestTenantUpgradeInterlock failed with artifacts on release-23.1 @ 11dd026fe67f08c609fc67428d7159854f95d1cd:

Fatal error:

panic: test timed out after 14m55s

Stack:

goroutine 1074742 [running]:
testing.(*M).startAlarm.func1()
	GOROOT/src/testing/testing.go:2036 +0x88
created by time.goFunc
	GOROOT/src/time/sleep.go:176 +0x38
Log preceding fatal error 

=== RUN   TestTenantUpgradeInterlock
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/ab29d1b80c7389017099436f61091ca9/logTestTenantUpgradeInterlock1396108167
    test_log_scope.go:79: use -show-logs to present logs inline
=== RUN   TestTenantUpgradeInterlock/current_binary_version_pause_after_version_bump_RPC
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "current binary version", configuration: "pause after version bump RPC"
=== RUN   TestTenantUpgradeInterlock/lagging_binary_version_pause_after_fence_RPC
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "lagging binary version", configuration: "pause after fence RPC"
=== RUN   TestTenantUpgradeInterlock/lagging_binary_version_pause_after_migration
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "lagging binary version", configuration: "pause after migration"
=== RUN   TestTenantUpgradeInterlock/lagging_binary_version_pause_after_second_check_of_instances
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "lagging binary version", configuration: "pause after second check of instances"
=== RUN   TestTenantUpgradeInterlock/current_binary_version_pause_after_first_check_for_instances
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "current binary version", configuration: "pause after first check for instances"
=== RUN   TestTenantUpgradeInterlock/current_binary_version_pause_after_second_check_of_instances
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "current binary version", configuration: "pause after second check of instances"
=== RUN   TestTenantUpgradeInterlock/lagging_binary_version_pause_after_fence_write_to_settings_table
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "lagging binary version", configuration: "pause after fence write to settings table"
=== RUN   TestTenantUpgradeInterlock/lagging_binary_version_pause_after_first_check_for_instances
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "lagging binary version", configuration: "pause after first check for instances"
=== RUN   TestTenantUpgradeInterlock/current_binary_version_pause_after_fence_RPC
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "current binary version", configuration: "pause after fence RPC"
=== RUN   TestTenantUpgradeInterlock/current_binary_version_pause_after_write_to_settings_table
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "current binary version", configuration: "pause after write to settings table"
=== RUN   TestTenantUpgradeInterlock/lagging_binary_version_pause_after_version_bump_RPC
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "lagging binary version", configuration: "pause after version bump RPC"

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/kvccl/kvtenantccl.TestTenantUpgradeInterlock failed with artifacts on release-23.1 @ 11dd026fe67f08c609fc67428d7159854f95d1cd:

Fatal error:

panic: test timed out after 14m55s

Stack:

goroutine 1086875 [running]:
testing.(*M).startAlarm.func1()
	GOROOT/src/testing/testing.go:2036 +0x88
created by time.goFunc
	GOROOT/src/time/sleep.go:176 +0x38
Log preceding fatal error 

=== RUN   TestTenantUpgradeInterlock
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/ab29d1b80c7389017099436f61091ca9/logTestTenantUpgradeInterlock711577095
    test_log_scope.go:79: use -show-logs to present logs inline
=== RUN   TestTenantUpgradeInterlock/lagging_binary_version_pause_after_second_check_of_instances
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "lagging binary version", configuration: "pause after second check of instances"
=== RUN   TestTenantUpgradeInterlock/current_binary_version_pause_after_migration
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "current binary version", configuration: "pause after migration"
=== RUN   TestTenantUpgradeInterlock/current_binary_version_pause_after_version_bump_RPC
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "current binary version", configuration: "pause after version bump RPC"
=== RUN   TestTenantUpgradeInterlock/lagging_binary_version_pause_after_fence_RPC
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "lagging binary version", configuration: "pause after fence RPC"
=== RUN   TestTenantUpgradeInterlock/lagging_binary_version_pause_after_fence_write_to_settings_table
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "lagging binary version", configuration: "pause after fence write to settings table"
=== RUN   TestTenantUpgradeInterlock/lagging_binary_version_pause_after_first_check_for_instances
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "lagging binary version", configuration: "pause after first check for instances"
=== RUN   TestTenantUpgradeInterlock/current_binary_version_pause_after_fence_RPC
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "current binary version", configuration: "pause after fence RPC"
=== RUN   TestTenantUpgradeInterlock/current_binary_version_pause_after_fence_write_to_settings_table
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "current binary version", configuration: "pause after fence write to settings table"
=== RUN   TestTenantUpgradeInterlock/current_binary_version_pause_after_first_check_for_instances
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "current binary version", configuration: "pause after first check for instances"
=== RUN   TestTenantUpgradeInterlock/current_binary_version_pause_after_second_check_of_instances
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "current binary version", configuration: "pause after second check of instances"
=== RUN   TestTenantUpgradeInterlock/current_binary_version_pause_after_write_to_settings_table
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "current binary version", configuration: "pause after write to settings table"
=== RUN   TestTenantUpgradeInterlock/lagging_binary_version_pause_after_migration
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "lagging binary version", configuration: "pause after migration"

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/kvccl/kvtenantccl.TestTenantUpgradeInterlock failed with artifacts on release-23.1 @ 244c452024226833fdf9f15939aebab68508b0e3:

Fatal error:

panic: test timed out after 14m55s

Stack:

goroutine 1040319 [running]:
testing.(*M).startAlarm.func1()
	GOROOT/src/testing/testing.go:2036 +0x88
created by time.goFunc
	GOROOT/src/time/sleep.go:176 +0x38
Log preceding fatal error 

=== RUN   TestTenantUpgradeInterlock
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/ab29d1b80c7389017099436f61091ca9/logTestTenantUpgradeInterlock3181946381
    test_log_scope.go:79: use -show-logs to present logs inline
=== RUN   TestTenantUpgradeInterlock/lagging_binary_version_pause_after_fence_RPC
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "lagging binary version", configuration: "pause after fence RPC"
=== RUN   TestTenantUpgradeInterlock/lagging_binary_version_pause_after_fence_write_to_settings_table
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "lagging binary version", configuration: "pause after fence write to settings table"
=== RUN   TestTenantUpgradeInterlock/lagging_binary_version_pause_after_second_check_of_instances
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "lagging binary version", configuration: "pause after second check of instances"
=== RUN   TestTenantUpgradeInterlock/current_binary_version_pause_after_fence_write_to_settings_table
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "current binary version", configuration: "pause after fence write to settings table"
=== RUN   TestTenantUpgradeInterlock/current_binary_version_pause_after_version_bump_RPC
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "current binary version", configuration: "pause after version bump RPC"
=== RUN   TestTenantUpgradeInterlock/current_binary_version_pause_after_fence_RPC
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "current binary version", configuration: "pause after fence RPC"
=== RUN   TestTenantUpgradeInterlock/lagging_binary_version_pause_after_first_check_for_instances
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "lagging binary version", configuration: "pause after first check for instances"
=== RUN   TestTenantUpgradeInterlock/lagging_binary_version_pause_after_version_bump_RPC
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "lagging binary version", configuration: "pause after version bump RPC"

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/kvccl/kvtenantccl.TestTenantUpgradeInterlock failed with artifacts on release-23.1 @ 46610508d2eaa9118ac415c47914f0b118ab8992:

Fatal error:

panic: test timed out after 14m55s

Stack:

goroutine 1064563 [running]:
testing.(*M).startAlarm.func1()
	GOROOT/src/testing/testing.go:2036 +0x88
created by time.goFunc
	GOROOT/src/time/sleep.go:176 +0x38
Log preceding fatal error 

=== RUN   TestTenantUpgradeInterlock
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/ab29d1b80c7389017099436f61091ca9/logTestTenantUpgradeInterlock170361422
    test_log_scope.go:79: use -show-logs to present logs inline
=== RUN   TestTenantUpgradeInterlock/current_binary_version_pause_after_fence_write_to_settings_table
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "current binary version", configuration: "pause after fence write to settings table"
=== RUN   TestTenantUpgradeInterlock/current_binary_version_pause_after_migration
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "current binary version", configuration: "pause after migration"
=== RUN   TestTenantUpgradeInterlock/lagging_binary_version_pause_after_fence_RPC
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "lagging binary version", configuration: "pause after fence RPC"
=== RUN   TestTenantUpgradeInterlock/lagging_binary_version_pause_after_second_check_of_instances
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "lagging binary version", configuration: "pause after second check of instances"
=== RUN   TestTenantUpgradeInterlock/current_binary_version_pause_after_first_check_for_instances
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "current binary version", configuration: "pause after first check for instances"
=== RUN   TestTenantUpgradeInterlock/current_binary_version_pause_after_version_bump_RPC
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "current binary version", configuration: "pause after version bump RPC"
=== RUN   TestTenantUpgradeInterlock/lagging_binary_version_pause_after_fence_write_to_settings_table
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "lagging binary version", configuration: "pause after fence write to settings table"
=== RUN   TestTenantUpgradeInterlock/current_binary_version_pause_after_fence_RPC
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "current binary version", configuration: "pause after fence RPC"
=== RUN   TestTenantUpgradeInterlock/current_binary_version_pause_after_second_check_of_instances
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "current binary version", configuration: "pause after second check of instances"
=== RUN   TestTenantUpgradeInterlock/lagging_binary_version_pause_after_migration
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "lagging binary version", configuration: "pause after migration"
=== RUN   TestTenantUpgradeInterlock/lagging_binary_version_pause_after_write_to_settings_table
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "lagging binary version", configuration: "pause after write to settings table"

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/kvccl/kvtenantccl.TestTenantUpgradeInterlock failed with artifacts on release-23.1 @ d6d73677266af0a87a744cee73d83dd1965a31cf:

Fatal error:

panic: test timed out after 14m55s

Stack:

goroutine 1033485 [running]:
testing.(*M).startAlarm.func1()
	GOROOT/src/testing/testing.go:2036 +0x88
created by time.goFunc
	GOROOT/src/time/sleep.go:176 +0x38
Log preceding fatal error 

=== RUN   TestTenantUpgradeInterlock
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/ab29d1b80c7389017099436f61091ca9/logTestTenantUpgradeInterlock2150302589
    test_log_scope.go:79: use -show-logs to present logs inline
=== RUN   TestTenantUpgradeInterlock/current_binary_version_pause_after_first_check_for_instances
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "current binary version", configuration: "pause after first check for instances"
=== RUN   TestTenantUpgradeInterlock/lagging_binary_version_pause_after_migration
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "lagging binary version", configuration: "pause after migration"
=== RUN   TestTenantUpgradeInterlock/current_binary_version_pause_after_fence_RPC
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "current binary version", configuration: "pause after fence RPC"
=== RUN   TestTenantUpgradeInterlock/current_binary_version_pause_after_fence_write_to_settings_table
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "current binary version", configuration: "pause after fence write to settings table"
=== RUN   TestTenantUpgradeInterlock/lagging_binary_version_pause_after_second_check_of_instances
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "lagging binary version", configuration: "pause after second check of instances"
=== RUN   TestTenantUpgradeInterlock/lagging_binary_version_pause_after_fence_write_to_settings_table
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "lagging binary version", configuration: "pause after fence write to settings table"
=== RUN   TestTenantUpgradeInterlock/lagging_binary_version_pause_after_version_bump_RPC
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "lagging binary version", configuration: "pause after version bump RPC"

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/kvccl/kvtenantccl.TestTenantUpgradeInterlock failed with artifacts on release-23.1 @ d124e411be3eedc1ee0911eba57eccba577f104e:

Fatal error:

panic: test timed out after 14m55s

Stack:

goroutine 1116643 [running]:
testing.(*M).startAlarm.func1()
	GOROOT/src/testing/testing.go:2036 +0x88
created by time.goFunc
	GOROOT/src/time/sleep.go:176 +0x38
Log preceding fatal error 

=== RUN   TestTenantUpgradeInterlock
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/ab29d1b80c7389017099436f61091ca9/logTestTenantUpgradeInterlock1923779585
    test_log_scope.go:79: use -show-logs to present logs inline
=== RUN   TestTenantUpgradeInterlock/current_binary_version_pause_after_fence_write_to_settings_table
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "current binary version", configuration: "pause after fence write to settings table"
=== RUN   TestTenantUpgradeInterlock/lagging_binary_version_pause_after_fence_RPC
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "lagging binary version", configuration: "pause after fence RPC"
=== RUN   TestTenantUpgradeInterlock/lagging_binary_version_pause_after_migration
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "lagging binary version", configuration: "pause after migration"
=== RUN   TestTenantUpgradeInterlock/lagging_binary_version_pause_after_write_to_settings_table
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "lagging binary version", configuration: "pause after write to settings table"

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/kvccl/kvtenantccl.TestTenantUpgradeInterlock failed with artifacts on release-23.1 @ 2eba36137983f5495350f6d29190a718255a7428:

Fatal error:

panic: test timed out after 14m55s

Stack:

goroutine 432326 [running]:
testing.(*M).startAlarm.func1()
	GOROOT/src/testing/testing.go:2036 +0x88
created by time.goFunc
	GOROOT/src/time/sleep.go:176 +0x38
Log preceding fatal error 

=== RUN   TestTenantUpgradeInterlock
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/ab29d1b80c7389017099436f61091ca9/logTestTenantUpgradeInterlock1653482821
    test_log_scope.go:79: use -show-logs to present logs inline
=== RUN   TestTenantUpgradeInterlock/lagging_binary_version_pause_after_fence_RPC
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "lagging binary version", configuration: "pause after fence RPC"

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/kvccl/kvtenantccl.TestTenantUpgradeInterlock failed with artifacts on release-23.1 @ d3b6cded68ee6227c0532f45256c379838329acd:

Fatal error:

panic: test timed out after 14m55s

Stack:

goroutine 1141150 [running]:
testing.(*M).startAlarm.func1()
	GOROOT/src/testing/testing.go:2036 +0x88
created by time.goFunc
	GOROOT/src/time/sleep.go:176 +0x38
Log preceding fatal error 

=== RUN   TestTenantUpgradeInterlock
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/ab29d1b80c7389017099436f61091ca9/logTestTenantUpgradeInterlock263860878
    test_log_scope.go:79: use -show-logs to present logs inline
=== RUN   TestTenantUpgradeInterlock/current_binary_version_pause_after_fence_RPC
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "current binary version", configuration: "pause after fence RPC"
=== RUN   TestTenantUpgradeInterlock/current_binary_version_pause_after_first_check_for_instances
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "current binary version", configuration: "pause after first check for instances"
=== RUN   TestTenantUpgradeInterlock/current_binary_version_pause_after_second_check_of_instances
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "current binary version", configuration: "pause after second check of instances"
=== RUN   TestTenantUpgradeInterlock/current_binary_version_pause_after_version_bump_RPC
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "current binary version", configuration: "pause after version bump RPC"
=== RUN   TestTenantUpgradeInterlock/current_binary_version_pause_after_write_to_settings_table
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "current binary version", configuration: "pause after write to settings table"

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/kvccl/kvtenantccl.TestTenantUpgradeInterlock failed with artifacts on release-23.1 @ d3b6cded68ee6227c0532f45256c379838329acd:

Fatal error:

panic: test timed out after 14m55s

Stack:

goroutine 667900 [running]:
testing.(*M).startAlarm.func1()
	GOROOT/src/testing/testing.go:2036 +0x8e
created by time.goFunc
	GOROOT/src/time/sleep.go:176 +0x32
Log preceding fatal error 

=== RUN   TestTenantUpgradeInterlock
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/e7ead71a4f5b50fae4d2996264db30c2/logTestTenantUpgradeInterlock1908587643
    test_log_scope.go:79: use -show-logs to present logs inline
=== RUN   TestTenantUpgradeInterlock/current_binary_version_pause_after_fence_write_to_settings_table
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "current binary version", configuration: "pause after fence write to settings table"
=== RUN   TestTenantUpgradeInterlock/current_binary_version_pause_after_first_check_for_instances
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "current binary version", configuration: "pause after first check for instances"
=== RUN   TestTenantUpgradeInterlock/current_binary_version_pause_after_migration
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "current binary version", configuration: "pause after migration"

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/kvccl/kvtenantccl.TestTenantUpgradeInterlock failed with artifacts on release-23.1 @ d3b6cded68ee6227c0532f45256c379838329acd:

Fatal error:

panic: test timed out after 14m55s

Stack:

goroutine 1180710 [running]:
testing.(*M).startAlarm.func1()
	GOROOT/src/testing/testing.go:2036 +0x88
created by time.goFunc
	GOROOT/src/time/sleep.go:176 +0x38
Log preceding fatal error 

=== RUN   TestTenantUpgradeInterlock
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/ab29d1b80c7389017099436f61091ca9/logTestTenantUpgradeInterlock2429876592
    test_log_scope.go:79: use -show-logs to present logs inline
=== RUN   TestTenantUpgradeInterlock/current_binary_version_pause_after_migration
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "current binary version", configuration: "pause after migration"
=== RUN   TestTenantUpgradeInterlock/current_binary_version_pause_after_second_check_of_instances
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "current binary version", configuration: "pause after second check of instances"
=== RUN   TestTenantUpgradeInterlock/current_binary_version_pause_after_write_to_settings_table
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "current binary version", configuration: "pause after write to settings table"
=== RUN   TestTenantUpgradeInterlock/lagging_binary_version_pause_after_fence_RPC
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "lagging binary version", configuration: "pause after fence RPC"
=== RUN   TestTenantUpgradeInterlock/current_binary_version_pause_after_fence_write_to_settings_table
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "current binary version", configuration: "pause after fence write to settings table"
=== RUN   TestTenantUpgradeInterlock/lagging_binary_version_pause_after_fence_write_to_settings_table
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "lagging binary version", configuration: "pause after fence write to settings table"
=== RUN   TestTenantUpgradeInterlock/lagging_binary_version_pause_after_first_check_for_instances
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "lagging binary version", configuration: "pause after first check for instances"
=== RUN   TestTenantUpgradeInterlock/current_binary_version_pause_after_first_check_for_instances
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "current binary version", configuration: "pause after first check for instances"
=== RUN   TestTenantUpgradeInterlock/current_binary_version_pause_after_version_bump_RPC
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "current binary version", configuration: "pause after version bump RPC"
=== RUN   TestTenantUpgradeInterlock/lagging_binary_version_pause_after_write_to_settings_table
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "lagging binary version", configuration: "pause after write to settings table"

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/kvccl/kvtenantccl.TestTenantUpgradeInterlock failed with artifacts on release-23.1 @ 22ac631d404984aac164ad50c2bc1fe51fc55451:

Fatal error:

panic: test timed out after 14m55s

Stack:

goroutine 1128379 [running]:
testing.(*M).startAlarm.func1()
	GOROOT/src/testing/testing.go:2036 +0x88
created by time.goFunc
	GOROOT/src/time/sleep.go:176 +0x38
Log preceding fatal error 

=== RUN   TestTenantUpgradeInterlock
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/ab29d1b80c7389017099436f61091ca9/logTestTenantUpgradeInterlock2586897734
    test_log_scope.go:79: use -show-logs to present logs inline
=== RUN   TestTenantUpgradeInterlock/current_binary_version_pause_after_first_check_for_instances
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "current binary version", configuration: "pause after first check for instances"
=== RUN   TestTenantUpgradeInterlock/current_binary_version_pause_after_migration
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "current binary version", configuration: "pause after migration"
=== RUN   TestTenantUpgradeInterlock/lagging_binary_version_pause_after_fence_write_to_settings_table
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "lagging binary version", configuration: "pause after fence write to settings table"
=== RUN   TestTenantUpgradeInterlock/lagging_binary_version_pause_after_first_check_for_instances
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "lagging binary version", configuration: "pause after first check for instances"
=== RUN   TestTenantUpgradeInterlock/lagging_binary_version_pause_after_migration
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "lagging binary version", configuration: "pause after migration"
=== RUN   TestTenantUpgradeInterlock/lagging_binary_version_pause_after_write_to_settings_table
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "lagging binary version", configuration: "pause after write to settings table"

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/kvccl/kvtenantccl.TestTenantUpgradeInterlock failed with artifacts on release-23.1 @ 5d5abe704b0551556c35d51aa673d3162f493aa8:

Fatal error:

panic: test timed out after 14m55s

Stack:

goroutine 1101289 [running]:
testing.(*M).startAlarm.func1()
	GOROOT/src/testing/testing.go:2036 +0x88
created by time.goFunc
	GOROOT/src/time/sleep.go:176 +0x38
Log preceding fatal error 

=== RUN   TestTenantUpgradeInterlock
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/ab29d1b80c7389017099436f61091ca9/logTestTenantUpgradeInterlock1422067922
    test_log_scope.go:79: use -show-logs to present logs inline
=== RUN   TestTenantUpgradeInterlock/current_binary_version_pause_after_fence_write_to_settings_table
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "current binary version", configuration: "pause after fence write to settings table"
=== RUN   TestTenantUpgradeInterlock/current_binary_version_pause_after_version_bump_RPC
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "current binary version", configuration: "pause after version bump RPC"

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/kvccl/kvtenantccl.TestTenantUpgradeInterlock failed with artifacts on release-23.1 @ 09e20c40188a87ddbfaad7ea50b79b0b6f08c2ae:

Fatal error:

panic: test timed out after 14m55s

Stack:

goroutine 1044546 [running]:
testing.(*M).startAlarm.func1()
	GOROOT/src/testing/testing.go:2036 +0x88
created by time.goFunc
	GOROOT/src/time/sleep.go:176 +0x38
Log preceding fatal error 

=== RUN   TestTenantUpgradeInterlock
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/ab29d1b80c7389017099436f61091ca9/logTestTenantUpgradeInterlock154175511
    test_log_scope.go:79: use -show-logs to present logs inline
=== RUN   TestTenantUpgradeInterlock/current_binary_version_pause_after_migration
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "current binary version", configuration: "pause after migration"
=== RUN   TestTenantUpgradeInterlock/current_binary_version_pause_after_second_check_of_instances
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "current binary version", configuration: "pause after second check of instances"
=== RUN   TestTenantUpgradeInterlock/lagging_binary_version_pause_after_fence_RPC
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "lagging binary version", configuration: "pause after fence RPC"
=== RUN   TestTenantUpgradeInterlock/lagging_binary_version_pause_after_first_check_for_instances
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "lagging binary version", configuration: "pause after first check for instances"
=== RUN   TestTenantUpgradeInterlock/lagging_binary_version_pause_after_second_check_of_instances
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "lagging binary version", configuration: "pause after second check of instances"
=== RUN   TestTenantUpgradeInterlock/current_binary_version_pause_after_first_check_for_instances
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "current binary version", configuration: "pause after first check for instances"
=== RUN   TestTenantUpgradeInterlock/current_binary_version_pause_after_version_bump_RPC
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "current binary version", configuration: "pause after version bump RPC"
=== RUN   TestTenantUpgradeInterlock/current_binary_version_pause_after_write_to_settings_table
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "current binary version", configuration: "pause after write to settings table"
=== RUN   TestTenantUpgradeInterlock/lagging_binary_version_pause_after_fence_write_to_settings_table
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "lagging binary version", configuration: "pause after fence write to settings table"
=== RUN   TestTenantUpgradeInterlock/lagging_binary_version_pause_after_migration
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "lagging binary version", configuration: "pause after migration"

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/kvccl/kvtenantccl.TestTenantUpgradeInterlock failed with artifacts on release-23.1 @ fe0aa4296d849289dcfda13ca74afa7862e22c80:

Fatal error:

panic: test timed out after 14m55s

Stack:

goroutine 1161885 [running]:
testing.(*M).startAlarm.func1()
	GOROOT/src/testing/testing.go:2036 +0x88
created by time.goFunc
	GOROOT/src/time/sleep.go:176 +0x38
Log preceding fatal error 

=== RUN   TestTenantUpgradeInterlock
I230403 15:16:12.368533 1 (gostd) rand.go:199  [-] 1  random seed: -2521041708731616950
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/ab29d1b80c7389017099436f61091ca9/logTestTenantUpgradeInterlock3149090702
    test_log_scope.go:79: use -show-logs to present logs inline
=== RUN   TestTenantUpgradeInterlock/current_binary_version_pause_after_fence_write_to_settings_table
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "current binary version", configuration: "pause after fence write to settings table"
=== RUN   TestTenantUpgradeInterlock/current_binary_version_pause_after_second_check_of_instances
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "current binary version", configuration: "pause after second check of instances"
=== RUN   TestTenantUpgradeInterlock/lagging_binary_version_pause_after_fence_RPC
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "lagging binary version", configuration: "pause after fence RPC"
=== RUN   TestTenantUpgradeInterlock/lagging_binary_version_pause_after_write_to_settings_table
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "lagging binary version", configuration: "pause after write to settings table"

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/kvccl/kvtenantccl.TestTenantUpgradeInterlock failed with artifacts on release-23.1 @ e8c451432b16df14aa85b43553229083a1fa34ab:

Fatal error:

panic: test timed out after 14m55s

Stack:

goroutine 974879 [running]:
testing.(*M).startAlarm.func1()
	GOROOT/src/testing/testing.go:2036 +0x88
created by time.goFunc
	GOROOT/src/time/sleep.go:176 +0x38
Log preceding fatal error 

=== RUN   TestTenantUpgradeInterlock
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/ab29d1b80c7389017099436f61091ca9/logTestTenantUpgradeInterlock544887419
    test_log_scope.go:79: use -show-logs to present logs inline
=== RUN   TestTenantUpgradeInterlock/current_binary_version_pause_after_fence_RPC
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "current binary version", configuration: "pause after fence RPC"
=== RUN   TestTenantUpgradeInterlock/current_binary_version_pause_after_fence_write_to_settings_table
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "current binary version", configuration: "pause after fence write to settings table"
=== RUN   TestTenantUpgradeInterlock/current_binary_version_pause_after_first_check_for_instances
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "current binary version", configuration: "pause after first check for instances"
=== RUN   TestTenantUpgradeInterlock/current_binary_version_pause_after_migration
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "current binary version", configuration: "pause after migration"
=== RUN   TestTenantUpgradeInterlock/current_binary_version_pause_after_second_check_of_instances
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "current binary version", configuration: "pause after second check of instances"
=== RUN   TestTenantUpgradeInterlock/current_binary_version_pause_after_version_bump_RPC
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "current binary version", configuration: "pause after version bump RPC"
=== RUN   TestTenantUpgradeInterlock/current_binary_version_pause_after_write_to_settings_table
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "current binary version", configuration: "pause after write to settings table"
=== RUN   TestTenantUpgradeInterlock/lagging_binary_version_pause_after_fence_RPC
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "lagging binary version", configuration: "pause after fence RPC"
=== RUN   TestTenantUpgradeInterlock/lagging_binary_version_pause_after_fence_write_to_settings_table
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "lagging binary version", configuration: "pause after fence write to settings table"
=== RUN   TestTenantUpgradeInterlock/lagging_binary_version_pause_after_first_check_for_instances
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "lagging binary version", configuration: "pause after first check for instances"
=== RUN   TestTenantUpgradeInterlock/lagging_binary_version_pause_after_migration
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "lagging binary version", configuration: "pause after migration"

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/kvccl/kvtenantccl.TestTenantUpgradeInterlock failed with artifacts on release-23.1 @ 671f0c231b72dd4b115a26036468fce28376de20:

Fatal error:

panic: test timed out after 14m55s

Stack:

goroutine 1161735 [running]:
testing.(*M).startAlarm.func1()
	GOROOT/src/testing/testing.go:2036 +0x88
created by time.goFunc
	GOROOT/src/time/sleep.go:176 +0x38
Log preceding fatal error 

=== RUN   TestTenantUpgradeInterlock
I230403 18:54:08.015957 1 (gostd) rand.go:199  [-] 1  random seed: -3625454191319887830
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/ab29d1b80c7389017099436f61091ca9/logTestTenantUpgradeInterlock3266532594
    test_log_scope.go:79: use -show-logs to present logs inline
=== RUN   TestTenantUpgradeInterlock/current_binary_version_pause_after_fence_RPC
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "current binary version", configuration: "pause after fence RPC"
=== RUN   TestTenantUpgradeInterlock/current_binary_version_pause_after_fence_write_to_settings_table
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "current binary version", configuration: "pause after fence write to settings table"
=== RUN   TestTenantUpgradeInterlock/current_binary_version_pause_after_write_to_settings_table
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "current binary version", configuration: "pause after write to settings table"
=== RUN   TestTenantUpgradeInterlock/lagging_binary_version_pause_after_fence_RPC
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "lagging binary version", configuration: "pause after fence RPC"
=== RUN   TestTenantUpgradeInterlock/lagging_binary_version_pause_after_fence_write_to_settings_table
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "lagging binary version", configuration: "pause after fence write to settings table"
=== RUN   TestTenantUpgradeInterlock/lagging_binary_version_pause_after_first_check_for_instances
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "lagging binary version", configuration: "pause after first check for instances"
=== RUN   TestTenantUpgradeInterlock/lagging_binary_version_pause_after_version_bump_RPC
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "lagging binary version", configuration: "pause after version bump RPC"
=== RUN   TestTenantUpgradeInterlock/current_binary_version_pause_after_migration
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "current binary version", configuration: "pause after migration"
=== RUN   TestTenantUpgradeInterlock/current_binary_version_pause_after_second_check_of_instances
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "current binary version", configuration: "pause after second check of instances"
=== RUN   TestTenantUpgradeInterlock/current_binary_version_pause_after_version_bump_RPC
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "current binary version", configuration: "pause after version bump RPC"
=== RUN   TestTenantUpgradeInterlock/lagging_binary_version_pause_after_migration
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "lagging binary version", configuration: "pause after migration"
=== RUN   TestTenantUpgradeInterlock/lagging_binary_version_pause_after_second_check_of_instances
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "lagging binary version", configuration: "pause after second check of instances"
=== RUN   TestTenantUpgradeInterlock/lagging_binary_version_pause_after_write_to_settings_table
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "lagging binary version", configuration: "pause after write to settings table"

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/kvccl/kvtenantccl.TestTenantUpgradeInterlock failed with artifacts on release-23.1 @ 29b734894672b1f122abd03f3d1526b2143fe314:

Fatal error:

panic: test timed out after 14m55s

Stack:

goroutine 1046588 [running]:
testing.(*M).startAlarm.func1()
	GOROOT/src/testing/testing.go:2036 +0x88
created by time.goFunc
	GOROOT/src/time/sleep.go:176 +0x38
Log preceding fatal error 

=== RUN   TestTenantUpgradeInterlock
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/ab29d1b80c7389017099436f61091ca9/logTestTenantUpgradeInterlock3381969430
    test_log_scope.go:79: use -show-logs to present logs inline
=== RUN   TestTenantUpgradeInterlock/lagging_binary_version_pause_after_first_check_for_instances
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "lagging binary version", configuration: "pause after first check for instances"
=== RUN   TestTenantUpgradeInterlock/lagging_binary_version_pause_after_migration
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "lagging binary version", configuration: "pause after migration"

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/kvccl/kvtenantccl.TestTenantUpgradeInterlock failed with artifacts on release-23.1 @ 90b2e4aaa985596604bc9218e011560f8101024f:

Fatal error:

panic: test timed out after 14m55s

Stack:

goroutine 1022043 [running]:
testing.(*M).startAlarm.func1()
	GOROOT/src/testing/testing.go:2036 +0x88
created by time.goFunc
	GOROOT/src/time/sleep.go:176 +0x38
Log preceding fatal error 

=== RUN   TestTenantUpgradeInterlock
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/ab29d1b80c7389017099436f61091ca9/logTestTenantUpgradeInterlock3864433482
    test_log_scope.go:79: use -show-logs to present logs inline
=== RUN   TestTenantUpgradeInterlock/current_binary_version_pause_after_first_check_for_instances
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "current binary version", configuration: "pause after first check for instances"
=== RUN   TestTenantUpgradeInterlock/current_binary_version_pause_after_second_check_of_instances
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "current binary version", configuration: "pause after second check of instances"
=== RUN   TestTenantUpgradeInterlock/current_binary_version_pause_after_version_bump_RPC
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "current binary version", configuration: "pause after version bump RPC"
=== RUN   TestTenantUpgradeInterlock/lagging_binary_version_pause_after_fence_write_to_settings_table
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "lagging binary version", configuration: "pause after fence write to settings table"
=== RUN   TestTenantUpgradeInterlock/current_binary_version_pause_after_write_to_settings_table
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "current binary version", configuration: "pause after write to settings table"
=== RUN   TestTenantUpgradeInterlock/lagging_binary_version_pause_after_fence_RPC
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "lagging binary version", configuration: "pause after fence RPC"
=== RUN   TestTenantUpgradeInterlock/lagging_binary_version_pause_after_first_check_for_instances
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "lagging binary version", configuration: "pause after first check for instances"
=== RUN   TestTenantUpgradeInterlock/lagging_binary_version_pause_after_migration
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "lagging binary version", configuration: "pause after migration"

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@knz knz mentioned this issue Apr 3, 2023
Merged
@cockroach-teamcity
Copy link
Member Author

ccl/kvccl/kvtenantccl.TestTenantUpgradeInterlock failed with artifacts on release-23.1 @ d4f070a9bf79eea22c6bf81536525ac2bff7d6ed:

Fatal error:

panic: test timed out after 14m55s

Stack:

goroutine 1058615 [running]:
testing.(*M).startAlarm.func1()
	GOROOT/src/testing/testing.go:2036 +0x88
created by time.goFunc
	GOROOT/src/time/sleep.go:176 +0x38
Log preceding fatal error 

=== RUN   TestTenantUpgradeInterlock
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/ab29d1b80c7389017099436f61091ca9/logTestTenantUpgradeInterlock2780866553
    test_log_scope.go:79: use -show-logs to present logs inline
=== RUN   TestTenantUpgradeInterlock/current_binary_version_pause_after_fence_RPC
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "current binary version", configuration: "pause after fence RPC"
=== RUN   TestTenantUpgradeInterlock/current_binary_version_pause_after_fence_write_to_settings_table
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "current binary version", configuration: "pause after fence write to settings table"
=== RUN   TestTenantUpgradeInterlock/lagging_binary_version_pause_after_migration
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "lagging binary version", configuration: "pause after migration"
=== RUN   TestTenantUpgradeInterlock/lagging_binary_version_pause_after_second_check_of_instances
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "lagging binary version", configuration: "pause after second check of instances"
=== RUN   TestTenantUpgradeInterlock/lagging_binary_version_pause_after_fence_write_to_settings_table
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "lagging binary version", configuration: "pause after fence write to settings table"
=== RUN   TestTenantUpgradeInterlock/lagging_binary_version_pause_after_first_check_for_instances
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "lagging binary version", configuration: "pause after first check for instances"
=== RUN   TestTenantUpgradeInterlock/current_binary_version_pause_after_migration
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "current binary version", configuration: "pause after migration"
=== RUN   TestTenantUpgradeInterlock/current_binary_version_pause_after_second_check_of_instances
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "current binary version", configuration: "pause after second check of instances"
=== RUN   TestTenantUpgradeInterlock/lagging_binary_version_pause_after_fence_RPC
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "lagging binary version", configuration: "pause after fence RPC"
=== RUN   TestTenantUpgradeInterlock/lagging_binary_version_pause_after_version_bump_RPC
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "lagging binary version", configuration: "pause after version bump RPC"

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/kvccl/kvtenantccl.TestTenantUpgradeInterlock failed with artifacts on release-23.1 @ d4f070a9bf79eea22c6bf81536525ac2bff7d6ed:

Fatal error:

panic: test timed out after 14m55s

Stack:

goroutine 1076648 [running]:
testing.(*M).startAlarm.func1()
	GOROOT/src/testing/testing.go:2036 +0x88
created by time.goFunc
	GOROOT/src/time/sleep.go:176 +0x38
Log preceding fatal error 

=== RUN   TestTenantUpgradeInterlock
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/ab29d1b80c7389017099436f61091ca9/logTestTenantUpgradeInterlock2970359680
    test_log_scope.go:79: use -show-logs to present logs inline
=== RUN   TestTenantUpgradeInterlock/lagging_binary_version_pause_after_fence_RPC
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "lagging binary version", configuration: "pause after fence RPC"
=== RUN   TestTenantUpgradeInterlock/lagging_binary_version_pause_after_version_bump_RPC
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "lagging binary version", configuration: "pause after version bump RPC"

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

craig bot pushed a commit that referenced this issue Apr 3, 2023
@knz
Merge #100523
a8f0efe 
100523: upgrades: ensure that create job_info synchronizes with jobs r=adityamaru a=knz

Probably needed for #99087 and #99836.

Prior to this patch, the migration that introduced the new job_info
table did not touch any other descriptor. As such, it couldn't
synchronize properly with any concurrent code that needed to access
the jobs table.

This patch fixes it by forcing a version upgrade on the jobs table
descriptor. This triggers a lease synchronization in the upgrade
manager and results in the proper ordering.

Release note: None
Epic: None

Co-authored-by: Raphael 'kena' Poss <[email protected]>
@knz knz removed release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. blocks-23.1.0-beta.1 labels Apr 3, 2023
@knz
Copy link
Contributor

knz commented Apr 3, 2023

I've verified that the test passes manually and reliably, so removing the release blocker.

@blathers-crl blathers-crl bot mentioned this issue Apr 3, 2023
Merged
@cockroach-teamcity
Copy link
Member Author

ccl/kvccl/kvtenantccl.TestTenantUpgradeInterlock failed with artifacts on release-23.1 @ 81490d67b6bdeaab667e1e900d1cdfbba360613f:

Fatal error:

panic: test timed out after 14m55s

Stack:

goroutine 1219142 [running]:
testing.(*M).startAlarm.func1()
	GOROOT/src/testing/testing.go:2036 +0x88
created by time.goFunc
	GOROOT/src/time/sleep.go:176 +0x38
Log preceding fatal error 

=== RUN   TestTenantUpgradeInterlock
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/ab29d1b80c7389017099436f61091ca9/logTestTenantUpgradeInterlock813584375
    test_log_scope.go:79: use -show-logs to present logs inline
=== RUN   TestTenantUpgradeInterlock/lagging_binary_version_pause_after_version_bump_RPC
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "lagging binary version", configuration: "pause after version bump RPC"
=== RUN   TestTenantUpgradeInterlock/current_binary_version_pause_after_migration
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "current binary version", configuration: "pause after migration"
=== RUN   TestTenantUpgradeInterlock/current_binary_version_pause_after_second_check_of_instances
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "current binary version", configuration: "pause after second check of instances"
=== RUN   TestTenantUpgradeInterlock/current_binary_version_pause_after_version_bump_RPC
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "current binary version", configuration: "pause after version bump RPC"
=== RUN   TestTenantUpgradeInterlock/lagging_binary_version_pause_after_write_to_settings_table
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "lagging binary version", configuration: "pause after write to settings table"

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/kvccl/kvtenantccl.TestTenantUpgradeInterlock failed with artifacts on release-23.1 @ 81490d67b6bdeaab667e1e900d1cdfbba360613f:

Fatal error:

panic: test timed out after 14m55s

Stack:

goroutine 1158345 [running]:
testing.(*M).startAlarm.func1()
	GOROOT/src/testing/testing.go:2036 +0x88
created by time.goFunc
	GOROOT/src/time/sleep.go:176 +0x38
Log preceding fatal error 

=== RUN   TestTenantUpgradeInterlock
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/ab29d1b80c7389017099436f61091ca9/logTestTenantUpgradeInterlock72239287
    test_log_scope.go:79: use -show-logs to present logs inline
=== RUN   TestTenantUpgradeInterlock/lagging_binary_version_pause_after_fence_RPC
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "lagging binary version", configuration: "pause after fence RPC"
=== RUN   TestTenantUpgradeInterlock/current_binary_version_pause_after_migration
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "current binary version", configuration: "pause after migration"
=== RUN   TestTenantUpgradeInterlock/current_binary_version_pause_after_write_to_settings_table
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "current binary version", configuration: "pause after write to settings table"
=== RUN   TestTenantUpgradeInterlock/lagging_binary_version_pause_after_fence_write_to_settings_table
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "lagging binary version", configuration: "pause after fence write to settings table"
=== RUN   TestTenantUpgradeInterlock/lagging_binary_version_pause_after_second_check_of_instances
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "lagging binary version", configuration: "pause after second check of instances"
=== RUN   TestTenantUpgradeInterlock/lagging_binary_version_pause_after_version_bump_RPC
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "lagging binary version", configuration: "pause after version bump RPC"
=== RUN   TestTenantUpgradeInterlock/current_binary_version_pause_after_fence_write_to_settings_table
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "current binary version", configuration: "pause after fence write to settings table"
=== RUN   TestTenantUpgradeInterlock/current_binary_version_pause_after_first_check_for_instances
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "current binary version", configuration: "pause after first check for instances"
=== RUN   TestTenantUpgradeInterlock/current_binary_version_pause_after_fence_RPC
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "current binary version", configuration: "pause after fence RPC"
=== RUN   TestTenantUpgradeInterlock/lagging_binary_version_pause_after_write_to_settings_table
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "lagging binary version", configuration: "pause after write to settings table"

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/kvccl/kvtenantccl.TestTenantUpgradeInterlock failed with artifacts on release-23.1 @ 81490d67b6bdeaab667e1e900d1cdfbba360613f:

Fatal error:

panic: test timed out after 14m55s

Stack:

goroutine 969334 [running]:
testing.(*M).startAlarm.func1()
	GOROOT/src/testing/testing.go:2036 +0x88
created by time.goFunc
	GOROOT/src/time/sleep.go:176 +0x38
Log preceding fatal error 

=== RUN   TestTenantUpgradeInterlock
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/ab29d1b80c7389017099436f61091ca9/logTestTenantUpgradeInterlock1765765510
    test_log_scope.go:79: use -show-logs to present logs inline
=== RUN   TestTenantUpgradeInterlock/current_binary_version_pause_after_second_check_of_instances
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "current binary version", configuration: "pause after second check of instances"
=== RUN   TestTenantUpgradeInterlock/lagging_binary_version_pause_after_fence_RPC
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "lagging binary version", configuration: "pause after fence RPC"
=== RUN   TestTenantUpgradeInterlock/lagging_binary_version_pause_after_first_check_for_instances
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "lagging binary version", configuration: "pause after first check for instances"
=== RUN   TestTenantUpgradeInterlock/lagging_binary_version_pause_after_migration
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "lagging binary version", configuration: "pause after migration"

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/kvccl/kvtenantccl.TestTenantUpgradeInterlock failed with artifacts on release-23.1 @ f539117b05ec532db085c204ef165bb9e349fad7:

Fatal error:

panic: test timed out after 14m55s

Stack:

goroutine 1148165 [running]:
testing.(*M).startAlarm.func1()
	GOROOT/src/testing/testing.go:2036 +0x88
created by time.goFunc
	GOROOT/src/time/sleep.go:176 +0x38
Log preceding fatal error 

=== RUN   TestTenantUpgradeInterlock
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/ab29d1b80c7389017099436f61091ca9/logTestTenantUpgradeInterlock717333650
    test_log_scope.go:79: use -show-logs to present logs inline
=== RUN   TestTenantUpgradeInterlock/lagging_binary_version_pause_after_first_check_for_instances
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "lagging binary version", configuration: "pause after first check for instances"
=== RUN   TestTenantUpgradeInterlock/lagging_binary_version_pause_after_second_check_of_instances
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "lagging binary version", configuration: "pause after second check of instances"
=== RUN   TestTenantUpgradeInterlock/lagging_binary_version_pause_after_write_to_settings_table
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "lagging binary version", configuration: "pause after write to settings table"

Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/kvccl/kvtenantccl.TestTenantUpgradeInterlock failed with artifacts on release-23.1 @ b32052092532fb91b9f4decd7007576f33b7293b:

Fatal error:

panic: test timed out after 14m55s

Stack:

goroutine 1140189 [running]:
testing.(*M).startAlarm.func1()
	GOROOT/src/testing/testing.go:2036 +0x88
created by time.goFunc
	GOROOT/src/time/sleep.go:176 +0x38
Log preceding fatal error 

=== RUN   TestTenantUpgradeInterlock
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/ab29d1b80c7389017099436f61091ca9/logTestTenantUpgradeInterlock1117071992
    test_log_scope.go:79: use -show-logs to present logs inline
=== RUN   TestTenantUpgradeInterlock/current_binary_version_pause_after_migration
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "current binary version", configuration: "pause after migration"
=== RUN   TestTenantUpgradeInterlock/lagging_binary_version_pause_after_first_check_for_instances
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "lagging binary version", configuration: "pause after first check for instances"
=== RUN   TestTenantUpgradeInterlock/current_binary_version_pause_after_fence_RPC
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "current binary version", configuration: "pause after fence RPC"
=== RUN   TestTenantUpgradeInterlock/current_binary_version_pause_after_fence_write_to_settings_table
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "current binary version", configuration: "pause after fence write to settings table"
=== RUN   TestTenantUpgradeInterlock/current_binary_version_pause_after_write_to_settings_table
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "current binary version", configuration: "pause after write to settings table"
=== RUN   TestTenantUpgradeInterlock/lagging_binary_version_pause_after_second_check_of_instances
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "lagging binary version", configuration: "pause after second check of instances"
=== RUN   TestTenantUpgradeInterlock/lagging_binary_version_pause_after_write_to_settings_table
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "lagging binary version", configuration: "pause after write to settings table"

Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

@knz knz mentioned this issue Apr 4, 2023
Merged
@cockroach-teamcity
Copy link
Member Author

ccl/kvccl/kvtenantccl.TestTenantUpgradeInterlock failed with artifacts on release-23.1 @ a2eaa46200eeb3f985d62cb0b875deeb55fad73c:

Fatal error:

panic: test timed out after 14m55s

Stack:

goroutine 1062062 [running]:
testing.(*M).startAlarm.func1()
	GOROOT/src/testing/testing.go:2036 +0x88
created by time.goFunc
	GOROOT/src/time/sleep.go:176 +0x38
Log preceding fatal error 

=== RUN   TestTenantUpgradeInterlock
I230404 17:25:38.519031 1 (gostd) rand.go:199  [-] 1  random seed: -2392368579699424709
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/ab29d1b80c7389017099436f61091ca9/logTestTenantUpgradeInterlock2368359836
    test_log_scope.go:79: use -show-logs to present logs inline
=== RUN   TestTenantUpgradeInterlock/lagging_binary_version_pause_after_version_bump_RPC
    tenant_upgrade_test.go:525: upgrade interlock test: running variant "lagging binary version", configuration: "pause after version bump RPC"

Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

@knz
Copy link
Contributor

knz commented Apr 4, 2023

(hopefully) fixed by #100603

@knz knz closed this as completed Apr 4, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
branch-release-23.1 Used to mark GA and release blockers, technical advisories, and bugs for 23.1 C-test-failure Broken test (automatically or manually discovered). GA-blocker O-robot Originated from a bot. skipped-test T-multitenant Issues owned by the multi-tenant virtual team
Projects
None yet
Milestone
23.1
Development

No branches or pull requests
6 participants
@knz @stevendanna @rafiss @celiala @ajstorm @cockroach-teamcity