ccl/kvccl/kvtenantccl: TestTenantUpgradeFailure failed #98555

cockroach-teamcity · 2023-03-14T05:39:28Z

ccl/kvccl/kvtenantccl.TestTenantUpgradeFailure failed with artifacts on master @ 4215b86090794a78649aa4463c7bf95d74c57c37:

=== RUN   TestTenantUpgradeFailure
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/16471d5fa37cec92efcb07e68ae77ada/logTestTenantUpgradeFailure1735758781
    test_log_scope.go:79: use -show-logs to present logs inline
=== CONT  TestTenantUpgradeFailure
    tenant_upgrade_test.go:366: -- test log scope end --
--- FAIL: TestTenantUpgradeFailure (2.76s)
=== RUN   TestTenantUpgradeFailure/upgrade_tenant_have_it_crash_then_resume
    tenant_upgrade_test.go:322: expected error '.*(database is closed|failed to connect|closed network connection)+', got: pq: migration-job-mark-job-succeeded: tenant-side-limiter pool closed: shutting down
    --- FAIL: TestTenantUpgradeFailure/upgrade_tenant_have_it_crash_then_resume (2.20s)

Parameters: TAGS=bazel,gss

Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/multi-tenant _{This test on roachdash | Improve this report!

Jira issue: CRDB-25327
Epic CRDB-27931}

The text was updated successfully, but these errors were encountered:

cockroach-teamcity · 2023-03-14T05:51:44Z

ccl/kvccl/kvtenantccl.TestTenantUpgradeFailure failed with artifacts on master @ 4215b86090794a78649aa4463c7bf95d74c57c37:

=== RUN   TestTenantUpgradeFailure
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/16471d5fa37cec92efcb07e68ae77ada/logTestTenantUpgradeFailure2113076791
    test_log_scope.go:79: use -show-logs to present logs inline
=== CONT  TestTenantUpgradeFailure
    tenant_upgrade_test.go:366: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/16471d5fa37cec92efcb07e68ae77ada/logTestTenantUpgradeFailure2113076791
--- FAIL: TestTenantUpgradeFailure (19.52s)
=== RUN   TestTenantUpgradeFailure/upgrade_tenant_have_it_crash_then_resume
    tenant_upgrade_test.go:322: expected error '.*(database is closed|failed to connect|closed network connection)+', got: ERROR: read tcp 127.0.0.1:43600->127.0.0.1:42143: read: connection reset by peer
    --- FAIL: TestTenantUpgradeFailure/upgrade_tenant_have_it_crash_then_resume (17.47s)

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2023-03-14T15:30:29Z

ccl/kvccl/kvtenantccl.TestTenantUpgradeFailure failed with artifacts on master @ ca5ae38022c45aec434fa1dd98f58c76e37643f5:

=== RUN   TestTenantUpgradeFailure
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/20a7c78ed60808351752b79bb905ee6b/logTestTenantUpgradeFailure573214503
    test_log_scope.go:79: use -show-logs to present logs inline
=== CONT  TestTenantUpgradeFailure
    tenant_upgrade_test.go:366: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/20a7c78ed60808351752b79bb905ee6b/logTestTenantUpgradeFailure573214503
--- FAIL: TestTenantUpgradeFailure (2.74s)
=== RUN   TestTenantUpgradeFailure/upgrade_tenant_have_it_crash_then_resume
    tenant_upgrade_test.go:322: expected error '.*(database is closed|failed to connect|closed network connection)+', got: ERROR: dial tcp 127.0.0.1:42625: connect: connection refused
    --- FAIL: TestTenantUpgradeFailure/upgrade_tenant_have_it_crash_then_resume (2.14s)

Help

See also: How To Investigate a Go Test Failure (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2023-03-15T05:49:35Z

ccl/kvccl/kvtenantccl.TestTenantUpgradeFailure failed with artifacts on master @ 9b43dc50d468171bc9919dc684b7274b63e13da2:

Fatal error:

panic: use of Span after Finish. Span: flow coordinator. Finish previously called at: <stack not captured. Set debugUseAfterFinish>

Stack:

goroutine 19490 [running]:
github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).detectUseAfterFinish(0xa527300?)
	github.com/cockroachdb/cockroach/pkg/util/tracing/span.go:181 +0x12b
github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).RecordingType(0xc006817400)
	github.com/cockroachdb/cockroach/pkg/util/tracing/span.go:441 +0x1e
github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).IsVerbose(...)
	github.com/cockroachdb/cockroach/pkg/util/tracing/span.go:449
github.com/cockroachdb/cockroach/pkg/util/log.getSpanOrEventLog({0x6f57d10, 0xc00aba39b0})
	github.com/cockroachdb/cockroach/pkg/util/log/trace.go:92 +0x3a
github.com/cockroachdb/cockroach/pkg/util/log.logfDepthInternal({0x6f57d10, 0xc00aba39b0}, 0x2, 0x3, 0x1, 0x0?, {0x5b49eed, 0x31}, {0xc0060f74f8, 0x1, ...})
	github.com/cockroachdb/cockroach/pkg/util/log/channels.go:101 +0x554
github.com/cockroachdb/cockroach/pkg/util/log.logfDepth(...)
	github.com/cockroachdb/cockroach/pkg/util/log/channels.go:39
github.com/cockroachdb/cockroach/pkg/util/log.loggerOps.Errorf(...)
	github.com/cockroachdb/cockroach/bazel-out/k8-fastbuild/bin/pkg/util/log/log_channels_generated.go:1281
github.com/cockroachdb/cockroach/pkg/sql/sqlinstance/instancestorage.newMigrationCache.func2()
	github.com/cockroachdb/cockroach/pkg/sql/sqlinstance/instancestorage/instancecache.go:316 +0x20a
sync.(*Once).doSlow(0xf425600000002?, 0x0?)
	GOROOT/src/sync/once.go:74 +0xc2
sync.(*Once).Do(...)
	GOROOT/src/sync/once.go:65
github.com/cockroachdb/cockroach/pkg/sql/sqlinstance/instancestorage.onVersionReached.func1({0x7fe5c9b4f518?, 0x6f57c68?}, {{0x724f0c0?, 0xc0?, 0x53bdec0?, 0xc0?}})
	github.com/cockroachdb/cockroach/pkg/sql/sqlinstance/instancestorage/instancecache.go:232 +0x55
github.com/cockroachdb/cockroach/pkg/clusterversion.(*handleImpl).SetOnChange.func1({0x6f57c68, 0xc00724f0c0})
	github.com/cockroachdb/cockroach/pkg/clusterversion/pkg/clusterversion/clusterversion.go:227 +0x6e
github.com/cockroachdb/cockroach/pkg/settings.(*Values).settingChanged(0xc00a634000?, {0x6f57c68, 0xc00724f0c0}, 0x53bdec0?)
	github.com/cockroachdb/cockroach/pkg/settings/pkg/settings/values.go:147 +0xf9
github.com/cockroachdb/cockroach/pkg/settings.(*Values).setGeneric(0xc00a634000, {0x6f57c68, 0xc00724f0c0}, 0x60f76e8?, {0x4f5dac0?, 0xc0053bdec0?})
	github.com/cockroachdb/cockroach/pkg/settings/pkg/settings/values.go:178 +0x5d
github.com/cockroachdb/cockroach/pkg/settings.(*VersionSetting).SetInternal(...)
	github.com/cockroachdb/cockroach/pkg/settings/pkg/settings/version.go:167
github.com/cockroachdb/cockroach/pkg/clusterversion.(*handleImpl).SetActiveVersion(0xc00434aba0, {0x6f57c68, 0xc00724f0c0}, {{0xf4256, 0x2, 0x0, 0x4e}})
	github.com/cockroachdb/cockroach/pkg/clusterversion/pkg/clusterversion/clusterversion.go:220 +0xeb
github.com/cockroachdb/cockroach/pkg/server/settingswatcher.(*SettingsWatcher).setLocked(0xc0052bfb00, {0x6f57c68, 0xc00724f0c0}, {0xc00ad34e80?, 0x6f57d10?}, {{0xc00ad34e90?, 0xc000d8de00?}, {0x9fc2d68?, 0x468dbd?}})
	github.com/cockroachdb/cockroach/pkg/server/settingswatcher/settings_watcher.go:325 +0x206
github.com/cockroachdb/cockroach/pkg/server/settingswatcher.(*SettingsWatcher).maybeSet(0xc0052bfb00, {0x6f57c68, 0xc00724f0c0}, {0xc00ad34e80, 0x7}, {{{0xc00ad34e90, 0xc}, {0x9fc2d68, 0x1}}, {0x174c81b4c62db657, ...}, ...})
	github.com/cockroachdb/cockroach/pkg/server/settingswatcher/settings_watcher.go:295 +0x325
github.com/cockroachdb/cockroach/pkg/server/settingswatcher.(*SettingsWatcher).handleKV(0xc0052bfb00, {0x6f57c68, 0xc00724f0c0}, 0xc009327260)
	github.com/cockroachdb/cockroach/pkg/server/settingswatcher/settings_watcher.go:261 +0x29b
github.com/cockroachdb/cockroach/pkg/server/settingswatcher.(*SettingsWatcher).Start.func4({0x6f57c68?, 0xc00724f0c0?}, 0xc003274000?)
	github.com/cockroachdb/cockroach/pkg/server/settingswatcher/settings_watcher.go:201 +0x33
github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeedcache.(*Watcher).Run.func3({0x6f57c68, 0xc00724f0c0}, 0xc008104880?)
	github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeedcache/watcher.go:261 +0x55
github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).processEvents(0xc004b38420, {0x6f57c68, 0xc00724f0c0}, 0x2?, 0xc0061ef140)
	github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:362 +0x3a9
github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run.func2({0x6f57c68?, 0xc00724f0c0?})
	github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:315 +0x38
github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.GoCtx.func1()
	github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:168 +0x25
golang.org/x/sync/errgroup.(*Group).Go.func1()
	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75 +0x64
created by golang.org/x/sync/errgroup.(*Group).Go
	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:72 +0xa5

Log preceding fatal error

=== RUN   TestTenantUpgradeFailure
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/16471d5fa37cec92efcb07e68ae77ada/logTestTenantUpgradeFailure2861072794
    test_log_scope.go:79: use -show-logs to present logs inline
=== RUN   TestTenantUpgradeFailure/upgrade_tenant_have_it_crash_then_resume

Parameters: TAGS=bazel,gss

Help

See also: How To Investigate a Go Test Failure (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2023-03-15T06:07:05Z

ccl/kvccl/kvtenantccl.TestTenantUpgradeFailure failed with artifacts on master @ 9b43dc50d468171bc9919dc684b7274b63e13da2:

=== RUN   TestTenantUpgradeFailure
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/16471d5fa37cec92efcb07e68ae77ada/logTestTenantUpgradeFailure415464810
    test_log_scope.go:79: use -show-logs to present logs inline
=== CONT  TestTenantUpgradeFailure
    tenant_upgrade_test.go:366: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/16471d5fa37cec92efcb07e68ae77ada/logTestTenantUpgradeFailure415464810
--- FAIL: TestTenantUpgradeFailure (20.12s)
=== RUN   TestTenantUpgradeFailure/upgrade_tenant_have_it_crash_then_resume
    tenant_upgrade_test.go:322: expected error '.*(database is closed|failed to connect|closed network connection)+', got: ERROR: dial tcp 127.0.0.1:40335: connect: connection refused
    --- FAIL: TestTenantUpgradeFailure/upgrade_tenant_have_it_crash_then_resume (18.06s)

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2023-03-15T06:23:44Z

ccl/kvccl/kvtenantccl.TestTenantUpgradeFailure failed with artifacts on master @ 9b43dc50d468171bc9919dc684b7274b63e13da2:

=== RUN   TestTenantUpgradeFailure
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/16471d5fa37cec92efcb07e68ae77ada/logTestTenantUpgradeFailure4249989184
    test_log_scope.go:79: use -show-logs to present logs inline
=== CONT  TestTenantUpgradeFailure
    tenant_upgrade_test.go:366: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/16471d5fa37cec92efcb07e68ae77ada/logTestTenantUpgradeFailure4249989184
--- FAIL: TestTenantUpgradeFailure (11.11s)
=== RUN   TestTenantUpgradeFailure/upgrade_tenant_have_it_crash_then_resume
    tenant_upgrade_test.go:322: expected error '.*(database is closed|failed to connect|closed network connection)+', got: ERROR: dial tcp 127.0.0.1:42549: connect: connection refused
    --- FAIL: TestTenantUpgradeFailure/upgrade_tenant_have_it_crash_then_resume (8.65s)

Parameters: TAGS=bazel,gss,race

Help

See also: How To Investigate a Go Test Failure (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2023-03-16T05:42:23Z

ccl/kvccl/kvtenantccl.TestTenantUpgradeFailure failed with artifacts on master @ d68f5cdfb25476aaf874bff0bb6515eba02efcd2:

Fatal error:

panic: use of Span after Finish. Span: commit sql txn. Finish previously called at: <stack not captured. Set debugUseAfterFinish>

Stack:

goroutine 21666 [running]:
github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).detectUseAfterFinish(0xa52e2c8?)
	github.com/cockroachdb/cockroach/pkg/util/tracing/span.go:181 +0x12b
github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).RecordingType(0xc003f05800)
	github.com/cockroachdb/cockroach/pkg/util/tracing/span.go:441 +0x1e
github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).IsVerbose(...)
	github.com/cockroachdb/cockroach/pkg/util/tracing/span.go:449
github.com/cockroachdb/cockroach/pkg/util/log.getSpanOrEventLog({0x6f5d470, 0xc009521fb0})
	github.com/cockroachdb/cockroach/pkg/util/log/trace.go:92 +0x3a
github.com/cockroachdb/cockroach/pkg/util/log.logfDepthInternal({0x6f5d470, 0xc009521fb0}, 0x2, 0x3, 0x1, 0x0?, {0x5b4cb68, 0x31}, {0xc00669b4f8, 0x1, ...})
	github.com/cockroachdb/cockroach/pkg/util/log/channels.go:101 +0x554
github.com/cockroachdb/cockroach/pkg/util/log.logfDepth(...)
	github.com/cockroachdb/cockroach/pkg/util/log/channels.go:39
github.com/cockroachdb/cockroach/pkg/util/log.loggerOps.Errorf(...)
	github.com/cockroachdb/cockroach/bazel-out/k8-fastbuild/bin/pkg/util/log/log_channels_generated.go:1281
github.com/cockroachdb/cockroach/pkg/sql/sqlinstance/instancestorage.newMigrationCache.func2()
	github.com/cockroachdb/cockroach/pkg/sql/sqlinstance/instancestorage/instancecache.go:316 +0x20a
sync.(*Once).doSlow(0xf425600000002?, 0x0?)
	GOROOT/src/sync/once.go:74 +0xc2
sync.(*Once).Do(...)
	GOROOT/src/sync/once.go:65
github.com/cockroachdb/cockroach/pkg/sql/sqlinstance/instancestorage.onVersionReached.func1({0x7f533e5f69d8?, 0x6f5d3c8?}, {{0x2a9a700?, 0xc0?, 0x4d98888?, 0xc0?}})
	github.com/cockroachdb/cockroach/pkg/sql/sqlinstance/instancestorage/instancecache.go:232 +0x55
github.com/cockroachdb/cockroach/pkg/clusterversion.(*handleImpl).SetOnChange.func1({0x6f5d3c8, 0xc002a9a700})
	github.com/cockroachdb/cockroach/pkg/clusterversion/pkg/clusterversion/clusterversion.go:227 +0x6e
github.com/cockroachdb/cockroach/pkg/settings.(*Values).settingChanged(0xc00976e000?, {0x6f5d3c8, 0xc002a9a700}, 0x4d98888?)
	github.com/cockroachdb/cockroach/pkg/settings/pkg/settings/values.go:147 +0xf9
github.com/cockroachdb/cockroach/pkg/settings.(*Values).setGeneric(0xc00976e000, {0x6f5d3c8, 0xc002a9a700}, 0x669b6e8?, {0x4f600c0?, 0xc004d98888?})
	github.com/cockroachdb/cockroach/pkg/settings/pkg/settings/values.go:178 +0x5d
github.com/cockroachdb/cockroach/pkg/settings.(*VersionSetting).SetInternal(...)
	github.com/cockroachdb/cockroach/pkg/settings/pkg/settings/version.go:167
github.com/cockroachdb/cockroach/pkg/clusterversion.(*handleImpl).SetActiveVersion(0xc003a0b2f0, {0x6f5d3c8, 0xc002a9a700}, {{0xf4256, 0x2, 0x0, 0x54}})
	github.com/cockroachdb/cockroach/pkg/clusterversion/pkg/clusterversion/clusterversion.go:220 +0xeb
github.com/cockroachdb/cockroach/pkg/server/settingswatcher.(*SettingsWatcher).setLocked(0xc0089237a0, {0x6f5d3c8, 0xc002a9a700}, {0xc005937b38?, 0x0?}, {{0xc005937b40?, 0xc003d8ece0?}, {0x9fc9ce8?, 0x468dbd?}})
	github.com/cockroachdb/cockroach/pkg/server/settingswatcher/settings_watcher.go:325 +0x206
github.com/cockroachdb/cockroach/pkg/server/settingswatcher.(*SettingsWatcher).maybeSet(0xc0089237a0, {0x6f5d3c8, 0xc002a9a700}, {0xc005937b38, 0x7}, {{{0xc005937b40, 0xc}, {0x9fc9ce8, 0x1}}, {0x174ccfe4cc31ea89, ...}, ...})
	github.com/cockroachdb/cockroach/pkg/server/settingswatcher/settings_watcher.go:295 +0x325
github.com/cockroachdb/cockroach/pkg/server/settingswatcher.(*SettingsWatcher).handleKV(0xc0089237a0, {0x6f5d3c8, 0xc002a9a700}, 0xc00506d6c0)
	github.com/cockroachdb/cockroach/pkg/server/settingswatcher/settings_watcher.go:261 +0x29b
github.com/cockroachdb/cockroach/pkg/server/settingswatcher.(*SettingsWatcher).Start.func4({0x6f5d3c8?, 0xc002a9a700?}, 0xc000d5f520?)
	github.com/cockroachdb/cockroach/pkg/server/settingswatcher/settings_watcher.go:201 +0x33
github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeedcache.(*Watcher).Run.func3({0x6f5d3c8, 0xc002a9a700}, 0xc000bef240?)
	github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeedcache/watcher.go:261 +0x55
github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).processEvents(0xc008204840, {0x6f5d3c8, 0xc002a9a700}, 0x511b9a0?, 0xc004c862a0)
	github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:362 +0x3a9
github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run.func2({0x6f5d3c8?, 0xc002a9a700?})
	github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:315 +0x38
github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.GoCtx.func1()
	github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:168 +0x25
golang.org/x/sync/errgroup.(*Group).Go.func1()
	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75 +0x64
created by golang.org/x/sync/errgroup.(*Group).Go
	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:72 +0xa5

Log preceding fatal error

=== RUN   TestTenantUpgradeFailure
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/16471d5fa37cec92efcb07e68ae77ada/logTestTenantUpgradeFailure686316603
    test_log_scope.go:79: use -show-logs to present logs inline
=== RUN   TestTenantUpgradeFailure/upgrade_tenant_have_it_crash_then_resume

Parameters: TAGS=bazel,gss

Help

See also: How To Investigate a Go Test Failure (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2023-03-16T05:49:02Z

ccl/kvccl/kvtenantccl.TestTenantUpgradeFailure failed with artifacts on master @ d68f5cdfb25476aaf874bff0bb6515eba02efcd2:

=== RUN   TestTenantUpgradeFailure
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/16471d5fa37cec92efcb07e68ae77ada/logTestTenantUpgradeFailure3240791991
    test_log_scope.go:79: use -show-logs to present logs inline
=== CONT  TestTenantUpgradeFailure
    tenant_upgrade_test.go:366: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/16471d5fa37cec92efcb07e68ae77ada/logTestTenantUpgradeFailure3240791991
--- FAIL: TestTenantUpgradeFailure (23.28s)
=== RUN   TestTenantUpgradeFailure/upgrade_tenant_have_it_crash_then_resume
    tenant_upgrade_test.go:322: expected error '.*(database is closed|failed to connect|closed network connection)+', got: ERROR: dial tcp 127.0.0.1:35669: connect: connection refused
    --- FAIL: TestTenantUpgradeFailure/upgrade_tenant_have_it_crash_then_resume (20.49s)

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2023-03-16T05:54:41Z

ccl/kvccl/kvtenantccl.TestTenantUpgradeFailure failed with artifacts on master @ d68f5cdfb25476aaf874bff0bb6515eba02efcd2:

=== RUN   TestTenantUpgradeFailure
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/16471d5fa37cec92efcb07e68ae77ada/logTestTenantUpgradeFailure1193447337
    test_log_scope.go:79: use -show-logs to present logs inline
=== CONT  TestTenantUpgradeFailure
    tenant_upgrade_test.go:366: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/16471d5fa37cec92efcb07e68ae77ada/logTestTenantUpgradeFailure1193447337
--- FAIL: TestTenantUpgradeFailure (13.06s)
=== RUN   TestTenantUpgradeFailure/upgrade_tenant_have_it_crash_then_resume
    tenant_upgrade_test.go:322: expected error '.*(database is closed|failed to connect|closed network connection)+', got: ERROR: read tcp 127.0.0.1:38196->127.0.0.1:46131: read: connection reset by peer
    --- FAIL: TestTenantUpgradeFailure/upgrade_tenant_have_it_crash_then_resume (10.04s)

Parameters: TAGS=bazel,gss,race

Help

See also: How To Investigate a Go Test Failure (internal)

_{This test on roachdash | Improve this report!}

mgartner · 2023-03-16T13:53:51Z

Saw this flake here: https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_BazelEssentialCi/9083486?showRootCauses=true&expandBuildChangesSection=true&expandBuildProblemsSection=true&expandBuildTestsSection=true

No pass/skip/fail event found for test
=== RUN   TestTenantUpgradeFailure/upgrade_tenant_have_it_crash_then_resume
panic: use of Span after Finish. Span: start-instance-reader. Finish previously called at: <stack not captured. Set debugUseAfterFinish>
goroutine 1521 [running]:
github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).detectUseAfterFinish(0xa527308?)
  github.com/cockroachdb/cockroach/pkg/util/tracing/span.go:181 +0x12b
github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).RecordingType(0xc003e07400)
  github.com/cockroachdb/cockroach/pkg/util/tracing/span.go:441 +0x1e
github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).IsVerbose(...)
  github.com/cockroachdb/cockroach/pkg/util/tracing/span.go:449
github.com/cockroachdb/cockroach/pkg/util/log.getSpanOrEventLog({0x6f58190, 0xc003eb9dd0})
  github.com/cockroachdb/cockroach/pkg/util/log/trace.go:92 +0x3a
github.com/cockroachdb/cockroach/pkg/util/log.logfDepthInternal({0x6f58190, 0xc003eb9dd0}, 0x2, 0x3, 0x1, 0x0?, {0x5b4a124, 0x31}, {0xc00493b4f8, 0x1, ...})
  github.com/cockroachdb/cockroach/pkg/util/log/channels.go:101 +0x554
github.com/cockroachdb/cockroach/pkg/util/log.logfDepth(...)
  github.com/cockroachdb/cockroach/pkg/util/log/channels.go:39
github.com/cockroachdb/cockroach/pkg/util/log.loggerOps.Errorf(...)
  github.com/cockroachdb/cockroach/bazel-out/k8-fastbuild/bin/pkg/util/log/log_channels_generated.go:1281
github.com/cockroachdb/cockroach/pkg/sql/sqlinstance/instancestorage.newMigrationCache.func2()
  github.com/cockroachdb/cockroach/pkg/sql/sqlinstance/instancestorage/instancecache.go:316 +0x20a
sync.(*Once).doSlow(0xf425600000002?, 0x0?)
  GOROOT/src/sync/once.go:74 +0xc2
sync.(*Once).Do(...)
  GOROOT/src/sync/once.go:65
github.com/cockroachdb/cockroach/pkg/sql/sqlinstance/instancestorage.onVersionReached.func1({0x7f851e6c4100?, 0x6f580e8?}, {{0x462a080?, 0xc0?, 0x4f46a80?, 0xc0?}})
  github.com/cockroachdb/cockroach/pkg/sql/sqlinstance/instancestorage/instancecache.go:232 +0x55
github.com/cockroachdb/cockroach/pkg/clusterversion.(*handleImpl).SetOnChange.func1({0x6f580e8, 0xc00462a080})
  github.com/cockroachdb/cockroach/pkg/clusterversion/pkg/clusterversion/clusterversion.go:227 +0x6e
github.com/cockroachdb/cockroach/pkg/settings.(*Values).settingChanged(0xc003390000?, {0x6f580e8, 0xc00462a080}, 0x4f46a80?)
  github.com/cockroachdb/cockroach/pkg/settings/pkg/settings/values.go:147 +0xf9
github.com/cockroachdb/cockroach/pkg/settings.(*Values).setGeneric(0xc003390000, {0x6f580e8, 0xc00462a080}, 0x493b6e8?, {0x4f5dbc0?, 0xc004f46a80?})
  github.com/cockroachdb/cockroach/pkg/settings/pkg/settings/values.go:178 +0x5d
github.com/cockroachdb/cockroach/pkg/settings.(*VersionSetting).SetInternal(...)
  github.com/cockroachdb/cockroach/pkg/settings/pkg/settings/version.go:167
github.com/cockroachdb/cockroach/pkg/clusterversion.(*handleImpl).SetActiveVersion(0xc004613200, {0x6f580e8, 0xc00462a080}, {{0xf4256, 0x2, 0x0, 0x50}})
  github.com/cockroachdb/cockroach/pkg/clusterversion/pkg/clusterversion/clusterversion.go:220 +0xeb
github.com/cockroachdb/cockroach/pkg/server/settingswatcher.(*SettingsWatcher).setLocked(0xc004af3200, {0x6f580e8, 0xc00462a080}, {0xc00465c8c8?, 0xc001c29888?}, {{0xc00465c8d0?, 0xc0012d42a0?}, {0x9fc2d68?, 0x468dbd?}})
  github.com/cockroachdb/cockroach/pkg/server/settingswatcher/settings_watcher.go:325 +0x206
github.com/cockroachdb/cockroach/pkg/server/settingswatcher.(*SettingsWatcher).maybeSet(0xc004af3200, {0x6f580e8, 0xc00462a080}, {0xc00465c8c8, 0x7}, {{{0xc00465c8d0, 0xc}, {0x9fc2d68, 0x1}}, {0x174cb1bcc87eea03, ...}, ...})
  github.com/cockroachdb/cockroach/pkg/server/settingswatcher/settings_watcher.go:295 +0x325
github.com/cockroachdb/cockroach/pkg/server/settingswatcher.(*SettingsWatcher).handleKV(0xc004af3200, {0x6f580e8, 0xc00462a080}, 0xc006fcc3f0)
  github.com/cockroachdb/cockroach/pkg/server/settingswatcher/settings_watcher.go:261 +0x29b
github.com/cockroachdb/cockroach/pkg/server/settingswatcher.(*SettingsWatcher).Start.func4({0x6f580e8?, 0xc00462a080?}, 0xc0035bc340?)
  github.com/cockroachdb/cockroach/pkg/server/settingswatcher/settings_watcher.go:201 +0x33
github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeedcache.(*Watcher).Run.func3({0x6f580e8, 0xc00462a080}, 0xc0043bce40?)
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeedcache/watcher.go:261 +0x55
github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).processEvents(0xc00476ef20, {0x6f580e8, 0xc00462a080}, 0xc000f33c60?, 0xc0042ccc60)
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:362 +0x3a9
github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run.func2({0x6f580e8?, 0xc00462a080?})
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:315 +0x38
github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.GoCtx.func1()
  github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:168 +0x25
golang.org/x/sync/errgroup.(*Group).Go.func1()
  golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75 +0x64
created by golang.org/x/sync/errgroup.(*Group).Go
  golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:72 +0xa5

rafiss · 2023-03-16T13:57:02Z

Added GA-blocker since this is new, just to make sure it gets looked at

knz · 2023-03-16T14:01:27Z

This stack trace tells me it's overlapping with @healthy-pod 's recent work and the assignment of tracers to different contexts. @abarganier could you perhaps help Ahmad investigating this?

panic: use of Span after Finish. Span: start-instance-reader. Finish previously called at: <stack not captured. Set debugUseAfterFinish>
goroutine 1521 [running]:
github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).detectUseAfterFinish(0xa527308?)
  github.com/cockroachdb/cockroach/pkg/util/tracing/span.go:181 +0x12b
github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).RecordingType(0xc003e07400)
  github.com/cockroachdb/cockroach/pkg/util/tracing/span.go:441 +0x1e
github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).IsVerbose(...)
  github.com/cockroachdb/cockroach/pkg/util/tracing/span.go:449
github.com/cockroachdb/cockroach/pkg/util/log.getSpanOrEventLog({0x6f58190, 0xc003eb9dd0})
  github.com/cockroachdb/cockroach/pkg/util/log/trace.go:92 +0x3a
github.com/cockroachdb/cockroach/pkg/util/log.logfDepthInternal({0x6f58190, 0xc003eb9dd0}, 0x2, 0x3, 0x1, 0x0?, {0x5b4a124, 0x31}, {0xc00493b4f8, 0x1, ...})
  github.com/cockroachdb/cockroach/pkg/util/log/channels.go:101 +0x554
github.com/cockroachdb/cockroach/pkg/util/log.logfDepth(...)
  github.com/cockroachdb/cockroach/pkg/util/log/channels.go:39
github.com/cockroachdb/cockroach/pkg/util/log.loggerOps.Errorf(...)
  github.com/cockroachdb/cockroach/bazel-out/k8-fastbuild/bin/pkg/util/log/log_channels_generated.go:1281
github.com/cockroachdb/cockroach/pkg/sql/sqlinstance/instancestorage.newMigrationCache.func2()
  github.com/cockroachdb/cockroach/pkg/sql/sqlinstance/instancestorage/instancecache.go:316 +0x20a
sync.(*Once).doSlow(0xf425600000002?, 0x0?)
  GOROOT/src/sync/once.go:74 +0xc2
sync.(*Once).Do(...)
  GOROOT/src/sync/once.go:65
github.com/cockroachdb/cockroach/pkg/sql/sqlinstance/instancestorage.onVersionReached.func1({0x7f851e6c4100?, 0x6f580e8?}, {{0x462a080?, 0xc0?, 0x4f46a80?, 0xc0?}})
  github.com/cockroachdb/cockroach/pkg/sql/sqlinstance/instancestorage/instancecache.go:232 +0x55
github.com/cockroachdb/cockroach/pkg/clusterversion.(*handleImpl).SetOnChange.func1({0x6f580e8, 0xc00462a080})
  github.com/cockroachdb/cockroach/pkg/clusterversion/pkg/clusterversion/clusterversion.go:227 +0x6e
github.com/cockroachdb/cockroach/pkg/settings.(*Values).settingChanged(0xc003390000?, {0x6f580e8, 0xc00462a080}, 0x4f46a80?)
  github.com/cockroachdb/cockroach/pkg/settings/pkg/settings/values.go:147 +0xf9
github.com/cockroachdb/cockroach/pkg/settings.(*Values).setGeneric(0xc003390000, {0x6f580e8, 0xc00462a080}, 0x493b6e8?, {0x4f5dbc0?, 0xc004f46a80?})
  github.com/cockroachdb/cockroach/pkg/settings/pkg/settings/values.go:178 +0x5d
github.com/cockroachdb/cockroach/pkg/settings.(*VersionSetting).SetInternal(...)
  github.com/cockroachdb/cockroach/pkg/settings/pkg/settings/version.go:167
github.com/cockroachdb/cockroach/pkg/clusterversion.(*handleImpl).SetActiveVersion(0xc004613200, {0x6f580e8, 0xc00462a080}, {{0xf4256, 0x2, 0x0, 0x50}})
  github.com/cockroachdb/cockroach/pkg/clusterversion/pkg/clusterversion/clusterversion.go:220 +0xeb
github.com/cockroachdb/cockroach/pkg/server/settingswatcher.(*SettingsWatcher).setLocked(0xc004af3200, {0x6f580e8, 0xc00462a080}, {0xc00465c8c8?, 0xc001c29888?}, {{0xc00465c8d0?, 0xc0012d42a0?}, {0x9fc2d68?, 0x468dbd?}})
  github.com/cockroachdb/cockroach/pkg/server/settingswatcher/settings_watcher.go:325 +0x206
github.com/cockroachdb/cockroach/pkg/server/settingswatcher.(*SettingsWatcher).maybeSet(0xc004af3200, {0x6f580e8, 0xc00462a080}, {0xc00465c8c8, 0x7}, {{{0xc00465c8d0, 0xc}, {0x9fc2d68, 0x1}}, {0x174cb1bcc87eea03, ...}, ...})
  github.com/cockroachdb/cockroach/pkg/server/settingswatcher/settings_watcher.go:295 +0x325
github.com/cockroachdb/cockroach/pkg/server/settingswatcher.(*SettingsWatcher).handleKV(0xc004af3200, {0x6f580e8, 0xc00462a080}, 0xc006fcc3f0)
  github.com/cockroachdb/cockroach/pkg/server/settingswatcher/settings_watcher.go:261 +0x29b
github.com/cockroachdb/cockroach/pkg/server/settingswatcher.(*SettingsWatcher).Start.func4({0x6f580e8?, 0xc00462a080?}, 0xc0035bc340?)
  github.com/cockroachdb/cockroach/pkg/server/settingswatcher/settings_watcher.go:201 +0x33
github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeedcache.(*Watcher).Run.func3({0x6f580e8, 0xc00462a080}, 0xc0043bce40?)
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeedcache/watcher.go:261 +0x55
github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).processEvents(0xc00476ef20, {0x6f580e8, 0xc00462a080}, 0xc000f33c60?, 0xc0042ccc60)
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:362 +0x3a9
github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run.func2({0x6f580e8?, 0xc00462a080?})
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:315 +0x38
github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.GoCtx.func1()
  github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:168 +0x25
golang.org/x/sync/errgroup.(*Group).Go.func1()
  golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75 +0x64
created by golang.org/x/sync/errgroup.(*Group).Go
  golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:72 +0xa5

cockroach-teamcity · 2023-03-17T05:18:50Z

ccl/kvccl/kvtenantccl.TestTenantUpgradeFailure failed with artifacts on master @ f0489334a0ee6980a9d365b361d2fce4b2cdc05b:

Fatal error:

panic: use of Span after Finish. Span: /cockroach.roachpb.Internal/Batch. Finish previously called at: <stack not captured. Set debugUseAfterFinish>

Stack:

goroutine 21704 [running]:
github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).detectUseAfterFinish(0xa54a7d8?)
	github.com/cockroachdb/cockroach/pkg/util/tracing/span.go:181 +0x12b
github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).RecordingType(0xc004bd1000)
	github.com/cockroachdb/cockroach/pkg/util/tracing/span.go:441 +0x1e
github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).IsVerbose(...)
	github.com/cockroachdb/cockroach/pkg/util/tracing/span.go:449
github.com/cockroachdb/cockroach/pkg/util/log.getSpanOrEventLog({0x6f70ad0, 0xc003d43dd0})
	github.com/cockroachdb/cockroach/pkg/util/log/trace.go:92 +0x3a
github.com/cockroachdb/cockroach/pkg/util/log.logfDepthInternal({0x6f70ad0, 0xc003d43dd0}, 0x2, 0x3, 0x1, 0x0?, {0x5b5b87e, 0x31}, {0xc00a6694f8, 0x1, ...})
	github.com/cockroachdb/cockroach/pkg/util/log/channels.go:101 +0x554
github.com/cockroachdb/cockroach/pkg/util/log.logfDepth(...)
	github.com/cockroachdb/cockroach/pkg/util/log/channels.go:39
github.com/cockroachdb/cockroach/pkg/util/log.loggerOps.Errorf(...)
	github.com/cockroachdb/cockroach/bazel-out/k8-fastbuild/bin/pkg/util/log/log_channels_generated.go:1281
github.com/cockroachdb/cockroach/pkg/sql/sqlinstance/instancestorage.newMigrationCache.func2()
	github.com/cockroachdb/cockroach/pkg/sql/sqlinstance/instancestorage/instancecache.go:316 +0x20a
sync.(*Once).doSlow(0xf425600000002?, 0x0?)
	GOROOT/src/sync/once.go:74 +0xc2
sync.(*Once).Do(...)
	GOROOT/src/sync/once.go:65
github.com/cockroachdb/cockroach/pkg/sql/sqlinstance/instancestorage.onVersionReached.func1({0x7fc7335847b0?, 0x6f70a28?}, {{0x75aadc0?, 0xc0?, 0xbf2eb70?, 0xc0?}})
	github.com/cockroachdb/cockroach/pkg/sql/sqlinstance/instancestorage/instancecache.go:232 +0x55
github.com/cockroachdb/cockroach/pkg/clusterversion.(*handleImpl).SetOnChange.func1({0x6f70a28, 0xc0075aadc0})
	github.com/cockroachdb/cockroach/pkg/clusterversion/pkg/clusterversion/clusterversion.go:227 +0x6e
github.com/cockroachdb/cockroach/pkg/settings.(*Values).settingChanged(0xc00914a000?, {0x6f70a28, 0xc0075aadc0}, 0xbf2eb70?)
	github.com/cockroachdb/cockroach/pkg/settings/pkg/settings/values.go:147 +0xf9
github.com/cockroachdb/cockroach/pkg/settings.(*Values).setGeneric(0xc00914a000, {0x6f70a28, 0xc0075aadc0}, 0xa6696e8?, {0x4f6cbe0?, 0xc00bf2eb70?})
	github.com/cockroachdb/cockroach/pkg/settings/pkg/settings/values.go:178 +0x5d
github.com/cockroachdb/cockroach/pkg/settings.(*VersionSetting).SetInternal(...)
	github.com/cockroachdb/cockroach/pkg/settings/pkg/settings/version.go:167
github.com/cockroachdb/cockroach/pkg/clusterversion.(*handleImpl).SetActiveVersion(0xc007f44240, {0x6f70a28, 0xc0075aadc0}, {{0xf4256, 0x2, 0x0, 0x58}})
	github.com/cockroachdb/cockroach/pkg/clusterversion/pkg/clusterversion/clusterversion.go:220 +0xeb
github.com/cockroachdb/cockroach/pkg/server/settingswatcher.(*SettingsWatcher).setLocked(0xc005f08a20, {0x6f70a28, 0xc0075aadc0}, {0xc009537868?, 0xc007cec888?}, {{0xc009537920?, 0xc006376000?}, {0x9fe5fc8?, 0x468dbd?}})
	github.com/cockroachdb/cockroach/pkg/server/settingswatcher/settings_watcher.go:325 +0x206
github.com/cockroachdb/cockroach/pkg/server/settingswatcher.(*SettingsWatcher).maybeSet(0xc005f08a20, {0x6f70a28, 0xc0075aadc0}, {0xc009537868, 0x7}, {{{0xc009537920, 0xc}, {0x9fe5fc8, 0x1}}, {0x174d1d302a143018, ...}, ...})
	github.com/cockroachdb/cockroach/pkg/server/settingswatcher/settings_watcher.go:295 +0x325
github.com/cockroachdb/cockroach/pkg/server/settingswatcher.(*SettingsWatcher).handleKV(0xc005f08a20, {0x6f70a28, 0xc0075aadc0}, 0xc007cfa770)
	github.com/cockroachdb/cockroach/pkg/server/settingswatcher/settings_watcher.go:261 +0x29b
github.com/cockroachdb/cockroach/pkg/server/settingswatcher.(*SettingsWatcher).Start.func4({0x6f70a28?, 0xc0075aadc0?}, 0xc00161c1a0?)
	github.com/cockroachdb/cockroach/pkg/server/settingswatcher/settings_watcher.go:201 +0x33
github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeedcache.(*Watcher).Run.func3({0x6f70a28, 0xc0075aadc0}, 0xc0070b5a40?)
	github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeedcache/watcher.go:261 +0x55
github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).processEvents(0xc0058d0c60, {0x6f70a28, 0xc0075aadc0}, 0x0?, 0xc00aedd140)
	github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:362 +0x3a9
github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run.func2({0x6f70a28?, 0xc0075aadc0?})
	github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:315 +0x38
github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.GoCtx.func1()
	github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:168 +0x25
golang.org/x/sync/errgroup.(*Group).Go.func1()
	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75 +0x64
created by golang.org/x/sync/errgroup.(*Group).Go
	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:72 +0xa5

Log preceding fatal error

=== RUN   TestTenantUpgradeFailure
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/16471d5fa37cec92efcb07e68ae77ada/logTestTenantUpgradeFailure2494440872
    test_log_scope.go:79: use -show-logs to present logs inline
=== RUN   TestTenantUpgradeFailure/upgrade_tenant_have_it_crash_then_resume

Parameters: TAGS=bazel,gss

Help

See also: How To Investigate a Go Test Failure (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2023-03-17T05:22:14Z

ccl/kvccl/kvtenantccl.TestTenantUpgradeFailure failed with artifacts on master @ f0489334a0ee6980a9d365b361d2fce4b2cdc05b:

=== RUN   TestTenantUpgradeFailure
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/16471d5fa37cec92efcb07e68ae77ada/logTestTenantUpgradeFailure1712018016
    test_log_scope.go:79: use -show-logs to present logs inline
=== CONT  TestTenantUpgradeFailure
    tenant_upgrade_test.go:366: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/16471d5fa37cec92efcb07e68ae77ada/logTestTenantUpgradeFailure1712018016
--- FAIL: TestTenantUpgradeFailure (18.70s)
=== RUN   TestTenantUpgradeFailure/upgrade_tenant_have_it_crash_then_resume
    tenant_upgrade_test.go:322: expected error '.*(database is closed|failed to connect|closed network connection)+', got: ERROR: dial tcp 127.0.0.1:42419: connect: connection refused
    --- FAIL: TestTenantUpgradeFailure/upgrade_tenant_have_it_crash_then_resume (16.90s)

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2023-03-17T05:27:58Z

ccl/kvccl/kvtenantccl.TestTenantUpgradeFailure failed with artifacts on master @ f0489334a0ee6980a9d365b361d2fce4b2cdc05b:

Fatal error:

panic: use of Span after Finish. Span: /cockroach.roachpb.Internal/Batch. Finish previously called at: <stack not captured. Set debugUseAfterFinish>

Stack:

goroutine 24277 [running]:
github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).detectUseAfterFinish(0xc002067c00)
	github.com/cockroachdb/cockroach/pkg/util/tracing/span.go:181 +0x24e
github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).RecordingType(0xc002067c00)
	github.com/cockroachdb/cockroach/pkg/util/tracing/span.go:441 +0x2f
github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).IsVerbose(...)
	github.com/cockroachdb/cockroach/pkg/util/tracing/span.go:449
github.com/cockroachdb/cockroach/pkg/util/log.getSpanOrEventLog({0xc3f2130, 0xc004a72660})
	github.com/cockroachdb/cockroach/pkg/util/log/trace.go:92 +0x4f
github.com/cockroachdb/cockroach/pkg/util/log.logfDepthInternal({0xc3f2130, 0xc004a72660}, 0x2, 0x3, 0x1, 0x0, {0x88d15ab, 0x31}, {0xc00343f198, 0x1, ...})
	github.com/cockroachdb/cockroach/pkg/util/log/channels.go:101 +0x60b
github.com/cockroachdb/cockroach/pkg/util/log.logfDepth(...)
	github.com/cockroachdb/cockroach/pkg/util/log/channels.go:39
github.com/cockroachdb/cockroach/pkg/util/log.loggerOps.Errorf(...)
	github.com/cockroachdb/cockroach/bazel-out/k8-fastbuild/bin/pkg/util/log/log_channels_generated.go:1281
github.com/cockroachdb/cockroach/pkg/sql/sqlinstance/instancestorage.newMigrationCache.func2()
	github.com/cockroachdb/cockroach/pkg/sql/sqlinstance/instancestorage/instancecache.go:316 +0x31d
sync.(*Once).doSlow(0xc000aeaff4, 0xc005bf7ef0)
	GOROOT/src/sync/once.go:74 +0x102
sync.(*Once).Do(0xc000aeaff4, 0xf425600000002?)
	GOROOT/src/sync/once.go:65 +0x47
github.com/cockroachdb/cockroach/pkg/sql/sqlinstance/instancestorage.onVersionReached.func1({0x4ff6da?, 0xc3f2088?}, {{0xd0e20c0?, 0xc0?, 0x4d8fc5?, 0x0?}})
	github.com/cockroachdb/cockroach/pkg/sql/sqlinstance/instancestorage/instancecache.go:232 +0x7b
github.com/cockroachdb/cockroach/pkg/clusterversion.(*handleImpl).SetOnChange.func1({0xc3f2088, 0xc00d0e20c0})
	github.com/cockroachdb/cockroach/pkg/clusterversion/pkg/clusterversion/clusterversion.go:227 +0xae
github.com/cockroachdb/cockroach/pkg/settings.(*Values).settingChanged(0xc004118000, {0xc3f2088, 0xc00d0e20c0}, 0x8)
	github.com/cockroachdb/cockroach/pkg/settings/pkg/settings/values.go:147 +0xf5
github.com/cockroachdb/cockroach/pkg/settings.(*Values).setGeneric(0xc004118000, {0xc3f2088, 0xc00d0e20c0}, 0x4118000?, {0x7cdbac0, 0xc0078eea50})
	github.com/cockroachdb/cockroach/pkg/settings/pkg/settings/values.go:178 +0x6d
github.com/cockroachdb/cockroach/pkg/settings.(*VersionSetting).SetInternal(...)
	github.com/cockroachdb/cockroach/pkg/settings/pkg/settings/version.go:167
github.com/cockroachdb/cockroach/pkg/clusterversion.(*handleImpl).SetActiveVersion(0xc00abe08a0, {0xc3f2088, 0xc00d0e20c0}, {{0xf4256, 0x2, 0x0, 0x58}})
	github.com/cockroachdb/cockroach/pkg/clusterversion/pkg/clusterversion/clusterversion.go:220 +0x239
github.com/cockroachdb/cockroach/pkg/server/settingswatcher.(*SettingsWatcher).setLocked(0xc005894000, {0xc3f2088, 0xc00d0e20c0}, {0xc001b88f78, 0x7}, {{0xc001b88f84?, 0x10000c002d96e38?}, {0x11edf0a8?, 0x3?}})
	github.com/cockroachdb/cockroach/pkg/server/settingswatcher/settings_watcher.go:325 +0x35d
github.com/cockroachdb/cockroach/pkg/server/settingswatcher.(*SettingsWatcher).maybeSet(0xc005894000, {0xc3f2088, 0xc00d0e20c0}, {0xc001b88f78, 0x7}, {{{0xc001b88f84, 0xc}, {0x11edf0a8, 0x1}}, {0x174d1dafd324156a, ...}, ...})
	github.com/cockroachdb/cockroach/pkg/server/settingswatcher/settings_watcher.go:295 +0x3b1
github.com/cockroachdb/cockroach/pkg/server/settingswatcher.(*SettingsWatcher).handleKV(0xc005894000, {0xc3f2088, 0xc00d0e20c0}, 0xc0020e9c00)
	github.com/cockroachdb/cockroach/pkg/server/settingswatcher/settings_watcher.go:261 +0x478
github.com/cockroachdb/cockroach/pkg/server/settingswatcher.(*SettingsWatcher).Start.func4({0xc3f2088, 0xc00d0e20c0}, 0xc00d0e20d8?)
	github.com/cockroachdb/cockroach/pkg/server/settingswatcher/settings_watcher.go:201 +0x4f
github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeedcache.(*Watcher).Run.func3({0xc3f2088, 0xc00d0e20c0}, 0xc001218b00?)
	github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeedcache/watcher.go:261 +0x9e
github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).processEvents(0xc005574160, {0xc3f2088, 0xc00d0e20c0}, 0xc0094f3fb0?, 0xc0009b9ce0)
	github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:362 +0x205
github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run.func2({0xc3f2088, 0xc00d0e20c0})
	github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:315 +0x65
github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.GoCtx.func1()
	github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:168 +0x4d
golang.org/x/sync/errgroup.(*Group).Go.func1()
	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75 +0x87
created by golang.org/x/sync/errgroup.(*Group).Go
	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:72 +0x12f

Log preceding fatal error

=== RUN   TestTenantUpgradeFailure
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/16471d5fa37cec92efcb07e68ae77ada/logTestTenantUpgradeFailure2948021845
    test_log_scope.go:79: use -show-logs to present logs inline
=== RUN   TestTenantUpgradeFailure/upgrade_tenant_have_it_crash_then_resume

Parameters: TAGS=bazel,gss,race

Help

See also: How To Investigate a Go Test Failure (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2023-03-17T11:47:23Z

ccl/kvccl/kvtenantccl.TestTenantUpgradeFailure failed with artifacts on master @ 0a4b38348c6d4874a619126e5273829d7a5bc894:

github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).RecordingType(0x4005111800)
	github.com/cockroachdb/cockroach/pkg/util/tracing/span.go:441 +0x20
github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).IsVerbose(...)
	github.com/cockroachdb/cockroach/pkg/util/tracing/span.go:449
github.com/cockroachdb/cockroach/pkg/util/log.getSpanOrEventLog({0x674efb0, 0x40043bb5c0})
	github.com/cockroachdb/cockroach/pkg/util/log/trace.go:92 +0x30
github.com/cockroachdb/cockroach/pkg/util/log.logfDepthInternal({0x674efb0, 0x40043bb5c0}, 0x2, 0x3, 0x1, 0x1e?, {0x533851f, 0x31}, {0x40062974a0, 0x1, ...})
	github.com/cockroachdb/cockroach/pkg/util/log/channels.go:101 +0x47c
github.com/cockroachdb/cockroach/pkg/util/log.logfDepth(...)
	github.com/cockroachdb/cockroach/pkg/util/log/channels.go:39
github.com/cockroachdb/cockroach/pkg/util/log.loggerOps.Errorf(...)
	github.com/cockroachdb/cockroach/bazel-out/aarch64-fastbuild/bin/pkg/util/log/log_channels_generated.go:1281
github.com/cockroachdb/cockroach/pkg/sql/sqlinstance/instancestorage.newMigrationCache.func2()
	github.com/cockroachdb/cockroach/pkg/sql/sqlinstance/instancestorage/instancecache.go:316 +0x190
sync.(*Once).doSlow(0x58?, 0xf425600000002?)
	GOROOT/src/sync/once.go:74 +0x104
sync.(*Once).Do(...)
	GOROOT/src/sync/once.go:65
github.com/cockroachdb/cockroach/pkg/sql/sqlinstance/instancestorage.onVersionReached.func1({0x40062975e8?, 0x674ef08?}, {{0x44eb800?, 0x40?, 0x5015728?, 0x40?}})
	github.com/cockroachdb/cockroach/pkg/sql/sqlinstance/instancestorage/instancecache.go:232 +0x60
github.com/cockroachdb/cockroach/pkg/clusterversion.(*handleImpl).SetOnChange.func1({0x674ef08, 0x40044eb800})
	github.com/cockroachdb/cockroach/pkg/clusterversion/pkg/clusterversion/clusterversion.go:227 +0x74
github.com/cockroachdb/cockroach/pkg/settings.(*Values).settingChanged(0x4003db2000?, {0x674ef08, 0x40044eb800}, 0x5015728?)
	github.com/cockroachdb/cockroach/pkg/settings/pkg/settings/values.go:147 +0x160
github.com/cockroachdb/cockroach/pkg/settings.(*Values).setGeneric(0x4003db2000, {0x674ef08, 0x40044eb800}, 0x1c369f8?, {0x4748a60?, 0x4005015728?})
	github.com/cockroachdb/cockroach/pkg/settings/pkg/settings/values.go:178 +0x58
github.com/cockroachdb/cockroach/pkg/settings.(*VersionSetting).SetInternal(...)
	github.com/cockroachdb/cockroach/pkg/settings/pkg/settings/version.go:167
github.com/cockroachdb/cockroach/pkg/clusterversion.(*handleImpl).SetActiveVersion(0x4004b04d80, {0x674ef08, 0x40044eb800}, {{0xf4256, 0x2, 0x0, 0x58}})
	github.com/cockroachdb/cockroach/pkg/clusterversion/pkg/clusterversion/clusterversion.go:220 +0xf8
github.com/cockroachdb/cockroach/pkg/server/settingswatcher.(*SettingsWatcher).setLocked(0x40037970e0, {0x674ef08, 0x40044eb800}, {0x4001b13f00?, 0x4006297868?}, {{0x4001b13f10?, 0x4006297898?}, {0x96fff28?, 0x4006297848?}})
	github.com/cockroachdb/cockroach/pkg/server/settingswatcher/settings_watcher.go:325 +0x19c
github.com/cockroachdb/cockroach/pkg/server/settingswatcher.(*SettingsWatcher).maybeSet(0x40037970e0, {0x674ef08, 0x40044eb800}, {0x4001b13f00, 0x7}, {{{0x4001b13f10, 0xc}, {0x96fff28, 0x1}}, {0x174d301f0dbbc2e9, ...}, ...})
	github.com/cockroachdb/cockroach/pkg/server/settingswatcher/settings_watcher.go:295 +0x2a0
github.com/cockroachdb/cockroach/pkg/server/settingswatcher.(*SettingsWatcher).handleKV(0x40037970e0, {0x674ef08, 0x40044eb800}, 0x40072d4230)
	github.com/cockroachdb/cockroach/pkg/server/settingswatcher/settings_watcher.go:261 +0x1c8
github.com/cockroachdb/cockroach/pkg/server/settingswatcher.(*SettingsWatcher).Start.func4({0x674ef08?, 0x40044eb800?}, 0x4006297d58?)
	github.com/cockroachdb/cockroach/pkg/server/settingswatcher/settings_watcher.go:201 +0x38
github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeedcache.(*Watcher).Run.func3({0x674ef08, 0x40044eb800}, 0x3?)
	github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeedcache/watcher.go:261 +0x44
github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).processEvents(0x40027db340, {0x674ef08, 0x40044eb800}, 0x4c0d74?, 0x4001bb90e0)
	github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:362 +0x244
github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run.func2({0x674ef08?, 0x40044eb800?})
	github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:315 +0x3c
github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.GoCtx.func1()
	github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:168 +0x2c
golang.org/x/sync/errgroup.(*Group).Go.func1()
	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75 +0x5c
created by golang.org/x/sync/errgroup.(*Group).Go
	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:72 +0xa4

Help

See also: How To Investigate a Go Test Failure (internal)

_{This test on roachdash | Improve this report!}

Refs: cockroachdb#98555 Reason: flaky test Generated by bin/skip-test. Release justification: non-production code changes Release note: None Epic: None

98853: ccl/kvccl/kvtenantccl: skip TestTenantUpgradeFailure r=smg260 a=smg260 Refs: #98555 Reason: flaky test Generated by bin/skip-test. Release justification: non-production code changes Release note: None Epic: None Co-authored-by: Miral Gadani <[email protected]>

knz · 2023-04-20T16:56:55Z

I have analyzed the test and I am confident this is not a GA blocker, so I'm removing the label.

However we still need to investigate it further.

healthy-pod · 2023-04-20T17:22:58Z

Update:

This change fixes the test by bumping slinstance.DefaultTTL.Override from 3s to 15s and ensuring v2 is at most TestingBinaryVersion (or the actual BinaryVersion).

Unanswered question:
We use
v1.Internal -= 2
but v1.Internal may be <= 2 and this will cause the test to fail. We need to find a way to keep v2 <= TestingBinaryVersion and v1.Internal < v2.Internal

The code change unskips `TestTenantUpgradeFailure` after making the following changes to fix it: 1. Bump `slinstance.DefaultTTL.Override` from 3s to 15s 2. Ensure v2 isn't higher than `TestingBinaryVersion` It also updates the registry override to let the test run all migrations. Release note: None Epic: none Closes cockroachdb#98555

101956: ugpradeccl: fix and unskip `TestTenantUpgradeFailure` r=knz a=healthy-pod The code change unskips `TestTenantUpgradeFailure` after making the following changes to fix it: 1. Bump `slinstance.DefaultTTL.Override` from 3s to 15s 2. Ensure v2 isn't higher than `TestingBinaryVersion` It also updates the registry override to let the test run all migrations. Release note: None Epic: none Closes #98555 Co-authored-by: healthy-pod <[email protected]>

cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. labels Mar 14, 2023

cockroach-teamcity added this to the 23.1 milestone Mar 14, 2023

blathers-crl bot added the T-kv KV Team label Mar 16, 2023

rafiss added the GA-blocker label Mar 16, 2023

knz added the A-multitenancy Related to multi-tenancy label Mar 16, 2023

knz assigned healthy-pod Mar 16, 2023

irfansharif mentioned this issue Mar 16, 2023

spanconfig: enable range coalescing by default #98820

Merged

smg260 mentioned this issue Mar 17, 2023

ccl/kvccl/kvtenantccl: skip TestTenantUpgradeFailure #98853

Merged

nvanbenschoten removed the T-kv KV Team label Mar 21, 2023

celiala added the T-multitenant Issues owned by the multi-tenant virtual team label Mar 23, 2023

knz removed the GA-blocker label Apr 20, 2023

healthy-pod mentioned this issue Apr 20, 2023

ugpradeccl: fix and unskip TestTenantUpgradeFailure #101956

Merged

craig bot closed this as completed in 4334dfb Apr 20, 2023

yuzefovich mentioned this issue Oct 17, 2023

multitenant: TestTenantUpgradeFailure fails when min version is bumped to 23.1 #112209

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ccl/kvccl/kvtenantccl: TestTenantUpgradeFailure failed #98555

ccl/kvccl/kvtenantccl: TestTenantUpgradeFailure failed #98555

cockroach-teamcity commented Mar 14, 2023 •

edited by exalate-issue-sync bot

Loading

cockroach-teamcity commented Mar 14, 2023

cockroach-teamcity commented Mar 14, 2023

cockroach-teamcity commented Mar 15, 2023

cockroach-teamcity commented Mar 15, 2023

cockroach-teamcity commented Mar 15, 2023

cockroach-teamcity commented Mar 16, 2023

cockroach-teamcity commented Mar 16, 2023

cockroach-teamcity commented Mar 16, 2023

mgartner commented Mar 16, 2023

rafiss commented Mar 16, 2023

knz commented Mar 16, 2023

cockroach-teamcity commented Mar 17, 2023

cockroach-teamcity commented Mar 17, 2023

cockroach-teamcity commented Mar 17, 2023

cockroach-teamcity commented Mar 17, 2023

knz commented Apr 20, 2023

healthy-pod commented Apr 20, 2023

ccl/kvccl/kvtenantccl: TestTenantUpgradeFailure failed #98555

ccl/kvccl/kvtenantccl: TestTenantUpgradeFailure failed #98555

Comments

cockroach-teamcity commented Mar 14, 2023 • edited by exalate-issue-sync bot Loading

cockroach-teamcity commented Mar 14, 2023

cockroach-teamcity commented Mar 14, 2023

cockroach-teamcity commented Mar 15, 2023

cockroach-teamcity commented Mar 15, 2023

cockroach-teamcity commented Mar 15, 2023

cockroach-teamcity commented Mar 16, 2023

cockroach-teamcity commented Mar 16, 2023

cockroach-teamcity commented Mar 16, 2023

mgartner commented Mar 16, 2023

rafiss commented Mar 16, 2023

knz commented Mar 16, 2023

cockroach-teamcity commented Mar 17, 2023

cockroach-teamcity commented Mar 17, 2023

cockroach-teamcity commented Mar 17, 2023

cockroach-teamcity commented Mar 17, 2023

knz commented Apr 20, 2023

healthy-pod commented Apr 20, 2023

cockroach-teamcity commented Mar 14, 2023 •

edited by exalate-issue-sync bot

Loading