Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ccl/sqlproxyccl/tenant: TestResume failed #106571

Closed
cockroach-teamcity opened this issue Jul 11, 2023 · 4 comments
Closed

ccl/sqlproxyccl/tenant: TestResume failed #106571

cockroach-teamcity opened this issue Jul 11, 2023 · 4 comments
Assignees
Labels
branch-release-23.1 Used to mark GA and release blockers, technical advisories, and bugs for 23.1 C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. T-serverless
Milestone

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Jul 11, 2023

ccl/sqlproxyccl/tenant.TestResume failed with artifacts on release-23.1 @ 17d14efc67f680945eb3a38dd0840ec70f766709:

github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/tenantdirsvr/test_directory_svr.go:55 tenantdirsvr.NewSubStopper.func1 ??? <<<<<
github.com/sasha-s/go-deadlock/external/com_github_sasha_s_go_deadlock/deadlock.go:84 go-deadlock.(*Mutex).Lock ???
github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:107 stop.CloserFn.Close ???
github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:530 stop.(*Stopper).Stop ???
github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:224 testcluster.(*TestCluster).stopServerLocked ???
github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:159 testcluster.(*TestCluster).stopServers ???
github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:440 testcluster.(*TestCluster).Start.func2 ???
github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:107 stop.CloserFn.Close ???
github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:530 stop.(*Stopper).Stop ???
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/tenant_test/pkg/ccl/sqlproxyccl/tenant/directory_cache_test.go:456 tenant_test.TestResume ???

happened after
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/tenantdirsvr/test_directory_svr.go:366 tenantdirsvr.(*TestDirectoryServer).deregisterInstance ??? <<<<<
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/tenantdirsvr/test_directory_svr.go:365 tenantdirsvr.(*TestDirectoryServer).deregisterInstance ???
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/tenantdirsvr/test_directory_svr.go:314 tenantdirsvr.(*TestDirectoryServer).EnsurePod.func1 ???
github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:107 stop.CloserFn.Close ???
github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:530 stop.(*Stopper).Stop ???
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/tenantdirsvr/test_directory_svr.go:60 tenantdirsvr.NewSubStopper.func1 ???
github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:107 stop.CloserFn.Close ???
github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:530 stop.(*Stopper).Stop ???
github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:224 testcluster.(*TestCluster).stopServerLocked ???
github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:159 testcluster.(*TestCluster).stopServers ???
github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:440 testcluster.(*TestCluster).Start.func2 ???
github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:107 stop.CloserFn.Close ???
github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:530 stop.(*Stopper).Stop ???
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/tenant_test/pkg/ccl/sqlproxyccl/tenant/directory_cache_test.go:456 tenant_test.TestResume ???

Other goroutines holding locks:
goroutine 654626 lock 0xc0005e2740
github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:139 testcluster.(*TestCluster).stopServers ??? <<<<<
github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:138 testcluster.(*TestCluster).stopServers ???
github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:440 testcluster.(*TestCluster).Start.func2 ???
github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:107 stop.CloserFn.Close ???
github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:530 stop.(*Stopper).Stop ???
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/tenant_test/pkg/ccl/sqlproxyccl/tenant/directory_cache_test.go:456 tenant_test.TestResume ???

goroutine 654626 lock 0xc003b6d388
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/tenantdirsvr/test_directory_svr.go:55 tenantdirsvr.NewSubStopper.func1 ??? <<<<<
github.com/sasha-s/go-deadlock/external/com_github_sasha_s_go_deadlock/deadlock.go:84 go-deadlock.(*Mutex).Lock ???
github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:107 stop.CloserFn.Close ???
github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:530 stop.(*Stopper).Stop ???
github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:224 testcluster.(*TestCluster).stopServerLocked ???
github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:159 testcluster.(*TestCluster).stopServers ???
github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:440 testcluster.(*TestCluster).Start.func2 ???
github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:107 stop.CloserFn.Close ???
github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:530 stop.(*Stopper).Stop ???
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/tenant_test/pkg/ccl/sqlproxyccl/tenant/directory_cache_test.go:456 tenant_test.TestResume ???



Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/sql-foundations @cockroachdb/server

This test on roachdash | Improve this report!

Jira issue: CRDB-29621

@cockroach-teamcity cockroach-teamcity added branch-release-23.1 Used to mark GA and release blockers, technical advisories, and bugs for 23.1 C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions) labels Jul 11, 2023
@cockroach-teamcity cockroach-teamcity added this to the 23.1 milestone Jul 11, 2023
@rafiss rafiss removed the T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions) label Jul 11, 2023
@cockroach-teamcity
Copy link
Member Author

ccl/sqlproxyccl/tenant.TestResume failed with artifacts on release-23.1 @ 7bfb0ce4cf4e5d3c6c51fd97ced445f4c05fbdc8:

    directory_cache_test.go:655: starting tenant 40
    directory_cache_test.go:660: tenant 40 started
POTENTIAL DEADLOCK: Inconsistent locking. saw this ordering in one goroutine:
happened before
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/tenantdirsvr/test_directory_svr.go:287 tenantdirsvr.(*TestDirectoryServer).EnsurePod.func1 ??? <<<<<
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/tenantdirsvr/test_directory_svr.go:286 tenantdirsvr.(*TestDirectoryServer).EnsurePod.func1 ???
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/tenantdirsvr/test_directory_svr.go:306 tenantdirsvr.(*TestDirectoryServer).EnsurePod ???
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/tenant/bazel-out/k8-fastbuild/bin/pkg/ccl/sqlproxyccl/tenant/tenant_go_proto_/github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/tenant/directory.pb.go:931 tenant._Directory_EnsurePod_Handler ???
google.golang.org/grpc/external/org_golang_google_grpc/server.go:1368 grpc.(*Server).processUnaryRPC ???
google.golang.org/grpc/external/org_golang_google_grpc/server.go:1713 grpc.(*Server).handleStream ???
google.golang.org/grpc/external/org_golang_google_grpc/server.go:965 grpc.(*Server).serveStreams.func1.2 ???

happened after
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/tenantdirsvr/test_directory_svr.go:47 tenantdirsvr.NewSubStopper ??? <<<<<
github.com/sasha-s/go-deadlock/external/com_github_sasha_s_go_deadlock/deadlock.go:84 go-deadlock.(*Mutex).Lock ???
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/tenantdirsvr/test_directory_svr.go:295 tenantdirsvr.(*TestDirectoryServer).EnsurePod.func1 ???
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/tenantdirsvr/test_directory_svr.go:306 tenantdirsvr.(*TestDirectoryServer).EnsurePod ???
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/tenant/bazel-out/k8-fastbuild/bin/pkg/ccl/sqlproxyccl/tenant/tenant_go_proto_/github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/tenant/directory.pb.go:931 tenant._Directory_EnsurePod_Handler ???
google.golang.org/grpc/external/org_golang_google_grpc/server.go:1368 grpc.(*Server).processUnaryRPC ???
google.golang.org/grpc/external/org_golang_google_grpc/server.go:1713 grpc.(*Server).handleStream ???
google.golang.org/grpc/external/org_golang_google_grpc/server.go:965 grpc.(*Server).serveStreams.func1.2 ???

in another goroutine: happened before
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/tenantdirsvr/test_directory_svr.go:40 tenantdirsvr.NewSubStopper.func1 ??? <<<<<
github.com/sasha-s/go-deadlock/external/com_github_sasha_s_go_deadlock/deadlock.go:84 go-deadlock.(*Mutex).Lock ???
github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:107 stop.CloserFn.Close ???
github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:530 stop.(*Stopper).Stop ???
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/tenant_test/pkg/ccl/sqlproxyccl/tenant/directory_cache_test.go:455 tenant_test.TestResume ???

happened after
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/tenantdirsvr/test_directory_svr.go:359 tenantdirsvr.(*TestDirectoryServer).deregisterInstance ??? <<<<<
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/tenantdirsvr/test_directory_svr.go:358 tenantdirsvr.(*TestDirectoryServer).deregisterInstance ???
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/tenantdirsvr/test_directory_svr.go:315 tenantdirsvr.(*TestDirectoryServer).EnsurePod.func2 ???
github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:107 stop.CloserFn.Close ???
github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:530 stop.(*Stopper).Stop ???
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/tenantdirsvr/test_directory_svr.go:45 tenantdirsvr.NewSubStopper.func1 ???
github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:107 stop.CloserFn.Close ???
github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:530 stop.(*Stopper).Stop ???
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/tenant_test/pkg/ccl/sqlproxyccl/tenant/directory_cache_test.go:455 tenant_test.TestResume ???

Other goroutines holding locks:
goroutine 661470 lock 0xc003f604a8
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/tenantdirsvr/test_directory_svr.go:40 tenantdirsvr.NewSubStopper.func1 ??? <<<<<
github.com/sasha-s/go-deadlock/external/com_github_sasha_s_go_deadlock/deadlock.go:84 go-deadlock.(*Mutex).Lock ???
github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:107 stop.CloserFn.Close ???
github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:530 stop.(*Stopper).Stop ???
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/tenant_test/pkg/ccl/sqlproxyccl/tenant/directory_cache_test.go:455 tenant_test.TestResume ???



Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/sqlproxyccl/tenant.TestResume failed with artifacts on release-23.1 @ 8897f4d13a030381997ff30aeaf633a9f8e87340:

    directory_cache_test.go:655: starting tenant 40
    directory_cache_test.go:660: tenant 40 started
POTENTIAL DEADLOCK: Inconsistent locking. saw this ordering in one goroutine:
happened before
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/tenantdirsvr/test_directory_svr.go:287 tenantdirsvr.(*TestDirectoryServer).EnsurePod.func1 ??? <<<<<
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/tenantdirsvr/test_directory_svr.go:286 tenantdirsvr.(*TestDirectoryServer).EnsurePod.func1 ???
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/tenantdirsvr/test_directory_svr.go:306 tenantdirsvr.(*TestDirectoryServer).EnsurePod ???
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/tenant/bazel-out/k8-fastbuild/bin/pkg/ccl/sqlproxyccl/tenant/tenant_go_proto_/github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/tenant/directory.pb.go:931 tenant._Directory_EnsurePod_Handler ???
google.golang.org/grpc/external/org_golang_google_grpc/server.go:1368 grpc.(*Server).processUnaryRPC ???
google.golang.org/grpc/external/org_golang_google_grpc/server.go:1713 grpc.(*Server).handleStream ???
google.golang.org/grpc/external/org_golang_google_grpc/server.go:965 grpc.(*Server).serveStreams.func1.2 ???

happened after
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/tenantdirsvr/test_directory_svr.go:47 tenantdirsvr.NewSubStopper ??? <<<<<
github.com/sasha-s/go-deadlock/external/com_github_sasha_s_go_deadlock/deadlock.go:84 go-deadlock.(*Mutex).Lock ???
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/tenantdirsvr/test_directory_svr.go:295 tenantdirsvr.(*TestDirectoryServer).EnsurePod.func1 ???
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/tenantdirsvr/test_directory_svr.go:306 tenantdirsvr.(*TestDirectoryServer).EnsurePod ???
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/tenant/bazel-out/k8-fastbuild/bin/pkg/ccl/sqlproxyccl/tenant/tenant_go_proto_/github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/tenant/directory.pb.go:931 tenant._Directory_EnsurePod_Handler ???
google.golang.org/grpc/external/org_golang_google_grpc/server.go:1368 grpc.(*Server).processUnaryRPC ???
google.golang.org/grpc/external/org_golang_google_grpc/server.go:1713 grpc.(*Server).handleStream ???
google.golang.org/grpc/external/org_golang_google_grpc/server.go:965 grpc.(*Server).serveStreams.func1.2 ???

in another goroutine: happened before
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/tenantdirsvr/test_directory_svr.go:40 tenantdirsvr.NewSubStopper.func1 ??? <<<<<
github.com/sasha-s/go-deadlock/external/com_github_sasha_s_go_deadlock/deadlock.go:84 go-deadlock.(*Mutex).Lock ???
github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:107 stop.CloserFn.Close ???
github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:530 stop.(*Stopper).Stop ???
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/tenant_test/pkg/ccl/sqlproxyccl/tenant/directory_cache_test.go:455 tenant_test.TestResume ???

happened after
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/tenantdirsvr/test_directory_svr.go:359 tenantdirsvr.(*TestDirectoryServer).deregisterInstance ??? <<<<<
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/tenantdirsvr/test_directory_svr.go:358 tenantdirsvr.(*TestDirectoryServer).deregisterInstance ???
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/tenantdirsvr/test_directory_svr.go:315 tenantdirsvr.(*TestDirectoryServer).EnsurePod.func2 ???
github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:107 stop.CloserFn.Close ???
github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:530 stop.(*Stopper).Stop ???
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/tenantdirsvr/test_directory_svr.go:45 tenantdirsvr.NewSubStopper.func1 ???
github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:107 stop.CloserFn.Close ???
github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:530 stop.(*Stopper).Stop ???
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/tenant_test/pkg/ccl/sqlproxyccl/tenant/directory_cache_test.go:455 tenant_test.TestResume ???

Other goroutines holding locks:
goroutine 590468 lock 0xc00315e6c0
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/tenantdirsvr/test_directory_svr.go:40 tenantdirsvr.NewSubStopper.func1 ??? <<<<<
github.com/sasha-s/go-deadlock/external/com_github_sasha_s_go_deadlock/deadlock.go:84 go-deadlock.(*Mutex).Lock ???
github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:107 stop.CloserFn.Close ???
github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:530 stop.(*Stopper).Stop ???
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/tenant_test/pkg/ccl/sqlproxyccl/tenant/directory_cache_test.go:455 tenant_test.TestResume ???



Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/sqlproxyccl/tenant.TestResume failed with artifacts on release-23.1 @ aaedb3dd0d97097cff7afeb2532c6265d3d62818:

    directory_cache_test.go:655: starting tenant 40
    directory_cache_test.go:660: tenant 40 started
POTENTIAL DEADLOCK: Inconsistent locking. saw this ordering in one goroutine:
happened before
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/tenantdirsvr/test_directory_svr.go:287 tenantdirsvr.(*TestDirectoryServer).EnsurePod.func1 ??? <<<<<
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/tenantdirsvr/test_directory_svr.go:286 tenantdirsvr.(*TestDirectoryServer).EnsurePod.func1 ???
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/tenantdirsvr/test_directory_svr.go:306 tenantdirsvr.(*TestDirectoryServer).EnsurePod ???
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/tenant/bazel-out/k8-fastbuild/bin/pkg/ccl/sqlproxyccl/tenant/tenant_go_proto_/github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/tenant/directory.pb.go:931 tenant._Directory_EnsurePod_Handler ???
google.golang.org/grpc/external/org_golang_google_grpc/server.go:1368 grpc.(*Server).processUnaryRPC ???
google.golang.org/grpc/external/org_golang_google_grpc/server.go:1713 grpc.(*Server).handleStream ???
google.golang.org/grpc/external/org_golang_google_grpc/server.go:965 grpc.(*Server).serveStreams.func1.2 ???

happened after
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/tenantdirsvr/test_directory_svr.go:47 tenantdirsvr.NewSubStopper ??? <<<<<
github.com/sasha-s/go-deadlock/external/com_github_sasha_s_go_deadlock/deadlock.go:84 go-deadlock.(*Mutex).Lock ???
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/tenantdirsvr/test_directory_svr.go:295 tenantdirsvr.(*TestDirectoryServer).EnsurePod.func1 ???
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/tenantdirsvr/test_directory_svr.go:306 tenantdirsvr.(*TestDirectoryServer).EnsurePod ???
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/tenant/bazel-out/k8-fastbuild/bin/pkg/ccl/sqlproxyccl/tenant/tenant_go_proto_/github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/tenant/directory.pb.go:931 tenant._Directory_EnsurePod_Handler ???
google.golang.org/grpc/external/org_golang_google_grpc/server.go:1368 grpc.(*Server).processUnaryRPC ???
google.golang.org/grpc/external/org_golang_google_grpc/server.go:1713 grpc.(*Server).handleStream ???
google.golang.org/grpc/external/org_golang_google_grpc/server.go:965 grpc.(*Server).serveStreams.func1.2 ???

in another goroutine: happened before
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/tenantdirsvr/test_directory_svr.go:40 tenantdirsvr.NewSubStopper.func1 ??? <<<<<
github.com/sasha-s/go-deadlock/external/com_github_sasha_s_go_deadlock/deadlock.go:84 go-deadlock.(*Mutex).Lock ???
github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:107 stop.CloserFn.Close ???
github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:530 stop.(*Stopper).Stop ???
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/tenant_test/pkg/ccl/sqlproxyccl/tenant/directory_cache_test.go:455 tenant_test.TestResume ???

happened after
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/tenantdirsvr/test_directory_svr.go:359 tenantdirsvr.(*TestDirectoryServer).deregisterInstance ??? <<<<<
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/tenantdirsvr/test_directory_svr.go:358 tenantdirsvr.(*TestDirectoryServer).deregisterInstance ???
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/tenantdirsvr/test_directory_svr.go:315 tenantdirsvr.(*TestDirectoryServer).EnsurePod.func2 ???
github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:107 stop.CloserFn.Close ???
github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:530 stop.(*Stopper).Stop ???
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/tenantdirsvr/test_directory_svr.go:45 tenantdirsvr.NewSubStopper.func1 ???
github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:107 stop.CloserFn.Close ???
github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:530 stop.(*Stopper).Stop ???
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/tenant_test/pkg/ccl/sqlproxyccl/tenant/directory_cache_test.go:455 tenant_test.TestResume ???

Other goroutines holding locks:
goroutine 668445 lock 0xc003dbf428
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/tenantdirsvr/test_directory_svr.go:40 tenantdirsvr.NewSubStopper.func1 ??? <<<<<
github.com/sasha-s/go-deadlock/external/com_github_sasha_s_go_deadlock/deadlock.go:84 go-deadlock.(*Mutex).Lock ???
github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:107 stop.CloserFn.Close ???
github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:530 stop.(*Stopper).Stop ???
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/tenant_test/pkg/ccl/sqlproxyccl/tenant/directory_cache_test.go:455 tenant_test.TestResume ???



Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/sqlproxyccl/tenant.TestResume failed with artifacts on release-23.1 @ 8b970683af4630cbb30b4c92f0be86cfadd97993:

    directory_cache_test.go:655: starting tenant 40
    directory_cache_test.go:660: tenant 40 started
POTENTIAL DEADLOCK: Inconsistent locking. saw this ordering in one goroutine:
happened before
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/tenantdirsvr/test_directory_svr.go:287 tenantdirsvr.(*TestDirectoryServer).EnsurePod.func1 ??? <<<<<
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/tenantdirsvr/test_directory_svr.go:286 tenantdirsvr.(*TestDirectoryServer).EnsurePod.func1 ???
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/tenantdirsvr/test_directory_svr.go:306 tenantdirsvr.(*TestDirectoryServer).EnsurePod ???
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/tenant/bazel-out/k8-fastbuild/bin/pkg/ccl/sqlproxyccl/tenant/tenant_go_proto_/github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/tenant/directory.pb.go:931 tenant._Directory_EnsurePod_Handler ???
google.golang.org/grpc/external/org_golang_google_grpc/server.go:1368 grpc.(*Server).processUnaryRPC ???
google.golang.org/grpc/external/org_golang_google_grpc/server.go:1713 grpc.(*Server).handleStream ???
google.golang.org/grpc/external/org_golang_google_grpc/server.go:965 grpc.(*Server).serveStreams.func1.2 ???

happened after
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/tenantdirsvr/test_directory_svr.go:47 tenantdirsvr.NewSubStopper ??? <<<<<
github.com/sasha-s/go-deadlock/external/com_github_sasha_s_go_deadlock/deadlock.go:84 go-deadlock.(*Mutex).Lock ???
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/tenantdirsvr/test_directory_svr.go:295 tenantdirsvr.(*TestDirectoryServer).EnsurePod.func1 ???
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/tenantdirsvr/test_directory_svr.go:306 tenantdirsvr.(*TestDirectoryServer).EnsurePod ???
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/tenant/bazel-out/k8-fastbuild/bin/pkg/ccl/sqlproxyccl/tenant/tenant_go_proto_/github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/tenant/directory.pb.go:931 tenant._Directory_EnsurePod_Handler ???
google.golang.org/grpc/external/org_golang_google_grpc/server.go:1368 grpc.(*Server).processUnaryRPC ???
google.golang.org/grpc/external/org_golang_google_grpc/server.go:1713 grpc.(*Server).handleStream ???
google.golang.org/grpc/external/org_golang_google_grpc/server.go:965 grpc.(*Server).serveStreams.func1.2 ???

in another goroutine: happened before
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/tenantdirsvr/test_directory_svr.go:40 tenantdirsvr.NewSubStopper.func1 ??? <<<<<
github.com/sasha-s/go-deadlock/external/com_github_sasha_s_go_deadlock/deadlock.go:84 go-deadlock.(*Mutex).Lock ???
github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:107 stop.CloserFn.Close ???
github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:530 stop.(*Stopper).Stop ???
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/tenant_test/pkg/ccl/sqlproxyccl/tenant/directory_cache_test.go:455 tenant_test.TestResume ???

happened after
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/tenantdirsvr/test_directory_svr.go:359 tenantdirsvr.(*TestDirectoryServer).deregisterInstance ??? <<<<<
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/tenantdirsvr/test_directory_svr.go:358 tenantdirsvr.(*TestDirectoryServer).deregisterInstance ???
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/tenantdirsvr/test_directory_svr.go:315 tenantdirsvr.(*TestDirectoryServer).EnsurePod.func2 ???
github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:107 stop.CloserFn.Close ???
github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:530 stop.(*Stopper).Stop ???
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/tenantdirsvr/test_directory_svr.go:45 tenantdirsvr.NewSubStopper.func1 ???
github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:107 stop.CloserFn.Close ???
github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:530 stop.(*Stopper).Stop ???
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/tenant_test/pkg/ccl/sqlproxyccl/tenant/directory_cache_test.go:455 tenant_test.TestResume ???

Other goroutines holding locks:
goroutine 673717 lock 0xc0027939a8
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/tenantdirsvr/test_directory_svr.go:40 tenantdirsvr.NewSubStopper.func1 ??? <<<<<
github.com/sasha-s/go-deadlock/external/com_github_sasha_s_go_deadlock/deadlock.go:84 go-deadlock.(*Mutex).Lock ???
github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:107 stop.CloserFn.Close ???
github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:530 stop.(*Stopper).Stop ???
github.com/cockroachdb/cockroach/pkg/ccl/sqlproxyccl/tenant_test/pkg/ccl/sqlproxyccl/tenant/directory_cache_test.go:455 tenant_test.TestResume ???



Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

craig bot pushed a commit that referenced this issue Aug 1, 2023
107294: ttl: show table name for jobs in SHOW SCHEDULES r=chrisseto a=chrisseto

Previously, `SHOW SCHEDULES` would only show a table's ID for row level TTL jobs. For the convenience of debugging, this commit upgrades the `label` column to contain the table's name instead of its ID.

Rather than performing table name resolution at query time, the job name is set at creation time and updated by the schema change during table renames. This minimizes the likelihood of `SHOW SCHEDULES` being rendered unusable when a cluster is in an unstable state.

Epic: CRDB-18322
Fixes: #93774

107316: sqlproxyccl: simplify NewSubStopper r=JeffSwenson a=JeffSwenson

The lock in NewSubStopper caused lock ordering warnings when run under deadlock detection. The justification given for the lock's existence is wrong. If a closer is added to an already stopped stopper, the closer is called immediately.

Release note: None
Part of: #106571

107941: storage: don't import `testing` for `DisableMetamorphicSimpleValueEncoding` r=erikgrinaker a=erikgrinaker

We shouldn't link `testing` in binaries.

Epic: none
Release note: None

107950: testutils: fix determinism in random predecessor history tests r=jayshrivastava a=renatolabs

Previously, the `rng` instance used by the tests was initialized once in the `var` block. That meant that every assertion that relied on that rng was order-dependent: running a single test in that file in isolation would lead to failed assertions.

This commit ensures that every test resets the rng before making any assertions, guaranteeing we get the same values regardless of execution order.

Epic: none

Release note: None

Co-authored-by: Chris Seto <[email protected]>
Co-authored-by: Jeff <[email protected]>
Co-authored-by: Erik Grinaker <[email protected]>
Co-authored-by: Renato Costa <[email protected]>
@craig craig bot closed this as completed in 6d94042 Aug 1, 2023
blathers-crl bot pushed a commit that referenced this issue Aug 1, 2023
The lock in NewSubStopper caused lock ordering warnings when run under
deadlock detection. The justification given for the lock's existence is
wrong. If a closer is added to an already stopped stopper, the closer is
called immediately.

Release note: None
Fixes: #106571
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
branch-release-23.1 Used to mark GA and release blockers, technical advisories, and bugs for 23.1 C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. T-serverless
Projects
None yet
Development

No branches or pull requests

3 participants