Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

sql: TestQueryCache failed #99570

Closed
cockroach-teamcity opened this issue Mar 25, 2023 · 3 comments
Closed

sql: TestQueryCache failed #99570

cockroach-teamcity opened this issue Mar 25, 2023 · 3 comments
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. T-kv KV Team
Milestone
23.1

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Mar 25, 2023 •
edited by cockroach-jira-scripts
Loading

sql.TestQueryCache failed with artifacts on master @ 0fcc33bc2870961b9387999d8a9fed97fccbb2ae:

goroutine 205520424 lock 0xc008d9b6a0
github.com/cockroachdb/cockroach/pkg/util/syncutil/singleflight/singleflight.go:395 singleflight.(*Group).doCall ??? <<<<<
github.com/cockroachdb/cockroach/pkg/util/syncutil/singleflight/singleflight.go:394 singleflight.(*Group).doCall ???

goroutine 194021442 lock 0xc00903e080
github.com/cockroachdb/cockroach/pkg/kv/kvserver/allocator/storepool/store_pool.go:971 storepool.(*StorePool).liveAndDeadReplicasWithLiveness ??? <<<<<
github.com/cockroachdb/cockroach/pkg/kv/kvserver/allocator/storepool/store_pool.go:970 storepool.(*StorePool).liveAndDeadReplicasWithLiveness ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/allocator/storepool/store_pool.go:960 storepool.(*StorePool).LiveAndDeadReplicas ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/allocator/allocatorimpl/allocator.go:708 allocatorimpl.LiveAndDeadVoterAndNonVoterReplicas ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replicate_queue.go:1049 kvserver.(*replicateQueue).PlanOneChange ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replicate_queue.go:937 kvserver.(*replicateQueue).processOneChange ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replicate_queue.go:848 kvserver.(*replicateQueue).processOneChangeWithTracing ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replicate_queue.go:731 kvserver.(*replicateQueue).process ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/queue.go:998 kvserver.(*baseQueue).processReplica.func1 ???
github.com/cockroachdb/cockroach/pkg/util/contextutil/context.go:91 contextutil.RunWithTimeout ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/queue.go:956 kvserver.(*baseQueue).processReplica ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/queue.go:1248 kvserver.(*baseQueue).processReplicasInPurgatory.func1.2 ???
github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:305 stop.(*Stopper).RunTask ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/queue.go:1250 kvserver.(*baseQueue).processReplicasInPurgatory.func1 ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/queue.go:1257 kvserver.(*baseQueue).processReplicasInPurgatory ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/queue.go:1193 kvserver.(*baseQueue).addToPurgatoryLocked.func2 ???
github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:470 stop.(*Stopper).RunAsyncTaskEx.func2 ???

goroutine 171809928 lock 0xc0046fb9d0
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:707 kvserver.(*Replica).handleRaftReady ??? <<<<<
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:706 kvserver.(*Replica).handleRaftReady ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:644 kvserver.(*Store).processReady ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:394 kvserver.(*raftSchedulerShard).worker ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:299 kvserver.(*raftScheduler).Start.func2 ???
github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:470 stop.(*Stopper).RunAsyncTaskEx.func2 ???



POTENTIAL DEADLOCK:
Previous place where the lock was grabbed
goroutine 193878508 lock 0xc0189eb2a0
github.com/cockroachdb/cockroach/pkg/server/node.go:1493 server.(*lockedMuxStream).Send ??? <<<<<
github.com/cockroachdb/cockroach/pkg/server/node.go:1492 server.(*lockedMuxStream).Send ???
github.com/cockroachdb/cockroach/pkg/server/node.go:1475 server.(*setRangeIDEventSink).Send ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_rangefeed.go:78 kvserver.(*lockedRangefeedStream).Send ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed/registry.go:330 rangefeed.(*registration).outputLoop ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed/registry.go:351 rangefeed.(*registration).runOutputLoop ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed/processor.go:321 rangefeed.(*Processor).run.func1 ???
github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:470 stop.(*Stopper).RunAsyncTaskEx.func2 ???

Have been trying to lock it again for more than 5m0s
goroutine 193653314 lock 0xc0189eb2a0
github.com/cockroachdb/cockroach/pkg/server/node.go:1493 server.(*lockedMuxStream).Send ??? <<<<<
github.com/cockroachdb/cockroach/pkg/server/node.go:1492 server.(*lockedMuxStream).Send ???
github.com/cockroachdb/cockroach/pkg/server/node.go:1475 server.(*setRangeIDEventSink).Send ???

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/sql-queries

This test on roachdash | Improve this report!

Jira issue: CRDB-25943
@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. labels Mar 25, 2023
@cockroach-teamcity cockroach-teamcity added this to the 23.1 milestone Mar 25, 2023
@blathers-crl blathers-crl bot added the T-sql-queries SQL Queries Team label Mar 25, 2023
@blathers-crl blathers-crl bot added the T-kv KV Team label Mar 27, 2023
@cockroach-teamcity
Copy link
Member Author

sql.TestQueryCache failed with artifacts on master @ 2bd2c806ab3044569b09e0a205b5bc0452ad4e2b:

github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1246 kvcoord.(*DistSender).divideAndSendBatchToRanges ???
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:867 kvcoord.(*DistSender).Send ???
github.com/cockroachdb/cockroach/pkg/kv/range_lookup.go:331 kv.lookupRangeFwdScan ???
github.com/cockroachdb/cockroach/pkg/kv/range_lookup.go:206 kv.RangeLookup ???
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:578 kvcoord.(*DistSender).RangeLookup ???
github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangecache/range_cache.go:1040 rangecache.(*RangeCache).performRangeLookup ???
github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangecache/range_cache.go:928 rangecache.tryLookupImpl.func1 ???
github.com/cockroachdb/cockroach/pkg/util/contextutil/context.go:91 contextutil.RunWithTimeout ???
github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangecache/range_cache.go:930 rangecache.tryLookupImpl ???
github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangecache/range_cache.go:834 rangecache.(*RangeCache).tryLookup.func3 ???
github.com/cockroachdb/cockroach/pkg/util/syncutil/singleflight/singleflight.go:386 singleflight.(*Group).doCall.func1 ???
github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:305 stop.(*Stopper).RunTask ???
github.com/cockroachdb/cockroach/pkg/util/syncutil/singleflight/singleflight.go:385 singleflight.(*Group).doCall ???



=== RUN   TestQueryCache/group/schemachange-parallel
=== PAUSE TestQueryCache/group/schemachange-parallel
=== CONT  TestQueryCache/group/schemachange-parallel
=== RUN   TestQueryCache/group/simple
=== PAUSE TestQueryCache/group/simple
=== CONT  TestQueryCache/group/simple
=== RUN   TestQueryCache/group
=== RUN   TestQueryCache/group/simple-prepare-with-args
=== PAUSE TestQueryCache/group/simple-prepare-with-args
=== CONT  TestQueryCache/group/simple-prepare-with-args
=== RUN   TestQueryCache/group/statschange
=== PAUSE TestQueryCache/group/statschange
=== CONT  TestQueryCache/group/statschange
=== RUN   TestQueryCache/group/multidb
=== PAUSE TestQueryCache/group/multidb
=== CONT  TestQueryCache/group/multidb
=== RUN   TestQueryCache/group/multidb-prepare
=== PAUSE TestQueryCache/group/multidb-prepare
=== CONT  TestQueryCache/group/multidb-prepare
=== RUN   TestQueryCache/group/schemachange
=== PAUSE TestQueryCache/group/schemachange
=== CONT  TestQueryCache/group/schemachange
=== RUN   TestQueryCache/group/schemachange-prepare
=== PAUSE TestQueryCache/group/schemachange-prepare
=== CONT  TestQueryCache/group/schemachange-prepare
=== RUN   TestQueryCache/group/parallel
=== PAUSE TestQueryCache/group/parallel
=== CONT  TestQueryCache/group/parallel
=== RUN   TestQueryCache/group/prepare-hints
=== PAUSE TestQueryCache/group/prepare-hints
=== CONT  TestQueryCache/group/prepare-hints
=== RUN   TestQueryCache/group/simple-prepare
=== PAUSE TestQueryCache/group/simple-prepare
=== CONT  TestQueryCache/group/simple-prepare

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity cockroach-teamcity mentioned this issue Mar 27, 2023
Closed
@cockroach-teamcity
Copy link
Member Author

sql.TestQueryCache failed with artifacts on master @ 37ac1e5f751d981ecced8f2bf94f12cb2ab7e150:

      github.com/cockroachdb/cockroach/pkg/util/admission/grant_coordinator.go:98 +0x329
  github.com/cockroachdb/cockroach/pkg/server.(*Server).PreStart()
      github.com/cockroachdb/cockroach/pkg/server/server.go:1804 +0x45bb
  github.com/cockroachdb/cockroach/pkg/server.(*TestServer).Start()
      github.com/cockroachdb/cockroach/pkg/server/testserver.go:612 +0x8f
  github.com/cockroachdb/cockroach/pkg/testutils/serverutils.StartServer()
      github.com/cockroachdb/cockroach/pkg/testutils/serverutils/test_server_shim.go:330 +0x1b6
  github.com/cockroachdb/cockroach/pkg/testutils/serverutils.StartServer()
      github.com/cockroachdb/cockroach/pkg/testutils/serverutils/test_server_shim.go:325 +0x96
  github.com/cockroachdb/cockroach/pkg/sql.makeQueryCacheTestHelper()
      github.com/cockroachdb/cockroach/pkg/sql/plan_opt_test.go:50 +0xf1
  github.com/cockroachdb/cockroach/pkg/sql.TestQueryCache.func1.13()
      github.com/cockroachdb/cockroach/pkg/sql/plan_opt_test.go:482 +0x64
  testing.tRunner()
      GOROOT/src/testing/testing.go:1446 +0x216
  testing.(*T).Run.func1()
      GOROOT/src/testing/testing.go:1493 +0x47

Goroutine 1287759 (running) created at:
  testing.(*T).Run()
      GOROOT/src/testing/testing.go:1493 +0x75d
  github.com/cockroachdb/cockroach/pkg/sql.TestQueryCache.func1()
      github.com/cockroachdb/cockroach/pkg/sql/plan_opt_test.go:480 +0x1c4
  testing.tRunner()
      GOROOT/src/testing/testing.go:1446 +0x216
  testing.(*T).Run.func1()
      GOROOT/src/testing/testing.go:1493 +0x47
==================
=== RUN   TestQueryCache/group/parallel-prepare
=== PAUSE TestQueryCache/group/parallel-prepare
=== CONT  TestQueryCache/group/parallel-prepare
=== RUN   TestQueryCache/group/schemachange-prepare
=== PAUSE TestQueryCache/group/schemachange-prepare
=== CONT  TestQueryCache/group/schemachange-prepare
=== RUN   TestQueryCache/group/statschange
=== PAUSE TestQueryCache/group/statschange
=== CONT  TestQueryCache/group/statschange
=== RUN   TestQueryCache/group/parallel
=== PAUSE TestQueryCache/group/parallel
=== CONT  TestQueryCache/group/parallel
=== RUN   TestQueryCache/group
=== RUN   TestQueryCache/group/schemachange
=== PAUSE TestQueryCache/group/schemachange
=== CONT  TestQueryCache/group/schemachange
=== RUN   TestQueryCache/group/simple
=== PAUSE TestQueryCache/group/simple
=== CONT  TestQueryCache/group/simple
=== RUN   TestQueryCache/group/simple-prepare-with-args
=== PAUSE TestQueryCache/group/simple-prepare-with-args
=== CONT  TestQueryCache/group/simple-prepare-with-args

Parameters: TAGS=bazel,gss,race

Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity cockroach-teamcity mentioned this issue Mar 30, 2023
Closed
@michae2
Copy link
Collaborator

michae2 commented Mar 30, 2023

Dupe of #99640

@michae2 michae2 closed this as completed Mar 30, 2023
@exalate-issue-sync exalate-issue-sync bot removed the T-sql-queries SQL Queries Team label Mar 30, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. T-kv KV Team
Projects
None yet
Milestone
23.1
Development

No branches or pull requests
2 participants
@cockroach-teamcity @michae2