test hang at package `table/tables` #29632

unconsolable · 2021-11-10T01:43:14Z

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

Run check_dev_2

2. What did you expect to see? (Required)

At least no test hang

3. What did you see instead (Required)

Test hang
https://ci.pingcap.net/blue/organizations/jenkins/tidb_ghpr_check_2/detail/tidb_ghpr_check_2/44389/pipeline/

[2021-11-10T00:09:21.408Z] panic: test timed out after 9m0s

[2021-11-10T00:09:21.408Z] 

[2021-11-10T00:09:21.408Z] goroutine 78408 [running]:

[2021-11-10T00:09:21.408Z] testing.(*M).startAlarm.func1()

[2021-11-10T00:09:21.408Z] 	/usr/local/go/src/testing/testing.go:1700 +0xe5

[2021-11-10T00:09:21.408Z] created by time.goFunc

[2021-11-10T00:09:21.408Z] 	/usr/local/go/src/time/sleep.go:180 +0x45

[2021-11-10T00:09:21.408Z] 

[2021-11-10T00:09:21.408Z] goroutine 1 [chan receive, 8 minutes]:

[2021-11-10T00:09:21.408Z] testing.tRunner.func1(0xc00058a600)

[2021-11-10T00:09:21.408Z] 	/usr/local/go/src/testing/testing.go:1159 +0x2bc

[2021-11-10T00:09:21.408Z] testing.tRunner(0xc00058a600, 0xc000dc7c60)

[2021-11-10T00:09:21.408Z] 	/usr/local/go/src/testing/testing.go:1197 +0x125

[2021-11-10T00:09:21.408Z] testing.runTests(0xc0010aa468, 0x58f2c60, 0x26, 0x26, 0xc05ae14b587facd8, 0x7dbdb64229, 0x5fd80a0, 0x38)

[2021-11-10T00:09:21.408Z] 	/usr/local/go/src/testing/testing.go:1509 +0x2fe

[2021-11-10T00:09:21.408Z] testing.(*M).Run(0xc001039900, 0x0)

[2021-11-10T00:09:21.408Z] 	/usr/local/go/src/testing/testing.go:1417 +0x1eb

[2021-11-10T00:09:21.408Z] go.uber.org/goleak.VerifyTestMain(0x41574c0, 0xc001039900, 0xc0014c7e60, 0x3, 0x3)

[2021-11-10T00:09:21.408Z] 	/nfs/cache/mod/go.uber.org/[email protected]/testmain.go:53 +0x38

[2021-11-10T00:09:21.408Z] github.com/pingcap/tidb/table/tables_test.TestMain(0xc001039900)

[2021-11-10T00:09:21.408Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/table/tables/main_test.go:31 +0x125

[2021-11-10T00:09:21.408Z] main.main()

[2021-11-10T00:09:21.408Z] 	_testmain.go:177 +0x1f6

[2021-11-10T00:09:21.408Z] 

[2021-11-10T00:09:21.408Z] goroutine 71 [select]:

[2021-11-10T00:09:21.408Z] go.opencensus.io/stats/view.(*worker).start(0xc000ae2b00)

[2021-11-10T00:09:21.408Z] 	/nfs/cache/mod/[email protected]/stats/view/worker.go:276 +0xcd

[2021-11-10T00:09:21.408Z] created by go.opencensus.io/stats/view.init.0

[2021-11-10T00:09:21.408Z] 	/nfs/cache/mod/[email protected]/stats/view/worker.go:34 +0x68

[2021-11-10T00:09:21.408Z] 

[2021-11-10T00:09:21.408Z] goroutine 149 [chan receive]:

[2021-11-10T00:09:21.408Z] go.etcd.io/etcd/pkg/logutil.(*MergeLogger).outputLoop(0xc000521aa0)

[2021-11-10T00:09:21.408Z] 	/nfs/cache/mod/go.etcd.io/[email protected]/pkg/logutil/merge_logger.go:173 +0x3ac

[2021-11-10T00:09:21.408Z] created by go.etcd.io/etcd/pkg/logutil.NewMergeLogger

[2021-11-10T00:09:21.408Z] 	/nfs/cache/mod/go.etcd.io/[email protected]/pkg/logutil/merge_logger.go:91 +0x85

[2021-11-10T00:09:21.408Z] 

[2021-11-10T00:09:21.408Z] goroutine 171 [chan receive]:

[2021-11-10T00:09:21.408Z] go.etcd.io/etcd/pkg/logutil.(*MergeLogger).outputLoop(0xc0005a3590)

[2021-11-10T00:09:21.408Z] 	/nfs/cache/mod/go.etcd.io/[email protected]/pkg/logutil/merge_logger.go:173 +0x3ac

[2021-11-10T00:09:21.408Z] created by go.etcd.io/etcd/pkg/logutil.NewMergeLogger

[2021-11-10T00:09:21.408Z] 	/nfs/cache/mod/go.etcd.io/[email protected]/pkg/logutil/merge_logger.go:91 +0x85

[2021-11-10T00:09:21.408Z] 

[2021-11-10T00:09:21.408Z] goroutine 49 [chan receive]:

[2021-11-10T00:09:21.408Z] go.etcd.io/etcd/pkg/logutil.(*MergeLogger).outputLoop(0xc00000f398)

[2021-11-10T00:09:21.408Z] 	/nfs/cache/mod/go.etcd.io/[email protected]/pkg/logutil/merge_logger.go:173 +0x3ac

[2021-11-10T00:09:21.408Z] created by go.etcd.io/etcd/pkg/logutil.NewMergeLogger

[2021-11-10T00:09:21.409Z] 	/nfs/cache/mod/go.etcd.io/[email protected]/pkg/logutil/merge_logger.go:91 +0x85

[2021-11-10T00:09:21.409Z] 

[2021-11-10T00:09:21.409Z] goroutine 1110 [chan receive, 8 minutes]:

[2021-11-10T00:09:21.409Z] github.com/pingcap/tidb/table/tables_test.TestCacheTableComplexRead(0xc00592c300)

[2021-11-10T00:09:21.409Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/table/tables/cache_test.go:235 +0x3a9

[2021-11-10T00:09:21.409Z] testing.tRunner(0xc00592c300, 0x3d80c08)

[2021-11-10T00:09:21.409Z] 	/usr/local/go/src/testing/testing.go:1193 +0xef

[2021-11-10T00:09:21.409Z] created by testing.(*T).Run

[2021-11-10T00:09:21.409Z] 	/usr/local/go/src/testing/testing.go:1238 +0x2b3

[2021-11-10T00:09:21.409Z] 

[2021-11-10T00:09:21.409Z] goroutine 23311 [chan send, 8 minutes]:

[2021-11-10T00:09:21.409Z] testing.tRunner.func1.1(0xc073a67ed5, 0x0, 0x0, 0xc004d72780, 0xc073a67e01)

[2021-11-10T00:09:21.409Z] 	/usr/local/go/src/testing/testing.go:1123 +0x59

[2021-11-10T00:09:21.409Z] testing.tRunner.func1(0xc004d72780)

[2021-11-10T00:09:21.409Z] 	/usr/local/go/src/testing/testing.go:1184 +0x405

[2021-11-10T00:09:21.409Z] testing.tRunner(0xc004d72780, 0x3d80ce0)

[2021-11-10T00:09:21.409Z] 	/usr/local/go/src/testing/testing.go:1197 +0x125

[2021-11-10T00:09:21.409Z] created by testing.(*T).Run

[2021-11-10T00:09:21.409Z] 	/usr/local/go/src/testing/testing.go:1238 +0x2b3

[2021-11-10T00:09:21.409Z] 

[2021-11-10T00:09:21.409Z] goroutine 24014 [select, 4 minutes]:

[2021-11-10T00:09:21.409Z] github.com/pingcap/tidb/store/mockstore/unistore/tikv.(*MVCCStore).StartDeadlockDetection.func1(0xc000ef6380)

[2021-11-10T00:09:21.409Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/store/mockstore/unistore/tikv/mvcc.go:1286 +0x8a

[2021-11-10T00:09:21.409Z] created by github.com/pingcap/tidb/store/mockstore/unistore/tikv.(*MVCCStore).StartDeadlockDetection

[2021-11-10T00:09:21.409Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/store/mockstore/unistore/tikv/mvcc.go:1284 +0x78

[2021-11-10T00:09:21.409Z] 

[2021-11-10T00:09:21.409Z] goroutine 723 [chan receive, 8 minutes]:

[2021-11-10T00:09:21.409Z] github.com/pingcap/tidb/table/tables.mockRemoteService(0xc0052776f8, 0xc0064f4ae0)

[2021-11-10T00:09:21.409Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/table/tables/state_remote.go:109 +0x7a

[2021-11-10T00:09:21.409Z] created by github.com/pingcap/tidb/table/tables.NewCachedTable

[2021-11-10T00:09:21.409Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/table/tables/cache.go:81 +0x1dd

[2021-11-10T00:09:21.409Z] 

[2021-11-10T00:09:21.409Z] goroutine 23377 [chan send, 8 minutes]:

[2021-11-10T00:09:21.409Z] testing.tRunner.func1.1(0xc07a549ed5, 0x0, 0x0, 0xc00074c900, 0xc07a549e01)

[2021-11-10T00:09:21.409Z] 	/usr/local/go/src/testing/testing.go:1123 +0x59

[2021-11-10T00:09:21.409Z] testing.tRunner.func1(0xc00074c900)

[2021-11-10T00:09:21.409Z] 	/usr/local/go/src/testing/testing.go:1184 +0x405

[2021-11-10T00:09:21.409Z] testing.tRunner(0xc00074c900, 0x3d80c70)

[2021-11-10T00:09:21.409Z] 	/usr/local/go/src/testing/testing.go:1197 +0x125

[2021-11-10T00:09:21.409Z] created by testing.(*T).Run

[2021-11-10T00:09:21.409Z] 	/usr/local/go/src/testing/testing.go:1238 +0x2b3

[2021-11-10T00:09:21.409Z] 

[2021-11-10T00:09:21.409Z] goroutine 23384 [chan send, 8 minutes]:

[2021-11-10T00:09:21.409Z] testing.tRunner.func1.1(0xc005b33ed5, 0x0, 0x0, 0xc00074d380, 0xc005b33e01)

[2021-11-10T00:09:21.409Z] 	/usr/local/go/src/testing/testing.go:1123 +0x59

[2021-11-10T00:09:21.409Z] testing.tRunner.func1(0xc00074d380)

[2021-11-10T00:09:21.409Z] 	/usr/local/go/src/testing/testing.go:1184 +0x405

[2021-11-10T00:09:21.409Z] testing.tRunner(0xc00074d380, 0x3d80ce8)

[2021-11-10T00:09:21.409Z] 	/usr/local/go/src/testing/testing.go:1197 +0x125

[2021-11-10T00:09:21.409Z] created by testing.(*T).Run

[2021-11-10T00:09:21.409Z] 	/usr/local/go/src/testing/testing.go:1238 +0x2b3

[2021-11-10T00:09:21.409Z] 

[2021-11-10T00:09:21.409Z] goroutine 23379 [chan send, 8 minutes]:

[2021-11-10T00:09:21.409Z] testing.tRunner.func1.1(0xc00206ded5, 0x0, 0x0, 0xc00074cc00, 0xc00206de01)

[2021-11-10T00:09:21.409Z] 	/usr/local/go/src/testing/testing.go:1123 +0x59

[2021-11-10T00:09:21.409Z] testing.tRunner.func1(0xc00074cc00)

[2021-11-10T00:09:21.409Z] 	/usr/local/go/src/testing/testing.go:1184 +0x405

[2021-11-10T00:09:21.409Z] testing.tRunner(0xc00074cc00, 0x3d80cf8)

[2021-11-10T00:09:21.409Z] 	/usr/local/go/src/testing/testing.go:1197 +0x125

[2021-11-10T00:09:21.409Z] created by testing.(*T).Run

[2021-11-10T00:09:21.409Z] 	/usr/local/go/src/testing/testing.go:1238 +0x2b3

[2021-11-10T00:09:21.409Z] 

[2021-11-10T00:09:21.409Z] goroutine 24179 [select, 8 minutes]:

[2021-11-10T00:09:21.409Z] github.com/dgraph-io/ristretto.(*defaultPolicy).processItems(0xc003460ae0)

[2021-11-10T00:09:21.409Z] 	/nfs/cache/mod/github.com/dgraph-io/[email protected]/policy.go:96 +0xa5

[2021-11-10T00:09:21.409Z] created by github.com/dgraph-io/ristretto.newDefaultPolicy

[2021-11-10T00:09:21.409Z] 	/nfs/cache/mod/github.com/dgraph-io/[email protected]/policy.go:80 +0x129

[2021-11-10T00:09:21.409Z] 

[2021-11-10T00:09:21.409Z] goroutine 23370 [chan send, 8 minutes]:

[2021-11-10T00:09:21.409Z] testing.tRunner.func1.1(0xc078911ed5, 0x0, 0x0, 0xc004d73c80, 0xc078911e01)

[2021-11-10T00:09:21.409Z] 	/usr/local/go/src/testing/testing.go:1123 +0x59

[2021-11-10T00:09:21.409Z] testing.tRunner.func1(0xc004d73c80)

[2021-11-10T00:09:21.409Z] 	/usr/local/go/src/testing/testing.go:1184 +0x405

[2021-11-10T00:09:21.409Z] testing.tRunner(0xc004d73c80, 0x3d80cf0)

[2021-11-10T00:09:21.409Z] 	/usr/local/go/src/testing/testing.go:1197 +0x125

[2021-11-10T00:09:21.409Z] created by testing.(*T).Run

[2021-11-10T00:09:21.409Z] 	/usr/local/go/src/testing/testing.go:1238 +0x2b3

[2021-11-10T00:09:21.409Z] 

[2021-11-10T00:09:21.409Z] goroutine 23312 [chan send, 8 minutes]:

[2021-11-10T00:09:21.409Z] testing.tRunner.func1.1(0xc00551bed5, 0x0, 0x0, 0xc004d72900, 0xc00551be01)

[2021-11-10T00:09:21.409Z] 	/usr/local/go/src/testing/testing.go:1123 +0x59

[2021-11-10T00:09:21.409Z] testing.tRunner.func1(0xc004d72900)

[2021-11-10T00:09:21.409Z] 	/usr/local/go/src/testing/testing.go:1184 +0x405

[2021-11-10T00:09:21.409Z] testing.tRunner(0xc004d72900, 0x3d80cb8)

[2021-11-10T00:09:21.409Z] 	/usr/local/go/src/testing/testing.go:1197 +0x125

[2021-11-10T00:09:21.409Z] created by testing.(*T).Run

[2021-11-10T00:09:21.409Z] 	/usr/local/go/src/testing/testing.go:1238 +0x2b3

[2021-11-10T00:09:21.409Z] 

[2021-11-10T00:09:21.409Z] goroutine 23313 [chan send, 8 minutes]:

[2021-11-10T00:09:21.409Z] testing.tRunner.func1.1(0xc005bb3ed5, 0x0, 0x0, 0xc004d72a80, 0xc005bb3e01)

[2021-11-10T00:09:21.409Z] 	/usr/local/go/src/testing/testing.go:1123 +0x59

[2021-11-10T00:09:21.409Z] testing.tRunner.func1(0xc004d72a80)

[2021-11-10T00:09:21.409Z] 	/usr/local/go/src/testing/testing.go:1184 +0x405

[2021-11-10T00:09:21.409Z] testing.tRunner(0xc004d72a80, 0x3d80cb0)

[2021-11-10T00:09:21.409Z] 	/usr/local/go/src/testing/testing.go:1197 +0x125

[2021-11-10T00:09:21.409Z] created by testing.(*T).Run

[2021-11-10T00:09:21.409Z] 	/usr/local/go/src/testing/testing.go:1238 +0x2b3

[2021-11-10T00:09:21.409Z] 

[2021-11-10T00:09:21.409Z] goroutine 23365 [chan send, 8 minutes]:

[2021-11-10T00:09:21.409Z] testing.tRunner.func1.1(0xc0746abed5, 0x0, 0x0, 0xc004d73200, 0xc0746abe01)

[2021-11-10T00:09:21.409Z] 	/usr/local/go/src/testing/testing.go:1123 +0x59

[2021-11-10T00:09:21.409Z] testing.tRunner.func1(0xc004d73200)

[2021-11-10T00:09:21.409Z] 	/usr/local/go/src/testing/testing.go:1184 +0x405

[2021-11-10T00:09:21.409Z] testing.tRunner(0xc004d73200, 0x3d80c90)

[2021-11-10T00:09:21.409Z] 	/usr/local/go/src/testing/testing.go:1197 +0x125

[2021-11-10T00:09:21.409Z] created by testing.(*T).Run

[2021-11-10T00:09:21.409Z] 	/usr/local/go/src/testing/testing.go:1238 +0x2b3

[2021-11-10T00:09:21.409Z] 

[2021-11-10T00:09:21.409Z] goroutine 31595 [select]:

[2021-11-10T00:09:21.409Z] github.com/pingcap/tidb/domain.(*Domain).handleEvolvePlanTasksLoop.func1(0xc07ee167e0, 0x41c16d8, 0xc07fb6eda0, 0x41ed530, 0xc0760efc00)

[2021-11-10T00:09:21.409Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/domain/domain.go:1106 +0xde

[2021-11-10T00:09:21.409Z] created by github.com/pingcap/tidb/domain.(*Domain).handleEvolvePlanTasksLoop

[2021-11-10T00:09:21.409Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/domain/domain.go:1099 +0x8f

[2021-11-10T00:09:21.409Z] 

[2021-11-10T00:09:21.409Z] goroutine 23369 [chan send, 8 minutes]:

[2021-11-10T00:09:21.409Z] testing.tRunner.func1.1(0xc075ab5ed5, 0x0, 0x0, 0xc004d73b00, 0xc075ab5e01)

[2021-11-10T00:09:21.409Z] 	/usr/local/go/src/testing/testing.go:1123 +0x59

[2021-11-10T00:09:21.409Z] testing.tRunner.func1(0xc004d73b00)

[2021-11-10T00:09:21.409Z] 	/usr/local/go/src/testing/testing.go:1184 +0x405

[2021-11-10T00:09:21.409Z] testing.tRunner(0xc004d73b00, 0x3d80c80)

[2021-11-10T00:09:21.410Z] 	/usr/local/go/src/testing/testing.go:1197 +0x125

[2021-11-10T00:09:21.410Z] created by testing.(*T).Run

[2021-11-10T00:09:21.410Z] 	/usr/local/go/src/testing/testing.go:1238 +0x2b3

[2021-11-10T00:09:21.410Z] 

[2021-11-10T00:09:21.410Z] goroutine 23599 [select]:

[2021-11-10T00:09:21.410Z] github.com/pingcap/badger.(*levelsController).runWorker(0xc001ec62a0, 0xc002982138, 0x0)

[2021-11-10T00:09:21.410Z] 	/nfs/cache/mod/github.com/pingcap/[email protected]/levels.go:212 +0x185

[2021-11-10T00:09:21.410Z] created by github.com/pingcap/badger.(*levelsController).startCompact

[2021-11-10T00:09:21.410Z] 	/nfs/cache/mod/github.com/pingcap/[email protected]/levels.go:180 +0x93

[2021-11-10T00:09:21.410Z] 

[2021-11-10T00:09:21.410Z] goroutine 23371 [chan send, 8 minutes]:

[2021-11-10T00:09:21.410Z] testing.tRunner.func1.1(0xc00284bed5, 0x0, 0x0, 0xc004d73e00, 0xc00284be01)

[2021-11-10T00:09:21.410Z] 	/usr/local/go/src/testing/testing.go:1123 +0x59

[2021-11-10T00:09:21.410Z] testing.tRunner.func1(0xc004d73e00)

[2021-11-10T00:09:21.410Z] 	/usr/local/go/src/testing/testing.go:1184 +0x405

[2021-11-10T00:09:21.410Z] testing.tRunner(0xc004d73e00, 0x3d80c28)

[2021-11-10T00:09:21.410Z] 	/usr/local/go/src/testing/testing.go:1197 +0x125

[2021-11-10T00:09:21.410Z] created by testing.(*T).Run

[2021-11-10T00:09:21.410Z] 	/usr/local/go/src/testing/testing.go:1238 +0x2b3

[2021-11-10T00:09:21.410Z] 

[2021-11-10T00:09:21.410Z] goroutine 23387 [chan send, 8 minutes]:

[2021-11-10T00:09:21.410Z] testing.tRunner.func1.1(0xc072c1ded5, 0x0, 0x0, 0xc00074d980, 0xc072c1de01)

[2021-11-10T00:09:21.410Z] 	/usr/local/go/src/testing/testing.go:1123 +0x59

[2021-11-10T00:09:21.410Z] testing.tRunner.func1(0xc00074d980)

[2021-11-10T00:09:21.410Z] 	/usr/local/go/src/testing/testing.go:1184 +0x405

[2021-11-10T00:09:21.410Z] testing.tRunner(0xc00074d980, 0x3d80bd0)

[2021-11-10T00:09:21.410Z] 	/usr/local/go/src/testing/testing.go:1197 +0x125

[2021-11-10T00:09:21.410Z] created by testing.(*T).Run

[2021-11-10T00:09:21.410Z] 	/usr/local/go/src/testing/testing.go:1238 +0x2b3

[2021-11-10T00:09:21.410Z] 

[2021-11-10T00:09:21.410Z] goroutine 23368 [chan send, 8 minutes]:

[2021-11-10T00:09:21.410Z] testing.tRunner.func1.1(0xc006759ed5, 0x0, 0x0, 0xc004d73980, 0xc006759e01)

[2021-11-10T00:09:21.410Z] 	/usr/local/go/src/testing/testing.go:1123 +0x59

[2021-11-10T00:09:21.410Z] testing.tRunner.func1(0xc004d73980)

[2021-11-10T00:09:21.410Z] 	/usr/local/go/src/testing/testing.go:1184 +0x405

[2021-11-10T00:09:21.410Z] testing.tRunner(0xc004d73980, 0x3d80ca8)

[2021-11-10T00:09:21.410Z] 	/usr/local/go/src/testing/testing.go:1197 +0x125

[2021-11-10T00:09:21.410Z] created by testing.(*T).Run

[2021-11-10T00:09:21.410Z] 	/usr/local/go/src/testing/testing.go:1238 +0x2b3

[2021-11-10T00:09:21.410Z] 

[2021-11-10T00:09:21.410Z] goroutine 23373 [chan send, 8 minutes]:

[2021-11-10T00:09:21.410Z] testing.tRunner.func1.1(0xc0792abed5, 0x0, 0x0, 0xc00074c300, 0xc0792abe01)

[2021-11-10T00:09:21.410Z] 	/usr/local/go/src/testing/testing.go:1123 +0x59

[2021-11-10T00:09:21.410Z] testing.tRunner.func1(0xc00074c300)

[2021-11-10T00:09:21.410Z] 	/usr/local/go/src/testing/testing.go:1184 +0x405

[2021-11-10T00:09:21.410Z] testing.tRunner(0xc00074c300, 0x3d80c60)

[2021-11-10T00:09:21.410Z] 	/usr/local/go/src/testing/testing.go:1197 +0x125

[2021-11-10T00:09:21.410Z] created by testing.(*T).Run

[2021-11-10T00:09:21.410Z] 	/usr/local/go/src/testing/testing.go:1238 +0x2b3

[2021-11-10T00:09:21.410Z] 

[2021-11-10T00:09:21.410Z] goroutine 23388 [chan send, 8 minutes]:

[2021-11-10T00:09:21.410Z] testing.tRunner.func1.1(0xc07b011ed5, 0x0, 0x0, 0xc00074db00, 0xc07b011e01)

[2021-11-10T00:09:21.410Z] 	/usr/local/go/src/testing/testing.go:1123 +0x59

[2021-11-10T00:09:21.410Z] testing.tRunner.func1(0xc00074db00)

[2021-11-10T00:09:21.410Z] 	/usr/local/go/src/testing/testing.go:1184 +0x405

[2021-11-10T00:09:21.410Z] testing.tRunner(0xc00074db00, 0x3d80c20)

[2021-11-10T00:09:21.410Z] 	/usr/local/go/src/testing/testing.go:1197 +0x125

[2021-11-10T00:09:21.410Z] created by testing.(*T).Run

[2021-11-10T00:09:21.410Z] 	/usr/local/go/src/testing/testing.go:1238 +0x2b3

[2021-11-10T00:09:21.410Z] 

[2021-11-10T00:09:21.410Z] goroutine 23364 [chan send, 8 minutes]:

[2021-11-10T00:09:21.410Z] testing.tRunner.func1.1(0xc073a25ed5, 0x0, 0x0, 0xc004d72f00, 0xc073a25e01)

[2021-11-10T00:09:21.410Z] 	/usr/local/go/src/testing/testing.go:1123 +0x59

[2021-11-10T00:09:21.410Z] testing.tRunner.func1(0xc004d72f00)

[2021-11-10T00:09:21.410Z] 	/usr/local/go/src/testing/testing.go:1184 +0x405

[2021-11-10T00:09:21.410Z] testing.tRunner(0xc004d72f00, 0x3d80c30)

[2021-11-10T00:09:21.410Z] 	/usr/local/go/src/testing/testing.go:1197 +0x125

[2021-11-10T00:09:21.410Z] created by testing.(*T).Run

[2021-11-10T00:09:21.410Z] 	/usr/local/go/src/testing/testing.go:1238 +0x2b3

[2021-11-10T00:09:21.410Z] 

[2021-11-10T00:09:21.410Z] goroutine 23600 [chan receive, 8 minutes]:

[2021-11-10T00:09:21.410Z] github.com/pingcap/badger.(*DB).runFlushMemTable(0xc002c3e900, 0xc00210e168, 0x0, 0x0)

[2021-11-10T00:09:21.410Z] 	/nfs/cache/mod/github.com/pingcap/[email protected]/db.go:920 +0x190

[2021-11-10T00:09:21.410Z] created by github.com/pingcap/badger.Open

[2021-11-10T00:09:21.410Z] 	/nfs/cache/mod/github.com/pingcap/[email protected]/db.go:361 +0x17b0

[2021-11-10T00:09:21.410Z] 

[2021-11-10T00:09:21.410Z] goroutine 24011 [select, 4 minutes]:

[2021-11-10T00:09:21.410Z] github.com/pingcap/tidb/store/mockstore/unistore/tikv.writeDBWorker.run(0xc002c9f7a0, 0xc002168800)

[2021-11-10T00:09:21.410Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/store/mockstore/unistore/tikv/write.go:96 +0x205

[2021-11-10T00:09:21.410Z] created by github.com/pingcap/tidb/store/mockstore/unistore/tikv.(*dbWriter).Open

[2021-11-10T00:09:21.410Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/store/mockstore/unistore/tikv/write.go:199 +0x8f

[2021-11-10T00:09:21.410Z] 

[2021-11-10T00:09:21.410Z] goroutine 23380 [chan send, 8 minutes]:

[2021-11-10T00:09:21.410Z] testing.tRunner.func1.1(0xc0722cfed5, 0x0, 0x0, 0xc00074cd80, 0xc0722cfe01)

[2021-11-10T00:09:21.410Z] 	/usr/local/go/src/testing/testing.go:1123 +0x59

[2021-11-10T00:09:21.410Z] testing.tRunner.func1(0xc00074cd80)

[2021-11-10T00:09:21.410Z] 	/usr/local/go/src/testing/testing.go:1184 +0x405

[2021-11-10T00:09:21.410Z] testing.tRunner(0xc00074cd80, 0x3d80d00)

[2021-11-10T00:09:21.410Z] 	/usr/local/go/src/testing/testing.go:1197 +0x125

[2021-11-10T00:09:21.410Z] created by testing.(*T).Run

[2021-11-10T00:09:21.410Z] 	/usr/local/go/src/testing/testing.go:1238 +0x2b3

[2021-11-10T00:09:21.410Z] 

[2021-11-10T00:09:21.410Z] goroutine 23598 [select, 8 minutes]:

[2021-11-10T00:09:21.410Z] github.com/pingcap/badger.(*blobGCHandler).run(0xc0011a81c0, 0xc002982120)

[2021-11-10T00:09:21.410Z] 	/nfs/cache/mod/github.com/pingcap/[email protected]/blob.go:468 +0xd4

[2021-11-10T00:09:21.410Z] created by github.com/pingcap/badger.(*blobManager).Open

[2021-11-10T00:09:21.410Z] 	/nfs/cache/mod/github.com/pingcap/[email protected]/blob.go:292 +0x618

[2021-11-10T00:09:21.410Z] 

[2021-11-10T00:09:21.410Z] goroutine 23375 [chan send, 8 minutes]:

[2021-11-10T00:09:21.410Z] testing.tRunner.func1.1(0xc079371ed5, 0x0, 0x0, 0xc00074c600, 0xc079371e01)

[2021-11-10T00:09:21.410Z] 	/usr/local/go/src/testing/testing.go:1123 +0x59

[2021-11-10T00:09:21.410Z] testing.tRunner.func1(0xc00074c600)

[2021-11-10T00:09:21.410Z] 	/usr/local/go/src/testing/testing.go:1184 +0x405

[2021-11-10T00:09:21.410Z] testing.tRunner(0xc00074c600, 0x3d80c48)

[2021-11-10T00:09:21.410Z] 	/usr/local/go/src/testing/testing.go:1197 +0x125

[2021-11-10T00:09:21.410Z] created by testing.(*T).Run

[2021-11-10T00:09:21.410Z] 	/usr/local/go/src/testing/testing.go:1238 +0x2b3

[2021-11-10T00:09:21.410Z] 

[2021-11-10T00:09:21.410Z] goroutine 23363 [chan send, 8 minutes]:

[2021-11-10T00:09:21.410Z] testing.tRunner.func1.1(0xc00526fed5, 0x0, 0x0, 0xc004d72d80, 0xc00526fe01)

[2021-11-10T00:09:21.410Z] 	/usr/local/go/src/testing/testing.go:1123 +0x59

[2021-11-10T00:09:21.410Z] testing.tRunner.func1(0xc004d72d80)

[2021-11-10T00:09:21.410Z] 	/usr/local/go/src/testing/testing.go:1184 +0x405

[2021-11-10T00:09:21.410Z] testing.tRunner(0xc004d72d80, 0x3d80cc0)

[2021-11-10T00:09:21.410Z] 	/usr/local/go/src/testing/testing.go:1197 +0x125

[2021-11-10T00:09:21.410Z] created by testing.(*T).Run

[2021-11-10T00:09:21.410Z] 	/usr/local/go/src/testing/testing.go:1238 +0x2b3

[2021-11-10T00:09:21.410Z] 

[2021-11-10T00:09:21.410Z] goroutine 32762 [select, 8 minutes]:

[2021-11-10T00:09:21.410Z] github.com/pingcap/tidb/domain.(*Domain).PlanReplayerLoop.func1(0xc07ee167e0)

[2021-11-10T00:09:21.410Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/domain/domain.go:1191 +0x11b

[2021-11-10T00:09:21.410Z] created by github.com/pingcap/tidb/domain.(*Domain).PlanReplayerLoop

[2021-11-10T00:09:21.410Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/domain/domain.go:1182 +0x5f

[2021-11-10T00:09:21.410Z] 

[2021-11-10T00:09:21.410Z] goroutine 23385 [chan send, 8 minutes]:

[2021-11-10T00:09:21.410Z] testing.tRunner.func1.1(0xc001377ed5, 0x0, 0x0, 0xc00074d680, 0xc001377e01)

[2021-11-10T00:09:21.410Z] 	/usr/local/go/src/testing/testing.go:1123 +0x59

[2021-11-10T00:09:21.410Z] testing.tRunner.func1(0xc00074d680)

[2021-11-10T00:09:21.410Z] 	/usr/local/go/src/testing/testing.go:1184 +0x405

[2021-11-10T00:09:21.410Z] testing.tRunner(0xc00074d680, 0x3d80cd8)

[2021-11-10T00:09:21.410Z] 	/usr/local/go/src/testing/testing.go:1197 +0x125

[2021-11-10T00:09:21.410Z] created by testing.(*T).Run

[2021-11-10T00:09:21.411Z] 	/usr/local/go/src/testing/testing.go:1238 +0x2b3

[2021-11-10T00:09:21.411Z] 

[2021-11-10T00:09:21.411Z] goroutine 24017 [select]:

[2021-11-10T00:09:21.411Z] github.com/tikv/client-go/v2/tikv.(*KVStore).runSafePointChecker(0xc0025cc480)

[2021-11-10T00:09:21.411Z] 	/nfs/cache/mod/github.com/tikv/client-go/[email protected]/tikv/kv.go:264 +0x136

[2021-11-10T00:09:21.411Z] created by github.com/tikv/client-go/v2/tikv.NewKVStore

[2021-11-10T00:09:21.411Z] 	/nfs/cache/mod/github.com/tikv/client-go/[email protected]/tikv/kv.go:187 +0x38a

[2021-11-10T00:09:21.411Z] 

[2021-11-10T00:09:21.411Z] goroutine 23381 [chan send, 8 minutes]:

[2021-11-10T00:09:21.411Z] testing.tRunner.func1.1(0xc0068d7ed5, 0x0, 0x0, 0xc00074cf00, 0xc0068d7e01)

[2021-11-10T00:09:21.411Z] 	/usr/local/go/src/testing/testing.go:1123 +0x59

[2021-11-10T00:09:21.411Z] testing.tRunner.func1(0xc00074cf00)

[2021-11-10T00:09:21.411Z] 	/usr/local/go/src/testing/testing.go:1184 +0x405

[2021-11-10T00:09:21.411Z] testing.tRunner(0xc00074cf00, 0x3d80cd0)

[2021-11-10T00:09:21.411Z] 	/usr/local/go/src/testing/testing.go:1197 +0x125

[2021-11-10T00:09:21.411Z] created by testing.(*T).Run

[2021-11-10T00:09:21.411Z] 	/usr/local/go/src/testing/testing.go:1238 +0x2b3

[2021-11-10T00:09:21.411Z] 

[2021-11-10T00:09:21.411Z] goroutine 23378 [chan send, 8 minutes]:

[2021-11-10T00:09:21.411Z] testing.tRunner.func1.1(0xc078f45ed5, 0x0, 0x0, 0xc00074ca80, 0xc078f45e01)

[2021-11-10T00:09:21.411Z] 	/usr/local/go/src/testing/testing.go:1123 +0x59

[2021-11-10T00:09:21.411Z] testing.tRunner.func1(0xc00074ca80)

[2021-11-10T00:09:21.411Z] 	/usr/local/go/src/testing/testing.go:1184 +0x405

[2021-11-10T00:09:21.411Z] testing.tRunner(0xc00074ca80, 0x3d80be0)

[2021-11-10T00:09:21.411Z] 	/usr/local/go/src/testing/testing.go:1197 +0x125

[2021-11-10T00:09:21.411Z] created by testing.(*T).Run

[2021-11-10T00:09:21.411Z] 	/usr/local/go/src/testing/testing.go:1238 +0x2b3

[2021-11-10T00:09:21.411Z] 

[2021-11-10T00:09:21.411Z] goroutine 23382 [chan send, 8 minutes]:

[2021-11-10T00:09:21.411Z] testing.tRunner.func1.1(0xc0776c1ed5, 0x0, 0x0, 0xc00074d080, 0xc0776c1e01)

[2021-11-10T00:09:21.411Z] 	/usr/local/go/src/testing/testing.go:1123 +0x59

[2021-11-10T00:09:21.411Z] testing.tRunner.func1(0xc00074d080)

[2021-11-10T00:09:21.411Z] 	/usr/local/go/src/testing/testing.go:1184 +0x405

[2021-11-10T00:09:21.411Z] testing.tRunner(0xc00074d080, 0x3d80d08)

[2021-11-10T00:09:21.411Z] 	/usr/local/go/src/testing/testing.go:1197 +0x125

[2021-11-10T00:09:21.411Z] created by testing.(*T).Run

[2021-11-10T00:09:21.411Z] 	/usr/local/go/src/testing/testing.go:1238 +0x2b3

[2021-11-10T00:09:21.411Z] 

[2021-11-10T00:09:21.411Z] goroutine 23376 [chan send, 8 minutes]:

[2021-11-10T00:09:21.411Z] testing.tRunner.func1.1(0xc005f77ed5, 0x0, 0x0, 0xc00074c780, 0xc005f77e01)

[2021-11-10T00:09:21.411Z] 	/usr/local/go/src/testing/testing.go:1123 +0x59

[2021-11-10T00:09:21.411Z] testing.tRunner.func1(0xc00074c780)

[2021-11-10T00:09:21.411Z] 	/usr/local/go/src/testing/testing.go:1184 +0x405

[2021-11-10T00:09:21.411Z] testing.tRunner(0xc00074c780, 0x3d80c68)

[2021-11-10T00:09:21.411Z] 	/usr/local/go/src/testing/testing.go:1197 +0x125

[2021-11-10T00:09:21.411Z] created by testing.(*T).Run

[2021-11-10T00:09:21.411Z] 	/usr/local/go/src/testing/testing.go:1238 +0x2b3

[2021-11-10T00:09:21.411Z] 

[2021-11-10T00:09:21.411Z] goroutine 23366 [chan send, 8 minutes]:

[2021-11-10T00:09:21.411Z] testing.tRunner.func1.1(0xc075e53ed5, 0x0, 0x0, 0xc004d73380, 0xc075e53e01)

[2021-11-10T00:09:21.411Z] 	/usr/local/go/src/testing/testing.go:1123 +0x59

[2021-11-10T00:09:21.411Z] testing.tRunner.func1(0xc004d73380)

[2021-11-10T00:09:21.411Z] 	/usr/local/go/src/testing/testing.go:1184 +0x405

[2021-11-10T00:09:21.411Z] testing.tRunner(0xc004d73380, 0x3d80c98)

[2021-11-10T00:09:21.411Z] 	/usr/local/go/src/testing/testing.go:1197 +0x125

[2021-11-10T00:09:21.411Z] created by testing.(*T).Run

[2021-11-10T00:09:21.411Z] 	/usr/local/go/src/testing/testing.go:1238 +0x2b3

[2021-11-10T00:09:21.411Z] 

[2021-11-10T00:09:21.411Z] goroutine 24015 [select]:

[2021-11-10T00:09:21.411Z] github.com/tikv/client-go/v2/oracle/oracles.(*pdOracle).updateTS(0xc0009480e0, 0x41a48a0, 0xc000134018, 0x77359400)

[2021-11-10T00:09:21.411Z] 	/nfs/cache/mod/github.com/tikv/client-go/[email protected]/oracle/oracles/pd.go:229 +0x131

[2021-11-10T00:09:21.411Z] created by github.com/tikv/client-go/v2/oracle/oracles.NewPdOracle

[2021-11-10T00:09:21.411Z] 	/nfs/cache/mod/github.com/tikv/client-go/[email protected]/oracle/oracles/pd.go:77 +0xd3

[2021-11-10T00:09:21.411Z] 

[2021-11-10T00:09:21.411Z] goroutine 23386 [chan send, 8 minutes]:

[2021-11-10T00:09:21.411Z] testing.tRunner.func1.1(0xc075819ed5, 0x0, 0x0, 0xc00074d800, 0xc075819e01)

[2021-11-10T00:09:21.411Z] 	/usr/local/go/src/testing/testing.go:1123 +0x59

[2021-11-10T00:09:21.411Z] testing.tRunner.func1(0xc00074d800)

[2021-11-10T00:09:21.411Z] 	/usr/local/go/src/testing/testing.go:1184 +0x405

[2021-11-10T00:09:21.411Z] testing.tRunner(0xc00074d800, 0x3d80c50)

[2021-11-10T00:09:21.411Z] 	/usr/local/go/src/testing/testing.go:1197 +0x125

[2021-11-10T00:09:21.411Z] created by testing.(*T).Run

[2021-11-10T00:09:21.411Z] 	/usr/local/go/src/testing/testing.go:1238 +0x2b3

[2021-11-10T00:09:21.411Z] 

[2021-11-10T00:09:21.411Z] goroutine 23389 [chan send, 8 minutes]:

[2021-11-10T00:09:21.411Z] testing.tRunner.func1.1(0xc09421fed5, 0x0, 0x0, 0xc00074dc80, 0xc09421fe01)

[2021-11-10T00:09:21.411Z] 	/usr/local/go/src/testing/testing.go:1123 +0x59

[2021-11-10T00:09:21.411Z] testing.tRunner.func1(0xc00074dc80)

[2021-11-10T00:09:21.411Z] 	/usr/local/go/src/testing/testing.go:1184 +0x405

[2021-11-10T00:09:21.411Z] testing.tRunner(0xc00074dc80, 0x3d80d10)

[2021-11-10T00:09:21.411Z] 	/usr/local/go/src/testing/testing.go:1197 +0x125

[2021-11-10T00:09:21.411Z] created by testing.(*T).Run

[2021-11-10T00:09:21.411Z] 	/usr/local/go/src/testing/testing.go:1238 +0x2b3

[2021-11-10T00:09:21.411Z] 

[2021-11-10T00:09:21.411Z] goroutine 23362 [chan send, 8 minutes]:

[2021-11-10T00:09:21.411Z] testing.tRunner.func1.1(0xc0774abed5, 0x0, 0x0, 0xc004d72c00, 0xc0774abe01)

[2021-11-10T00:09:21.411Z] 	/usr/local/go/src/testing/testing.go:1123 +0x59

[2021-11-10T00:09:21.411Z] testing.tRunner.func1(0xc004d72c00)

[2021-11-10T00:09:21.411Z] 	/usr/local/go/src/testing/testing.go:1184 +0x405

[2021-11-10T00:09:21.411Z] testing.tRunner(0xc004d72c00, 0x3d80c38)

[2021-11-10T00:09:21.411Z] 	/usr/local/go/src/testing/testing.go:1197 +0x125

[2021-11-10T00:09:21.411Z] created by testing.(*T).Run

[2021-11-10T00:09:21.411Z] 	/usr/local/go/src/testing/testing.go:1238 +0x2b3

[2021-11-10T00:09:21.411Z] 

[2021-11-10T00:09:21.411Z] goroutine 31163 [select, 8 minutes]:

[2021-11-10T00:09:21.411Z] github.com/pingcap/tidb/ddl.(*delRange).startEmulator(0xc07c9f31d0)

[2021-11-10T00:09:21.411Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/ddl/delete_range.go:144 +0x128

[2021-11-10T00:09:21.411Z] created by github.com/pingcap/tidb/ddl.(*delRange).start

[2021-11-10T00:09:21.411Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/ddl/delete_range.go:126 +0x6a

[2021-11-10T00:09:21.411Z] 

[2021-11-10T00:09:21.411Z] goroutine 24178 [select]:

[2021-11-10T00:09:21.411Z] github.com/tikv/client-go/v2/tikv.(*KVStore).safeTSUpdater(0xc0025cc480)

[2021-11-10T00:09:21.411Z] 	/nfs/cache/mod/github.com/tikv/client-go/[email protected]/tikv/kv.go:544 +0x194

[2021-11-10T00:09:21.411Z] created by github.com/tikv/client-go/v2/tikv.NewKVStore

[2021-11-10T00:09:21.411Z] 	/nfs/cache/mod/github.com/tikv/client-go/[email protected]/tikv/kv.go:188 +0x3ac

[2021-11-10T00:09:21.411Z] 

[2021-11-10T00:09:21.411Z] goroutine 23374 [chan send, 8 minutes]:

[2021-11-10T00:09:21.411Z] testing.tRunner.func1.1(0xc004e23ed5, 0x0, 0x0, 0xc00074c480, 0xc004e23e01)

[2021-11-10T00:09:21.411Z] 	/usr/local/go/src/testing/testing.go:1123 +0x59

[2021-11-10T00:09:21.411Z] testing.tRunner.func1(0xc00074c480)

[2021-11-10T00:09:21.411Z] 	/usr/local/go/src/testing/testing.go:1184 +0x405

[2021-11-10T00:09:21.411Z] testing.tRunner(0xc00074c480, 0x3d80c40)

[2021-11-10T00:09:21.411Z] 	/usr/local/go/src/testing/testing.go:1197 +0x125

[2021-11-10T00:09:21.411Z] created by testing.(*T).Run

[2021-11-10T00:09:21.411Z] 	/usr/local/go/src/testing/testing.go:1238 +0x2b3

[2021-11-10T00:09:21.411Z] 

[2021-11-10T00:09:21.411Z] goroutine 23383 [chan send, 8 minutes]:

[2021-11-10T00:09:21.411Z] testing.tRunner.func1.1(0xc0771a3ed5, 0x0, 0x0, 0xc00074d200, 0xc0771a3e01)

[2021-11-10T00:09:21.411Z] 	/usr/local/go/src/testing/testing.go:1123 +0x59

[2021-11-10T00:09:21.411Z] testing.tRunner.func1(0xc00074d200)

[2021-11-10T00:09:21.411Z] 	/usr/local/go/src/testing/testing.go:1184 +0x405

[2021-11-10T00:09:21.411Z] testing.tRunner(0xc00074d200, 0x3d80c78)

[2021-11-10T00:09:21.411Z] 	/usr/local/go/src/testing/testing.go:1197 +0x125

[2021-11-10T00:09:21.411Z] created by testing.(*T).Run

[2021-11-10T00:09:21.411Z] 	/usr/local/go/src/testing/testing.go:1238 +0x2b3

[2021-11-10T00:09:21.411Z] 

[2021-11-10T00:09:21.411Z] goroutine 24012 [select, 4 minutes]:

[2021-11-10T00:09:21.411Z] github.com/pingcap/tidb/store/mockstore/unistore/tikv.writeLockWorker.run(0xc002c9f800, 0xc002168800)

[2021-11-10T00:09:21.412Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/store/mockstore/unistore/tikv/write.go:148 +0x3e5

[2021-11-10T00:09:21.412Z] created by github.com/pingcap/tidb/store/mockstore/unistore/tikv.(*dbWriter).Open

[2021-11-10T00:09:21.412Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/store/mockstore/unistore/tikv/write.go:206 +0xdc

[2021-11-10T00:09:21.412Z] 

[2021-11-10T00:09:21.412Z] goroutine 23367 [chan send, 8 minutes]:

[2021-11-10T00:09:21.412Z] testing.tRunner.func1.1(0xc0721f5ed5, 0x0, 0x0, 0xc004d73680, 0xc0721f5e01)

[2021-11-10T00:09:21.412Z] 	/usr/local/go/src/testing/testing.go:1123 +0x59

[2021-11-10T00:09:21.412Z] testing.tRunner.func1(0xc004d73680)

[2021-11-10T00:09:21.412Z] 	/usr/local/go/src/testing/testing.go:1184 +0x405

[2021-11-10T00:09:21.412Z] testing.tRunner(0xc004d73680, 0x3d80c88)

[2021-11-10T00:09:21.412Z] 	/usr/local/go/src/testing/testing.go:1197 +0x125

[2021-11-10T00:09:21.412Z] created by testing.(*T).Run

[2021-11-10T00:09:21.412Z] 	/usr/local/go/src/testing/testing.go:1238 +0x2b3

[2021-11-10T00:09:21.412Z] 

[2021-11-10T00:09:21.412Z] goroutine 23390 [chan receive, 8 minutes]:

[2021-11-10T00:09:21.412Z] testing.runTests.func1.1(0xc00058a600)

[2021-11-10T00:09:21.412Z] 	/usr/local/go/src/testing/testing.go:1516 +0x3b

[2021-11-10T00:09:21.412Z] created by testing.runTests.func1

[2021-11-10T00:09:21.412Z] 	/usr/local/go/src/testing/testing.go:1516 +0xac

[2021-11-10T00:09:21.412Z] 

[2021-11-10T00:09:21.412Z] goroutine 24180 [select, 8 minutes]:

[2021-11-10T00:09:21.412Z] github.com/dgraph-io/ristretto.(*Cache).processItems(0xc002c9fbc0)

[2021-11-10T00:09:21.412Z] 	/nfs/cache/mod/github.com/dgraph-io/[email protected]/cache.go:257 +0x9d

[2021-11-10T00:09:21.412Z] created by github.com/dgraph-io/ristretto.NewCache

[2021-11-10T00:09:21.412Z] 	/nfs/cache/mod/github.com/dgraph-io/[email protected]/cache.go:155 +0x205

[2021-11-10T00:09:21.412Z] 

[2021-11-10T00:09:21.412Z] goroutine 24013 [select, 2 minutes]:

[2021-11-10T00:09:21.412Z] github.com/pingcap/tidb/store/mockstore/unistore/tikv.(*MVCCStore).runUpdateSafePointLoop(0xc000ef6380)

[2021-11-10T00:09:21.412Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/store/mockstore/unistore/tikv/mvcc.go:1550 +0x28a

[2021-11-10T00:09:21.412Z] created by github.com/pingcap/tidb/store/mockstore/unistore/tikv.NewMVCCStore

[2021-11-10T00:09:21.412Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/store/mockstore/unistore/tikv/mvcc.go:85 +0x269

[2021-11-10T00:09:21.412Z] 

[2021-11-10T00:09:21.412Z] goroutine 23372 [chan send, 8 minutes]:

[2021-11-10T00:09:21.412Z] testing.tRunner.func1.1(0xc071d07ed5, 0x0, 0x0, 0xc00074c180, 0xc071d07e01)

[2021-11-10T00:09:21.412Z] 	/usr/local/go/src/testing/testing.go:1123 +0x59

[2021-11-10T00:09:21.412Z] testing.tRunner.func1(0xc00074c180)

[2021-11-10T00:09:21.412Z] 	/usr/local/go/src/testing/testing.go:1184 +0x405

[2021-11-10T00:09:21.412Z] testing.tRunner(0xc00074c180, 0x3d80cc8)

[2021-11-10T00:09:21.412Z] 	/usr/local/go/src/testing/testing.go:1197 +0x125

[2021-11-10T00:09:21.412Z] created by testing.(*T).Run

[2021-11-10T00:09:21.412Z] 	/usr/local/go/src/testing/testing.go:1238 +0x2b3

[2021-11-10T00:09:21.412Z] 

[2021-11-10T00:09:21.412Z] goroutine 24016 [select, 2 minutes]:

[2021-11-10T00:09:21.412Z] github.com/tikv/client-go/v2/internal/locate.(*RegionCache).asyncCheckAndResolveLoop(0xc000ef6480, 0xdf8475800)

[2021-11-10T00:09:21.412Z] 	/nfs/cache/mod/github.com/tikv/client-go/[email protected]/internal/locate/region_cache.go:399 +0xdc

[2021-11-10T00:09:21.412Z] created by github.com/tikv/client-go/v2/internal/locate.NewRegionCache

[2021-11-10T00:09:21.412Z] 	/nfs/cache/mod/github.com/tikv/client-go/[email protected]/internal/locate/region_cache.go:370 +0x259

[2021-11-10T00:09:21.412Z] 

[2021-11-10T00:09:21.412Z] goroutine 23418 [select, 2 minutes]:

[2021-11-10T00:09:21.412Z] github.com/pingcap/badger.(*DB).updateSize(0xc002c3e900, 0xc0019e0198)

[2021-11-10T00:09:21.412Z] 	/nfs/cache/mod/github.com/pingcap/[email protected]/db.go:1039 +0x108

[2021-11-10T00:09:21.412Z] created by github.com/pingcap/badger.Open

[2021-11-10T00:09:21.412Z] 	/nfs/cache/mod/github.com/pingcap/[email protected]/db.go:328 +0xe14

[2021-11-10T00:09:21.412Z] 

[2021-11-10T00:09:21.412Z] goroutine 23419 [select]:

[2021-11-10T00:09:21.412Z] github.com/pingcap/badger/epoch.(*ResourceManager).collectLoop(0xc001271740, 0xc0019e01b0)

[2021-11-10T00:09:21.412Z] 	/nfs/cache/mod/github.com/pingcap/[email protected]/epoch/manager.go:101 +0xdc

[2021-11-10T00:09:21.412Z] created by github.com/pingcap/badger/epoch.NewResourceManager

[2021-11-10T00:09:21.412Z] 	/nfs/cache/mod/github.com/pingcap/[email protected]/epoch/manager.go:79 +0xa7

[2021-11-10T00:09:21.412Z] 

[2021-11-10T00:09:21.412Z] goroutine 23560 [select, 8 minutes]:

[2021-11-10T00:09:21.412Z] github.com/pingcap/badger.Open.func4(0xc005cb0000)

[2021-11-10T00:09:21.412Z] 	/nfs/cache/mod/github.com/pingcap/[email protected]/db.go:342 +0x17c

[2021-11-10T00:09:21.412Z] created by github.com/pingcap/badger.Open

[2021-11-10T00:09:21.412Z] 	/nfs/cache/mod/github.com/pingcap/[email protected]/db.go:339 +0x10a8

[2021-11-10T00:09:21.412Z] 

[2021-11-10T00:09:21.412Z] goroutine 24010 [chan receive, 4 minutes]:

[2021-11-10T00:09:21.412Z] github.com/pingcap/badger.(*writeWorker).runMergeLSM(0xc002c96460, 0xc001d3c180)

[2021-11-10T00:09:21.412Z] 	/nfs/cache/mod/github.com/pingcap/[email protected]/writer.go:154 +0xad

[2021-11-10T00:09:21.412Z] created by github.com/pingcap/badger.startWriteWorker

[2021-11-10T00:09:21.412Z] 	/nfs/cache/mod/github.com/pingcap/[email protected]/writer.go:66 +0x1e8

[2021-11-10T00:09:21.412Z] 

[2021-11-10T00:09:21.412Z] goroutine 24009 [chan receive, 4 minutes, locked to thread]:

[2021-11-10T00:09:21.412Z] github.com/pingcap/badger.(*writeWorker).runWriteLSM(0xc002c96460, 0xc001d3c180)

[2021-11-10T00:09:21.412Z] 	/nfs/cache/mod/github.com/pingcap/[email protected]/writer.go:141 +0x195

[2021-11-10T00:09:21.412Z] created by github.com/pingcap/badger.startWriteWorker

[2021-11-10T00:09:21.412Z] 	/nfs/cache/mod/github.com/pingcap/[email protected]/writer.go:65 +0x1bc

[2021-11-10T00:09:21.412Z] 

[2021-11-10T00:09:21.412Z] goroutine 24008 [select, 4 minutes]:

[2021-11-10T00:09:21.412Z] github.com/pingcap/badger.(*writeWorker).runWriteVLog(0xc002c96460, 0xc001d3c180)

[2021-11-10T00:09:21.412Z] 	/nfs/cache/mod/github.com/pingcap/[email protected]/writer.go:94 +0x1dc

[2021-11-10T00:09:21.412Z] created by github.com/pingcap/badger.startWriteWorker

[2021-11-10T00:09:21.412Z] 	/nfs/cache/mod/github.com/pingcap/[email protected]/writer.go:64 +0x190

[2021-11-10T00:09:21.412Z] 

[2021-11-10T00:09:21.412Z] goroutine 33250 [select]:

[2021-11-10T00:09:21.412Z] github.com/pingcap/tidb/domain.(*Domain).LoadSysVarCacheLoop.func1(0xc07ee167e0, 0xc073b59818, 0x6fc23ac00, 0x41ed530, 0xc07ee09a00)

[2021-11-10T00:09:21.412Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/domain/domain.go:989 +0x125

[2021-11-10T00:09:21.412Z] created by github.com/pingcap/tidb/domain.(*Domain).LoadSysVarCacheLoop

[2021-11-10T00:09:21.412Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/domain/domain.go:980 +0x10e

[2021-11-10T00:09:21.412Z] 

[2021-11-10T00:09:21.412Z] goroutine 31594 [select]:

[2021-11-10T00:09:21.412Z] github.com/pingcap/tidb/domain.(*Domain).globalBindHandleWorkerLoop.func1(0xc07ee167e0, 0x41c16d8, 0xc07fb6eda0)

[2021-11-10T00:09:21.412Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/domain/domain.go:1071 +0x157

[2021-11-10T00:09:21.412Z] created by github.com/pingcap/tidb/domain.(*Domain).globalBindHandleWorkerLoop

[2021-11-10T00:09:21.412Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/domain/domain.go:1058 +0x73

[2021-11-10T00:09:21.412Z] 

[2021-11-10T00:09:21.412Z] goroutine 32760 [select, 8 minutes]:

[2021-11-10T00:09:21.412Z] github.com/pingcap/tidb/domain.(*Domain).TelemetryReportLoop.func1(0xc07ee167e0, 0x41ed530, 0xc086018000)

[2021-11-10T00:09:21.412Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/domain/domain.go:1140 +0x12f

[2021-11-10T00:09:21.412Z] created by github.com/pingcap/tidb/domain.(*Domain).TelemetryReportLoop

[2021-11-10T00:09:21.412Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/domain/domain.go:1132 +0x2c6

[2021-11-10T00:09:21.412Z] 

[2021-11-10T00:09:21.412Z] goroutine 31164 [select]:

[2021-11-10T00:09:21.412Z] github.com/pingcap/tidb/ddl.(*worker).start(0xc07cfddc30, 0xc07c9c6300)

[2021-11-10T00:09:21.412Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/ddl/ddl_worker.go:178 +0x315

[2021-11-10T00:09:21.412Z] created by github.com/pingcap/tidb/ddl.(*ddl).Start

[2021-11-10T00:09:21.412Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/ddl/ddl.go:369 +0x6a5

[2021-11-10T00:09:21.412Z] 

[2021-11-10T00:09:21.412Z] goroutine 31162 [select, 8 minutes]:

[2021-11-10T00:09:21.412Z] github.com/pingcap/tidb/ddl.(*ddl).limitDDLJobs(0xc07c9e27e0)

[2021-11-10T00:09:21.412Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/ddl/ddl_worker.go:263 +0x2d9

[2021-11-10T00:09:21.413Z] created by github.com/pingcap/tidb/ddl.(*ddl).Start

[2021-11-10T00:09:21.413Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/ddl/ddl.go:351 +0x2d5

[2021-11-10T00:09:21.413Z] 

[2021-11-10T00:09:21.413Z] goroutine 31169 [select, 8 minutes]:

[2021-11-10T00:09:21.413Z] github.com/pingcap/tidb/domain.(*Domain).globalConfigSyncerKeeper(0xc07ee167e0)

[2021-11-10T00:09:21.413Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/domain/domain.go:510 +0xf4

[2021-11-10T00:09:21.413Z] created by github.com/pingcap/tidb/domain.(*Domain).Init

[2021-11-10T00:09:21.413Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/domain/domain.go:830 +0x7af

[2021-11-10T00:09:21.413Z] 

[2021-11-10T00:09:21.413Z] goroutine 32761 [select]:

[2021-11-10T00:09:21.413Z] github.com/pingcap/tidb/domain.(*Domain).TelemetryRotateSubWindowLoop.func1(0xc07ee167e0)

[2021-11-10T00:09:21.413Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/domain/domain.go:1169 +0xd8

[2021-11-10T00:09:21.413Z] created by github.com/pingcap/tidb/domain.(*Domain).TelemetryRotateSubWindowLoop

[2021-11-10T00:09:21.413Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/domain/domain.go:1162 +0x7f

[2021-11-10T00:09:21.413Z] 

[2021-11-10T00:09:21.413Z] goroutine 31165 [select]:

[2021-11-10T00:09:21.413Z] github.com/pingcap/tidb/ddl.(*worker).start(0xc07cfddb80, 0xc07c9c6300)

[2021-11-10T00:09:21.413Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/ddl/ddl_worker.go:178 +0x315

[2021-11-10T00:09:21.413Z] created by github.com/pingcap/tidb/ddl.(*ddl).Start

[2021-11-10T00:09:21.413Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/ddl/ddl.go:369 +0x6a5

[2021-11-10T00:09:21.413Z] 

[2021-11-10T00:09:21.413Z] goroutine 31202 [select]:

[2021-11-10T00:09:21.413Z] github.com/pingcap/tidb/domain.(*Domain).topologySyncerKeeper(0xc07ee167e0)

[2021-11-10T00:09:21.413Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/domain/domain.go:533 +0x193

[2021-11-10T00:09:21.413Z] created by github.com/pingcap/tidb/domain.(*Domain).Init

[2021-11-10T00:09:21.413Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/domain/domain.go:834 +0x815

[2021-11-10T00:09:21.413Z] 

[2021-11-10T00:09:21.413Z] goroutine 31168 [select]:

[2021-11-10T00:09:21.413Z] github.com/pingcap/tidb/domain.(*Domain).infoSyncerKeeper(0xc07ee167e0)

[2021-11-10T00:09:21.413Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/domain/domain.go:487 +0x138

[2021-11-10T00:09:21.413Z] created by github.com/pingcap/tidb/domain.(*Domain).Init

[2021-11-10T00:09:21.413Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/domain/domain.go:829 +0x78a

[2021-11-10T00:09:21.413Z] 

[2021-11-10T00:09:21.413Z] goroutine 31167 [select, 8 minutes]:

[2021-11-10T00:09:21.413Z] github.com/pingcap/tidb/domain.(*Domain).topNSlowQueryLoop(0xc07ee167e0)

[2021-11-10T00:09:21.413Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/domain/domain.go:455 +0x1e5

[2021-11-10T00:09:21.413Z] created by github.com/pingcap/tidb/domain.(*Domain).Init

[2021-11-10T00:09:21.413Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/domain/domain.go:828 +0x765

[2021-11-10T00:09:21.413Z] 
[2021-11-10T00:09:21.413Z] goroutine 32495 [select, 4 minutes]:
[2021-11-10T00:09:21.413Z] github.com/pingcap/tidb/domain.(*Domain).LoadPrivilegeLoop.func1(0xc07ee167e0, 0xc073b596b8, 0x45d964b800, 0x41ed530, 0xc07a9ca200)
[2021-11-10T00:09:21.413Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/domain/domain.go:939 +0x125
[2021-11-10T00:09:21.413Z] created by github.com/pingcap/tidb/domain.(*Domain).LoadPrivilegeLoop
[2021-11-10T00:09:21.413Z] 	/home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/domain/domain.go:930 +0x1f8
[2021-11-10T00:09:21.413Z] FAIL	github.com/pingcap/tidb/table/tables	540.138s

https://ci.pingcap.net/blue/organizations/jenkins/tidb_ghpr_check_2/detail/tidb_ghpr_check_2/44348/pipeline/
https://ci.pingcap.net/blue/organizations/jenkins/tidb_ghpr_check_2/detail/tidb_ghpr_check_2/44340/pipeline/

4. What is your TiDB version? (Required)

master, commit dbea989

The text was updated successfully, but these errors were encountered:

JayLZhou · 2021-11-10T03:05:35Z

/assign

tiancaiamao · 2021-11-10T04:54:17Z

Because you're using t.Parallel()...
And the mockService is only one, so test case conflict with each other. @jayl-zxl

JayLZhou · 2021-11-10T05:13:46Z

Because you're using t.Parallel()... And the mockService is only one, so test case conflict with each other. @jayl-zxl
sorry. my mistake

JayLZhou · 2021-11-10T05:54:27Z

it should solved in #29443 @unconsolable

unconsolable · 2021-11-10T05:57:10Z

it should solved in #29443 @unconsolable

Please add a close issue, thanks.

github-actions · 2021-11-11T07:29:00Z

Please check whether the issue should be labeled with 'affects-x.y' or 'fixes-x.y.z', and then remove 'needs-more-info' label.

unconsolable added the type/bug The issue is confirmed as a bug. label Nov 10, 2021

unconsolable mentioned this issue Nov 10, 2021

executor: fix error msg of granting non-table level privilege #29321

Merged

12 tasks

ChenPeng2013 added component/test severity/major labels Nov 10, 2021

ti-chi-bot assigned JayLZhou Nov 10, 2021

JayLZhou mentioned this issue Nov 10, 2021

test : fix TestCacheTableComplexRead bug #29644

Closed

JayLZhou mentioned this issue Nov 10, 2021

Time out test table/tables #29647

Closed

tiancaiamao mentioned this issue Nov 11, 2021

table/tables: make CI stable for table/tables package #29688

Merged

12 tasks

lcwangchao closed this as completed in #29688 Nov 11, 2021

github-actions bot added the needs-more-info label Nov 11, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

test hang at package `table/tables` #29632

test hang at package `table/tables` #29632

unconsolable commented Nov 10, 2021

JayLZhou commented Nov 10, 2021

tiancaiamao commented Nov 10, 2021

JayLZhou commented Nov 10, 2021

JayLZhou commented Nov 10, 2021 •

edited

Loading

unconsolable commented Nov 10, 2021

github-actions bot commented Nov 11, 2021

test hang at package table/tables #29632

test hang at package table/tables #29632

Comments

unconsolable commented Nov 10, 2021

Bug Report

1. Minimal reproduce step (Required)

2. What did you expect to see? (Required)

3. What did you see instead (Required)

4. What is your TiDB version? (Required)

JayLZhou commented Nov 10, 2021

tiancaiamao commented Nov 10, 2021

JayLZhou commented Nov 10, 2021

JayLZhou commented Nov 10, 2021 • edited Loading

unconsolable commented Nov 10, 2021

github-actions bot commented Nov 11, 2021

test hang at package `table/tables` #29632

test hang at package `table/tables` #29632

JayLZhou commented Nov 10, 2021 •

edited

Loading