internal sql related to stats block building new connection

[xuyifangreeneyes]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

When building new connection, it needs to get h.mu.Lock, which is not reasonable.

goroutine 6473293526 [semacquire]:
sync.runtime_SemacquireMutex(0xd0?, 0xb?, 0x20300b?)
        /usr/local/go/src/runtime/sema.go:77 +0x25
sync.(*Mutex).lockSlow(0xc0007a8140)
        /usr/local/go/src/sync/mutex.go:171 +0x165
sync.(*Mutex).Lock(...)
        /usr/local/go/src/sync/mutex.go:90
sync.(*RWMutex).Lock(0xc0880f1202?)
        /usr/local/go/src/sync/rwmutex.go:147 +0x36
github.com/pingcap/tidb/statistics/handle.(*Handle).NewSessionStatsCollector(0xc0007a8140)
        /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/statistics/handle/update.go:237 +0x3b
github.com/pingcap/tidb/session.CreateSessionWithOpt({0x4fdf670, 0xc000a78780}, 0xc04daea580?)
        /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/session.go:3051 +0x1e5
github.com/pingcap/tidb/session.CreateSession(...)
        /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/session.go:3025
github.com/pingcap/tidb/server.(*TiDBDriver).OpenCtx(0xc03079b080?, 0x0?, 0x55339a8?, 0x2e, {0x32f8b69?, 0xc039c5aae0?}, 0xc025ef1810?, 0x0?)
        /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/driver_tidb.go:196 +0x47
github.com/pingcap/tidb/server.(*clientConn).openSession(0xc0510bea80)
        /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/conn.go:801 +0x119
github.com/pingcap/tidb/server.(*clientConn).writeInitialHandshake(0xc0510bea80, {0x4fb5c30, 0xc0771cecf0})
        /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/conn.go:404 +0x54a
github.com/pingcap/tidb/server.(*clientConn).handshake(0xc0510bea80, {0x4fb5c30, 0xc0771cecf0})
        /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/conn.go:284 +0x46
github.com/pingcap/tidb/server.(*Server).onConn(0xc00794a900, 0xc0510bea80)
        /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/server.go:563 +0x428
created by github.com/pingcap/tidb/server.(*Server).startNetworkListener
        /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/server.go:482 +0x5ca

h.mu may be held when executing some internal sql related to stats, such as dumpTableStatCountToKV.

goroutine 566 [select]:
github.com/tikv/client-go/v2/internal/client.sendBatchRequest({0x4fb5c30?, 0xc07fcb6e10}, {0xc08b752400, 0x38}, {0x0, 0x0}, 0xc00054ad00, 0xc04c6dc9f0, 0xc058e64bc0?)
        /go/pkg/mod/github.com/tikv/client-go/[email protected]/internal/client/client_batch.go:786 +0x79a
github.com/tikv/client-go/v2/internal/client.(*RPCClient).sendRequest(0xc000efa930, {0x4fb5c30, 0xc07fcb6e10}, {0xc08b752400, 0x38}, 0xc02166be40, 0x10000c0203158d8?)
        /go/pkg/mod/github.com/tikv/client-go/[email protected]/internal/client/client.go:497 +0x605
github.com/tikv/client-go/v2/internal/client.(*RPCClient).SendRequest(0xc007b3e000?, {0x4fb5c30, 0xc07fcb6e10}, {0xc08b752400, 0x38}, 0xc03773fcb0?, 0x18?)
        /go/pkg/mod/github.com/tikv/client-go/[email protected]/internal/client/client.go:540 +0x78
github.com/tikv/client-go/v2/internal/client.interceptedClient.SendRequest.func1({0xc08b752400?, 0xc08b752400?}, 0x38?)
        /go/pkg/mod/github.com/tikv/client-go/[email protected]/internal/client/client_interceptor.go:39 +0x45
github.com/pingcap/tidb/util/topsql/stmtstats.(*KvExecCounter).RPCInterceptor.func1.1({0xc08b752400?, 0x38?}, 0x76f6788?)
        /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/util/topsql/stmtstats/kv_exec_count.go:57 +0x6d
github.com/tikv/client-go/v2/internal/client.interceptedClient.SendRequest({{0x4fafc20?, 0xc000efa930?}}, {0x4fb5c30, 0xc07fcb6e10}, {0xc08b752400, 0x38}, 0x63102d0?, 0x6fc23ac00)
        /go/pkg/mod/github.com/tikv/client-go/[email protected]/internal/client/client_interceptor.go:40 +0x142
github.com/tikv/client-go/v2/internal/client.reqCollapse.SendRequest({{0x4fb3c10?, 0xc000a72520?}}, {0x4fb5c30, 0xc07fcb6e10}, {0xc08b752400, 0x38}, 0xc020315a78?, 0x17b9af7?)
        /go/pkg/mod/github.com/tikv/client-go/[email protected]/internal/client/client_collapse.go:74 +0xd6
github.com/tikv/client-go/v2/internal/locate.(*RegionRequestSender).sendReqToRegion(0xc020316ad0, 0xc05845fb00, 0xc071aaf110, 0xc02166be40, 0xc0203161f0?)
        /go/pkg/mod/github.com/tikv/client-go/[email protected]/internal/locate/region_request.go:1190 +0x894
github.com/tikv/client-go/v2/internal/locate.(*RegionRequestSender).SendReqCtx(0xc020316ad0, 0xc05845fb00, 0xc02166be40, {0x25b777, 0xb, 0x42ff}, 0x6fc23ac00, 0x0, {0x0, 0x0, ...})
        /go/pkg/mod/github.com/tikv/client-go/[email protected]/internal/locate/region_request.go:1023 +0x1274
github.com/tikv/client-go/v2/internal/locate.(*RegionRequestSender).SendReq(...)
        /go/pkg/mod/github.com/tikv/client-go/[email protected]/internal/locate/region_request.go:233
github.com/tikv/client-go/v2/txnkv/transaction.actionPrewrite.handleSingleBatch({0x0?}, 0xc07c739520, 0xc05845fb00, {{0x25b777, 0xb, 0x42ff}, {0x4fd5df8, 0xc07d81aba0}, 0x1})
        /go/pkg/mod/github.com/tikv/client-go/[email protected]/txnkv/transaction/prewrite.go:264 +0xc45
github.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).doActionOnBatches(0xc07c739520, 0xc05845fb00, {0x4fb3d30?, 0x7451d60}, {0xc04c1cf0e0?, 0x1, 0xc020316fa0?})
        /go/pkg/mod/github.com/tikv/client-go/[email protected]/txnkv/transaction/2pc.go:1035 +0x329
github.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).doActionOnGroupMutations(0xc07c739520, 0xc05845fb00, {0x4fb3d30, 0x7451d60?}, {0xc04c1cf0b0, 0x1, 0x0?})
        /go/pkg/mod/github.com/tikv/client-go/[email protected]/txnkv/transaction/2pc.go:1016 +0xfa7
github.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).doActionOnMutations(0xc07c739520, 0xc07fcb6e10?, {0x4fb3d30?, 0x7451d60}, {0x4fd5df8, 0xc01f1fee40})
        /go/pkg/mod/github.com/tikv/client-go/[email protected]/txnkv/transaction/2pc.go:787 +0xc7
github.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).prewriteMutations(0xc07c739520?, 0xc05845fb00, {0x4fd5df8, 0xc01f1fee40})
        /go/pkg/mod/github.com/tikv/client-go/[email protected]/txnkv/transaction/prewrite.go:438 +0x225
github.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).execute(0xc07c739520, {0x4fb5c30?, 0xc07fcb6e10})
        /go/pkg/mod/github.com/tikv/client-go/[email protected]/txnkv/transaction/2pc.go:1480 +0x56e
github.com/tikv/client-go/v2/txnkv/transaction.(*KVTxn).Commit(0xc075dcf8c0, {0x4fb5c30, 0xc07fcb6db0?})
        /go/pkg/mod/github.com/tikv/client-go/[email protected]/txnkv/transaction/txn.go:502 +0xf05
github.com/pingcap/tidb/store/driver/txn.(*tikvTxn).Commit(0xc068e48de0, {0x4fb5c30?, 0xc07fcb6db0?})
        /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/store/driver/txn/txn_driver.go:80 +0x2e
github.com/pingcap/tidb/session.(*LazyTxn).Commit(0xc00144c510, {0x4fb5c30, 0xc07fcb6db0})
        /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/txn.go:408 +0x477
github.com/pingcap/tidb/session.(*session).commitTxnWithTemporaryData(0x4fb5c30?, {0x4fb5c30?, 0xc07fcb6db0?}, {0x4ff73e0?, 0xc00144c510?})
        /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/session.go:840 +0x1b5
github.com/pingcap/tidb/session.(*session).doCommit(0xc00144c500, {0x4fb5c30, 0xc07fcb6d50})
        /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/session.go:726 +0xd78
github.com/pingcap/tidb/session.(*session).doCommitWithRetry(0xc00144c500, {0x4fb5c30, 0xc07fcb6d50})
        /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/session.go:972 +0x2c9
github.com/pingcap/tidb/session.(*session).CommitTxn(0xc00144c500, {0x4fb5c30, 0xc07fcb6c30})
        /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/session.go:1102 +0x227
github.com/pingcap/tidb/session.autoCommitAfterStmt({0x4fb5c30?, 0xc07fcb6c30?}, 0xc00144c500, {0x0?, 0x0?}, {0x4fc77c0?, 0xc07bcdcb40})
        /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/tidb.go:283 +0x40c
github.com/pingcap/tidb/session.finishStmt({0x4fb5c30, 0xc07fcb6c30}, 0xc00144c500, {0x0, 0x0}, {0x4fc77c0, 0xc07bcdcb40?})
        /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/tidb.go:249 +0x25f
github.com/pingcap/tidb/session.runStmt({0x4fb5c30, 0xc07fcb6c30}, 0xc00144c500, {0x4fc77c0, 0xc07bcdcb40?})
        /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/session.go:2368 +0x80c
github.com/pingcap/tidb/session.(*session).ExecuteStmt(0xc00144c500, {0x4fb5c30, 0xc04c1ceba0}, {0x4fcdcb8?, 0xc003d17040})
        /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/session.go:2213 +0xaed
github.com/pingcap/tidb/session.(*session).ExecuteInternal(0xc00144c500, {0x4fb5c30, 0xc04c1ceba0}, {0x4834545, 0x6}, {0x0, 0x0, 0x0})
        /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/session.go:1674 +0x3f2
github.com/pingcap/tidb/statistics/handle.finishTransaction({0x4fb5c30?, 0xc04c1ceba0?}, {0x7fa0f80dc558?, 0xc00144c500?}, {0x0?, 0x0?})
        /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/statistics/handle/ddl.go:340 +0xb5
github.com/pingcap/tidb/statistics/handle.(*Handle).dumpTableStatCountToKV.func2()
        /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/statistics/handle/update.go:542 +0x39
github.com/pingcap/tidb/statistics/handle.(*Handle).dumpTableStatCountToKV(0xc0007a8140, 0x142e, {0x24, 0x24, 0xc04be0c870, {0x0, 0x0, 0x0}, 0x0})
        /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/statistics/handle/update.go:586 +0x51a
github.com/pingcap/tidb/statistics/handle.(*Handle).DumpStatsDeltaToKV(0xc0007a8140, 0x0)
        /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/statistics/handle/update.go:500 +0x2f2
github.com/pingcap/tidb/domain.(*Domain).updateStatsWorker(0xc000a0e900, {0x5024898?, 0xc00144c500?}, {0x4fd5b88, 0xc0017050e0})
        /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/domain/domain.go:1969 +0x61a
github.com/pingcap/tidb/domain.(*Domain).UpdateTableStatsLoop.func1()
        /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/domain/domain.go:1833 +0x2d
github.com/pingcap/tidb/util.(*WaitGroupWrapper).Run.func1()
        /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/util/wait_group_wrapper.go:33 +0x5a
created by github.com/pingcap/tidb/util.(*WaitGroupWrapper).Run
        /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/util/wait_group_wrapper.go:31 +0x85

2. What did you expect to see? (Required)

Internal sql related to stats should not block building new connection

3. What did you see instead (Required)

Internal sql related to stats blocks building new connection.

4. What is your TiDB version? (Required)

v6.5.0, master

[xuyifangreeneyes]

/cherry-pick release-6.5-20230122-v6.5.0

[jackysp]

The implementation of creating internal session during handshake comes from #24991