v5.1.0 fatal error: concurrent map writes

[darren]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

2. What did you expect to see? (Required)

no panic

3. What did you see instead (Required)

fatal error: concurrent map writes
fatal error: concurrent map writes

goroutine 61844729 [running]:
runtime.throw(0x37361b3, 0x15)
        /usr/local/go/src/runtime/panic.go:1117 +0x72 fp=0xc005639718 sp=0xc0056396e8 pc=0x10f91b2
runtime.mapassign_faststr(0x331d4a0, 0xc039c8a570, 0x3715fa8, 0xa, 0x57abba0)
        /usr/local/go/src/runtime/map_faststr.go:291 +0x3d8 fp=0xc005639780 sp=0xc005639718 pc=0x10d54f8
github.com/pingcap/tidb/store/tikv/retry.(*Backoffer).BackoffWithCfgAndMaxSleep(0xc04fda0180, 0xc0003ed9e0, 0xffffffffffffffff, 0x3b9f0e0, 0xc04feb62e8, 0x34552c0, 0xc0258b2000)
        /go/src/github.com/pingcap/tidb/store/tikv/retry/backoff.go:187 +0x1213 fp=0xc005639ca0 sp=0xc005639780 pc=0x1be5653
github.com/pingcap/tidb/store/tikv/retry.(*Backoffer).Backoff(0xc04fda0180, 0xc0003ed9e0, 0x3b9f0e0, 0xc04feb62e8, 0x0, 0x0)
        /go/src/github.com/pingcap/tidb/store/tikv/retry/backoff.go:99 +0xda fp=0xc005639d68 sp=0xc005639ca0 pc=0x1be3b9a
github.com/pingcap/tidb/store/tikv.(*LockResolver).checkSecondaries(0xc0001d2880, 0xc04fda0180, 0x5e92fee12bc0002, 0xc04feb6210, 0x1, 0x1, 0x688ab93, 0xb96d, 0x17b6, 0xc039c8a330, ...)
        /go/src/github.com/pingcap/tidb/store/tikv/lock_resolver.go:735 +0x351 fp=0xc005639f08 sp=0xc005639d68 pc=0x1c29351
github.com/pingcap/tidb/store/tikv.(*LockResolver).checkAllSecondaries.func1(0xc01ba27f80, 0xc0001d2880, 0xc04fda0180, 0xc01ba27d40, 0xc04feb6210, 0x1, 0x1, 0x688ab93, 0xb96d, 0x17b6, ...)
        /go/src/github.com/pingcap/tidb/store/tikv/lock_resolver.go:831 +0xab fp=0xc005639f88 sp=0xc005639f08 pc=0x1c6b66b
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc005639f90 sp=0xc005639f88 pc=0x1134461
created by github.com/pingcap/tidb/store/tikv.(*LockResolver).checkAllSecondaries
        /go/src/github.com/pingcap/tidb/store/tikv/lock_resolver.go:830 +0x29a

goroutine 1 [IO wait]:
internal/poll.runtime_pollWait(0x7f0e003119f0, 0x72, 0x0)
        /usr/local/go/src/runtime/netpoll.go:222 +0x55
internal/poll.(*pollDesc).wait(0xc001d17198, 0x72, 0x0, 0x0, 0x370d800)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Accept(0xc001d17180, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
        /usr/local/go/src/internal/poll/fd_unix.go:401 +0x212
net.(*netFD).accept(0xc001d17180, 0x34045e0, 0x1, 0xc0836e65a8)
        /usr/local/go/src/net/fd_unix.go:172 +0x45
net.(*TCPListener).accept(0xc000477710, 0xc00346cc10, 0xc00346cc18, 0x10)
        /usr/local/go/src/net/tcpsock_posix.go:139 +0x32
net.(*TCPListener).Accept(0xc000477710, 0x381bc08, 0xc000bb4f70, 0xc03293a300, 0x0)
        /usr/local/go/src/net/tcpsock.go:261 +0x65
github.com/pingcap/tidb/server.(*Server).Run(0xc000bb4f70, 0xc0022a72f0, 0xc00214b7a0)
        /go/src/github.com/pingcap/tidb/server/server.go:340 +0xeb
main.main()
        /go/src/github.com/pingcap/tidb/tidb-server/main.go:192 +0x31d

goroutine 12 [select]:
go.opencensus.io/stats/view.(*worker).start(0xc0000eaf00)
        /go/pkg/mod/[email protected]/stats/view/worker.go:154 +0xcd
created by go.opencensus.io/stats/view.init.0
        /go/pkg/mod/[email protected]/stats/view/worker.go:32 +0x57

goroutine 143 [chan receive]:
go.etcd.io/etcd/pkg/logutil.(*MergeLogger).outputLoop(0xc000119a28)
        /go/pkg/mod/go.etcd.io/[email protected]/pkg/logutil/merge_logger.go:173 +0x3ac
created by go.etcd.io/etcd/pkg/logutil.NewMergeLogger
        /go/pkg/mod/go.etcd.io/[email protected]/pkg/logutil/merge_logger.go:91 +0x85

goroutine 12273 [IO wait]:
internal/poll.runtime_pollWait(0x7f0df8378d80, 0x72, 0xffffffffffffffff)
        /usr/local/go/src/runtime/netpoll.go:222 +0x55
internal/poll.(*pollDesc).wait(0xc0267c4098, 0x72, 0x8000, 0x8000, 0xffffffffffffffff)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc0267c4080, 0xc010ec0000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
        /usr/local/go/src/internal/poll/fd_unix.go:166 +0x1d5

4. What is your TiDB version? (Required)

Release Version: v5.1.0
Edition: Community
Git Commit Hash: 8acd5c88471cb7b4d4c4a8ed73b4d53d6833f13e
Git Branch: release-5.1
UTC Build Time: 2021-06-25 03:29:22
GoVersion: go1.16.4
Race Enabled: false
TiKV Min Version: v3.0.0-60965b006877ca7234adaced7890d7b029ed1306
Check Table Before Drop: false

[ti-srebot]

Please edit this comment or add a new comment to complete the following information

Bug

Note: Make Sure that 'component', and 'severity' labels are added
Example for how to fill out the template: #20100

1. Root Cause Analysis (RCA) (optional)

2. Symptom (optional)

TiDB panics and the goroutine stack is related to resolving async-commit locks.

3. All Trigger Conditions (optional)

TiDB resolves async-commit locks on different regions and backoff on different goroutines.

4. Workaround (optional)

No.

5. Affected versions

[v5.0.0:v5.0.3], v5.1

6. Fixed versions

v5.0.4, v5.1.1

[ti-srebot]

The values in ( FixedVersions ) fields are incorrect.

[youjiali1995]

@darren Thanks for your report!