Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fatal error: sync: unlock of unlocked mutex #57798

Closed
tiancaiamao opened this issue Nov 28, 2024 · 1 comment · Fixed by #57799
Closed

fatal error: sync: unlock of unlocked mutex #57798

tiancaiamao opened this issue Nov 28, 2024 · 1 comment · Fixed by #57799
Labels
affects-5.4 This bug affects the 5.4.x(LTS) versions. affects-6.1 This bug affects the 6.1.x(LTS) versions. affects-6.5 This bug affects the 6.5.x(LTS) versions. affects-7.1 This bug affects the 7.1.x(LTS) versions. affects-7.5 This bug affects the 7.5.x(LTS) versions. affects-8.1 This bug affects the 8.1.x(LTS) versions. affects-8.5 This bug affects the 8.5.x(LTS) versions. report/customer Customers have encountered this bug. severity/major sig/sql-infra SIG: SQL Infra type/bug The issue is confirmed as a bug.

Comments

@tiancaiamao
Copy link
Contributor

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

2. What did you expect to see? (Required)

3. What did you see instead (Required)

fatal error: sync: unlock of unlocked mutex

goroutine 7349387639 [running]:
sync.fatal({0x5a4e38e?, 0x30?})
	/usr/local/go/src/runtime/panic.go:1061 +0x18
sync.(*Mutex).unlockSlow(0xc09de4f2d0, 0xffffffff)
	/usr/local/go/src/sync/mutex.go:229 +0x35
sync.(*Mutex).Unlock(0x0?)
	/usr/local/go/src/sync/mutex.go:223 +0x25
github.com/pingcap/tidb/pkg/sessionctx/variable.(*SessionVars).GetRelatedTableForMDL(0xc15716a000)
	/workspace/source/tidb/pkg/sessionctx/variable/session.go:3558 +0x106
github.com/pingcap/tidb/pkg/session.(*session).TxnInfo(0xc05925d180)
	/workspace/source/tidb/pkg/session/session.go:528 +0x209
github.com/pingcap/tidb/pkg/server.(*Server).ShowTxnList(0xc002603100)
	/workspace/source/tidb/pkg/server/server.go:849 +0x1a8
github.com/pingcap/tidb/pkg/executor.(*tidbTrxTableRetriever).retrieve(0xc09e604d20, {0x63a0500?, 0xc06dc9a570}, {0x6419780?, 0xc098691680})
	/workspace/source/tidb/pkg/executor/infoschema_reader.go:2492 +0xee
github.com/pingcap/tidb/pkg/executor.(*MemTableReaderExec).Next(0xc09e6106e0, {0x63a0500, 0xc06dc9a570}, 0xc0a2d0fc20)
	/workspace/source/tidb/pkg/executor/memtable_reader.go:119 +0x2a6
github.com/pingcap/tidb/pkg/executor/internal/exec.Next({0x63a0500, 0xc06dc9a570}, {0x63b5a40, 0xc09e6106e0}, 0xc0a2d0fc20)
	/workspace/source/tidb/pkg/executor/internal/exec/executor.go:283 +0x2fa
github.com/pingcap/tidb/pkg/executor.(*SelectionExec).Next(0xc09e6140e0, {0x63a0500, 0xc06dc9a570}, 0xc0a2d0fc70)
	/workspace/source/tidb/pkg/executor/executor.go:1598 +0xde
github.com/pingcap/tidb/pkg/executor/internal/exec.Next({0x63a0500, 0xc06dc9a570}, {0x63b6d80, 0xc09e6140e0}, 0xc0a2d0fc70)
	/workspace/source/tidb/pkg/executor/internal/exec/executor.go:283 +0x2fa
github.com/pingcap/tidb/pkg/executor.(*CoprocessorDAGHandler).HandleRequest(0xc09e65d8f0, {0x63a0500?, 0xc06dc9a150?}, 0xc000f75900?)
	/workspace/source/tidb/pkg/executor/coprocessor.go:90 +0x2b7
github.com/pingcap/tidb/pkg/server.(*rpcServer).handleCopRequest(0x1c19d5a?, {0x63a0500, 0xc06dc9a150}, 0x0?)
	/workspace/source/tidb/pkg/server/rpc_server.go:214 +0x216
github.com/pingcap/tidb/pkg/server.(*rpcServer).Coprocessor(0x589fa20?, {0x63a0500, 0xc06dc9a150}, 0x0?)
	/workspace/source/tidb/pkg/server/rpc_server.go:103 +0xa5
github.com/pingcap/kvproto/pkg/tikvpb._Tikv_Coprocessor_Handler({0x59b3e80?, 0xc01413f0e0}, {0x63a0500, 0xc06dc9a150}, 0xc1b33c4c00, 0x0)
	/root/go/pkg/mod/github.com/pingcap/[email protected]/pkg/tikvpb/tikvpb.pb.go:3512 +0x169
google.golang.org/grpc.(*Server).processUnaryRPC(0xc013fde780, {0x63a0500, 0xc06dc9a090}, {0x63bf7e0, 0xc00f1bd1e0}, 0xc09dcec7e0, 0xc01413f200, 0x8b6c0b0, 0x0)
	/root/go/pkg/mod/google.golang.org/[email protected]/server.go:1343 +0xe03
google.golang.org/grpc.(*Server).handleStream(0xc013fde780, {0x63bf7e0, 0xc00f1bd1e0}, 0xc09dcec7e0)
	/root/go/pkg/mod/google.golang.org/[email protected]/server.go:1737 +0xc4c
google.golang.org/grpc.(*Server).serveStreams.func1.1()
	/root/go/pkg/mod/google.golang.org/[email protected]/server.go:986 +0x86
created by google.golang.org/grpc.(*Server).serveStreams.func1 in goroutine 52285
	/root/go/pkg/mod/google.golang.org/[email protected]/server.go:997 +0x145

4. What is your TiDB version? (Required)

v7.5.3
@tiancaiamao tiancaiamao added type/bug The issue is confirmed as a bug. affects-7.5 This bug affects the 7.5.x(LTS) versions. labels Nov 28, 2024
@tiancaiamao tiancaiamao mentioned this issue Nov 28, 2024
Merged
13 tasks
@jebter jebter added the sig/sql-infra SIG: SQL Infra label Nov 29, 2024
@ti-chi-bot ti-chi-bot bot added the report/customer Customers have encountered this bug. label Nov 29, 2024
@tiancaiamao tiancaiamao added the affects-8.5 This bug affects the 8.5.x(LTS) versions. label Dec 2, 2024
@tiancaiamao tiancaiamao added affects-6.1 This bug affects the 6.1.x(LTS) versions. affects-6.5 This bug affects the 6.5.x(LTS) versions. and removed may-affects-5.4 This bug maybe affects 5.4.x versions. may-affects-6.1 may-affects-6.5 may-affects-7.1 may-affects-8.1 labels Dec 3, 2024
@tiancaiamao
Copy link
Contributor Author

Finally, I get the trigger condition.
It should some query on the "information_schema.tidb_trx" table ...that's a tidb coprocessor request.

When tidb receive the request, it checks the transaction info for each session.
If at the same time the session is closing a query, then the DATA RACE happen.

A similiar one can be found here:

==================
WARNING: DATA RACE
Write at 0x00c049ccc310 by goroutine 164804:
  github.com/pingcap/tidb/pkg/sessionctx/variable.(*TransactionContext).Cleanup()
      pkg/sessionctx/variable/session.go:482 +0x4c4
  github.com/pingcap/tidb/pkg/session.(*session).CommitTxn()
      pkg/session/session.go:937 +0x3aa
  github.com/pingcap/tidb/pkg/session.autoCommitAfterStmt()
      pkg/session/tidb.go:273 +0x40c
  github.com/pingcap/tidb/pkg/session.finishStmt()
      pkg/session/tidb.go:235 +0x47b
  github.com/pingcap/tidb/pkg/session.runStmt()
      pkg/session/session.go:2321 +0xb35
  github.com/pingcap/tidb/pkg/session.(*session).ExecuteStmt()
      pkg/session/session.go:2152 +0x1cd8
  github.com/pingcap/tidb/pkg/testkit.(*TestKit).ExecWithContext()
      pkg/testkit/testkit.go:425 +0x1030
  github.com/pingcap/tidb/pkg/testkit.(*TestKit).MustExecWithContext()
      pkg/testkit/testkit.go:159 +0xab
  github.com/pingcap/tidb/pkg/testkit.(*TestKit).MustExec()
      pkg/testkit/testkit.go:154 +0x117
  tests/realtikvtest/txntest/txntest_test.TestRollbackTxnState.func1()
      tests/realtikvtest/txntest/txn_state_test.go:243 +0xd0

Previous write at 0x00c049ccc310 by goroutine 163128:
  github.com/pingcap/tidb/pkg/sessionctx/variable.(*SessionVars).GetRelatedTableForMDL()
      pkg/sessionctx/variable/session.go:3830 +0x41e
  github.com/pingcap/tidb/pkg/session.(*session).TxnInfo()
      pkg/session/session.go:487 +0x498
  tests/realtikvtest/txntest/txntest_test.TestRollbackTxnState()
      tests/realtikvtest/txntest/txn_state_test.go:248 +0x20e
  github.com/pingcap/tidb/pkg/sessionctx/variable.init.func620()
      pkg/sessionctx/variable/sysvar.go:3318 +0x67
  github.com/pingcap/tidb/pkg/sessionctx/variable.(*SysVar).SetGlobalFromHook()
      pkg/sessionctx/variable/variable.go:265 +0x138
  github.com/pingcap/tidb/pkg/session.(*session).SetGlobalSysVar()
      pkg/session/session.go:1333 +0x12e
  github.com/pingcap/tidb/pkg/sessionctx/variable.init.func619()
      pkg/sessionctx/variable/sysvar.go:3310 +0x52
  github.com/pingcap/tidb/pkg/sessionctx/variable.(*SysVar).Validate()
      pkg/sessionctx/variable/variable.go:317 +0x12b
  github.com/pingcap/tidb/pkg/session.(*session).SetGlobalSysVar()
      pkg/session/session.go:1330 +0xd2
  github.com/pingcap/tidb/pkg/executor.(*SetExecutor).setSysVariable()
      pkg/executor/set.go:155 +0x69d
  github.com/pingcap/tidb/pkg/executor.(*SetExecutor).Next()
      pkg/executor/set.go:105 +0xee6
  github.com/pingcap/tidb/pkg/executor/internal/exec.Next()
      pkg/executor/internal/exec/executor.go:460 +0x3a8
  github.com/pingcap/tidb/pkg/executor.(*ExecStmt).next()
      pkg/executor/adapter.go:1269 +0x84
  github.com/pingcap/tidb/pkg/executor.(*ExecStmt).handleNoDelayExecutor()
      pkg/executor/adapter.go:1018 +0x624
  github.com/pingcap/tidb/pkg/executor.(*ExecStmt).handleNoDelay()
      pkg/executor/adapter.go:851 +0x3e7
  github.com/pingcap/tidb/pkg/executor.(*ExecStmt).Exec()
      pkg/executor/adapter.go:614 +0x1824
  github.com/pingcap/tidb/pkg/session.runStmt()
      pkg/session/session.go:2290 +0x62b
  github.com/pingcap/tidb/pkg/session.(*session).ExecuteStmt()
      pkg/session/session.go:2152 +0x1cd8
  github.com/pingcap/tidb/pkg/testkit.(*TestKit).ExecWithContext()
      pkg/testkit/testkit.go:425 +0x1030
  github.com/pingcap/tidb/pkg/testkit.(*TestKit).MustExecWithContext()
      pkg/testkit/testkit.go:159 +0xab
  github.com/pingcap/tidb/pkg/testkit.(*TestKit).MustExec()
      pkg/testkit/testkit.go:154 +0x117
  github.com/pingcap/tidb/pkg/testkit.(*TestKit).RefreshSession()
      pkg/testkit/testkit.go:126 +0x3b2
  github.com/pingcap/tidb/pkg/testkit.NewTestKit()
      pkg/testkit/testkit.go:86 +0x4dd
  tests/realtikvtest/txntest/txntest_test.TestRollbackTxnState()
      tests/realtikvtest/txntest/txn_state_test.go:234 +0x4d
  github.com/pingcap/tidb/pkg/session.(*session).SetGlobalSysVar()
      pkg/session/session.go:1342 +0x2f0
  github.com/pingcap/tidb/pkg/executor.(*SetExecutor).setSysVariable()
      pkg/executor/set.go:155 +0x69d
  github.com/pingcap/tidb/pkg/executor.(*SetExecutor).Next()
      pkg/executor/set.go:105 +0xee6
  github.com/pingcap/tidb/pkg/executor/internal/exec.Next()
      pkg/executor/internal/exec/executor.go:460 +0x3a8
  github.com/pingcap/tidb/pkg/executor.(*ExecStmt).next()
      pkg/executor/adapter.go:1269 +0x84
  github.com/pingcap/tidb/pkg/executor.(*ExecStmt).handleNoDelayExecutor()
      pkg/executor/adapter.go:1018 +0x624
  github.com/pingcap/tidb/pkg/executor.(*ExecStmt).handleNoDelay()
      pkg/executor/adapter.go:851 +0x3e7
  github.com/pingcap/tidb/pkg/executor.(*ExecStmt).Exec()
      pkg/executor/adapter.go:614 +0x1824
  github.com/pingcap/tidb/pkg/session.runStmt()
      pkg/session/session.go:2290 +0x62b
  github.com/pingcap/tidb/pkg/session.(*session).ExecuteStmt()
      pkg/session/session.go:2152 +0x1cd8
  github.com/pingcap/tidb/pkg/testkit.(*TestKit).ExecWithContext()
      pkg/testkit/testkit.go:425 +0x1030
      pkg/testkit/testkit.go:154 +0x117
  github.com/pingcap/tidb/pkg/testkit.(*TestKit).RefreshSession()
      pkg/testkit/testkit.go:126 +0x3b2
  github.com/pingcap/tidb/pkg/testkit.NewTestKit()
      pkg/testkit/testkit.go:86 +0x4dd
  tests/realtikvtest/txntest/txntest_test.TestRollbackTxnState()
      tests/realtikvtest/txntest/txn_state_test.go:234 +0x4d
  github.com/pingcap/tidb/pkg/session.(*session).SetGlobalSysVar()
      pkg/session/session.go:1342 +0x2f0
  github.com/pingcap/tidb/pkg/executor.(*SetExecutor).setSysVariable()
      pkg/executor/set.go:155 +0x69d
  github.com/pingcap/tidb/pkg/executor.(*SetExecutor).Next()
      pkg/executor/set.go:105 +0xee6
  github.com/pingcap/tidb/pkg/executor/internal/exec.Next()
      pkg/executor/internal/exec/executor.go:460 +0x3a8
  github.com/pingcap/tidb/pkg/executor.(*ExecStmt).next()
      pkg/executor/adapter.go:1269 +0x84
  github.com/pingcap/tidb/pkg/executor.(*ExecStmt).handleNoDelayExecutor()
      pkg/executor/adapter.go:1018 +0x624
  github.com/pingcap/tidb/pkg/executor.(*ExecStmt).handleNoDelay()
      pkg/executor/adapter.go:851 +0x3e7
  github.com/pingcap/tidb/pkg/executor.(*ExecStmt).Exec()
      pkg/executor/adapter.go:614 +0x1824
  github.com/pingcap/tidb/pkg/session.runStmt()
      pkg/session/session.go:2290 +0x62b
  github.com/pingcap/tidb/pkg/session.(*session).ExecuteStmt()
      pkg/session/session.go:2152 +0x1cd8
  github.com/pingcap/tidb/pkg/testkit.(*TestKit).ExecWithContext()
      pkg/testkit/testkit.go:425 +0x1030
...

@ti-chi-bot ti-chi-bot bot closed this as completed in 12b9421 Dec 5, 2024
@ti-chi-bot ti-chi-bot mentioned this issue Dec 5, 2024
Merged
13 tasks
ti-chi-bot bot pushed a commit that referenced this issue Dec 5, 2024
@ti-chi-bot
*: avoid unlock of unlocked mutex panic on TableDeltaMap (#57799) (#5…
2f9739a 
…7997)

close #57798
@ti-chi-bot ti-chi-bot bot added affects-8.1 This bug affects the 8.1.x(LTS) versions. affects-5.4 This bug affects the 5.4.x(LTS) versions. affects-7.1 This bug affects the 7.1.x(LTS) versions. labels Dec 9, 2024
ti-chi-bot pushed a commit to ti-chi-bot/tidb that referenced this issue Dec 10, 2024
@tiancaiamao @ti-chi-bot
*: avoid unlock of unlocked mutex panic on TableDeltaMap (pingcap#57799)
4df91b3 
close pingcap#57798
@ti-chi-bot ti-chi-bot mentioned this issue Dec 10, 2024
Merged
13 tasks
ti-chi-bot bot pushed a commit that referenced this issue Dec 10, 2024
@ti-chi-bot
*: avoid unlock of unlocked mutex panic on TableDeltaMap (#57799) (#5…
972d702 
…8122)

close #57798
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
affects-5.4 This bug affects the 5.4.x(LTS) versions. affects-6.1 This bug affects the 6.1.x(LTS) versions. affects-6.5 This bug affects the 6.5.x(LTS) versions. affects-7.1 This bug affects the 7.1.x(LTS) versions. affects-7.5 This bug affects the 7.5.x(LTS) versions. affects-8.1 This bug affects the 8.1.x(LTS) versions. affects-8.5 This bug affects the 8.5.x(LTS) versions. report/customer Customers have encountered this bug. severity/major sig/sql-infra SIG: SQL Infra type/bug The issue is confirmed as a bug.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

*: avoid unlock of unlocked mutex panic on TableDeltaMap tiancaiamao/tidb
2 participants
@tiancaiamao @jebter