Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fatal error: concurrent map read and map write from (*RuntimeStatsColl).GetRootStats #39094

Closed
AkiraXie opened this issue Nov 12, 2022 · 2 comments
Labels
affects-6.4 severity/major type/bug The issue is confirmed as a bug.

Comments

@AkiraXie
Copy link

AkiraXie commented Nov 12, 2022

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

  1. ch workload
  2. cc workload
  3. set tidb_memory_usage_alarm_ratio = 0.1

2. What did you expect to see? (Required)

no error

3. What did you see instead (Required)

fatal error: concurrent map read and map write
goroutine 26697 [running]:
github.com/pingcap/tidb/util/execdetails.(*RuntimeStatsColl).GetRootStats(0xc0e976ee40, 0x0?)
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/util/execdetails/execdetails.go:699 +0xb2
github.com/pingcap/tidb/planner/core.getRuntimeInfo({0x4eb7f68, 0xc025b4ea00}, {0x4e6f850, 0xc13da90000}, 0x7fc98eaf05b8?)
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/planner/core/common_plans.go:867 +0xac
github.com/pingcap/tidb/planner/core.getRuntimeInfoStr({0x4eb7f68?, 0xc025b4ea00?}, {0x4e6f850?, 0xc13da90000?}, 0xc13da90000?)
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/planner/core/common_plans.go:830 +0xa5
github.com/pingcap/tidb/planner/core.(*Explain).prepareOperatorInfo(0xc139cecf28, {0x4e6f850, 0xc13da90000}, {0x46f4ad3, 0x4}, {0xc13d5efec0, 0xb})
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/planner/core/common_plans.go:895 +0x3c9
github.com/pingcap/tidb/planner/core.(*Explain).explainFlatOpInRowFormat(0xc139cecf28, 0xc13d852600)
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/planner/core/common_plans.go:815 +0x21d
github.com/pingcap/tidb/planner/core.(*Explain).explainFlatPlanInRowFormat(0x4e6f850?, 0xc13d8454f0)
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/planner/core/common_plans.go:796 +0x5f
github.com/pingcap/tidb/planner/core.(*Explain).RenderResult(0xc139cecf28)
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/planner/core/common_plans.go:763 +0xd4e
github.com/pingcap/tidb/planner/core.GetExplainAnalyzeRowsForPlan(0xc139cecf28)
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/planner/core/common_plans.go:666 +0x2f
github.com/pingcap/tidb/session.(*session).getCurrentAnalyzePlan(0xc025b4ea00, {0x4631720?, 0xc13da90000}, 0xc0e976ee40)
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/session.go:1647 +0xd7
github.com/pingcap/tidb/util/memoryusagealarm.getCurrentAnalyzePlan(0x4ed4d2?)
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/util/memoryusagealarm/memoryusagealarm.go:249 +0x53
github.com/pingcap/tidb/util/memoryusagealarm.(*memoryUsageAlarm).getTop10SqlInfo(0xc139cedf20, 0x36?, {0xc13d83aa80?, 0x3, 0x38?})
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/util/memoryusagealarm/memoryusagealarm.go:291 +0x9e6
github.com/pingcap/tidb/util/memoryusagealarm.(*memoryUsageAlarm).getTop10SqlInfoByMemoryUsage(0x8?, {0xc13d83aa80?, 0x38?, 0x3c?})
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/util/memoryusagealarm/memoryusagealarm.go:311 +0x2e
github.com/pingcap/tidb/util/memoryusagealarm.(*memoryUsageAlarm).printTop10SqlInfo(0xc0e80833b0?, {0xc13d83aa80, 0x3, 0xe}, 0xc13d6b7840?)
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/util/memoryusagealarm/memoryusagealarm.go:261 +0x1f3
github.com/pingcap/tidb/util/memoryusagealarm.(*memoryUsageAlarm).recordSQL(0xc07760dbc0?, {0x4e700c0?, 0xc01b2f4100?}, {0xc07760dbc0, 0x3c})
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/util/memoryusagealarm/memoryusagealarm.go:342 +0x43a
github.com/pingcap/tidb/util/memoryusagealarm.(*memoryUsageAlarm).doRecord(0xc139cedf20, 0x1e1ef8000?, 0xffffffffffffffff?, {0x4e700c0, 0xc01b2f4100}, 0x1)
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/util/memoryusagealarm/memoryusagealarm.go:228 +0x103e
github.com/pingcap/tidb/util/memoryusagealarm.(*memoryUsageAlarm).alarm4ExcessiveMemUsage(0xc139cedf20, {0x4e700c0, 0xc01b2f4100})
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/util/memoryusagealarm/memoryusagealarm.go:165 +0x18d
github.com/pingcap/tidb/util/memoryusagealarm.(*Handle).Run(0xc0006a0330)
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/util/memoryusagealarm/memoryusagealarm.go:66 +0x113
created by main.createServer
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/tidb-server/main.go:762 +0x4d0...

tidb0-1112-err.log

4. What is your TiDB version? (Required)

6.4.0

@AkiraXie AkiraXie added the type/bug The issue is confirmed as a bug. label Nov 12, 2022
@AkiraXie
Copy link
Author

/severity major

@ti-chi-bot ti-chi-bot added severity/major may-affects-4.0 This bug maybe affects 4.0.x versions. may-affects-5.0 This bug maybe affects 5.0.x versions. may-affects-5.1 This bug maybe affects 5.1.x versions. may-affects-5.2 This bug maybe affects 5.2.x versions. may-affects-5.3 This bug maybe affects 5.3.x versions. may-affects-5.4 This bug maybe affects 5.4.x versions. may-affects-6.0 may-affects-6.1 may-affects-6.2 may-affects-6.3 labels Nov 12, 2022
@hawkingrei
Copy link
Member

@AkiraXie I cannot see two data race gorotine in the dump. Can you enable race to run this case?

@XuHuaiyu XuHuaiyu added affects-6.4 and removed may-affects-4.0 This bug maybe affects 4.0.x versions. may-affects-5.1 This bug maybe affects 5.1.x versions. may-affects-5.2 This bug maybe affects 5.2.x versions. may-affects-5.3 This bug maybe affects 5.3.x versions. may-affects-5.4 This bug maybe affects 5.4.x versions. may-affects-5.0 This bug maybe affects 5.0.x versions. may-affects-6.0 may-affects-6.1 may-affects-6.2 may-affects-6.3 labels Nov 13, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
affects-6.4 severity/major type/bug The issue is confirmed as a bug.
Projects
None yet
Development

No branches or pull requests

4 participants