Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

data race found in the kv client #10095

Closed
3AceShowHand opened this issue Nov 14, 2023 · 1 comment
Closed

data race found in the kv client #10095

3AceShowHand opened this issue Nov 14, 2023 · 1 comment
Assignees
@3AceShowHand
Labels
affects-6.5 This bug affects the 6.5.x(LTS) versions. affects-7.1 This bug affects the 7.1.x(LTS) versions. affects-7.5 This bug affects the 7.5.x(LTS) versions. area/ticdc Issues or PRs related to TiCDC. severity/minor type/bug The issue is confirmed as a bug.

Comments

@3AceShowHand
Copy link
Contributor

What did you do?

this is found by the integration test

What did you expect to see?

nothing

What did you see instead?

WARNING: DATA RACE
Read at 0x00000f4a9840 by goroutine 806:
  github.com/pingcap/tiflow/cdc/kv.(*regionWorker).initPoolHandles()
      /home/jenkins/agent/workspace/pingcap/tiflow/release-6.5/pull_cdc_integration_test/tiflow/cdc/kv/region_worker.go:400 +0x1b0
  github.com/pingcap/tiflow/cdc/kv.(*regionWorker).run()
      /home/jenkins/agent/workspace/pingcap/tiflow/release-6.5/pull_cdc_integration_test/tiflow/cdc/kv/region_worker.go:578 +0x17d
  github.com/pingcap/tiflow/cdc/kv.(*eventFeedSession).receiveFromStream.func3()
      /home/jenkins/agent/workspace/pingcap/tiflow/release-6.5/pull_cdc_integration_test/tiflow/cdc/kv/client.go:1017 +0x86
  golang.org/x/sync/errgroup.(*Group).Go.func1()
      /go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 +0x86

Previous write at 0x00000f4a9840 by goroutine 524:
  github.com/pingcap/tiflow/cdc/kv.InitWorkerPool.func1()
      /home/jenkins/agent/workspace/pingcap/tiflow/release-6.5/pull_cdc_integration_test/tiflow/cdc/kv/region_worker.go:856 +0x5a
  sync.(*Once).doSlow()
      /usr/local/go/src/sync/once.go:74 +0x101
  sync.(*Once).Do()
      /usr/local/go/src/sync/once.go:65 +0x46
  github.com/pingcap/tiflow/cdc/kv.InitWorkerPool()
      /home/jenkins/agent/workspace/pingcap/tiflow/release-6.5/pull_cdc_integration_test/tiflow/cdc/kv/region_worker.go:854 +0x44
  github.com/pingcap/tiflow/cdc/kv.RunWorkerPool()
      /home/jenkins/agent/workspace/pingcap/tiflow/release-6.5/pull_cdc_integration_test/tiflow/cdc/kv/region_worker.go:863 +0x44
  github.com/pingcap/tiflow/cdc/server.(*server).run.func3()
      /home/jenkins/agent/workspace/pingcap/tiflow/release-6.5/pull_cdc_integration_test/tiflow/cdc/server/server.go:398 +0x53
  golang.org/x/sync/errgroup.(*Group).Go.func1()
      /go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 +0x86

Goroutine 806 (running) created at:
  golang.org/x/sync/errgroup.(*Group).Go()
      /go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:72 +0x12e
  github.com/pingcap/tiflow/cdc/kv.(*eventFeedSession).receiveFromStream()
      /home/jenkins/agent/workspace/pingcap/tiflow/release-6.5/pull_cdc_integration_test/tiflow/cdc/kv/client.go:1016 +0xa05
  github.com/pingcap/tiflow/cdc/kv.(*eventFeedSession).requestRegionToStore.func1()
      /home/jenkins/agent/workspace/pingcap/tiflow/release-6.5/pull_cdc_integration_test/tiflow/cdc/kv/client.go:669 +0x167
  golang.org/x/sync/errgroup.(*Group).Go.func1()
      /go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 +0x86

Goroutine 524 (running) created at:
  golang.org/x/sync/errgroup.(*Group).Go()
      /go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:72 +0x12e
  github.com/pingcap/tiflow/cdc/server.(*server).run()
      /home/jenkins/agent/workspace/pingcap/tiflow/release-6.5/pull_cdc_integration_test/tiflow/cdc/server/server.go:397 +0x431
  github.com/pingcap/tiflow/cdc/server.(*server).Run()
      /home/jenkins/agent/workspace/pingcap/tiflow/release-6.5/pull_cdc_integration_test/tiflow/cdc/server/server.go:301 +0x124
  github.com/pingcap/tiflow/pkg/cmd/server.(*options).run()
      /home/jenkins/agent/workspace/pingcap/tiflow/release-6.5/pull_cdc_integration_test/tiflow/pkg/cmd/server/server.go:141 +0x8a4
  github.com/pingcap/tiflow/pkg/cmd/server.NewCmdServer.func1()
      /home/jenkins/agent/workspace/pingcap/tiflow/release-6.5/pull_cdc_integration_test/tiflow/pkg/cmd/server/server.go:280 +0x9c
  github.com/spf13/cobra.(*Command).execute()
      /go/pkg/mod/github.com/spf13/[email protected]/command.go:916 +0xbf4
  github.com/spf13/cobra.(*Command).ExecuteC()
      /go/pkg/mod/github.com/spf13/[email protected]/command.go:1044 +0x5fa
  github.com/spf13/cobra.(*Command).Execute()
      /go/pkg/mod/github.com/spf13/[email protected]/command.go:968 +0x318
  github.com/pingcap/tiflow/pkg/cmd.Run()
      /home/jenkins/agent/workspace/pingcap/tiflow/release-6.5/pull_cdc_integration_test/tiflow/pkg/cmd/cmd.go:50 +0x320
  github.com/pingcap/tiflow/cmd/cdc.main()
      /home/jenkins/agent/workspace/pingcap/tiflow/release-6.5/pull_cdc_integration_test/tiflow/cmd/cdc/main.go:22 +0x33
  github.com/pingcap/tiflow/cmd/cdc.TestRunMain()
      /home/jenkins/agent/workspace/pingcap/tiflow/release-6.5/pull_cdc_integration_test/tiflow/cmd/cdc/main_test.go:35 +0xca
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:1446 +0x216
  testing.(*T).Run.func1()
      /usr/local/go/src/testing/testing.go:1493 +0x47

Versions of the cluster

Upstream TiDB cluster version (execute SELECT tidb_version(); in a MySQL client):

(paste TiDB cluster version here)

Upstream TiKV version (execute tikv-server --version):

(paste TiKV version here)

TiCDC version (execute cdc version):

release-6.5
@3AceShowHand 3AceShowHand added type/bug The issue is confirmed as a bug. area/ticdc Issues or PRs related to TiCDC. labels Nov 14, 2023
@3AceShowHand 3AceShowHand self-assigned this Nov 30, 2023
@3AceShowHand 3AceShowHand added severity/minor needs-cherry-pick-release-6.5 Should cherry pick this PR to release-6.5 branch. affects-6.5 This bug affects the 6.5.x(LTS) versions. and removed severity/moderate needs-cherry-pick-release-6.5 Should cherry pick this PR to release-6.5 branch. labels Nov 30, 2023
@3AceShowHand 3AceShowHand mentioned this issue Nov 30, 2023
Merged
@3AceShowHand 3AceShowHand added affects-7.1 This bug affects the 7.1.x(LTS) versions. affects-7.5 This bug affects the 7.5.x(LTS) versions. labels Nov 30, 2023
@3AceShowHand 3AceShowHand mentioned this issue Nov 30, 2023
Merged
ti-chi-bot bot pushed a commit that referenced this issue Nov 30, 2023
@3AceShowHand
kvclient(ticdc): fix the workerpool data race (#10195)
6c1515c 
close #10095
ti-chi-bot bot pushed a commit that referenced this issue Nov 30, 2023
@3AceShowHand
kvclient(ticdc): fix the workerpool data race (#10196)
a78bd01 
ref #10095
This was referenced Nov 30, 2023
Closed
Merged
ti-chi-bot bot pushed a commit that referenced this issue Feb 19, 2024
@ti-chi-bot
kvclient(ticdc): fix the workerpool data race (#10196) (#10201)
2d9a053 
ref #10095
@3AceShowHand
Copy link
Contributor Author

closed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@3AceShowHand 3AceShowHand

Labels
affects-6.5 This bug affects the 6.5.x(LTS) versions. affects-7.1 This bug affects the 7.1.x(LTS) versions. affects-7.5 This bug affects the 7.5.x(LTS) versions. area/ticdc Issues or PRs related to TiCDC. severity/minor type/bug The issue is confirmed as a bug.
Projects
Question and Bug Reports
Status: Done
Milestone
No milestone
Development

No branches or pull requests
1 participant
@3AceShowHand