unstable tests etcdSuite.TestRegionWorkerExitWhenIsIdle

[overvenus]

Bug Report

Please answer these questions before submitting your issue. Thanks!

[2021-09-05T06:49:51.323Z] panic: close of closed channel
[2021-09-05T06:49:51.323Z] 
[2021-09-05T06:49:51.323Z] goroutine 4770 [running]:
[2021-09-05T06:49:51.323Z] github.com/pingcap/ticdc/cdc/kv.(*etcdSuite).TestRegionWorkerExitWhenIsIdle.func1.1(0xc0017842a0, 0xc000d48600, 0xc001806000)
[2021-09-05T06:49:51.323Z] 	/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client_test.go:3305 +0x39
[2021-09-05T06:49:51.323Z] github.com/pingcap/ticdc/cdc/kv.(*etcdSuite).TestRegionWorkerExitWhenIsIdle.func1(0x6208e50, 0xc0014e0260)
[2021-09-05T06:49:51.323Z] 	/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client_test.go:3316 +0x26e
[2021-09-05T06:49:51.323Z] github.com/pingcap/ticdc/cdc/kv.(*mockChangeDataService).EventFeed.func1(0xc0005066e0, 0x6208e50, 0xc0014e0260)
[2021-09-05T06:49:51.323Z] 	/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client_test.go:237 +0x62
[2021-09-05T06:49:51.323Z] created by github.com/pingcap/ticdc/cdc/kv.(*mockChangeDataService).EventFeed
[2021-09-05T06:49:51.323Z] 	/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client_test.go:236 +0x38f

https://ci2.pingcap.net/blue/organizations/jenkins/cdc_ghpr_integration_test/detail/cdc_ghpr_integration_test/3869/pipeline/108

TiCDC: https://github.com/pingcap/ticdc/tree/97df39e9bcd14e827480e2ad89cfddb1548ee4a8

[asddongmen]

another case: https://ci2.pingcap.net/blue/organizations/jenkins/cdc_ghpr_test/detail/cdc_ghpr_test/7508/pipeline

[maxshuang]

A common solution has been pr in #3612. If you find this issue happens again after this pr, pls reopen the this issue and @ me.

[glorv]

https://ci.pingcap.net/blue/organizations/jenkins/atom-ut/detail/atom-ut/8411/pipeline meet this issue again. @maxshuang

[maxshuang]

https://ci2.pingcap.net/blue/organizations/jenkins/cdc_ghpr_leak_test/detail/cdc_ghpr_leak_test/18923/pipeline/

[2022-01-21T05:38:04.314Z] ----------------------------------------------------------------------

[2022-01-21T05:38:04.314Z] FAIL: client_test.go:3330: clientSuite.TestRegionWorkerExitWhenIsIdle

[2022-01-21T05:38:04.314Z]

[2022-01-21T05:38:04.314Z] client_test.go:3406:

[2022-01-21T05:38:04.314Z] ...

[2022-01-21T05:38:04.314Z] /home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/tiflow/pkg/util/testleak/leaktest.go:165:

[2022-01-21T05:38:04.314Z] c.Errorf("Test %s check-count %d appears to have leaked: %v", c.TestName(), cnt, g)

[2022-01-21T05:38:04.314Z] ... Error: Test clientSuite.TestRegionWorkerExitWhenIsIdle check-count 50 appears to have leaked: github.com/pingcap/tiflow/cdc/kv.(*clientSuite).TestRegionWorkerExitWhenIsIdle.func1.1(0xc000066c60)

[2022-01-21T05:38:04.314Z] /home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/tiflow/cdc/kv/client_test.go:3346 +0x45

[2022-01-21T05:38:04.314Z] github.com/pingcap/tiflow/cdc/kv.(*clientSuite).TestRegionWorkerExitWhenIsIdle.func1(0x6dab790, 0xc000b668f0)

[2022-01-21T05:38:04.314Z] /home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/tiflow/cdc/kv/client_test.go:3355 +0x260

[2022-01-21T05:38:04.314Z] github.com/pingcap/tiflow/cdc/kv.(*mockChangeDataService).EventFeed.func1(0xc0007fc4b0, 0x6dab790, 0xc000b668f0)

[2022-01-21T05:38:04.314Z] /home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/tiflow/cdc/kv/client_test.go:256 +0x62

[2022-01-21T05:38:04.314Z] created by github.com/pingcap/tiflow/cdc/kv.(*mockChangeDataService).EventFeed

[2022-01-21T05:38:04.314Z] /home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/tiflow/cdc/kv/client_test.go:255 +0x38f

[2022-01-21T05:38:04.314Z]

[2022-01-21T05:38:04.314Z] [2022/01/21 13:37:42.056 +08:00] [ERROR] [client_test.go:1524] ["mock server error"] [error="rpc error: code = Canceled desc = context canceled"] [stack="github.com/pingcap/tiflow/cdc/kv.(*clientSuite).TestStreamRecvWithErrorAndResolvedGoBack.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/tiflow/cdc/kv/client_test.go:1524\ngithub.com/pingcap/tiflow/cdc/kv.(*mockChangeDataService).EventFeed.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/tiflow/cdc/kv/client_test.go:256"]

[2022-01-21T05:38:04.314Z] [2022/01/21 13:37:42.079 +08:00] [ERROR] [client_test.go:1524] ["mock server error"] [error="rpc error: code = Canceled desc = context canceled"] [stack="github.com/pingcap/tiflow/cdc/kv.(*clientSuite).TestStreamRecvWithErrorAndResolvedGoBack.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/tiflow/cdc/kv/client_test.go:1524\ngithub.com/pingcap/tiflow/cdc/kv.(*mockChangeDataService).EventFeed.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/tiflow/cdc/kv/client_test.go:256"]

[2022-01-21T05:38:04.314Z] [2022/01/21 13:37:50.357 +08:00] [ERROR] [client_test.go:1334] ["mock server error"] [error="rpc error: code = Canceled desc = context canceled"] [stack="github.com/pingcap/tiflow/cdc/kv.(*clientSuite).TestStreamSendWithError.func3\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/tiflow/cdc/kv/client_test.go:1334\ngithub.com/pingcap/tiflow/cdc/kv.(*mockChangeDataService).EventFeed.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_leak_test/go/src/github.com/pingcap/tiflow/cdc/kv/client_test.go:256"]

[2022-01-21T05:38:04.314Z] OOPS: 48 passed, 1 FAILED

[2022-01-21T05:38:04.314Z] --- FAIL: Test (114.12s)

[3AceShowHand]

https://ci2.pingcap.net/blue/organizations/jenkins/cdc_ghpr_leak_test/detail/cdc_ghpr_leak_test/19861/pipeline

[maxshuang]

panic: close of closed channel has been fixed.