Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

unit test etcdSuite.TestKVClientForceReconnect is not stable #2349

Closed
amyangfei opened this issue Jul 22, 2021 · 10 comments
Closed

unit test etcdSuite.TestKVClientForceReconnect is not stable #2349

amyangfei opened this issue Jul 22, 2021 · 10 comments
Assignees
@overvenus @maxshuang
Labels
area/ticdc Issues or PRs related to TiCDC. component/test Unit tests and integration tests component. difficulty/medium Medium task. severity/major type/bug The issue is confirmed as a bug.

Comments

@amyangfei
Copy link
Contributor

Bug Report

Please answer these questions before submitting your issue. Thanks!

  1. What did you do? If possible, provide a recipe for reproducing the error.
[2021-07-21T06:26:24.989Z] [2021/07/21 14:23:21.529 +08:00] [ERROR] [client.go:1282] ["failed to receive from stream"] [addr=127.0.0.1:42035] [storeID=1] [error="injected stream recv error"] [errorVerbose="injected stream recv error\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).receiveFromStream\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:1271\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).requestRegionToStore.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:857\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:57\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357"] [stack="github.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).receiveFromStream\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:1282\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).requestRegionToStore.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:857\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:57"]
[2021-07-21T06:26:24.990Z] [2021/07/21 14:23:27.113 +08:00] [ERROR] [client.go:1282] ["failed to receive from stream"] [addr=127.0.0.1:36796] [storeID=1] [error="injected stream recv error"] [errorVerbose="injected stream recv error\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).receiveFromStream\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:1271\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).requestRegionToStore.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:857\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:57\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357"] [stack="github.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).receiveFromStream\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:1282\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).requestRegionToStore.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:857\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:57"]
[2021-07-21T06:26:24.990Z] [2021/07/21 14:23:29.125 +08:00] [ERROR] [client.go:462] ["check tikv version failed"] [error="[CDC:ErrGetAllStoresFailed]context canceled"] [errorVerbose="[CDC:ErrGetAllStoresFailed]context canceled\ngithub.com/pingcap/errors.AddStack\n\t/go/pkg/mod/github.com/pingcap/[email protected]/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByCause\n\t/go/pkg/mod/github.com/pingcap/[email protected]/normalize.go:279\ngithub.com/pingcap/ticdc/pkg/errors.WrapError\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/pkg/errors/helper.go:28\ngithub.com/pingcap/ticdc/pkg/version.CheckStoreVersion\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/pkg/version/check.go:128\ngithub.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:456\ngithub.com/pingcap/ticdc/pkg/retry.Run.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:32\ngithub.com/cenkalti/backoff.RetryNotify\n\t/go/pkg/mod/github.com/cenkalti/[email protected]+incompatible/retry.go:37\ngithub.com/cenkalti/backoff.Retry\n\t/go/pkg/mod/github.com/cenkalti/[email protected]+incompatible/retry.go:24\ngithub.com/pingcap/ticdc/pkg/retry.Run\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:31\ngithub.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:450\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).requestRegionToStore\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:825\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).eventFeed.func2\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:610\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:57\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357"] [storeID=1] [stack="github.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:462\ngithub.com/pingcap/ticdc/pkg/retry.Run.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:32\ngithub.com/cenkalti/backoff.RetryNotify\n\t/go/pkg/mod/github.com/cenkalti/[email protected]+incompatible/retry.go:37\ngithub.com/cenkalti/backoff.Retry\n\t/go/pkg/mod/github.com/cenkalti/[email protected]+incompatible/retry.go:24\ngithub.com/pingcap/ticdc/pkg/retry.Run\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:31\ngithub.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:450\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).requestRegionToStore\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:825\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).eventFeed.func2\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:610\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:57"]
[2021-07-21T06:26:24.991Z] [2021/07/21 14:23:30.081 +08:00] [ERROR] [client.go:1635] ["The CommitTs must be greater than the resolvedTs"] ["Event Type"=COMMIT] [CommitTs=98] [resolvedTs=100] [regionID=3] [stack="github.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).singleEventFeed\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:1635\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).partialRegionFeed\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:1018\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).sendRegionChangeEvent.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:1391\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:57"]
[2021-07-21T06:26:24.991Z] [2021/07/21 14:23:32.380 +08:00] [ERROR] [client.go:1614] ["The CommitTs must be greater than the resolvedTs"] ["Event Type"=COMMITTED] [CommitTs=98] [resolvedTs=100] [regionID=3] [stack="github.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).singleEventFeed\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:1614\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).partialRegionFeed\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:1018\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).sendRegionChangeEvent.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:1391\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:57"]
[2021-07-21T06:26:24.991Z] [2021/07/21 14:23:35.258 +08:00] [ERROR] [client.go:1189] ["tikv reported compatibility error, which is not expected"] [rpcCtx="region ID: 3, meta: id:3 peers:<id:4 store_id:1 > , peer: id:4 store_id:1 , addr: 127.0.0.1:40157, idx: 0, reqStoreType: TiKvOnly, runStoreType: tikv"] [error="required_version:\"v4.0.7\" "] [stack="github.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).handleError\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:1189\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).eventFeed.func4\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:642\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:57"]
[2021-07-21T06:26:24.991Z] [2021/07/21 14:23:51.317 +08:00] [ERROR] [client.go:1185] ["tikv reported duplicated request to the same region, which is not expected"] [regionID=3] [stack="github.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).handleError\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:1185\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).eventFeed.func4\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:642\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:57"]
[2021-07-21T06:26:24.992Z] [2021/07/21 14:24:19.051 +08:00] [ERROR] [client.go:462] ["check tikv version failed"] [error="[CDC:ErrVersionIncompatible]version is incompatible: TiKV 3.0.10 is not supported, require minimal version 4.0.0-rc.1"] [errorVerbose="[CDC:ErrVersionIncompatible]version is incompatible: TiKV 3.0.10 is not supported, require minimal version 4.0.0-rc.1\ngithub.com/pingcap/errors.AddStack\n\t/go/pkg/mod/github.com/pingcap/[email protected]/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByArgs\n\t/go/pkg/mod/github.com/pingcap/[email protected]/normalize.go:156\ngithub.com/pingcap/ticdc/pkg/version.CheckStoreVersion\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/pkg/version/check.go:140\ngithub.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:456\ngithub.com/pingcap/ticdc/pkg/retry.Run.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:32\ngithub.com/cenkalti/backoff.RetryNotify\n\t/go/pkg/mod/github.com/cenkalti/[email protected]+incompatible/retry.go:37\ngithub.com/cenkalti/backoff.Retry\n\t/go/pkg/mod/github.com/cenkalti/[email protected]+incompatible/retry.go:24\ngithub.com/pingcap/ticdc/pkg/retry.Run\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:31\ngithub.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:450\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).requestRegionToStore\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:825\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).eventFeed.func2\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:610\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:57\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357"] [storeID=1] [stack="github.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:462\ngithub.com/pingcap/ticdc/pkg/retry.Run.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:32\ngithub.com/cenkalti/backoff.RetryNotify\n\t/go/pkg/mod/github.com/cenkalti/[email protected]+incompatible/retry.go:37\ngithub.com/cenkalti/backoff.Retry\n\t/go/pkg/mod/github.com/cenkalti/[email protected]+incompatible/retry.go:24\ngithub.com/pingcap/ticdc/pkg/retry.Run\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:31\ngithub.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:450\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).requestRegionToStore\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:825\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).eventFeed.func2\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:610\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:57"]
[2021-07-21T06:26:24.992Z] [2021/07/21 14:24:19.122 +08:00] [ERROR] [client.go:462] ["check tikv version failed"] [error="[CDC:ErrVersionIncompatible]version is incompatible: TiKV 3.1.0 is not supported, require minimal version 4.0.0-rc.1"] [errorVerbose="[CDC:ErrVersionIncompatible]version is incompatible: TiKV 3.1.0 is not supported, require minimal version 4.0.0-rc.1\ngithub.com/pingcap/errors.AddStack\n\t/go/pkg/mod/github.com/pingcap/[email protected]/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByArgs\n\t/go/pkg/mod/github.com/pingcap/[email protected]/normalize.go:156\ngithub.com/pingcap/ticdc/pkg/version.CheckStoreVersion\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/pkg/version/check.go:140\ngithub.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:456\ngithub.com/pingcap/ticdc/pkg/retry.Run.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:32\ngithub.com/cenkalti/backoff.RetryNotify\n\t/go/pkg/mod/github.com/cenkalti/[email protected]+incompatible/retry.go:37\ngithub.com/cenkalti/backoff.Retry\n\t/go/pkg/mod/github.com/cenkalti/[email protected]+incompatible/retry.go:24\ngithub.com/pingcap/ticdc/pkg/retry.Run\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:31\ngithub.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:450\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).requestRegionToStore\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:825\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).eventFeed.func2\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:610\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:57\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357"] [storeID=1] [stack="github.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:462\ngithub.com/pingcap/ticdc/pkg/retry.Run.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:32\ngithub.com/cenkalti/backoff.RetryNotify\n\t/go/pkg/mod/github.com/cenkalti/[email protected]+incompatible/retry.go:37\ngithub.com/cenkalti/backoff.Retry\n\t/go/pkg/mod/github.com/cenkalti/[email protected]+incompatible/retry.go:24\ngithub.com/pingcap/ticdc/pkg/retry.Run\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:31\ngithub.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:450\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).requestRegionToStore\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:825\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).eventFeed.func2\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:610\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:57"]
[2021-07-21T06:26:24.993Z] [2021/07/21 14:24:19.201 +08:00] [ERROR] [client.go:462] ["check tikv version failed"] [error="[CDC:ErrVersionIncompatible]version is incompatible: TiKV 4.0.0-rc is not supported, require minimal version 4.0.0-rc.1"] [errorVerbose="[CDC:ErrVersionIncompatible]version is incompatible: TiKV 4.0.0-rc is not supported, require minimal version 4.0.0-rc.1\ngithub.com/pingcap/errors.AddStack\n\t/go/pkg/mod/github.com/pingcap/[email protected]/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByArgs\n\t/go/pkg/mod/github.com/pingcap/[email protected]/normalize.go:156\ngithub.com/pingcap/ticdc/pkg/version.CheckStoreVersion\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/pkg/version/check.go:140\ngithub.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:456\ngithub.com/pingcap/ticdc/pkg/retry.Run.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:32\ngithub.com/cenkalti/backoff.RetryNotify\n\t/go/pkg/mod/github.com/cenkalti/[email protected]+incompatible/retry.go:37\ngithub.com/cenkalti/backoff.Retry\n\t/go/pkg/mod/github.com/cenkalti/[email protected]+incompatible/retry.go:24\ngithub.com/pingcap/ticdc/pkg/retry.Run\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:31\ngithub.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:450\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).requestRegionToStore\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:825\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).eventFeed.func2\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:610\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:57\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357"] [storeID=1] [stack="github.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:462\ngithub.com/pingcap/ticdc/pkg/retry.Run.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:32\ngithub.com/cenkalti/backoff.RetryNotify\n\t/go/pkg/mod/github.com/cenkalti/[email protected]+incompatible/retry.go:37\ngithub.com/cenkalti/backoff.Retry\n\t/go/pkg/mod/github.com/cenkalti/[email protected]+incompatible/retry.go:24\ngithub.com/pingcap/ticdc/pkg/retry.Run\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:31\ngithub.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:450\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).requestRegionToStore\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:825\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).eventFeed.func2\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:610\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:57"]
[2021-07-21T06:26:24.994Z] [2021/07/21 14:24:19.292 +08:00] [ERROR] [client.go:462] ["check tikv version failed"] [error="[CDC:ErrVersionIncompatible]version is incompatible: TiKV 2.1.10 is not supported, require minimal version 4.0.0-rc.1"] [errorVerbose="[CDC:ErrVersionIncompatible]version is incompatible: TiKV 2.1.10 is not supported, require minimal version 4.0.0-rc.1\ngithub.com/pingcap/errors.AddStack\n\t/go/pkg/mod/github.com/pingcap/[email protected]/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByArgs\n\t/go/pkg/mod/github.com/pingcap/[email protected]/normalize.go:156\ngithub.com/pingcap/ticdc/pkg/version.CheckStoreVersion\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/pkg/version/check.go:140\ngithub.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:456\ngithub.com/pingcap/ticdc/pkg/retry.Run.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:32\ngithub.com/cenkalti/backoff.RetryNotify\n\t/go/pkg/mod/github.com/cenkalti/[email protected]+incompatible/retry.go:37\ngithub.com/cenkalti/backoff.Retry\n\t/go/pkg/mod/github.com/cenkalti/[email protected]+incompatible/retry.go:24\ngithub.com/pingcap/ticdc/pkg/retry.Run\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:31\ngithub.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:450\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).requestRegionToStore\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:825\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).eventFeed.func2\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:610\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:57\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357"] [storeID=1] [stack="github.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:462\ngithub.com/pingcap/ticdc/pkg/retry.Run.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:32\ngithub.com/cenkalti/backoff.RetryNotify\n\t/go/pkg/mod/github.com/cenkalti/[email protected]+incompatible/retry.go:37\ngithub.com/cenkalti/backoff.Retry\n\t/go/pkg/mod/github.com/cenkalti/[email protected]+incompatible/retry.go:24\ngithub.com/pingcap/ticdc/pkg/retry.Run\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:31\ngithub.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:450\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).requestRegionToStore\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:825\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).eventFeed.func2\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:610\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:57"]
[2021-07-21T06:26:24.994Z] [2021/07/21 14:24:19.394 +08:00] [ERROR] [client.go:462] ["check tikv version failed"] [error="[CDC:ErrVersionIncompatible]version is incompatible: TiKV 3.1.0 is not supported, require minimal version 4.0.0-rc.1"] [errorVerbose="[CDC:ErrVersionIncompatible]version is incompatible: TiKV 3.1.0 is not supported, require minimal version 4.0.0-rc.1\ngithub.com/pingcap/errors.AddStack\n\t/go/pkg/mod/github.com/pingcap/[email protected]/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByArgs\n\t/go/pkg/mod/github.com/pingcap/[email protected]/normalize.go:156\ngithub.com/pingcap/ticdc/pkg/version.CheckStoreVersion\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/pkg/version/check.go:140\ngithub.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:456\ngithub.com/pingcap/ticdc/pkg/retry.Run.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:32\ngithub.com/cenkalti/backoff.RetryNotify\n\t/go/pkg/mod/github.com/cenkalti/[email protected]+incompatible/retry.go:37\ngithub.com/cenkalti/backoff.Retry\n\t/go/pkg/mod/github.com/cenkalti/[email protected]+incompatible/retry.go:24\ngithub.com/pingcap/ticdc/pkg/retry.Run\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:31\ngithub.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:450\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).requestRegionToStore\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:825\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).eventFeed.func2\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:610\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:57\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357"] [storeID=1] [stack="github.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:462\ngithub.com/pingcap/ticdc/pkg/retry.Run.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:32\ngithub.com/cenkalti/backoff.RetryNotify\n\t/go/pkg/mod/github.com/cenkalti/[email protected]+incompatible/retry.go:37\ngithub.com/cenkalti/backoff.Retry\n\t/go/pkg/mod/github.com/cenkalti/[email protected]+incompatible/retry.go:24\ngithub.com/pingcap/ticdc/pkg/retry.Run\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:31\ngithub.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:450\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).requestRegionToStore\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:825\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).eventFeed.func2\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:610\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:57"]
[2021-07-21T06:26:24.995Z] [2021/07/21 14:24:19.469 +08:00] [ERROR] [client.go:462] ["check tikv version failed"] [error="[CDC:ErrVersionIncompatible]version is incompatible: TiKV 4.0.0-rc is not supported, require minimal version 4.0.0-rc.1"] [errorVerbose="[CDC:ErrVersionIncompatible]version is incompatible: TiKV 4.0.0-rc is not supported, require minimal version 4.0.0-rc.1\ngithub.com/pingcap/errors.AddStack\n\t/go/pkg/mod/github.com/pingcap/[email protected]/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByArgs\n\t/go/pkg/mod/github.com/pingcap/[email protected]/normalize.go:156\ngithub.com/pingcap/ticdc/pkg/version.CheckStoreVersion\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/pkg/version/check.go:140\ngithub.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:456\ngithub.com/pingcap/ticdc/pkg/retry.Run.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:32\ngithub.com/cenkalti/backoff.RetryNotify\n\t/go/pkg/mod/github.com/cenkalti/[email protected]+incompatible/retry.go:37\ngithub.com/cenkalti/backoff.Retry\n\t/go/pkg/mod/github.com/cenkalti/[email protected]+incompatible/retry.go:24\ngithub.com/pingcap/ticdc/pkg/retry.Run\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:31\ngithub.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:450\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).requestRegionToStore\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:825\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).eventFeed.func2\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:610\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:57\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357"] [storeID=1] [stack="github.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:462\ngithub.com/pingcap/ticdc/pkg/retry.Run.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:32\ngithub.com/cenkalti/backoff.RetryNotify\n\t/go/pkg/mod/github.com/cenkalti/[email protected]+incompatible/retry.go:37\ngithub.com/cenkalti/backoff.Retry\n\t/go/pkg/mod/github.com/cenkalti/[email protected]+incompatible/retry.go:24\ngithub.com/pingcap/ticdc/pkg/retry.Run\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:31\ngithub.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:450\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).requestRegionToStore\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:825\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).eventFeed.func2\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:610\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:57"]
[2021-07-21T06:26:24.995Z] [2021/07/21 14:24:19.581 +08:00] [ERROR] [client.go:462] ["check tikv version failed"] [error="[CDC:ErrVersionIncompatible]version is incompatible: TiKV 2.1.10 is not supported, require minimal version 4.0.0-rc.1"] [errorVerbose="[CDC:ErrVersionIncompatible]version is incompatible: TiKV 2.1.10 is not supported, require minimal version 4.0.0-rc.1\ngithub.com/pingcap/errors.AddStack\n\t/go/pkg/mod/github.com/pingcap/[email protected]/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByArgs\n\t/go/pkg/mod/github.com/pingcap/[email protected]/normalize.go:156\ngithub.com/pingcap/ticdc/pkg/version.CheckStoreVersion\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/pkg/version/check.go:140\ngithub.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:456\ngithub.com/pingcap/ticdc/pkg/retry.Run.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:32\ngithub.com/cenkalti/backoff.RetryNotify\n\t/go/pkg/mod/github.com/cenkalti/[email protected]+incompatible/retry.go:37\ngithub.com/cenkalti/backoff.Retry\n\t/go/pkg/mod/github.com/cenkalti/[email protected]+incompatible/retry.go:24\ngithub.com/pingcap/ticdc/pkg/retry.Run\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:31\ngithub.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:450\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).requestRegionToStore\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:825\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).eventFeed.func2\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:610\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:57\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357"] [storeID=1] [stack="github.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:462\ngithub.com/pingcap/ticdc/pkg/retry.Run.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:32\ngithub.com/cenkalti/backoff.RetryNotify\n\t/go/pkg/mod/github.com/cenkalti/[email protected]+incompatible/retry.go:37\ngithub.com/cenkalti/backoff.Retry\n\t/go/pkg/mod/github.com/cenkalti/[email protected]+incompatible/retry.go:24\ngithub.com/pingcap/ticdc/pkg/retry.Run\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:31\ngithub.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:450\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).requestRegionToStore\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:825\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).eventFeed.func2\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:610\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:57"]
[2021-07-21T06:26:24.996Z] [2021/07/21 14:24:19.709 +08:00] [ERROR] [client.go:462] ["check tikv version failed"] [error="[CDC:ErrVersionIncompatible]version is incompatible: TiKV 3.0.10 is not supported, require minimal version 4.0.0-rc.1"] [errorVerbose="[CDC:ErrVersionIncompatible]version is incompatible: TiKV 3.0.10 is not supported, require minimal version 4.0.0-rc.1\ngithub.com/pingcap/errors.AddStack\n\t/go/pkg/mod/github.com/pingcap/[email protected]/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByArgs\n\t/go/pkg/mod/github.com/pingcap/[email protected]/normalize.go:156\ngithub.com/pingcap/ticdc/pkg/version.CheckStoreVersion\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/pkg/version/check.go:140\ngithub.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:456\ngithub.com/pingcap/ticdc/pkg/retry.Run.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:32\ngithub.com/cenkalti/backoff.RetryNotify\n\t/go/pkg/mod/github.com/cenkalti/[email protected]+incompatible/retry.go:37\ngithub.com/cenkalti/backoff.Retry\n\t/go/pkg/mod/github.com/cenkalti/[email protected]+incompatible/retry.go:24\ngithub.com/pingcap/ticdc/pkg/retry.Run\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:31\ngithub.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:450\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).requestRegionToStore\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:825\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).eventFeed.func2\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:610\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:57\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357"] [storeID=1] [stack="github.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:462\ngithub.com/pingcap/ticdc/pkg/retry.Run.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:32\ngithub.com/cenkalti/backoff.RetryNotify\n\t/go/pkg/mod/github.com/cenkalti/[email protected]+incompatible/retry.go:37\ngithub.com/cenkalti/backoff.Retry\n\t/go/pkg/mod/github.com/cenkalti/[email protected]+incompatible/retry.go:24\ngithub.com/pingcap/ticdc/pkg/retry.Run\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:31\ngithub.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:450\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).requestRegionToStore\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:825\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).eventFeed.func2\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:610\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:57"]
[2021-07-21T06:26:24.997Z] [2021/07/21 14:24:19.812 +08:00] [ERROR] [client.go:462] ["check tikv version failed"] [error="[CDC:ErrVersionIncompatible]version is incompatible: TiKV 4.0.0-rc is not supported, require minimal version 4.0.0-rc.1"] [errorVerbose="[CDC:ErrVersionIncompatible]version is incompatible: TiKV 4.0.0-rc is not supported, require minimal version 4.0.0-rc.1\ngithub.com/pingcap/errors.AddStack\n\t/go/pkg/mod/github.com/pingcap/[email protected]/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByArgs\n\t/go/pkg/mod/github.com/pingcap/[email protected]/normalize.go:156\ngithub.com/pingcap/ticdc/pkg/version.CheckStoreVersion\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/pkg/version/check.go:140\ngithub.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:456\ngithub.com/pingcap/ticdc/pkg/retry.Run.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:32\ngithub.com/cenkalti/backoff.RetryNotify\n\t/go/pkg/mod/github.com/cenkalti/[email protected]+incompatible/retry.go:37\ngithub.com/cenkalti/backoff.Retry\n\t/go/pkg/mod/github.com/cenkalti/[email protected]+incompatible/retry.go:24\ngithub.com/pingcap/ticdc/pkg/retry.Run\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:31\ngithub.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:450\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).requestRegionToStore\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:825\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).eventFeed.func2\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:610\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:57\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357"] [storeID=1] [stack="github.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:462\ngithub.com/pingcap/ticdc/pkg/retry.Run.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:32\ngithub.com/cenkalti/backoff.RetryNotify\n\t/go/pkg/mod/github.com/cenkalti/[email protected]+incompatible/retry.go:37\ngithub.com/cenkalti/backoff.Retry\n\t/go/pkg/mod/github.com/cenkalti/[email protected]+incompatible/retry.go:24\ngithub.com/pingcap/ticdc/pkg/retry.Run\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:31\ngithub.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:450\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).requestRegionToStore\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:825\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).eventFeed.func2\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:610\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:57"]
[2021-07-21T06:26:24.997Z] [2021/07/21 14:24:19.839 +08:00] [ERROR] [client.go:462] ["check tikv version failed"] [error="[CDC:ErrVersionIncompatible]version is incompatible: TiKV 2.1.10 is not supported, require minimal version 4.0.0-rc.1"] [errorVerbose="[CDC:ErrVersionIncompatible]version is incompatible: TiKV 2.1.10 is not supported, require minimal version 4.0.0-rc.1\ngithub.com/pingcap/errors.AddStack\n\t/go/pkg/mod/github.com/pingcap/[email protected]/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByArgs\n\t/go/pkg/mod/github.com/pingcap/[email protected]/normalize.go:156\ngithub.com/pingcap/ticdc/pkg/version.CheckStoreVersion\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/pkg/version/check.go:140\ngithub.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:456\ngithub.com/pingcap/ticdc/pkg/retry.Run.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:32\ngithub.com/cenkalti/backoff.RetryNotify\n\t/go/pkg/mod/github.com/cenkalti/[email protected]+incompatible/retry.go:37\ngithub.com/cenkalti/backoff.Retry\n\t/go/pkg/mod/github.com/cenkalti/[email protected]+incompatible/retry.go:24\ngithub.com/pingcap/ticdc/pkg/retry.Run\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:31\ngithub.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:450\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).requestRegionToStore\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:825\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).eventFeed.func2\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:610\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:57\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357"] [storeID=1] [stack="github.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:462\ngithub.com/pingcap/ticdc/pkg/retry.Run.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:32\ngithub.com/cenkalti/backoff.RetryNotify\n\t/go/pkg/mod/github.com/cenkalti/[email protected]+incompatible/retry.go:37\ngithub.com/cenkalti/backoff.Retry\n\t/go/pkg/mod/github.com/cenkalti/[email protected]+incompatible/retry.go:24\ngithub.com/pingcap/ticdc/pkg/retry.Run\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:31\ngithub.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:450\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).requestRegionToStore\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:825\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).eventFeed.func2\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:610\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:57"]
[2021-07-21T06:26:24.998Z] [2021/07/21 14:24:19.895 +08:00] [ERROR] [client.go:462] ["check tikv version failed"] [error="[CDC:ErrVersionIncompatible]version is incompatible: TiKV 3.0.10 is not supported, require minimal version 4.0.0-rc.1"] [errorVerbose="[CDC:ErrVersionIncompatible]version is incompatible: TiKV 3.0.10 is not supported, require minimal version 4.0.0-rc.1\ngithub.com/pingcap/errors.AddStack\n\t/go/pkg/mod/github.com/pingcap/[email protected]/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByArgs\n\t/go/pkg/mod/github.com/pingcap/[email protected]/normalize.go:156\ngithub.com/pingcap/ticdc/pkg/version.CheckStoreVersion\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/pkg/version/check.go:140\ngithub.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:456\ngithub.com/pingcap/ticdc/pkg/retry.Run.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:32\ngithub.com/cenkalti/backoff.RetryNotify\n\t/go/pkg/mod/github.com/cenkalti/[email protected]+incompatible/retry.go:37\ngithub.com/cenkalti/backoff.Retry\n\t/go/pkg/mod/github.com/cenkalti/[email protected]+incompatible/retry.go:24\ngithub.com/pingcap/ticdc/pkg/retry.Run\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:31\ngithub.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:450\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).requestRegionToStore\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:825\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).eventFeed.func2\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:610\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:57\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357"] [storeID=1] [stack="github.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:462\ngithub.com/pingcap/ticdc/pkg/retry.Run.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:32\ngithub.com/cenkalti/backoff.RetryNotify\n\t/go/pkg/mod/github.com/cenkalti/[email protected]+incompatible/retry.go:37\ngithub.com/cenkalti/backoff.Retry\n\t/go/pkg/mod/github.com/cenkalti/[email protected]+incompatible/retry.go:24\ngithub.com/pingcap/ticdc/pkg/retry.Run\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:31\ngithub.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:450\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).requestRegionToStore\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:825\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).eventFeed.func2\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:610\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:57"]
[2021-07-21T06:26:24.999Z] [2021/07/21 14:24:20.049 +08:00] [ERROR] [client.go:462] ["check tikv version failed"] [error="[CDC:ErrVersionIncompatible]version is incompatible: TiKV 3.1.0 is not supported, require minimal version 4.0.0-rc.1"] [errorVerbose="[CDC:ErrVersionIncompatible]version is incompatible: TiKV 3.1.0 is not supported, require minimal version 4.0.0-rc.1\ngithub.com/pingcap/errors.AddStack\n\t/go/pkg/mod/github.com/pingcap/[email protected]/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByArgs\n\t/go/pkg/mod/github.com/pingcap/[email protected]/normalize.go:156\ngithub.com/pingcap/ticdc/pkg/version.CheckStoreVersion\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/pkg/version/check.go:140\ngithub.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:456\ngithub.com/pingcap/ticdc/pkg/retry.Run.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:32\ngithub.com/cenkalti/backoff.RetryNotify\n\t/go/pkg/mod/github.com/cenkalti/[email protected]+incompatible/retry.go:37\ngithub.com/cenkalti/backoff.Retry\n\t/go/pkg/mod/github.com/cenkalti/[email protected]+incompatible/retry.go:24\ngithub.com/pingcap/ticdc/pkg/retry.Run\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:31\ngithub.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:450\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).requestRegionToStore\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:825\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).eventFeed.func2\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:610\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:57\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357"] [storeID=1] [stack="github.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:462\ngithub.com/pingcap/ticdc/pkg/retry.Run.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:32\ngithub.com/cenkalti/backoff.RetryNotify\n\t/go/pkg/mod/github.com/cenkalti/[email protected]+incompatible/retry.go:37\ngithub.com/cenkalti/backoff.Retry\n\t/go/pkg/mod/github.com/cenkalti/[email protected]+incompatible/retry.go:24\ngithub.com/pingcap/ticdc/pkg/retry.Run\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:31\ngithub.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:450\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).requestRegionToStore\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:825\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).eventFeed.func2\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:610\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:57"]
[2021-07-21T06:26:24.999Z] [2021/07/21 14:24:20.152 +08:00] [ERROR] [client.go:462] ["check tikv version failed"] [error="[CDC:ErrVersionIncompatible]version is incompatible: TiKV 2.1.10 is not supported, require minimal version 4.0.0-rc.1"] [errorVerbose="[CDC:ErrVersionIncompatible]version is incompatible: TiKV 2.1.10 is not supported, require minimal version 4.0.0-rc.1\ngithub.com/pingcap/errors.AddStack\n\t/go/pkg/mod/github.com/pingcap/[email protected]/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByArgs\n\t/go/pkg/mod/github.com/pingcap/[email protected]/normalize.go:156\ngithub.com/pingcap/ticdc/pkg/version.CheckStoreVersion\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/pkg/version/check.go:140\ngithub.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:456\ngithub.com/pingcap/ticdc/pkg/retry.Run.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:32\ngithub.com/cenkalti/backoff.RetryNotify\n\t/go/pkg/mod/github.com/cenkalti/[email protected]+incompatible/retry.go:37\ngithub.com/cenkalti/backoff.Retry\n\t/go/pkg/mod/github.com/cenkalti/[email protected]+incompatible/retry.go:24\ngithub.com/pingcap/ticdc/pkg/retry.Run\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:31\ngithub.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:450\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).requestRegionToStore\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:825\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).eventFeed.func2\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:610\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:57\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357"] [storeID=1] [stack="github.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:462\ngithub.com/pingcap/ticdc/pkg/retry.Run.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:32\ngithub.com/cenkalti/backoff.RetryNotify\n\t/go/pkg/mod/github.com/cenkalti/[email protected]+incompatible/retry.go:37\ngithub.com/cenkalti/backoff.Retry\n\t/go/pkg/mod/github.com/cenkalti/[email protected]+incompatible/retry.go:24\ngithub.com/pingcap/ticdc/pkg/retry.Run\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:31\ngithub.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:450\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).requestRegionToStore\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:825\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).eventFeed.func2\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:610\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:57"]
[2021-07-21T06:26:25.000Z] [2021/07/21 14:24:20.191 +08:00] [ERROR] [client.go:462] ["check tikv version failed"] [error="[CDC:ErrVersionIncompatible]version is incompatible: TiKV 3.0.10 is not supported, require minimal version 4.0.0-rc.1"] [errorVerbose="[CDC:ErrVersionIncompatible]version is incompatible: TiKV 3.0.10 is not supported, require minimal version 4.0.0-rc.1\ngithub.com/pingcap/errors.AddStack\n\t/go/pkg/mod/github.com/pingcap/[email protected]/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByArgs\n\t/go/pkg/mod/github.com/pingcap/[email protected]/normalize.go:156\ngithub.com/pingcap/ticdc/pkg/version.CheckStoreVersion\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/pkg/version/check.go:140\ngithub.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:456\ngithub.com/pingcap/ticdc/pkg/retry.Run.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:32\ngithub.com/cenkalti/backoff.RetryNotify\n\t/go/pkg/mod/github.com/cenkalti/[email protected]+incompatible/retry.go:37\ngithub.com/cenkalti/backoff.Retry\n\t/go/pkg/mod/github.com/cenkalti/[email protected]+incompatible/retry.go:24\ngithub.com/pingcap/ticdc/pkg/retry.Run\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:31\ngithub.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:450\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).requestRegionToStore\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:825\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).eventFeed.func2\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:610\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:57\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357"] [storeID=1] [stack="github.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:462\ngithub.com/pingcap/ticdc/pkg/retry.Run.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:32\ngithub.com/cenkalti/backoff.RetryNotify\n\t/go/pkg/mod/github.com/cenkalti/[email protected]+incompatible/retry.go:37\ngithub.com/cenkalti/backoff.Retry\n\t/go/pkg/mod/github.com/cenkalti/[email protected]+incompatible/retry.go:24\ngithub.com/pingcap/ticdc/pkg/retry.Run\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:31\ngithub.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:450\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).requestRegionToStore\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:825\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).eventFeed.func2\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:610\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:57"]
[2021-07-21T06:26:25.000Z] [2021/07/21 14:24:20.256 +08:00] [ERROR] [client.go:462] ["check tikv version failed"] [error="[CDC:ErrVersionIncompatible]version is incompatible: TiKV 3.1.0 is not supported, require minimal version 4.0.0-rc.1"] [errorVerbose="[CDC:ErrVersionIncompatible]version is incompatible: TiKV 3.1.0 is not supported, require minimal version 4.0.0-rc.1\ngithub.com/pingcap/errors.AddStack\n\t/go/pkg/mod/github.com/pingcap/[email protected]/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByArgs\n\t/go/pkg/mod/github.com/pingcap/[email protected]/normalize.go:156\ngithub.com/pingcap/ticdc/pkg/version.CheckStoreVersion\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/pkg/version/check.go:140\ngithub.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:456\ngithub.com/pingcap/ticdc/pkg/retry.Run.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:32\ngithub.com/cenkalti/backoff.RetryNotify\n\t/go/pkg/mod/github.com/cenkalti/[email protected]+incompatible/retry.go:37\ngithub.com/cenkalti/backoff.Retry\n\t/go/pkg/mod/github.com/cenkalti/[email protected]+incompatible/retry.go:24\ngithub.com/pingcap/ticdc/pkg/retry.Run\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:31\ngithub.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:450\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).requestRegionToStore\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:825\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).eventFeed.func2\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:610\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:57\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357"] [storeID=1] [stack="github.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:462\ngithub.com/pingcap/ticdc/pkg/retry.Run.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:32\ngithub.com/cenkalti/backoff.RetryNotify\n\t/go/pkg/mod/github.com/cenkalti/[email protected]+incompatible/retry.go:37\ngithub.com/cenkalti/backoff.Retry\n\t/go/pkg/mod/github.com/cenkalti/[email protected]+incompatible/retry.go:24\ngithub.com/pingcap/ticdc/pkg/retry.Run\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry.go:31\ngithub.com/pingcap/ticdc/cdc/kv.(*CDCClient).newStream\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:450\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).requestRegionToStore\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:825\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).eventFeed.func2\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:610\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:57"]
[2021-07-21T06:26:25.001Z] [2021/07/21 14:24:21.030 +08:00] [ERROR] [client_test.go:1582] ["mock server error"] [error="rpc error: code = Canceled desc = context canceled"] [stack="github.com/pingcap/ticdc/cdc/kv.(*etcdSuite).TestIncompatibleTiKV.func2\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client_test.go:1582\ngithub.com/pingcap/ticdc/cdc/kv.(*mockChangeDataService).EventFeed.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client_test.go:232"]
[2021-07-21T06:26:25.001Z] {"level":"error","ts":"2021-07-21T14:25:01.282+0800","caller":"grpclog/grpclog.go:75","msg":"transport: Got too many pings from the client, closing the connection.","stacktrace":"google.golang.org/grpc/grpclog.Errorf\n\t/go/pkg/mod/google.golang.org/[email protected]/grpclog/grpclog.go:75\ngoogle.golang.org/grpc/internal/transport.errorf\n\t/go/pkg/mod/google.golang.org/[email protected]/internal/transport/log.go:42\ngoogle.golang.org/grpc/internal/transport.(*http2Server).handlePing\n\t/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:727\ngoogle.golang.org/grpc/internal/transport.(*http2Server).HandleStreams\n\t/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:499\ngoogle.golang.org/grpc.(*Server).serveStreams\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:718\ngoogle.golang.org/grpc.(*Server).handleRawConn.func1\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:679"}
[2021-07-21T06:26:25.001Z] {"level":"error","ts":"2021-07-21T14:25:01.282+0800","caller":"grpclog/grpclog.go:75","msg":"transport: Got too many pings from the client, closing the connection.","stacktrace":"google.golang.org/grpc/grpclog.Errorf\n\t/go/pkg/mod/google.golang.org/[email protected]/grpclog/grpclog.go:75\ngoogle.golang.org/grpc/internal/transport.errorf\n\t/go/pkg/mod/google.golang.org/[email protected]/internal/transport/log.go:42\ngoogle.golang.org/grpc/internal/transport.(*http2Server).handlePing\n\t/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:727\ngoogle.golang.org/grpc/internal/transport.(*http2Server).HandleStreams\n\t/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:499\ngoogle.golang.org/grpc.(*Server).serveStreams\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:718\ngoogle.golang.org/grpc.(*Server).handleRawConn.func1\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:679"}
[2021-07-21T06:26:25.001Z] {"level":"error","ts":"2021-07-21T14:25:01.282+0800","caller":"grpclog/grpclog.go:75","msg":"transport: loopyWriter.run returning. Err: transport: Connection closing","stacktrace":"google.golang.org/grpc/grpclog.Errorf\n\t/go/pkg/mod/google.golang.org/[email protected]/grpclog/grpclog.go:75\ngoogle.golang.org/grpc/internal/transport.errorf\n\t/go/pkg/mod/google.golang.org/[email protected]/internal/transport/log.go:42\ngoogle.golang.org/grpc/internal/transport.newHTTP2Server.func2\n\t/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:297"}
[2021-07-21T06:26:25.001Z] {"level":"error","ts":"2021-07-21T14:25:01.282+0800","caller":"grpclog/grpclog.go:75","msg":"transport: loopyWriter.run returning. Err: transport: Connection closing","stacktrace":"google.golang.org/grpc/grpclog.Errorf\n\t/go/pkg/mod/google.golang.org/[email protected]/grpclog/grpclog.go:75\ngoogle.golang.org/grpc/internal/transport.errorf\n\t/go/pkg/mod/google.golang.org/[email protected]/internal/transport/log.go:42\ngoogle.golang.org/grpc/internal/transport.newHTTP2Server.func2\n\t/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:297"}
[2021-07-21T06:26:25.002Z] [2021/07/21 14:25:01.283 +08:00] [ERROR] [client_test.go:2762] ["mock server error"] [error="rpc error: code = Canceled desc = context canceled"] [stack="github.com/pingcap/ticdc/cdc/kv.(*etcdSuite).testKVClientForceReconnect.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client_test.go:2762\ngithub.com/pingcap/ticdc/cdc/kv.(*mockChangeDataService).EventFeed.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client_test.go:232"]
[2021-07-21T06:26:25.002Z] [2021/07/21 14:25:01.283 +08:00] [ERROR] [client.go:1282] ["failed to receive from stream"] [addr=127.0.0.1:40785] [storeID=1] [error="rpc error: code = Unavailable desc = transport is closing"] [stack="github.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).receiveFromStream\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:1282\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).requestRegionToStore.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:857\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:57"]
[2021-07-21T06:26:25.002Z] 
[2021-07-21T06:26:25.002Z] ----------------------------------------------------------------------
[2021-07-21T06:26:25.002Z] PANIC: client_test.go:2734: etcdSuite.TestKVClientForceReconnect
[2021-07-21T06:26:25.002Z] 
[2021-07-21T06:26:25.002Z] ... Panic: send on closed channel (PC=0x1410FA1)
[2021-07-21T06:26:25.002Z] 
[2021-07-21T06:26:25.002Z] /usr/local/go/src/runtime/panic.go:679
[2021-07-21T06:26:25.002Z]   in gopanic
[2021-07-21T06:26:25.002Z] /usr/local/go/src/runtime/chan.go:187
[2021-07-21T06:26:25.002Z]   in chansend
[2021-07-21T06:26:25.002Z] /usr/local/go/src/runtime/chan.go:127
[2021-07-21T06:26:25.002Z]   in chansend1
[2021-07-21T06:26:25.002Z] client_test.go:2804
[2021-07-21T06:26:25.002Z]   in etcdSuite.testKVClientForceReconnect
[2021-07-21T06:26:25.002Z] client_test.go:2739
[2021-07-21T06:26:25.002Z]   in etcdSuite.TestKVClientForceReconnect
[2021-07-21T06:26:25.002Z] /usr/local/go/src/reflect/value.go:321
[2021-07-21T06:26:25.002Z]   in Value.Call
[2021-07-21T06:26:25.002Z] /go/pkg/mod/github.com/pingcap/[email protected]/check.go:850
[2021-07-21T06:26:25.002Z]   in suiteRunner.forkTest.func1
[2021-07-21T06:26:25.002Z] /go/pkg/mod/github.com/pingcap/[email protected]/check.go:739
[2021-07-21T06:26:25.002Z]   in suiteRunner.forkCall.func1
[2021-07-21T06:26:25.002Z] /usr/local/go/src/runtime/asm_amd64.s:1357
[2021-07-21T06:26:25.002Z]   in goexit
[2021-07-21T06:26:25.002Z] [2021/07/21 14:25:27.211 +08:00] [ERROR] [client.go:1380] ["received an event but neither pending region nor running region was found"] [regionID=3] [requestID=74] [addr=127.0.0.1:40547] [stack="github.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).sendRegionChangeEvent\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:1380\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).receiveFromStream\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:1326\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).requestRegionToStore.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:857\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:57"]
[2021-07-21T06:26:25.002Z] [2021/07/21 14:26:05.226 +08:00] [ERROR] [client_test.go:1371] ["mock server error"] [error="rpc error: code = Canceled desc = context canceled"] [stack="github.com/pingcap/ticdc/cdc/kv.(*etcdSuite).TestStreamRecvWithErrorAndResolvedGoBack.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client_test.go:1371\ngithub.com/pingcap/ticdc/cdc/kv.(*mockChangeDataService).EventFeed.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client_test.go:232"]
[2021-07-21T06:26:25.002Z] [2021/07/21 14:26:05.226 +08:00] [ERROR] [client.go:1282] ["failed to receive from stream"] [addr=127.0.0.1:39815] [storeID=1] [error=] [errorVerbose="\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).receiveFromStream\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:1271\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).requestRegionToStore.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:857\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:57\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357"] [stack="github.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).receiveFromStream\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:1282\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).requestRegionToStore.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:857\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:57"]
[2021-07-21T06:26:25.003Z] [2021/07/21 14:26:05.300 +08:00] [ERROR] [client_test.go:1371] ["mock server error"] [error="rpc error: code = Canceled desc = context canceled"] [stack="github.com/pingcap/ticdc/cdc/kv.(*etcdSuite).TestStreamRecvWithErrorAndResolvedGoBack.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client_test.go:1371\ngithub.com/pingcap/ticdc/cdc/kv.(*mockChangeDataService).EventFeed.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client_test.go:232"]
[2021-07-21T06:26:25.003Z] [2021/07/21 14:26:07.858 +08:00] [ERROR] [client.go:1282] ["failed to receive from stream"] [addr=127.0.0.1:43230] [storeID=1] [error=EOF] [stack="github.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).receiveFromStream\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:1282\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).requestRegionToStore.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:857\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:57"]
[2021-07-21T06:26:25.003Z] [2021/07/21 14:26:11.870 +08:00] [ERROR] [client.go:1282] ["failed to receive from stream"] [addr=127.0.0.1:40169] [storeID=1] [error="injected stream recv error"] [errorVerbose="injected stream recv error\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).receiveFromStream\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:1271\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).requestRegionToStore.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:857\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:57\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357"] [stack="github.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).receiveFromStream\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:1282\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).requestRegionToStore.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:857\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:57"]
[2021-07-21T06:26:25.003Z] [2021/07/21 14:26:15.420 +08:00] [ERROR] [client.go:1282] ["failed to receive from stream"] [addr=127.0.0.1:43654] [storeID=1] [error="rpc error: code = Unavailable desc = transport is closing"] [stack="github.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).receiveFromStream\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:1282\ngithub.com/pingcap/ticdc/cdc/kv.(*eventFeedSession).requestRegionToStore.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client.go:857\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:57"]
[2021-07-21T06:26:25.003Z] [2021/07/21 14:26:15.651 +08:00] [ERROR] [client_test.go:1190] ["mock server error"] [error="rpc error: code = Canceled desc = context canceled"] [stack="github.com/pingcap/ticdc/cdc/kv.(*etcdSuite).TestStreamSendWithError.func3\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client_test.go:1190\ngithub.com/pingcap/ticdc/cdc/kv.(*mockChangeDataService).EventFeed.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client_test.go:232"]
[2021-07-21T06:26:25.003Z] OOPS: 61 passed, 1 PANICKED
[2021-07-21T06:26:25.003Z] --- FAIL: Test (184.50s)
[2021-07-21T06:26:25.003Z] FAIL
[2021-07-21T06:26:25.003Z] coverage: 55.7% of statements
[2021-07-21T06:26:25.003Z] FAIL	github.com/pingcap/ticdc/cdc/kv	184.959s

  1. Versions of the cluster
@amyangfei amyangfei added the type/bug The issue is confirmed as a bug. label Jul 22, 2021
@amyangfei amyangfei mentioned this issue Jul 22, 2021
Closed
@asddongmen asddongmen added component/test Unit tests and integration tests component. difficulty/easy Easy task. labels Jul 27, 2021
@overvenus overvenus self-assigned this Jul 29, 2021
@Rustin170506
Copy link
Member

Another fail: https://ci2.pingcap.net/blue/organizations/jenkins/cdc_ghpr_test/detail/cdc_ghpr_test/5072/pipeline

@overvenus
Copy link
Member

Another failed case: https://ci2.pingcap.net/blue/organizations/jenkins/cdc_ghpr_integration_test/detail/cdc_ghpr_integration_test/2812/pipeline#step-435-log-37

[2021-08-12T09:21:19.499Z] FAIL: client_test.go:2833: etcdSuite.TestKVClientForceReconnect
[2021-08-12T09:21:19.499Z] 
[2021-08-12T09:21:19.499Z] client_test.go:2838:
[2021-08-12T09:21:19.499Z]     // test kv client v1
[2021-08-12T09:21:19.499Z]     s.testKVClientForceReconnect(c)
[2021-08-12T09:21:19.499Z] client_test.go:297:
[2021-08-12T09:21:19.499Z]     c.Assert(err, check.IsNil)
[2021-08-12T09:21:19.499Z] ... value *errors.Error = &errors.Error{code:0, codeText:"CDC:ErrReachMaxTry", message:"reach maximum try: %d", redactArgsPos:[]int(nil), cause:(*errors.fundamental)(0xc001140ac0), args:[]interface {}{20}, file:"/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry_with_opt.go", line:65} ("[CDC:ErrReachMaxTry]reach maximum try: 20")

@asddongmen
Copy link
Contributor

https://ci2.pingcap.net/blue/organizations/jenkins/cdc_ghpr_test/detail/cdc_ghpr_test/5901/pipeline
Another failed case:

AIL: client_test.go:3030: etcdSuite.TestKVClientForceReconnect

[2021-08-18T09:06:59.989Z] 

[2021-08-18T09:06:59.989Z] client_test.go:3044:

[2021-08-18T09:06:59.989Z]     s.testKVClientForceReconnect(c)

[2021-08-18T09:06:59.989Z] client_test.go:303:

[2021-08-18T09:06:59.989Z]     c.Assert(err, check.IsNil)

[2021-08-18T09:06:59.989Z] ... value *errors.Error = &errors.Error{code:0, codeText:"CDC:ErrReachMaxTry", message:"reach maximum try: %d", redactArgsPos:[]int(nil), cause:(*errors.fundamental)(0xc000e96270), args:[]interface {}{20}, file:"/home/jenkins/agent/workspace/cdc_ghpr_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry_with_opt.go", line:65} ("[CDC:ErrReachMaxTry]reach maximum try: 20")

@Rustin170506
Copy link
Member

Another failed case: https://ci2.pingcap.net/blue/organizations/jenkins/cdc_ghpr_test/detail/cdc_ghpr_test/6215/pipeline

@amyangfei amyangfei added difficulty/medium Medium task. and removed difficulty/easy Easy task. labels Aug 27, 2021
@ben1009
Copy link
Contributor

ben1009 commented Sep 14, 2021

https://ci2.pingcap.net/blue/organizations/jenkins/cdc_ghpr_integration_test/detail/cdc_ghpr_integration_test/4208/pipeline

[2021-09-13T10:34:12.204Z] ----------------------------------------------------------------------

[2021-09-13T10:34:12.204Z] FAIL: client_test.go:3026: etcdSuite.TestKVClientForceReconnect

[2021-09-13T10:34:12.204Z] 

[2021-09-13T10:34:12.204Z] client_test.go:3037:

[2021-09-13T10:34:12.204Z]     s.testKVClientForceReconnect(c)

[2021-09-13T10:34:12.204Z] client_test.go:301:

[2021-09-13T10:34:12.204Z]     c.Assert(err, check.IsNil)

[2021-09-13T10:34:12.205Z] ... value *errors.Error = &errors.Error{code:0, codeText:"CDC:ErrReachMaxTry", message:"reach maximum try: %d", redactArgsPos:[]int(nil), cause:(*errors.fundamental)(0xc001214168), args:[]interface {}{20}, file:"/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry_with_opt.go", line:65} ("[CDC:ErrReachMaxTry]reach maximum try: 20: request id 64 is not larger than 64")

@Rustin170506
Copy link
Member

https://ci2.pingcap.net/blue/organizations/jenkins/cdc_ghpr_kafka_integration_test/detail/cdc_ghpr_kafka_integration_test/3677/pipeline/112

[2021-09-14T08:41:32.242Z] 

[2021-09-14T08:41:32.242Z] ----------------------------------------------------------------------

[2021-09-14T08:41:32.242Z] FAIL: client_test.go:3026: etcdSuite.TestKVClientForceReconnect

[2021-09-14T08:41:32.242Z] 

[2021-09-14T08:41:32.242Z] client_test.go:3040:

[2021-09-14T08:41:32.242Z]     s.testKVClientForceReconnect(c)

[2021-09-14T08:41:32.242Z] client_test.go:301:

[2021-09-14T08:41:32.242Z]     c.Assert(err, check.IsNil)

[2021-09-14T08:41:32.242Z] ... value *errors.Error = &errors.Error{code:0, codeText:"CDC:ErrReachMaxTry", message:"reach maximum try: %d", redactArgsPos:[]int(nil), cause:(*errors.fundamental)(0xc000988168), args:[]interface {}{20}, file:"/home/jenkins/agent/workspace/cdc_ghpr_kafka_integration_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry_with_opt.go", line:65} ("[CDC:ErrReachMaxTry]reach maximum try: 20: request id 69 is not larger than 69")

[2021-09-14T08:41:32.242Z] 

[2021-09-14T08:41:32.242Z] [2021/09/14 16:40:42.395 +08:00] [ERROR] [client_test.go:3311] ["mock server error"] [error="rpc error: code = Canceled desc = context canceled"] [stack="github.com/pingcap/ticdc/cdc/kv.(*etcdSuite).TestRegionWorkerExitWhenIsIdle.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_kafka_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client_test.go:3311\ngithub.com/pingcap/ticdc/cdc/kv.(*mockChangeDataService).EventFeed.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_kafka_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client_test.go:236"]

[2021-09-14T08:41:32.242Z] [2021/09/14 16:41:05.049 +08:00] [ERROR] [client_test.go:1411] ["mock server error"] [error="rpc error: code = Canceled desc = context canceled"] [stack="github.com/pingcap/ticdc/cdc/kv.(*etcdSuite).TestStreamRecvWithErrorAndResolvedGoBack.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_kafka_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client_test.go:1411\ngithub.com/pingcap/ticdc/cdc/kv.(*mockChangeDataService).EventFeed.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_kafka_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client_test.go:236"]

[2021-09-14T08:41:32.242Z] [2021/09/14 16:41:05.154 +08:00] [ERROR] [client_test.go:1411] ["mock server error"] [error="rpc error: code = Canceled desc = context canceled"] [stack="github.com/pingcap/ticdc/cdc/kv.(*etcdSuite).TestStreamRecvWithErrorAndResolvedGoBack.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_kafka_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client_test.go:1411\ngithub.com/pingcap/ticdc/cdc/kv.(*mockChangeDataService).EventFeed.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_kafka_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client_test.go:236"]

[2021-09-14T08:41:32.242Z] [2021/09/14 16:41:16.006 +08:00] [ERROR] [client_test.go:1221] ["mock server error"] [error="rpc error: code = Canceled desc = context canceled"] [stack="github.com/pingcap/ticdc/cdc/kv.(*etcdSuite).TestStreamSendWithError.func3\n\t/home/jenkins/agent/workspace/cdc_ghpr_kafka_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client_test.go:1221\ngithub.com/pingcap/ticdc/cdc/kv.(*mockChangeDataService).EventFeed.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_kafka_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client_test.go:236"]

[2021-09-14T08:41:32.243Z] OOPS: 62 passed, 2 FAILED

[2021-09-14T08:41:32.243Z] --- FAIL: Test (99.32s)

@overvenus
Copy link
Member

https://ci2.pingcap.net/blue/organizations/jenkins/cdc_ghpr_integration_test/detail/cdc_ghpr_integration_test/4257/pipeline#step-236-log-38

[2021-09-14T16:22:27.292Z] FAIL: client_test.go:3026: etcdSuite.TestKVClientForceReconnect

[2021-09-14T16:22:27.292Z] 

[2021-09-14T16:22:27.292Z] client_test.go:3037:

[2021-09-14T16:22:27.292Z]     s.testKVClientForceReconnect(c)

[2021-09-14T16:22:27.292Z] client_test.go:301:

[2021-09-14T16:22:27.292Z]     c.Assert(err, check.IsNil)

[2021-09-14T16:22:27.292Z] ... value *errors.Error = &errors.Error{code:0, codeText:"CDC:ErrReachMaxTry", message:"reach maximum try: %d", redactArgsPos:[]int(nil), cause:(*errors.fundamental)(0xc000ce03d8), args:[]interface {}{20}, file:"/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry_with_opt.go", line:65} ("[CDC:ErrReachMaxTry]reach maximum try: 20: request id 64 is not larger than 65")

@asddongmen
Copy link
Contributor

https://ci2.pingcap.net/blue/organizations/jenkins/cdc_ghpr_integration_test/detail/cdc_ghpr_integration_test/4612/pipeline/111

----------------------------------------------------------------------

[2021-09-24T09:42:00.425Z] FAIL: client_test.go:3027: etcdSuite.TestKVClientForceReconnect

[2021-09-24T09:42:00.425Z] 

[2021-09-24T09:42:00.425Z] client_test.go:3041:

[2021-09-24T09:42:00.425Z]     s.testKVClientForceReconnect(c)

[2021-09-24T09:42:00.425Z] client_test.go:302:

[2021-09-24T09:42:00.425Z]     c.Assert(err, check.IsNil)

[2021-09-24T09:42:00.425Z] ... value *errors.Error = &errors.Error{code:0, codeText:"CDC:ErrReachMaxTry", message:"reach maximum try: %d", redactArgsPos:[]int(nil), cause:(*errors.fundamental)(0xc000242480), args:[]interface {}{20}, file:"/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry_with_opt.go", line:65} ("[CDC:ErrReachMaxTry]reach maximum try: 20")

[2021-09-24T09:42:00.425Z] 

[2021-09-24T09:42:00.425Z] [2021/09/24 17:41:15.689 +08:00] [ERROR] [client_test.go:3312] ["mock server error"] [error="rpc error: code = Canceled desc = context canceled"] [stack="github.com/pingcap/ticdc/cdc/kv.(*etcdSuite).TestRegionWorkerExitWhenIsIdle.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client_test.go:3312\ngithub.com/pingcap/ticdc/cdc/kv.(*mockChangeDataService).EventFeed.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client_test.go:237"]

[2021-09-24T09:42:00.425Z] [2021/09/24 17:41:38.385 +08:00] [ERROR] [client_test.go:1412] ["mock server error"] [error="rpc error: code = Canceled desc = context canceled"] [stack="github.com/pingcap/ticdc/cdc/kv.(*etcdSuite).TestStreamRecvWithErrorAndResolvedGoBack.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client_test.go:1412\ngithub.com/pingcap/ticdc/cdc/kv.(*mockChangeDataService).EventFeed.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client_test.go:237"]

[2021-09-24T09:42:00.425Z] [2021/09/24 17:41:38.478 +08:00] [ERROR] [client_test.go:1412] ["mock server error"] [error="rpc error: code = Canceled desc = context canceled"] [stack="github.com/pingcap/ticdc/cdc/kv.(*etcdSuite).TestStreamRecvWithErrorAndResolvedGoBack.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client_test.go:1412\ngithub.com/pingcap/ticdc/cdc/kv.(*mockChangeDataService).EventFeed.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client_test.go:237"]

[2021-09-24T09:42:00.425Z] [2021/09/24 17:41:49.496 +08:00] [ERROR] [client_test.go:1222] ["mock server error"] [error="rpc error: code = Canceled desc = context canceled"] [stack="github.com/pingcap/ticdc/cdc/kv.(*etcdSuite).TestStreamSendWithError.func3\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client_test.go:1222\ngithub.com/pingcap/ticdc/cdc/kv.(*mockChangeDataService).EventFeed.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_integration_test/go/src/github.com/pingcap/ticdc/cdc/kv/client_test.go:237"]

[2021-09-24T09:42:00.425Z] OOPS: 62 passed, 2 FAILED

@Rustin170506
Copy link
Member

https://ci2.pingcap.net/blue/organizations/jenkins/cdc_ghpr_test/detail/cdc_ghpr_test/9430/pipeline

[2021-10-21T08:12:37.415Z] ----------------------------------------------------------------------

[2021-10-21T08:12:37.415Z] FAIL: client_test.go:3067: clientSuite.TestKVClientForceReconnect

[2021-10-21T08:12:37.415Z] 

[2021-10-21T08:12:37.415Z] client_test.go:3071:

[2021-10-21T08:12:37.415Z]     s.testKVClientForceReconnect(c)

[2021-10-21T08:12:37.415Z] client_test.go:317:

[2021-10-21T08:12:37.415Z]     c.Assert(err, check.IsNil)

[2021-10-21T08:12:37.415Z] ... value *errors.Error = &errors.Error{code:0, codeText:"CDC:ErrReachMaxTry", message:"reach maximum try: %d", redactArgsPos:[]int(nil), cause:(*errors.fundamental)(0xc0005d4288), args:[]interface {}{20}, file:"/home/jenkins/agent/workspace/cdc_ghpr_test/go/src/github.com/pingcap/ticdc/pkg/retry/retry_with_opt.go", line:65} ("[CDC:ErrReachMaxTry]reach maximum try: 20: request id 63 is not larger than 63")

[2021-10-21T08:12:37.415Z] 

[2021-10-21T08:12:37.415Z] [2021/10/21 16:11:41.206 +08:00] [ERROR] [client_test.go:3425] ["mock server error"] [error="rpc error: code = Canceled desc = context canceled"] [stack="github.com/pingcap/ticdc/cdc/kv.(*clientSuite).TestPrewriteNotMatchError.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_test/go/src/github.com/pingcap/ticdc/cdc/kv/client_test.go:3425\ngithub.com/pingcap/ticdc/cdc/kv.(*mockChangeDataService).EventFeed.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_test/go/src/github.com/pingcap/ticdc/cdc/kv/client_test.go:252"]

[2021-10-21T08:12:37.415Z] [2021/10/21 16:11:42.213 +08:00] [ERROR] [client_test.go:3510] ["mock server error"] [error="rpc error: code = Canceled desc = context canceled"] [stack="github.com/pingcap/ticdc/cdc/kv.(*clientSuite).TestPrewriteNotMatchError.func4\n\t/home/jenkins/agent/workspace/cdc_ghpr_test/go/src/github.com/pingcap/ticdc/cdc/kv/client_test.go:3510\ngithub.com/pingcap/ticdc/cdc/kv.(*mockChangeDataService).EventFeed.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_test/go/src/github.com/pingcap/ticdc/cdc/kv/client_test.go:252"]

[2021-10-21T08:12:37.415Z] [2021/10/21 16:11:47.979 +08:00] [ERROR] [client_test.go:3337] ["mock server error"] [error="rpc error: code = Canceled desc = context canceled"] [stack="github.com/pingcap/ticdc/cdc/kv.(*clientSuite).TestRegionWorkerExitWhenIsIdle.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_test/go/src/github.com/pingcap/ticdc/cdc/kv/client_test.go:3337\ngithub.com/pingcap/ticdc/cdc/kv.(*mockChangeDataService).EventFeed.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_test/go/src/github.com/pingcap/ticdc/cdc/kv/client_test.go:252"]

[2021-10-21T08:12:37.415Z] [2021/10/21 16:12:14.686 +08:00] [ERROR] [client_test.go:1499] ["mock server error"] [error="rpc error: code = Canceled desc = context canceled"] [stack="github.com/pingcap/ticdc/cdc/kv.(*clientSuite).TestStreamRecvWithErrorAndResolvedGoBack.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_test/go/src/github.com/pingcap/ticdc/cdc/kv/client_test.go:1499\ngithub.com/pingcap/ticdc/cdc/kv.(*mockChangeDataService).EventFeed.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_test/go/src/github.com/pingcap/ticdc/cdc/kv/client_test.go:252"]

[2021-10-21T08:12:37.415Z] [2021/10/21 16:12:14.831 +08:00] [ERROR] [client_test.go:1499] ["mock server error"] [error="rpc error: code = Canceled desc = context canceled"] [stack="github.com/pingcap/ticdc/cdc/kv.(*clientSuite).TestStreamRecvWithErrorAndResolvedGoBack.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_test/go/src/github.com/pingcap/ticdc/cdc/kv/client_test.go:1499\ngithub.com/pingcap/ticdc/cdc/kv.(*mockChangeDataService).EventFeed.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_test/go/src/github.com/pingcap/ticdc/cdc/kv/client_test.go:252"]

[2021-10-21T08:12:37.415Z] [2021/10/21 16:12:23.663 +08:00] [ERROR] [client_test.go:1309] ["mock server error"] [error="rpc error: code = Canceled desc = context canceled"] [stack="github.com/pingcap/ticdc/cdc/kv.(*clientSuite).TestStreamSendWithError.func3\n\t/home/jenkins/agent/workspace/cdc_ghpr_test/go/src/github.com/pingcap/ticdc/cdc/kv/client_test.go:1309\ngithub.com/pingcap/ticdc/cdc/kv.(*mockChangeDataService).EventFeed.func1\n\t/home/jenkins/agent/workspace/cdc_ghpr_test/go/src/github.com/pingcap/ticdc/cdc/kv/client_test.go:252"]

[2021-10-21T08:12:37.415Z] OOPS: 47 passed, 1 FAILED

[2021-10-21T08:12:37.415Z] --- FAIL: Test (96.48s)

@maxshuang maxshuang mentioned this issue Nov 29, 2021
Merged
This was referenced Nov 29, 2021
Closed
Closed
@maxshuang maxshuang linked a pull request Nov 29, 2021 that will close this issue
kvclient(ticdc): fix kvclient takes too long time to recover (#3612) #3659
Closed
This was referenced Nov 29, 2021
Merged
Merged
Merged
@maxshuang
Copy link
Contributor

A common solution has been pr in https://github.com/pingcap/ticdc/pull/3612. If you find this issue happens again after this pr, pls reopen the this issue and @ me.

@AkiraXie AkiraXie added the area/ticdc Issues or PRs related to TiCDC. label Mar 9, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@overvenus overvenus

@maxshuang maxshuang

Labels
area/ticdc Issues or PRs related to TiCDC. component/test Unit tests and integration tests component. difficulty/medium Medium task. severity/major type/bug The issue is confirmed as a bug.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

kvclient(ticdc): fix kvclient takes too long time to recover (#3612) ti-chi-bot/tiflow
7 participants
@amyangfei @ben1009 @overvenus @maxshuang @asddongmen @Rustin170506 @AkiraXie