cluster: Support TiKV-CDC component #2022

pingyu · 2022-08-15T12:28:30Z

Issue Number: #1999

Signed-off-by: pingyu [email protected]

What problem does this PR solve?

#1999

What is changed and how it works?

TiUP cluster supports the TiKV-CDC component.

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)

➜  tiup-cluster deploy up v6.2.0 up.yaml
...
...
Cluster `up` deployed successfully, you can start it with command: `tiup-cluster start up --init`
➜  tiup-cluster start up -y
Starting cluster up...
...
...
Started cluster `up` successfully
➜  tiup-cluster display up
Cluster type:       tidb
Cluster name:       up
Cluster version:    v6.2.0
Deploy user:        root
SSH type:           builtin
...
...
Total nodes: 14
➜ tikv-cdc cli --pd 10.2.x.x:2379 capture list
[
  {
    "id": "3e2ede2a-13ad-4ded-8ac6-1c5f315dfd5e",
    "is-owner": false,
    "address": "10.2.x.x:8600"
  },
  {
    "id": "7b68a84e-63d5-4fd2-89a6-90aa15a7069c",
    "is-owner": false,
    "address": "10.2.x.x:8600"
  },
  {
    "id": "a05ae66d-6dcf-4c28-b0d2-0cab3a221dfc",
    "is-owner": true,
    "address": "10.2.x.x:8600"
  }
]
➜

Code changes

Has exported variable/fields change
- spec.TiKVCDCSpec, scripts.TiKVCDCScript, api.TiKVCDCOpenAPIClient
- spec.ServerConfigs.TiKVCDC, spec.Specification.TiKVCDCServers, instance.ComponentTiKVCDC

Side effects

Increased code complexity

Related changes

Need to update the documentation
- TBD

Release notes:

TiUP cluster supports the `TiKV-CDC` component.

Issue Number: pingcap#1999 Signed-off-by: pingyu <[email protected]>

ti-chi-bot · 2022-08-15T12:28:32Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

nexustar

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

Signed-off-by: pingyu <[email protected]>

codecov-commenter · 2022-08-15T12:55:33Z

Codecov Report

Base: 47.67% // Head: 51.46% // Increases project coverage by +3.79% 🎉

Coverage data is based on head (5304c07) compared to base (fb81eac).
Patch coverage: 36.87% of modified lines in pull request are covered.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #2022      +/-   ##
==========================================
+ Coverage   47.67%   51.46%   +3.79%     
==========================================
  Files         309      312       +3     
  Lines       36042    36379     +337     
==========================================
+ Hits        17182    18720    +1538     
+ Misses      16836    15436    -1400     
- Partials     2024     2223     +199

Flag	Coverage Δ
cluster	`39.88% <36.87%> (+6.64%)`	⬆️
dm	`23.82% <0.00%> (-0.22%)`	⬇️
playground	`13.01% <0.00%> (-0.19%)`	⬇️
tiup	`14.01% <0.00%> (-0.23%)`	⬇️
unittest	`20.57% <3.30%> (-0.19%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
pkg/cluster/api/tikv_cdc.go	`0.00% <0.00%> (ø)`
pkg/cluster/spec/tikv_cdc.go	`45.81% <45.81%> (ø)`
pkg/cluster/template/scripts/tikv_cdc.go	`72.09% <72.09%> (ø)`
pkg/cluster/spec/bindversion.go	`90.91% <100.00%> (ø)`
pkg/cluster/spec/monitoring.go	`73.86% <100.00%> (-1.68%)`	⬇️
pkg/cluster/spec/spec.go	`83.37% <100.00%> (+3.52%)`	⬆️
pkg/cluster/task/update_meta.go	`74.07% <100.00%> (-12.06%)`	⬇️
pkg/cluster/template/config/prometheus.go	`81.44% <100.00%> (+0.59%)`	⬆️
pkg/tidbver/tidbver.go	`95.12% <100.00%> (+0.39%)`	⬆️
pkg/cluster/api/cdcapi.go	`0.00% <0.00%> (-56.12%)`	⬇️
... and 80 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

Signed-off-by: pingyu <[email protected]>

pingyu · 2022-08-17T02:09:02Z

@nexustar PTAL, thanks~

pingyu · 2022-08-18T07:46:28Z

/cc @haojinming @zeminzhou

ti-chi-bot · 2022-08-18T07:46:30Z

@pingyu: GitHub didn't allow me to request PR reviews from the following users: haojinming, zeminzhou.

Note that only pingcap members and repo collaborators can review this PR, and authors cannot review their own PRs.

In response to this:

/cc @haojinming @zeminzhou

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

haojinming

LGTM~

ti-chi-bot · 2022-08-23T08:06:44Z

@haojinming: Thanks for your review. The bot only counts approvals from reviewers and higher roles in list, but you're still welcome to leave your comments.

In response to this:

LGTM~

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

zeminzhou · 2022-08-23T10:48:05Z

LGTM~

Signed-off-by: pingyu <[email protected]>

ti-chi-bot · 2022-09-07T03:08:51Z

@zeminzhou: Thanks for your review. The bot only counts approvals from reviewers and higher roles in list, but you're still welcome to leave your comments.

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

AstroProfundis · 2022-09-08T08:43:39Z

pkg/cluster/spec/bindversion.go

@@ -28,7 +28,8 @@ func TiDBComponentVersion(comp, version string) string {
 		ComponentPushwaygate,
 		ComponentCheckCollector,
 		ComponentSpark,
-		ComponentTiSpark:
+		ComponentTiSpark,
+		ComponentTiKVCDC: // TiKV-CDC use individual version.


Is there any compatibility requirement or the user can always use the latest version of TiKV-CDC regardless of the TiDB version?

No requirement. User can always use the latest version by now, and we will maintain the compatibility.

AstroProfundis · 2022-09-08T08:44:58Z

pkg/cluster/spec/spec.go

@@ -112,6 +112,7 @@ type (
 		Pump           map[string]interface{} `yaml:"pump"`
 		Drainer        map[string]interface{} `yaml:"drainer"`
 		CDC            map[string]interface{} `yaml:"cdc"`
+		TiKVCDC        map[string]interface{} `yaml:"tikv-cdc"`


Better use tikv_cdc to keep the same naming format with other components.

The reason why I chose tikv-cdc is that:

I refer to this PR that using tidb-dashboard & tidb-dashboard_servers.

The binary is named as tikv-cdc, so using tikv_cdc will be error-prone, e.g. run_tikv-cdc.sh.tpl will has both tikv-cdc & tikv_cdc.

Or how about using kvcdc ? Just as what we add to playground (#2000) ?

I think #2017 should also change to use identical name formatting, I'm absolutely sure that mixing - and _ in the same key name will cause chaos to the end users.

kvcdc would be a good choice IMO.

Okay~ I will change to kvcdc.

Done. PTAL~

AstroProfundis · 2022-09-08T08:45:12Z

pkg/cluster/spec/spec.go

@@ -127,6 +128,7 @@ type (
 		PumpServers      []*PumpSpec          `yaml:"pump_servers,omitempty"`
 		Drainers         []*DrainerSpec       `yaml:"drainer_servers,omitempty"`
 		CDCServers       []*CDCSpec           `yaml:"cdc_servers,omitempty"`
+		TiKVCDCServers   []*TiKVCDCSpec       `yaml:"tikv-cdc_servers,omitempty"`


ditto, tikv_cdc_servers

AstroProfundis · 2022-09-08T08:52:13Z

tests/tiup-cluster/script/cmd_subtest.sh

+                native_ssh=true
+                shift
+                ;;
+            --staging)


This staging mirror is not used anywhere in this PR

AstroProfundis · 2022-09-08T08:52:45Z

tests/tiup-cluster/script/tikv_cdc.sh

+	version="nightly"
+	topo_name="tikv_cdc"
+	test_tls=false
+	mirror="https://tiup-mirrors.pingcap.com/"


tiup mirror set --reset will do the same as tiup mirror set https://tiup-mirrors.pingcap.com

AstroProfundis · 2022-09-08T08:53:33Z

tests/tiup-cluster/script/tikv_cdc.sh

+	done
+
+	# set mirror
+	tiup mirror set $mirror


as the staging mirror is not used, i think it's not necessary to build tiup and use it to manually set the mirror.

tiup is also used for installing tikv-cdc component for test case of tiup cluster patch. I remove the tiup mirror here, but keep the tiup target & function.

AstroProfundis · 2022-09-08T08:56:59Z

pkg/cluster/spec/tikv_cdc.go

+	Imported        bool                   `yaml:"imported,omitempty"`
+	Patched         bool                   `yaml:"patched,omitempty"`
+	IgnoreExporter  bool                   `yaml:"ignore_exporter,omitempty"`
+	Port            int                    `yaml:"port" default:"8600"`


The port should also be added to portTypes list in pkg/cluster/spec/validate.go at around L619, other wise the port conflict check is not working for it.

I found that the Port is already in the list for portTypes where portConflictsDetect get port value from. So the port conflict check should work.

And I add a test case to verify port conflict check in integrated test for tikv-cdc. PTAL~

Signed-off-by: pingyu <[email protected]>

pingyu · 2022-09-09T10:41:43Z

@AstroProfundis Thanks for your review ! All comments are addressed or replied, PTAL~

nexustar · 2022-09-13T03:22:43Z

/merge

ti-chi-bot · 2022-09-13T03:22:46Z

This pull request has been accepted and is ready to merge.

Commit hash: c1e81fd

ti-chi-bot · 2022-09-13T03:31:07Z

@pingyu: Your PR was out of date, I have automatically updated it for you.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

Signed-off-by: pingyu <[email protected]>

pingyu · 2022-09-14T08:37:20Z

Added cluster version check. PTAL, thanks~

nexustar · 2022-09-14T17:38:43Z

/merge

ti-chi-bot · 2022-09-14T17:38:46Z

This pull request has been accepted and is ready to merge.

Commit hash: 5304c07

cluster: Support TiKV-CDC component.

30cb753

Issue Number: pingcap#1999 Signed-off-by: pingyu <[email protected]>

ti-chi-bot requested review from lucklove and nexustar August 15, 2022 12:28

ti-chi-bot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Aug 15, 2022

fix default port

c8b2170

Signed-off-by: pingyu <[email protected]>

pingyu added 4 commits August 15, 2022 21:25

bug fix

4b32ee0

Signed-off-by: pingyu <[email protected]>

add integration test cases

978ddc9

Signed-off-by: pingyu <[email protected]>

fix CI. add unit tests

103474c

Signed-off-by: pingyu <[email protected]>

fix prom

caaa0a5

Signed-off-by: pingyu <[email protected]>

pingyu changed the title ~~[Draft] cluster: Support TiKV-CDC component~~ cluster: Support TiKV-CDC component Aug 16, 2022

trigger CI

fb169e5

Signed-off-by: pingyu <[email protected]>

haojinming approved these changes Aug 23, 2022

View reviewed changes

nexustar added this to the v1.11.0 milestone Aug 25, 2022

pingyu added 2 commits September 7, 2022 10:45

Merge branch 'master' into cluster-tikv-cdc

d3a11c0

remove staging

68a4054

Signed-off-by: pingyu <[email protected]>

zeminzhou approved these changes Sep 7, 2022

View reviewed changes

pingyu mentioned this pull request Sep 7, 2022

playground: Supports TiKV-CDC component #2000

Merged

Merge branch 'master' into cluster-tikv-cdc

e1a0a77

AstroProfundis reviewed Sep 8, 2022

View reviewed changes

pingyu added 3 commits September 9, 2022 10:46

address comment

bec14dc

Signed-off-by: pingyu <[email protected]>

address comment; add test case for port conflict detection

e70d89a

Signed-off-by: pingyu <[email protected]>

add tiup back

e4e3da7

Signed-off-by: pingyu <[email protected]>

pingyu added 2 commits September 9, 2022 17:38

change to kvcdc

c41a24a

Signed-off-by: pingyu <[email protected]>

simplify integrated test

c1e81fd

Signed-off-by: pingyu <[email protected]>

nexustar approved these changes Sep 13, 2022

View reviewed changes

ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Sep 13, 2022

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Sep 13, 2022

Merge branch 'master' into cluster-tikv-cdc

752d421

fix CI error; TiKV-CDC requires >= v6.2.0

2a4f063

Signed-off-by: pingyu <[email protected]>

ti-chi-bot removed the status/can-merge Indicates a PR has been approved by a committer. label Sep 13, 2022

This was referenced Sep 13, 2022

docs: Change TiUP related docs tikv/migration#222

Closed

[to #222] Change docs for TiUP tikv/migration#223

Merged

add cluster version check

5304c07

Signed-off-by: pingyu <[email protected]>

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Sep 14, 2022

ti-chi-bot merged commit d6a2826 into pingcap:master Sep 14, 2022

pingyu mentioned this pull request Nov 10, 2022

cdc: Add TiUP support for TiKV-CDC tikv/migration#151

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cluster: Support TiKV-CDC component #2022

cluster: Support TiKV-CDC component #2022

pingyu commented Aug 15, 2022 •

edited

Loading

ti-chi-bot commented Aug 15, 2022 •

edited

Loading

codecov-commenter commented Aug 15, 2022 •

edited

Loading

pingyu commented Aug 17, 2022

pingyu commented Aug 18, 2022

ti-chi-bot commented Aug 18, 2022

haojinming left a comment

ti-chi-bot commented Aug 23, 2022

zeminzhou commented Aug 23, 2022

ti-chi-bot commented Sep 7, 2022

AstroProfundis Sep 8, 2022

pingyu Sep 8, 2022

AstroProfundis Sep 8, 2022

pingyu Sep 8, 2022

AstroProfundis Sep 9, 2022

pingyu Sep 9, 2022

pingyu Sep 9, 2022

AstroProfundis Sep 8, 2022

AstroProfundis Sep 8, 2022

AstroProfundis Sep 8, 2022

AstroProfundis Sep 8, 2022

pingyu Sep 9, 2022

AstroProfundis Sep 8, 2022

pingyu Sep 9, 2022

pingyu commented Sep 9, 2022

nexustar commented Sep 13, 2022

ti-chi-bot commented Sep 13, 2022

ti-chi-bot commented Sep 13, 2022

pingyu commented Sep 14, 2022

nexustar commented Sep 14, 2022

ti-chi-bot commented Sep 14, 2022

cluster: Support TiKV-CDC component #2022

cluster: Support TiKV-CDC component #2022

Conversation

pingyu commented Aug 15, 2022 • edited Loading

What problem does this PR solve?

What is changed and how it works?

Check List

ti-chi-bot commented Aug 15, 2022 • edited Loading

codecov-commenter commented Aug 15, 2022 • edited Loading

Codecov Report

pingyu commented Aug 17, 2022

pingyu commented Aug 18, 2022

ti-chi-bot commented Aug 18, 2022

haojinming left a comment

Choose a reason for hiding this comment

ti-chi-bot commented Aug 23, 2022

zeminzhou commented Aug 23, 2022

ti-chi-bot commented Sep 7, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pingyu commented Sep 9, 2022

nexustar commented Sep 13, 2022

ti-chi-bot commented Sep 13, 2022

ti-chi-bot commented Sep 13, 2022

pingyu commented Sep 14, 2022

nexustar commented Sep 14, 2022

ti-chi-bot commented Sep 14, 2022

pingyu commented Aug 15, 2022 •

edited

Loading

ti-chi-bot commented Aug 15, 2022 •

edited

Loading

codecov-commenter commented Aug 15, 2022 •

edited

Loading