BR: add stream backup meta client #30088

YuJuncen · 2021-11-24T03:30:01Z

What problem does this PR solve?

Issue Number: TiDB (BR) part of #29969

Added the client at BR side for stream backup metadata.

Generally, provided a stream.MetadataClient, which provides operations over the tasks.

What is changed and how it works?

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No code

Release note

None

ti-chi-bot · 2021-11-24T03:30:03Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

3pointer

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

sre-bot · 2021-11-24T03:30:35Z

Please follow PR Title Format:

pkg [, pkg2, pkg3]: what is changed

Or if the count of mainly changed packages are more than 3, use

*: what is changed

After you have format title, you can leave a comment /run-check_title to recheck it

joccau · 2021-11-24T06:17:32Z

br/pkg/stream/models.go

+
+// Ranges is a vector of [start_key, end_key) pairs.
+type Ranges = []Range
+type Range = [2][]byte


use kv.KeyRange directly ?
https://github.com/pingcap/tidb/blob/master/kv/key.go#L91

…ort it in need)

3pointer · 2021-11-24T08:05:11Z

go.mod

@@ -102,3 +102,5 @@ replace github.com/pingcap/tidb/parser => ./parser

 // fix potential security issue(CVE-2020-26160) introduced by indirect dependency.
 replace github.com/dgrijalva/jwt-go => github.com/form3tech-oss/jwt-go v3.2.6-0.20210809144907-32ab6a8243d7+incompatible
+
+replace github.com/pingcap/kvproto => /Users/hillium/Developer/kvproto


we can use br-stream branch kvproto in github.

3pointer · 2021-11-25T08:07:53Z

br/pkg/stream/client.go

+	return nil
+}
+
+func (c *MetaDataClient) PauseTask(ctx context.Context, taskName string) error {


how we control TiKV pause observe task.🤔

According to kenny, pausing / resuming is out of our current scope. 🤔

to clarify, it is out of scope because we aren't sure if we want the non-destructive pause/resume or the destructive stop/start, so we delay the feature as it is not essential anyway.

joccau · 2021-12-01T11:26:20Z

br/pkg/stream/client.go

+
+// MinNextBackupTS query the all next backup ts of a store, returning the minimal next backup ts of the store.
+func (t *Task) MinNextBackupTS(ctx context.Context, store uint64) (uint64, error) {
+	min := uint64(0xffffffff)


Maybe 0xFFFFFFFFFFFFFFFF ?

I just found math.MaxUint64 🦉

YuJuncen · 2021-12-06T03:35:20Z

util/memory/meminfo.go:142:12: undefined: os.ReadFile 🤔

YuJuncen · 2021-12-06T05:37:27Z

/build
/run-check_dev

…am-meta

YuJuncen · 2021-12-06T07:51:07Z

/build

YuJuncen · 2021-12-06T07:51:15Z

/run-check_dev

joccau

LGTM

ti-chi-bot · 2021-12-06T13:05:29Z

@joccau: Thanks for your review. The bot only counts approvals from reviewers and higher roles in list, but you're still welcome to leave your comments.

In response to this:

LGTM

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

3pointer

Rest LGTM

3pointer · 2021-12-07T08:26:28Z

br/pkg/stream/client.go

@@ -0,0 +1,171 @@
+package stream


add the license

3pointer · 2021-12-07T08:26:36Z

br/pkg/stream/integration_test.go

@@ -0,0 +1,230 @@
+// This package tests the login in MetaClient with a embed etcd.


3pointer · 2021-12-07T08:26:45Z

br/pkg/stream/models.go

@@ -0,0 +1,165 @@
+package stream


3pointer · 2021-12-07T08:26:51Z

br/pkg/stream/prefix_scanner.go

@@ -0,0 +1,96 @@
+package stream


* stream: initial commit of stream backup * br/stream: basic logic of stream backup metadata * br/stream: add prefix scanner * br/stream: use types from kv package * br/stream: unexport prefix scanner(remove it to utils package and export it in need) * br/stream: move kvproto to br-stream branch, added a playground * br/stream: added some new functions for playground * br/stream: added a builder for task * br/stream: remove playground, use math.MaxUInt64 * br: replace kvproto with br-stream * br/stream: update error code * br/stream: fix tests * br/stream: added copyright

YuJuncen added 3 commits November 22, 2021 14:02

stream: initial commit of stream backup

69c8021

br/stream: basic logic of stream backup metadata

103c130

br/stream: add prefix scanner

f473bbf

YuJuncen added sig/migrate component/br This issue is related to BR of TiDB. labels Nov 24, 2021

YuJuncen requested review from kennytm, 3pointer and joccau November 24, 2021 03:30

ti-chi-bot added release-note-none Denotes a PR that doesn't merit a release note. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Nov 24, 2021

YuJuncen changed the title ~~Stream meta~~ BR: add stream backup meta client Nov 24, 2021

YuJuncen marked this pull request as draft November 24, 2021 03:37

ti-chi-bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Nov 24, 2021

joccau reviewed Nov 24, 2021

View reviewed changes

YuJuncen added 2 commits November 24, 2021 14:31

br/stream: use types from kv package

6f6842a

br/stream: unexport prefix scanner(remove it to utils package and exp…

26056fd

…ort it in need)

YuJuncen mentioned this pull request Nov 24, 2021

Register Log-based Backup Task to PD. #29969

Open

3pointer reviewed Nov 24, 2021

View reviewed changes

3pointer reviewed Nov 25, 2021

View reviewed changes

YuJuncen and others added 3 commits November 26, 2021 01:08

br/stream: move kvproto to br-stream branch, added a playground

9ddb4d1

br/stream: added some new functions for playground

f5b26c4

br/stream: added a builder for task

bea4263

joccau reviewed Dec 1, 2021

View reviewed changes

br/stream: remove playground, use math.MaxUInt64

60b1382

YuJuncen marked this pull request as ready for review December 6, 2021 03:18

ti-chi-bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Dec 6, 2021

YuJuncen added 2 commits December 6, 2021 14:18

Merge branch 'br-stream' of https://github.com/pingcap/tidb into stre…

03871ee

…am-meta

br: replace kvproto with br-stream

6a46907

br/stream: update error code

ecf6a74

joccau approved these changes Dec 6, 2021

View reviewed changes

br/stream: fix tests

fe2e9dc

3pointer reviewed Dec 7, 2021

View reviewed changes

br/stream: added copyright

1e4f772

3pointer approved these changes Dec 7, 2021

View reviewed changes

ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Dec 7, 2021

YuJuncen added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Dec 7, 2021

ti-chi-bot added status/LGT1 Indicates that a PR has LGTM 1. and removed status/LGT2 Indicates that a PR has LGTM 2. labels Dec 7, 2021

YuJuncen merged commit bcdee32 into pingcap:br-stream Dec 7, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BR: add stream backup meta client #30088

BR: add stream backup meta client #30088

YuJuncen commented Nov 24, 2021

ti-chi-bot commented Nov 24, 2021 •

edited

Loading

sre-bot commented Nov 24, 2021

joccau Nov 24, 2021

3pointer Nov 24, 2021

3pointer Nov 25, 2021

YuJuncen Nov 25, 2021

kennytm Nov 29, 2021

joccau Dec 1, 2021

YuJuncen Dec 6, 2021

YuJuncen commented Dec 6, 2021

YuJuncen commented Dec 6, 2021

YuJuncen commented Dec 6, 2021

YuJuncen commented Dec 6, 2021

joccau left a comment

ti-chi-bot commented Dec 6, 2021

3pointer left a comment

3pointer Dec 7, 2021

3pointer Dec 7, 2021

3pointer Dec 7, 2021

3pointer Dec 7, 2021

		@@ -0,0 +1,230 @@
		// This package tests the login in MetaClient with a embed etcd.

BR: add stream backup meta client #30088

BR: add stream backup meta client #30088

Conversation

YuJuncen commented Nov 24, 2021

What problem does this PR solve?

What is changed and how it works?

Check List

Release note

ti-chi-bot commented Nov 24, 2021 • edited Loading

sre-bot commented Nov 24, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

YuJuncen commented Dec 6, 2021

YuJuncen commented Dec 6, 2021

YuJuncen commented Dec 6, 2021

YuJuncen commented Dec 6, 2021

joccau left a comment

Choose a reason for hiding this comment

ti-chi-bot commented Dec 6, 2021

3pointer left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ti-chi-bot commented Nov 24, 2021 •

edited

Loading