Skip to content

Commit

Permalink
glue: add GlueCheckpointDB and remove external TiDB usage (pingcap#478)
Browse files Browse the repository at this point in the history 
* save my work

add notes

save work

save work

fix unit test

remove tidbMgr in RestoreController

remove some comments

remove some comments

change logger in SQLWithRetry

revert replace log.Logger to *zap.Logger

dep: update uuid dependency to latest google/uuid (pingcap#452)

* dep: update satori/go.uuid to latest

* fix tests

* change to google/uuid

* fix build

* try fix test

* get familiar with google/uuid

* address comment

tidb-lightning-ctl: change default of -d to 'noop://' (pingcap#453)

also add noop:// to supported storage types (to represent an empty store)

replace tab to space

try another port to fix CI

remove some comment

*: more glue

restore: fix the bug that gc life time ttl does not take effect (pingcap#448)

* fix gc ttl loop

* resolve comment and add tests

fix CI

report info to host TiDB

config: filter out all system schemas by default (pingcap#459)

backend: fix auto random default value for primary key (pingcap#457)

* fix auto generate auto random primary key column

* fix default for auto random primary key

* fix test

* use prev row id for auto random and add a test

* replace chunck with session opt

* fix

* fix

mydumper: fix parquet data parser (pingcap#435)

* fix parquet

* reorder imports

* fix test

* use empty collation

* fix a error and add more test cases

* add pointer type tests

* resolve comments

Co-authored-by: kennytm <[email protected]>

address comment

backend/local: use range properties to optimize region range estimate (pingcap#422)

* use range propreties to estimate region range

* post-restore: add optional level for post-restore operations (pingcap#421)

* add optional level for opst-restore operations

* trim leading and suffix '"

* use UnmarshalTOML to unmarshal post restore op level

* resolve comments and fix unit test

* backend/local: do not retry epochNotMatch error when ingest sst (pingcap#419)

* do not retry epochNotMatch error when ingest sst

* add retry ingest for 'Raft raft: proposal dropped' error in ingest

* change some retryable error log level from Error to Warn

* fix nextKey

* add a comment for nextKey

* fix comment and add a unit test

* wrap time.Sleep in select

Co-authored-by: kennytm <[email protected]>

* update

* use range properties to optimze region range estimate

* update pebble

* change the default value for batch-size

* add unit tests and reslove comments

* add a comment to range properties test

* add a comment

* add a test for range property with pebble

* rename const variable

Co-authored-by: kennytm <[email protected]>

fix pd service id is empty (pingcap#460)

fix s3 parquet reader (pingcap#461)

Co-authored-by: Neil Shen <[email protected]>

fix service gc ttl again (pingcap#465)

address comment

mydumper: verify file routing config (pingcap#470)

* fix file routing

* remove useless line

* remove redundant if check

rename a method in interface

save work

try fix CI

could work

change ctx usage

try fix CI

try fix CI

refine function interface

refine some fucntion interface

debug CI

address comment

config: allow four byte-size config to be specified using human-readable units ("100 GiB") (pingcap#471)

* Makefile: add `make finish-prepare` action

* config: accept human-readable size for most byte-related config

e.g. allow `region-split-size = '96M'` in additional to `= 100663296`

(known issue: these values' precisions will be truncated to 53 bits
instead of supporting all 63 bits)

* restore: reduce chance of spurious errors from TestGcTTLManagerSingle

Co-authored-by: glorv <[email protected]>

remove debug log

test: change double type syntax (pingcap#474)

address comment

checkpoint: add glue checkpoint

resolve cycle import

expose Retry

refine

change interface to cope with TiDB

fix SQL string

fix SQL

adjust interface to embedded in TiDB

could import now

reduce TLS

restore: add `glue.Glue` interface and other function (pingcap#456)

* save my work

* add notes

* save work

* save work

* fix unit test

* remove tidbMgr in RestoreController

* remove some comments

* remove some comments

* change logger in SQLWithRetry

* revert replace log.Logger to *zap.Logger

* replace tab to space

* try another port to fix CI

* remove some comment

* *: more glue

* report info to host TiDB

* fix CI

* address comment

* address comment

* rename a method in interface

* save work

* try fix CI

* could work

* change ctx usage

* try fix CI

* try fix CI

* refine function interface

* refine some fucntion interface

* debug CI

* address comment

* remove debug log

* address comment

modify code

add comment

refine some code

* address comment

* add some comments

* fix CI and change CREATE TABLE
  • Loading branch information
@lance6716
lance6716 authored Nov 23, 2020
1 parent a78137f commit da8b24e
Show file tree
Hide file tree
Showing 10 changed files with 958 additions and 200 deletions.
17 changes: 17 additions & 0 deletions lightning/backend/importer.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ import (
"github.com/pingcap/errors"
kv "github.com/pingcap/kvproto/pkg/import_kvpb"
"github.com/pingcap/parser/model"
"github.com/pingcap/tidb-lightning/lightning/glue"
"github.com/pingcap/tidb/table"
"go.uber.org/zap"
"google.golang.org/grpc"
Expand Down Expand Up @@ -267,6 +268,22 @@ func checkTiDBVersion(tls *common.TLS, requiredVersion semver.Version) error {
return checkVersion("TiDB", requiredVersion, *version)
}

func checkTiDBVersionBySQL(g glue.Glue, requiredVersion semver.Version) error {
versionStr, err := g.GetSQLExecutor().ObtainStringWithLog(
context.Background(),
"SELECT version();",
"check TiDB version",
log.L())
if err != nil {
return errors.Trace(err)
}
version, err := common.ExtractTiDBVersion(versionStr)
if err != nil {
return errors.Trace(err)
}
return checkVersion("TiDB", requiredVersion, *version)
}

func checkPDVersion(tls *common.TLS, pdAddr string, requiredVersion semver.Version) error {
version, err := common.FetchPDVersion(tls, pdAddr)
if err != nil {
Expand Down
6 changes: 5 additions & 1 deletion lightning/backend/local.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,7 @@ import (
"github.com/pingcap/kvproto/pkg/kvrpcpb"
"github.com/pingcap/kvproto/pkg/metapb"
"github.com/pingcap/parser/model"
"github.com/pingcap/tidb-lightning/lightning/glue"
"github.com/pingcap/tidb/table"
"github.com/pingcap/tidb/util/codec"
"github.com/pingcap/tidb/util/hack"
Expand Down Expand Up @@ -167,6 +168,7 @@ type local struct {
splitCli split.SplitClient
tls *common.TLS
pdAddr string
g glue.Glue

localStoreDir string
regionSplitSize int64
Expand Down Expand Up @@ -248,6 +250,7 @@ func NewLocalBackend(
rangeConcurrency int,
sendKVPairs int,
enableCheckpoint bool,
g glue.Glue,
) (Backend, error) {
pdCli, err := pd.NewClient([]string{pdAddr}, tls.ToPDSecurityOption())
if err != nil {
Expand Down Expand Up @@ -278,6 +281,7 @@ func NewLocalBackend(
splitCli: splitCli,
tls: tls,
pdAddr: pdAddr,
g: g,

localStoreDir: localFile,
regionSplitSize: regionSplitSize,
Expand Down Expand Up @@ -1170,7 +1174,7 @@ func (local *local) CleanupEngine(ctx context.Context, engineUUID uuid.UUID) err
}

func (local *local) CheckRequirements() error {
if err := checkTiDBVersion(local.tls, localMinTiDBVersion); err != nil {
if err := checkTiDBVersionBySQL(local.g, localMinTiDBVersion); err != nil {
return err
}
if err := checkPDVersion(local.tls, local.pdAddr, localMinPDVersion); err != nil {
Expand Down
Loading

0 comments on commit da8b24e

Please sign in to comment.