Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

validator(dm): full mode validator, part 1 #4539

Merged
merged 50 commits into from
Mar 3, 2022
Merged
Show file tree
Hide file tree
Changes from 42 commits
Commits
Show all changes
50 commits
Select commit Hold shift + click to select a range
f37a133
feat: implement validator
buchuitoudegou Feb 9, 2022
a67c580
feat: cond ut
buchuitoudegou Feb 9, 2022
cf136e0
feat: validator unit test
buchuitoudegou Feb 10, 2022
3235bb3
feat: validator dovalidate ut
buchuitoudegou Feb 11, 2022
f6c6906
feat: add test case and fix bugs
buchuitoudegou Feb 14, 2022
e718a3b
rename, close db on err, move some init to ctor
D3Hunter Feb 14, 2022
95156d9
fix: conflict import
buchuitoudegou Feb 14, 2022
c330483
fix
D3Hunter Feb 14, 2022
14b5252
validation
D3Hunter Feb 14, 2022
c019bee
use info from syncer:ddl, loc
D3Hunter Feb 15, 2022
7a058c3
wait until syncer synced
D3Hunter Feb 16, 2022
6c98343
split worker
D3Hunter Feb 17, 2022
86cbced
refactor
D3Hunter Feb 18, 2022
25106da
Merge remote-tracking branch 'upstream/master' into incr-validator
D3Hunter Feb 22, 2022
c35dd4c
genColData
D3Hunter Feb 22, 2022
e288c53
fix
D3Hunter Feb 22, 2022
61821c9
fix ut
D3Hunter Feb 22, 2022
f623835
gen rowkey
D3Hunter Feb 23, 2022
64f9474
gen rowkey
D3Hunter Feb 23, 2022
aa889e5
ut and fix
D3Hunter Feb 24, 2022
50c81ac
add ut and fix
D3Hunter Feb 25, 2022
c9be548
Merge remote-tracking branch 'upstream/master' into incr-validator
D3Hunter Feb 25, 2022
e8ca5a1
ut and fix
D3Hunter Feb 25, 2022
8eb156d
remove useless field
D3Hunter Feb 25, 2022
70bea8d
move genColData
D3Hunter Feb 25, 2022
9dd8047
remove worker,make pr smaller
D3Hunter Feb 25, 2022
1128634
Merge branch 'master' into incr-validator
D3Hunter Feb 25, 2022
d4e3ab2
Merge remote-tracking branch 'upstream/master' into incr-validator
D3Hunter Feb 28, 2022
5fe4564
add log
D3Hunter Feb 28, 2022
0a45ff9
Merge remote-tracking branch 'origin/incr-validator' into incr-validator
D3Hunter Feb 28, 2022
ce93b8d
Merge remote-tracking branch 'upstream/master' into incr-validator
D3Hunter Feb 28, 2022
28f9d3c
add defer
D3Hunter Feb 28, 2022
ee3a4ab
make err channel bigger
D3Hunter Feb 28, 2022
f8f6b11
fix it
D3Hunter Feb 28, 2022
0ac1ce6
Merge remote-tracking branch 'upstream/master' into incr-validator
D3Hunter Feb 28, 2022
24dc127
revert syncer stage, use isRunning
D3Hunter Feb 28, 2022
c1bffff
Merge remote-tracking branch 'upstream/master' into incr-validator
D3Hunter Mar 1, 2022
c70c560
close error chan after all possible sender goroutines stopped
D3Hunter Mar 1, 2022
d374d38
fix comments
D3Hunter Mar 2, 2022
2317d64
timer to time.after
D3Hunter Mar 2, 2022
d968aa9
fix comments
D3Hunter Mar 2, 2022
a4b807d
fix comments
D3Hunter Mar 2, 2022
0c75d9f
fix comments
D3Hunter Mar 3, 2022
5edcc94
Merge remote-tracking branch 'upstream/master' into incr-validator
D3Hunter Mar 3, 2022
9df2d93
Merge branch 'master' into incr-validator
D3Hunter Mar 3, 2022
6277f06
fix comments
D3Hunter Mar 3, 2022
9533c4d
Merge remote-tracking branch 'origin/incr-validator' into incr-validator
D3Hunter Mar 3, 2022
603c8be
Merge branch 'master' into incr-validator
ti-chi-bot Mar 3, 2022
56529b5
Merge branch 'master' into incr-validator
ti-chi-bot Mar 3, 2022
9bf7f5b
Merge branch 'master' into incr-validator
D3Hunter Mar 3, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 7 additions & 1 deletion dm/dm/config/task.go
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,8 @@ const (
ValidationNone = "none"
ValidationFast = "fast"
ValidationFull = "full"

DefaultValidatorWorkerCount = 4
)

// default config item values.
Expand Down Expand Up @@ -337,7 +339,8 @@ func (m *SyncerConfig) UnmarshalYAML(unmarshal func(interface{}) error) error {
}

type ValidatorConfig struct {
Mode string `yaml:"mode" toml:"mode" json:"mode"`
Mode string `yaml:"mode" toml:"mode" json:"mode"`
WorkerCount int `yaml:"worker-count" toml:"worker-count" json:"worker-count"`
D3Hunter marked this conversation as resolved.
Show resolved Hide resolved
}

func (v *ValidatorConfig) adjust() error {
Expand All @@ -347,6 +350,9 @@ func (v *ValidatorConfig) adjust() error {
if v.Mode != ValidationNone && v.Mode != ValidationFast && v.Mode != ValidationFull {
return terror.ErrConfigValidationMode
}
if v.WorkerCount <= 0 {
v.WorkerCount = DefaultValidatorWorkerCount
}
return nil
}

Expand Down
4 changes: 4 additions & 0 deletions dm/pkg/binlog/position.go
Original file line number Diff line number Diff line change
Expand Up @@ -428,3 +428,7 @@ func (l *Location) SetGTID(gset gmysql.GTIDSet) error {
func (l *Location) GetGTID() gtid.Set {
return l.gtidSet
}

func (l *Location) Update(gtidStr string) error {
Ehco1996 marked this conversation as resolved.
Show resolved Hide resolved
return l.gtidSet.Update(gtidStr)
}
13 changes: 13 additions & 0 deletions dm/pkg/conn/mockdb.go
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,19 @@ func InitVersionDB(c *check.C) sqlmock.Sqlmock {
return mock
}

func InitMockDBFull() (*sql.DB, sqlmock.Sqlmock, error) {
db, mock, err := sqlmock.New()
if err != nil {
return nil, nil, err
}
if mdbp, ok := DefaultDBProvider.(*mockDBProvider); ok {
mdbp.db = db
} else {
DefaultDBProvider = &mockDBProvider{db: db}
}
return db, mock, err
}

// TODO: export Config in https://github.com/pingcap/tidb/blob/a8fa29b56d633b1ec843e21cb89131dd4fd601db/br/pkg/mock/mock_cluster.go#L35
// Cluster is mock tidb cluster.
type Cluster struct {
Expand Down
Loading