syncer: use downstream PK/UK to generate DML

[WizardXiao]

What problem does this PR solve?

#1895

What is changed and how it works?

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)
• No code

Code changes

• Has exported function/method change
• Has exported variable/fields change
• Has interface methods change
• Has persistent data change

Side effects

• Possible performance regression
• Increased code complexity
• Breaking backward compatibility

Related changes

• Need to cherry-pick to the release branch
• Need to update the documentation
• Need to update the dm/dm-ansible
• Need to be included in the release note

[ti-chi-bot]

[REVIEW NOTIFICATION]

This pull request has not been approved.

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

[CLAassistant]

All committers have signed the CLA.

[ti-chi-bot]

Welcome @WizardXiao!

It looks like this is your first PR to pingcap/dm 🎉.

I'm the bot to help you request reviewers, add labels and more, See available commands.

We want to make sure your contribution gets all the attention it needs!



Thank you, and welcome to pingcap/dm. 😃

[Ehco1996]

@WizardXiao Thank you for your contribution 🎉🎉 is this PR ready for review?

a few more suggestions：

• is there a publicly available spec design document? If so, please post it in the description of the PR
• please fix the errors in the CI to ensure that it passes.

Misoperation

[lichunzhu]

Will review later

[lance6716]

/run-unit-tests

[Ehco1996]

some ut failed and panic

----------------------------------------------------------------------
FAIL: syncer_test.go:1222: testSyncerSuite.TestExitSafeModeByConfig

syncer_test.go:1370:
    checkJobs(c, testJobs.jobs, expectJobs)
syncer_test.go:1604:
    c.Assert(len(jobs), Equals, len(expectJobs), Commentf("jobs = %q", jobs))
... obtained int = 6
... expected int = 12
... jobs = ["tp: flush, sql: , args: [], key: , ddls: [], last_location: position: (, 0), gtid-set: , start_location: position: (, 0), gtid-set: , current_location: position: (, 0), gtid-set: " "tp: ddl, sql: , args: [], key: , ddls: [CREATE DATABASE IF NOT EXISTS `test_1`], last_location: position: (, 1938), gtid-set: , start_location: position: (, 1871), gtid-set: , current_location: position: (, 1938), gtid-set: " "tp: flush, sql: , args: [], key: , ddls: [], last_location: position: (, 0), gtid-set: , start_location: position: (, 0), gtid-set: , current_location: position: (, 0), gtid-set: " "tp: ddl, sql: , args: [], key: , ddls: [CREATE TABLE IF NOT EXISTS `test_1`.`t_1` (`id` INT PRIMARY KEY,`name` VARCHAR(24))], last_location: position: (, 2107), gtid-set: , start_location: position: (, 2003), gtid-set: , current_location: position: (, 2107), gtid-set: " "tp: insert, sql: REPLACE INTO `test_1`.`t_1` (`id`,`name`) VALUES (?,?), args: [1 a], key: , ddls: [], last_location: position: (, 2214), gtid-set: , start_location: position: (, 0), gtid-set: , current_location: position: (, 2307), gtid-set: " "tp: flush, sql: , args: [], key: , ddls: [], last_location: position: (, 0), gtid-set: , start_location: position: (, 0), gtid-set: , current_location: position: (, 0), gtid-set: "]


----------------------------------------------------------------------
PANIC: syncer_test.go:714: testSyncerSuite.TestGeneratedColumn

... Panic: runtime error: invalid memory address or nil pointer dereference (PC=0x1415466)

/usr/local/go/src/runtime/panic.go:971
  in gopanic
/usr/local/go/src/runtime/panic.go:212
  in panicmem
/usr/local/go/src/runtime/signal_unix.go:734
  in sigpanic
/usr/local/go/src/database/sql/sql.go:1197
  in DB.conn
/usr/local/go/src/database/sql/sql.go:1553
  in DB.exec
/usr/local/go/src/database/sql/sql.go:1535
  in DB.ExecContext
/usr/local/go/src/database/sql/sql.go:1549
  in DB.Exec
syncer_test.go:716
  in testSyncerSuite.TestGeneratedColumn
/usr/local/go/src/reflect/value.go:337
  in Value.Call
/nfs/cache/mod/github.com/pingcap/[email protected]/check.go:850
  in suiteRunner.forkTest.func1
/nfs/cache/mod/github.com/pingcap/[email protected]/check.go:739
  in suiteRunner.forkCall.func1
/usr/local/go/src/runtime/asm_amd64.s:1371
  in goexit

[WizardXiao]

/run-unit-test

[lance6716]

will review later

[Ehco1996]

i think this work is almost done 🎉
only some light changes need to be applied after your merge master code, remeber to update the genSql logic ( prepare index/table info before gensql in case of accessing tracker concurrently )

btw, please reslove some outdated comment for better reveiw 😉

[WizardXiao]

i think this work is almost done 🎉 only some light changes need to be applied after your merge master code, remeber to update the genSql logic ( prepare index/table info before gensql in case of accessing tracker concurrently )

btw, please reslove some outdated comment for better reveiw 😉

thanks, i have merged master code and fixed unit-test/integration-test.

[lance6716]

/cc @Ehco1996

[lance6716]

/run-dm-integration-tests

[WizardXiao]

This PR has been moved to https://github.com/pingcap/ticdc/pull/3168