pessimistic shard ddl block in GetOperation

[GMHDBJD]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. What did you do? If possible, provide a recipe for reproducing the error.

source1 ddl1
source2 ddl1
source1 ddl2
source2 ddl2

2. What did you expect to see?
   DM works well

3. What did you see instead?
   both worker block at ddl2

4. Versions of the cluster

   • DM version (run dmctl -V or dm-worker -V or dm-master -V):

     master
     

5. root cause
   jenkins link
   master log:

[2020-10-30T02:25:09.879Z] [2020/10/30 10:24:51.715 +08:00] [INFO] [pessimist.go:502] ["receive a shard DDL lock operation"] [component="shard DDL pessimist"] [operation="{\"id\":\"test-`shardddl`.`tb`\",\"task\":\"test\",\"source\":\"mysql-replica-02\",\"ddls\":[\"ALTER TABLE `shardddl`.`tb` ADD UNIQUE `idx_a`(`a`)\"],\"exec\":false,\"done\":true}"]
[2020-10-30T02:25:09.879Z] [2020/10/30 10:24:51.715 +08:00] [INFO] [pessimist.go:522] ["the lock for the shard DDL lock operation has been resolved"] [component="shard DDL pessimist"] [operation="{\"id\":\"test-`shardddl`.`tb`\",\"task\":\"test\",\"source\":\"mysql-replica-02\",\"ddls\":[\"ALTER TABLE `shardddl`.`tb` ADD UNIQUE `idx_a`(`a`)\"],\"exec\":false,\"done\":true}"]

[2020-10-30T02:25:09.879Z] [2020/10/30 10:24:51.818 +08:00] [INFO] [pessimist.go:466] ["receive a shard DDL info"] [component="shard DDL pessimist"] [info="{\"task\":\"test\",\"source\":\"mysql-replica-01\",\"schema\":\"shardddl\",\"table\":\"tb\",\"ddls\":[\"ALTER TABLE `shardddl`.`tb` ADD UNIQUE `idx_b`(`b`)\"]}"]
[2020-10-30T02:25:09.880Z] [2020/10/30 10:24:51.818 +08:00] [ERROR] [pessimist.go:472] ["fail to try sync shard DDL lock"] [component="shard DDL pessimist"] [info="{\"task\":\"test\",\"source\":\"mysql-replica-01\",\"schema\":\"shardddl\",\"table\":\"tb\",\"ddls\":[\"ALTER TABLE `shardddl`.`tb` ADD UNIQUE `idx_b`(`b`)\"]}"] [error="[code=38016:class=dm-master:scope=internal:level=high], Message: sharding ddls in ddl lock [ALTER TABLE `shardddl`.`tb` ADD UNIQUE `idx_a`(`a`)] is different with [ALTER TABLE `shardddl`.`tb` ADD UNIQUE `idx_b`(`b`)], Workaround: Please use show-ddl-locks command for more details."]
[2020-10-30T02:25:09.880Z] [2020/10/30 10:24:51.825 +08:00] [DEBUG] [interceptor.go:175] ["request stats"] [component="embed etcd"] ["start time"=2020/10/30 10:24:51.716 +08:00] ["time spent"=989.308µs] [remote=127.0.0.1:52956] ["response type"=/etcdserverpb.KV/Txn] ["request count"=2] ["request size"=170] ["response count"=0] ["response size"=49] ["request content"="compare:<> success:<request_delete_range:<key:\"/dm-master/shardddl-pessimism/operation/74657374/6d7973716c2d7265706c6963612d3031\" >  request_delete_range:<key:\"/dm-master/shardddl-pessimism/operation/74657374/6d7973716c2d7265706c6963612d3032\" > > failure:<>"]
[2020-10-30T02:25:09.880Z] [2020/10/30 10:24:51.828 +08:00] [INFO] [pessimist.go:652] ["delete shard DDL lock operations"] [component="shard DDL pessimist"] [lock=test-`shardddl`.`tb`] [revision=114]
[2020-10-30T02:25:09.880Z] [2020/10/30 10:24:51.829 +08:00] [INFO] [pessimist.go:529] ["the lock info for the shard DDL lock operation has been cleared"] [component="shard DDL pessimist"] [operation="{\"id\":\"test-`shardddl`.`tb`\",\"task\":\"test\",\"source\":\"mysql-replica-02\",\"ddls\":[\"ALTER TABLE `shardddl`.`tb` ADD UNIQUE `idx_a`(`a`)\"],\"exec\":false,\"done\":true}"]

worker log:

[2020-10-30T02:25:09.948Z] [2020/10/30 10:24:51.819 +08:00] [INFO] [syncer.go:2013] ["putted shard DDL info"] [task=test] [unit="binlog replication"] [info="{\"task\":\"test\",\"source\":\"mysql-replica-01\",\"schema\":\"shardddl\",\"table\":\"tb\",\"ddls\":[\"ALTER TABLE `shardddl`.`tb` ADD UNIQUE `idx_b`(`b`)\"]}"] [revision=115]

TrySync for DDL2 but DDL1's lock hasn't been removed
Similar to #936 , We also need to add lock in pessimistic. 😢