task paused frequently because of "invalid connection"

[csuzhangxc]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. What did you do? If possible, provide a recipe for reproducing the error.

running the task.

2. What did you expect to see?

the task keep running.

3. What did you see instead?

task paused frequently because of "invalid connection".

4. Versions of the cluster

   • DM version (run dmctl -V or dm-worker -V or dm-master -V):

     until the version below, the problem exists.

     Release Version: v1.0.0-alpha-10-g4d01d79
     Git Commit Hash: 4d01d798415e835417bb6db7250c56fa2d964d47
     Git Branch: master
     UTC Build Time: 2019-02-11 14:50:57
     Go Version: go version go1.11.2 linux/amd64
     

5. Operation logs
   DM-worker's log

       2019/01/29 05:27:54.883 subtask.go:213: [error] [subtask] vccs_sharding_merge dm-unit Sync process error with type ExecSQL:
       invalid connection
       github.com/pingcap/errors.AddStack
               /home/jenkins/workspace/build_dm/go/pkg/mod/github.com/pingcap/[email protected]/errors.go:174
       github.com/pingcap/errors.Trace
               /home/jenkins/workspace/build_dm/go/pkg/mod/github.com/pingcap/[email protected]/juju_adaptor.go:12
       github.com/pingcap/dm/syncer.(*Conn).executeSQLJobImp
               /home/jenkins/workspace/build_dm/go/src/github.com/pingcap/dm/syncer/db.go:242
       github.com/pingcap/dm/syncer.(*Conn).executeSQLJob
               /home/jenkins/workspace/build_dm/go/src/github.com/pingcap/dm/syncer/db.go:195
       github.com/pingcap/dm/syncer.(*Syncer).sync.func3
               /home/jenkins/workspace/build_dm/go/src/github.com/pingcap/dm/syncer/syncer.go:733
       github.com/pingcap/dm/syncer.(*Syncer).sync
       github.com/pingcap/errors.Trace
               /home/jenkins/workspace/build_dm/go/pkg/mod/github.com/pingcap/[email protected]/juju_adaptor.go:12
       github.com/pingcap/dm/syncer.(*Conn).executeSQLJobImp
               /home/jenkins/workspace/build_dm/go/src/github.com/pingcap/dm/syncer/db.go:242
       github.com/pingcap/dm/syncer.(*Conn).executeSQLJob
               /home/jenkins/workspace/build_dm/go/src/github.com/pingcap/dm/syncer/db.go:195
       github.com/pingcap/dm/syncer.(*Syncer).sync.func3
               /home/jenkins/workspace/build_dm/go/src/github.com/pingcap/dm/syncer/syncer.go:733
       github.com/pingcap/dm/syncer.(*Syncer).sync
               /home/jenkins/workspace/build_dm/go/src/github.com/pingcap/dm/syncer/syncer.go:806
       github.com/pingcap/dm/syncer.(*Syncer).Run.func2
               /home/jenkins/workspace/build_dm/go/src/github.com/pingcap/dm/syncer/syncer.go:867
       runtime.goexit
               /usr/local/go/src/runtime/asm_amd64.s:1333
   

[csuzhangxc]

from @WangXiangUSTC, sync_diff_inspector seems solved invalid connection error through set SetMaxOpenConns/SetMaxIdleConns. ref https://github.com/pingcap/tidb-tools/blob/1e17541441200466ec9ca2f59c70e45c35bef827/sync_diff_inspector/diff.go#L101

an issue at go-sql-driver(go-sql-driver/mysql#654) maybe useful.

[csuzhangxc]

another log but with the error when doing BEGIN

2019/02/19 10:30:23 db.go:206: [error] exec sqls[***, key: 3176, last_pos: (mysql-3306|000001.000022, 1072473564), cmd_pos: (mysql-3306|000001.000022, 1072473914), gtid:<nil>]] begin failed invalid connection

[csuzhangxc]

add retry in #66