Skip to content
This repository has been archived by the owner on Nov 24, 2023. It is now read-only.

dm-worker: refine conn (#266) #325

Merged
merged 1 commit into from
Oct 19, 2019
Merged

Conversation

3pointer
Copy link
Contributor

cherry-pick this PR to release-1.0

What problem does this PR solve?

Optimize Database connections in dm-worker units, after unify DB operations

What is changed and how it works?

  1. one worker use one individual connection
  2. same functional connection comes from one DB object
  3. define some package for DB object and connections, all DB object generate in conn.DBProvider.Apply
  4. split upstream DB connection and downstream(worker) DB connection, because they have totally different DB operations, it's better use *sql.DB in upstream, and use individual connection for each worker in downstream.
  5. define two basic object, BaseDB and BaseConn,
    BaseDB generates BaseConn, BaseConn wraps sql.Conn with own retry policy
    BaseDB -> BaseConn correspond to sql.DB -> sql.Conn
    In our scenario, there are two main reasons why we need BaseConn
    1. we often need one fixed DB connection to execute sql
    2. we need own retry policy during execute failed
      So we split a fixed sql.Conn out of sql.DB, and wraps it to BaseConn
      And Similar with sql.Conn, all BaseConn generated from one BaseDB shares this BaseDB to reset

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No code

Code changes

  • Has exported function/method change
  • Has exported variable/fields change
  • Has interface methods change
  • Has persistent data change

Side effects

  • Possible performance regression
  • Increased code complexity
  • Breaking backward compatibility

Related changes

  • Need to cherry-pick to the release branch
  • Need to update the documentation
  • Need to update the dm/dm-ansible
  • Need to be included in the release note

* define conn
split workerConn and upstreamConn in syncer
split conn and db
unify conn in syncer and loader
* change upstream DB to BaseDB
* update raw db config
@csuzhangxc
Copy link
Member

how about changing the title (and the commit message later) to dm-worker: refine conn (#266)

@3pointer 3pointer changed the title cherry-pick: refine conn (#266) dm-worker: refine conn (#266) Oct 18, 2019
@3pointer
Copy link
Contributor Author

how about changing the title (and the commit message later) to dm-worker: refine conn (#266)

sure, and the commit message already be dm-worker: refine conn (#266)

@codecov
Copy link

codecov bot commented Oct 18, 2019

Codecov Report

❗ No coverage uploaded for pull request base (release-1.0@a4412d5). Click here to learn what that means.
The diff coverage is 21.8487%.

@@               Coverage Diff                @@
##             release-1.0       #325   +/-   ##
================================================
  Coverage               ?   59.9333%           
================================================
  Files                  ?        135           
  Lines                  ?      15010           
  Branches               ?          0           
================================================
  Hits                   ?       8996           
  Misses                 ?       5144           
  Partials               ?        870

Copy link
Member

@csuzhangxc csuzhangxc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@csuzhangxc csuzhangxc added priority/normal Minor change, requires approval from ≥1 primary reviewer priority/release-blocker This PR blocks a release. Please review it ASAP. status/LGT1 One reviewer already commented LGTM type/cherry-pick This PR is just a cherry-pick (backport) type/enhancement Performance improvement or refactoring labels Oct 18, 2019
@csuzhangxc
Copy link
Member

@WangXiangUSTC @amyangfei PTAL

Copy link
Contributor

@WangXiangUSTC WangXiangUSTC left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@WangXiangUSTC WangXiangUSTC added status/LGT2 Two reviewers already commented LGTM, ready for merge and removed status/LGT1 One reviewer already commented LGTM labels Oct 18, 2019
@IANTHEREAL IANTHEREAL merged commit f4cf10e into pingcap:release-1.0 Oct 19, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
priority/normal Minor change, requires approval from ≥1 primary reviewer priority/release-blocker This PR blocks a release. Please review it ASAP. status/LGT2 Two reviewers already commented LGTM, ready for merge type/cherry-pick This PR is just a cherry-pick (backport) type/enhancement Performance improvement or refactoring
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants