Skip to content
This repository has been archived by the owner on Nov 24, 2023. It is now read-only.

relay: use Reader interface read binlog events #92

Merged
merged 9 commits into from
Mar 28, 2019

Conversation

csuzhangxc
Copy link
Member

@csuzhangxc csuzhangxc commented Mar 25, 2019

What problem does this PR solve?

This PR is a part work of #91

Introduced a binlog event Reader interface, so we can implement at least 3 kinds of Reader later:

  1. TCPReader: read binlog events from a master server through TCP stream
  2. FileReader: read binlog events from binlog files (or relay log files)
  3. InMemoryReader: read binlog events from memory, used as a mock master to do testing

In this PR, I only implement a TCPReader and refactor a litter code for relay writer to make review easier.

Later, the following things will be done:

  1. split relay writer to reader --> translator --> writer
  2. add an InMemoryReader to test 3 parts above
  3. refactor sync unit (relay reader) to read through a FileReader and use an InMemoryReader to test it

What is changed and how it works?

  • add a TCPReader
  • refactor relay writer to use the TCPReader

Check List

Tests

  • Unit test

Code changes

  • Has interface methods change

@csuzhangxc csuzhangxc added priority/normal Minor change, requires approval from ≥1 primary reviewer status/DNM Do not merge, test is failing or blocked by another PR type/enhancement Performance improvement or refactoring status/WIP This PR is still work in progress labels Mar 25, 2019
@csuzhangxc
Copy link
Member Author

/run-all-tests

@csuzhangxc csuzhangxc changed the title binlog: refactor binlog event reader relay: use Reader interface read binlog events Mar 26, 2019
@csuzhangxc csuzhangxc added status/PTAL This PR is ready for review. Add this label back after committing new changes and removed status/DNM Do not merge, test is failing or blocked by another PR status/WIP This PR is still work in progress labels Mar 26, 2019
@csuzhangxc
Copy link
Member Author

@GregoryIan @amyangfei PTAL
I want to open several PRs for #91, this one is a part of it.

// TCPReaderStatus represents the status of a TCPReader.
type TCPReaderStatus struct {
Stage string `json:"stage"`
Connection uint32 `json:"connection"`
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what's meaning of Connection?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

renamed to ConnID, but keep the json flag as connection.

defer r.stageMu.Unlock()

if r.stage != stageNew {
return errors.NotValidf("stage %d, expect %d", r.stage, stageNew)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is log stage %d, expect %d is not valid?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

changed to stage %s, expect %s only.

r.syncerCfg.User, r.syncerCfg.Password, r.syncerCfg.Host, r.syncerCfg.Port)
db, err := sql.Open("mysql", dsn)
if err != nil {
return errors.Annotate(err, "open connection to the master")
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can output which master

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added

defer db.Close()
err = utils.KillConn(db, connID)
if err != nil {
return errors.Annotatef(err, "kill connection %d", connID)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added

func (r *TCPReader) GetEvent(ctx context.Context, checkStage bool) (*replication.BinlogEvent, error) {
if checkStage {
r.stageMu.Lock()
if r.stage != stagePrepared {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

would atomic operation be better?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

changed to use atomic operation, and removed checkStage.


// StartSyncByGTID implements Reader.StartSyncByGTID.
func (r *TCPReader) StartSyncByGTID(gSet gtid.Set) error {
r.stageMu.Lock()
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in fact this lock prevent more than stage field

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, rename to mu.

@IANTHEREAL
Copy link
Collaborator

Rest LGTM. well done

@csuzhangxc
Copy link
Member Author

/run-all-tests

func (r *TCPReader) Status() interface{} {
r.mu.Lock()
stage := r.stage
r.mu.Unlock()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why only lock for stage getter? besides I find some redundant lock protection, such as

r.mu.Lock()
defer r.mu.Unlock()
r.stage.Get()

Is there any consideration to do that?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I change to use stage = r.stage.Get without lock.

We use stage both in comparing(L150) and recording(L154).
Saving stage in L148, so we can get the same value both in L150 and L154.

@IANTHEREAL
Copy link
Collaborator

LGTM

@IANTHEREAL IANTHEREAL added status/LGT1 One reviewer already commented LGTM and removed status/PTAL This PR is ready for review. Add this label back after committing new changes labels Mar 28, 2019
Copy link
Contributor

@amyangfei amyangfei left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@csuzhangxc csuzhangxc added the status/LGT2 Two reviewers already commented LGTM, ready for merge label Mar 28, 2019
@csuzhangxc csuzhangxc removed the status/LGT1 One reviewer already commented LGTM label Mar 28, 2019
@csuzhangxc csuzhangxc merged commit ae83c32 into pingcap:master Mar 28, 2019
@csuzhangxc csuzhangxc deleted the binlog-interface branch March 28, 2019 12:22
lichunzhu pushed a commit to lichunzhu/dm that referenced this pull request Apr 6, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
priority/normal Minor change, requires approval from ≥1 primary reviewer status/LGT2 Two reviewers already commented LGTM, ready for merge type/enhancement Performance improvement or refactoring
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants