Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: 多个 destination 时的 zk 竞争问题 #68

Merged
merged 44 commits into from
Mar 26, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
44 commits
Select commit Hold shift + click to select a range
6257ef7
Merge pull request #26 from CanalClient/dev
withlin Feb 18, 2019
a31b8db
fix issue #23
withlin Feb 18, 2019
cc16ab4
fix issue #30
withlin May 8, 2019
ff9105f
update readme.md
withlin May 8, 2019
75793b5
Merge pull request #31 from withlin/master
withlin May 8, 2019
b80893a
update readme
withlin May 8, 2019
691cdbf
Merge pull request #32 from withlin/master
withlin May 8, 2019
af79f50
Update README.md
withlin May 9, 2019
3e946ae
Update README.md
withlin May 9, 2019
f4ee096
Update README.md
withlin May 9, 2019
35b8283
Update README.md
withlin Jul 24, 2019
0289443
Update README.md
withlin Jul 24, 2019
fce6a27
Update README.md
withlin Jul 24, 2019
4a3c5a9
Update README.md
withlin Jul 24, 2019
3a7ee23
拆封receiveMessages
jingyugao Oct 10, 2019
2db66ad
Merge pull request #38 from jingyugao/split_receiveMessages
withlin Oct 11, 2019
e1c5383
fix: subscribe filter not work expected
yinheli Nov 8, 2019
b68e735
Merge pull request #41 from yinheli/bugfix/subscribe-filter
withlin Nov 10, 2019
a5be0df
添加.gitignore文件
geange Dec 12, 2019
d7acb7c
1. 修改过滤器的传入参数
geange Dec 12, 2019
4a761e2
变动:
geange Dec 12, 2019
33ac8c2
修改sample的代码 (#45)
withlin Dec 12, 2019
0bb1a3f
ack时判断成功状态以避免一个会引起崩溃的队列顺序问题
Feb 20, 2020
96bf8b8
Merge pull request #48 from open-source-go/del_dev
withlin Feb 27, 2020
10f49c9
重写cluster模式
Mar 22, 2020
1083968
重写cluster模式
Mar 22, 2020
5df0df2
重写cluster模式
Mar 22, 2020
d552e74
Merge pull request #50 from Xsxdot/master
withlin Mar 23, 2020
3d17120
fixed lazyParseEntry模式Message.Raw
yntor Apr 2, 2020
37b9e9d
Merge pull request #52 from yntor/master
withlin Apr 5, 2020
958ec01
fix:import error packege
Apr 17, 2020
5678e80
fix: del unuse code
Apr 20, 2020
030ecaa
Merge pull request #54 from wangfeiso/master
withlin Apr 21, 2020
b21f232
Fix package URL not working
withlin Apr 25, 2020
fbdaa41
Merge pull request #55 from withlin/package
withlin Apr 25, 2020
91a8adf
修改CanalConnector接口定义
Jul 31, 2020
1df6805
Merge pull request #59 from go-xman/master
withlin Aug 3, 2020
0f57720
fix #60
withlin Aug 8, 2020
2e53b24
upgrade golang version
withlin Aug 8, 2020
9540de9
fix ci
withlin Aug 8, 2020
72ff485
Update README.md
withlin Aug 8, 2020
98f8bee
增加密码Auth #46
stulzq Nov 5, 2020
b7c5331
Merge pull request #63 from stulzq/master
withlin Nov 5, 2020
5ae733f
fix: 多个 destination 时的 zk 竞争问题
Mar 19, 2021
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 22 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
# Created by .ignore support plugin (hsz.mobi)
### Go template
# Binaries for programs and plugins
*.exe
*.exe~
*.dll
*.so
*.dylib

# Test binary, built with `go test -c`
*.test

# Output of the go coverage tool, specifically when used with LiteIDE
*.out

# Dependency directories (remove the comment below to include it)
# vendor/

.vscode
.idea

vendor
5 changes: 2 additions & 3 deletions .travis.yml
Original file line number Diff line number Diff line change
@@ -1,7 +1,6 @@
language: go

go:
- "1.10.x"
- "1.11.x"
- "1.14.x"

go_import_path: github.com/CanalClient/canal-go
go_import_path: github.com/withlin/canal-go
24 changes: 0 additions & 24 deletions .vscode/launch.json

This file was deleted.

17 changes: 0 additions & 17 deletions Gopkg.lock

This file was deleted.

34 changes: 0 additions & 34 deletions Gopkg.toml

This file was deleted.

177 changes: 3 additions & 174 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,176 +1,5 @@

# canal-go |[中文](https://github.com/CanalSharp/canal-go/blob/master/README.zh-cn.md)
# canal-go

[![Build Status](https://travis-ci.org/CanalClient/canal-go.svg?branch=master)](https://travis-ci.org/CanalSharp/canal-go)

## Ⅰ.What is canal-go

canal-go is a golang Client of alibaba's open source component `Canal`,and it provides golang developer a much easier way to use Canal.

`Cancal` is a incremental Publish&Subscription component based on mysql's `binlog`.To learn more a bout Cancal,please visit ` https://github.com/alibaba/canal/wiki`

log based incremental pub&sub model can be applied to many business cases:

1) Database Mirroring
2) Realtime Database backup
3) Multi-level index
4) search build
5) Busienss cache refresh
6) Notification on price change(in e-commerce site) and so on

## Ⅱ.Using scenarios

`canal-go` as a golang client of canal,it can be used in situations where Canal is being used.We have listed some using scenarios above,here are some more detailed use cases:

1) As a substitution of polling Database to monitor Database changes,it is much more effective than polling.

2) Realtime Update data in search engine when data in mysql database changed.For example,in a E-commerce site,when Product information changed,it can be updated to Elasticsearch、solr,etc in a realtime way.

3) Realtime update data in cache,Also take E-commerce site as an example,when price,inventory and other product information changed,it can be updated to Redis in a realtime way.

4) Database Backup,synchronization.

5) To trigger some other business logics on data change,for example,when a customer place an order without payment in a certain period of time and so the order is cancelled automatically,we get the changed status of the order and then we can push a notification to the customer(or take some other actions)

6) Push changed data to RabbitMq,kafak or other message queues for consuming.

## Ⅲ.Working mechanism

As a golang client of Canal,it use `Socket` for communication,the transfer Protocol is `TCP`,and the data exchange Protocol is Google's Protocol Buffer 3.0

## Ⅳ.Workflow

1) Canal connect to Mysql database ,pretented to be a slave of mysql

2) canal-go(as mentioned it is a client of Cancal) connect to Canal

3) When data in the database changed,it write the changes to binlog

4) Canal send dump request to mysql database,get binlog and then parse it.

5) canal-go send request to Canal for data.

6) Canal send parsed data to canal-go

7) canal-go get data successfully and then send an acknowledgement to Canal(optional)

8) Canal record the consumption position

The following picture shows how it works:


![1537860226808](assets/668104-20180925182816462-2110152563.png)

## Ⅴ.Quick Start

### 1. Install Canal

To Install and configure Canal,please visit `https://github.com/alibaba/canal/wiki/QuickStart` for information

### 2.Create a golang Console Project

### 3. Install go get from github source

````shell
go get github.com/CanalSharp/canal-go
````

or you can install it from nuget manager(a graphic ui) in visual studio

### 4. Connect to Canal


````golang

connector := client.NewSimpleCanalConnector("192.168.199.17", 11111, "", "", "example", 60000, 60*60*1000)
err :=connector.Connect()
if err != nil {
log.Println(err)
os.Exit(1)
}
err = connector.Subscribe(".*\\\\..*")
if err != nil {
log.Println(err)
os.Exit(1)
}

for {

message,err := connector.Get(100, nil, nil)
if err != nil {
log.Println(err)
os.Exit(1)
}
batchId := message.Id
if batchId == -1 || len(message.Entries) <= 0 {
time.Sleep(300 * time.Millisecond)
fmt.Println("===没有数据了===")
continue
}

printEntry(message.Entries)

}

````

for more detailed information please visit [Sample](https://github.com/CanalSharp/canal-go/tree/master/samples)

## Ⅵ Run canal-go via Docker quickly

### 1. Run mysql and canal by command

````shell
git clone https://github.com/CanalClient/canal-go.git
cd canal-go
cd docker
docker-compose up -d
````

### 2. Use navicat or orther client to connect to mysql

ip:the ip address of the server where docker running

mysql username:root

mysql password:000000

mysql port:4406

There is a default database named `test`,which contains a table named `test`

![1537866852816](assets/668104-20180925182815646-1209020640.png)

### 3.run the sample project

### 4.Test it

Execute the following sql:

````sql
insert into test values(1000,'111');
update test set name='222' where id=1000;
delete from test where id=1000;
````

![](assets/ys.gif)

we can see that after executing `insrt,update,delete` sql,Our canal-go get the changed data.



## Ⅶ What we will do next

canal-go clustering support

## Ⅷ Contribute

We gladly accept community contributions.

1.fork the canal-go Project

2.make changes to it

3.make a pull request

Please do not hesitate to make a pull request,your effort will not in vain.
[![Build Status](https://travis-ci.org/withlin/canal-go.svg?branch=master)](https://travis-ci.org/withlin/canal-go)
[![Go Report Card](https://goreportcard.com/badge/github.com/withlin/canal-go)](https://goreportcard.com/badge/github.com/withlin/canal-go)
Loading