-
Notifications
You must be signed in to change notification settings - Fork 169
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
fix: 多个 destination 时的 zk 竞争问题 (#68)
- Loading branch information
1 parent
077c474
commit 8135ff3
Showing
36 changed files
with
564 additions
and
12,443 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,22 @@ | ||
# Created by .ignore support plugin (hsz.mobi) | ||
### Go template | ||
# Binaries for programs and plugins | ||
*.exe | ||
*.exe~ | ||
*.dll | ||
*.so | ||
*.dylib | ||
|
||
# Test binary, built with `go test -c` | ||
*.test | ||
|
||
# Output of the go coverage tool, specifically when used with LiteIDE | ||
*.out | ||
|
||
# Dependency directories (remove the comment below to include it) | ||
# vendor/ | ||
|
||
.vscode | ||
.idea | ||
|
||
vendor |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,7 +1,6 @@ | ||
language: go | ||
|
||
go: | ||
- "1.10.x" | ||
- "1.11.x" | ||
- "1.14.x" | ||
|
||
go_import_path: github.com/CanalClient/canal-go | ||
go_import_path: github.com/withlin/canal-go |
This file was deleted.
Oops, something went wrong.
This file was deleted.
Oops, something went wrong.
This file was deleted.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,176 +1,5 @@ | ||
|
||
# canal-go |[中文](https://github.com/CanalSharp/canal-go/blob/master/README.zh-cn.md) | ||
# canal-go | ||
|
||
[![Build Status](https://travis-ci.org/CanalClient/canal-go.svg?branch=master)](https://travis-ci.org/CanalSharp/canal-go) | ||
|
||
## Ⅰ.What is canal-go | ||
|
||
canal-go is a golang Client of alibaba's open source component `Canal`,and it provides golang developer a much easier way to use Canal. | ||
|
||
`Cancal` is a incremental Publish&Subscription component based on mysql's `binlog`.To learn more a bout Cancal,please visit ` https://github.com/alibaba/canal/wiki` | ||
|
||
log based incremental pub&sub model can be applied to many business cases: | ||
|
||
1) Database Mirroring | ||
2) Realtime Database backup | ||
3) Multi-level index | ||
4) search build | ||
5) Busienss cache refresh | ||
6) Notification on price change(in e-commerce site) and so on | ||
|
||
## Ⅱ.Using scenarios | ||
|
||
`canal-go` as a golang client of canal,it can be used in situations where Canal is being used.We have listed some using scenarios above,here are some more detailed use cases: | ||
|
||
1) As a substitution of polling Database to monitor Database changes,it is much more effective than polling. | ||
|
||
2) Realtime Update data in search engine when data in mysql database changed.For example,in a E-commerce site,when Product information changed,it can be updated to Elasticsearch、solr,etc in a realtime way. | ||
|
||
3) Realtime update data in cache,Also take E-commerce site as an example,when price,inventory and other product information changed,it can be updated to Redis in a realtime way. | ||
|
||
4) Database Backup,synchronization. | ||
|
||
5) To trigger some other business logics on data change,for example,when a customer place an order without payment in a certain period of time and so the order is cancelled automatically,we get the changed status of the order and then we can push a notification to the customer(or take some other actions) | ||
|
||
6) Push changed data to RabbitMq,kafak or other message queues for consuming. | ||
|
||
## Ⅲ.Working mechanism | ||
|
||
As a golang client of Canal,it use `Socket` for communication,the transfer Protocol is `TCP`,and the data exchange Protocol is Google's Protocol Buffer 3.0 | ||
|
||
## Ⅳ.Workflow | ||
|
||
1) Canal connect to Mysql database ,pretented to be a slave of mysql | ||
|
||
2) canal-go(as mentioned it is a client of Cancal) connect to Canal | ||
|
||
3) When data in the database changed,it write the changes to binlog | ||
|
||
4) Canal send dump request to mysql database,get binlog and then parse it. | ||
|
||
5) canal-go send request to Canal for data. | ||
|
||
6) Canal send parsed data to canal-go | ||
|
||
7) canal-go get data successfully and then send an acknowledgement to Canal(optional) | ||
|
||
8) Canal record the consumption position | ||
|
||
The following picture shows how it works: | ||
|
||
|
||
![1537860226808](assets/668104-20180925182816462-2110152563.png) | ||
|
||
## Ⅴ.Quick Start | ||
|
||
### 1. Install Canal | ||
|
||
To Install and configure Canal,please visit `https://github.com/alibaba/canal/wiki/QuickStart` for information | ||
|
||
### 2.Create a golang Console Project | ||
|
||
### 3. Install go get from github source | ||
|
||
````shell | ||
go get github.com/CanalSharp/canal-go | ||
```` | ||
|
||
or you can install it from nuget manager(a graphic ui) in visual studio | ||
|
||
### 4. Connect to Canal | ||
|
||
|
||
````golang | ||
|
||
connector := client.NewSimpleCanalConnector("192.168.199.17", 11111, "", "", "example", 60000, 60*60*1000) | ||
err :=connector.Connect() | ||
if err != nil { | ||
log.Println(err) | ||
os.Exit(1) | ||
} | ||
err = connector.Subscribe(".*\\\\..*") | ||
if err != nil { | ||
log.Println(err) | ||
os.Exit(1) | ||
} | ||
|
||
for { | ||
|
||
message,err := connector.Get(100, nil, nil) | ||
if err != nil { | ||
log.Println(err) | ||
os.Exit(1) | ||
} | ||
batchId := message.Id | ||
if batchId == -1 || len(message.Entries) <= 0 { | ||
time.Sleep(300 * time.Millisecond) | ||
fmt.Println("===没有数据了===") | ||
continue | ||
} | ||
|
||
printEntry(message.Entries) | ||
|
||
} | ||
|
||
```` | ||
|
||
for more detailed information please visit [Sample](https://github.com/CanalSharp/canal-go/tree/master/samples) | ||
|
||
## Ⅵ Run canal-go via Docker quickly | ||
|
||
### 1. Run mysql and canal by command | ||
|
||
````shell | ||
git clone https://github.com/CanalClient/canal-go.git | ||
cd canal-go | ||
cd docker | ||
docker-compose up -d | ||
```` | ||
|
||
### 2. Use navicat or orther client to connect to mysql | ||
|
||
ip:the ip address of the server where docker running | ||
|
||
mysql username:root | ||
|
||
mysql password:000000 | ||
|
||
mysql port:4406 | ||
|
||
There is a default database named `test`,which contains a table named `test` | ||
|
||
![1537866852816](assets/668104-20180925182815646-1209020640.png) | ||
|
||
### 3.run the sample project | ||
|
||
### 4.Test it | ||
|
||
Execute the following sql: | ||
|
||
````sql | ||
insert into test values(1000,'111'); | ||
update test set name='222' where id=1000; | ||
delete from test where id=1000; | ||
```` | ||
|
||
![](assets/ys.gif) | ||
|
||
we can see that after executing `insrt,update,delete` sql,Our canal-go get the changed data. | ||
|
||
|
||
|
||
## Ⅶ What we will do next | ||
|
||
canal-go clustering support | ||
|
||
## Ⅷ Contribute | ||
|
||
We gladly accept community contributions. | ||
|
||
1.fork the canal-go Project | ||
|
||
2.make changes to it | ||
|
||
3.make a pull request | ||
|
||
Please do not hesitate to make a pull request,your effort will not in vain. | ||
[![Build Status](https://travis-ci.org/withlin/canal-go.svg?branch=master)](https://travis-ci.org/withlin/canal-go) | ||
[![Go Report Card](https://goreportcard.com/badge/github.com/withlin/canal-go)](https://goreportcard.com/badge/github.com/withlin/canal-go) |
Oops, something went wrong.