Skip to content

Commit

Permalink
Merge remote-tracking branch 'upstream/dev' into dev
Browse files Browse the repository at this point in the history
  • Loading branch information
beckhampu committed Oct 9, 2018
2 parents 45eafeb + a738d1b commit d86a31d
Show file tree
Hide file tree
Showing 23 changed files with 309 additions and 209 deletions.
69 changes: 37 additions & 32 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,24 +21,26 @@ Official website: http://shardingsphere.io/

## Overview

Sharding-Sphere is an open source distributed database middleware solution suite, which consists of Sharding-JDBC, Sharding-Proxy and Sharding-Sidecar(planning). These three sub-projects are suitable for various scenarios and provide consistent functions including data sharding, distributed transaction and database orchestration.
Sharding-Sphere is an open-source ecosystem consisted of a set of distributed database middleware solution, including 3 independent products, Sharding-JDBC, Sharding-Proxy & Sharding-Sidecar (todo).
They all provide functions of data sharding, distributed transaction and database orchestration, applicable in a variety of situations such as Java isomorphism, heterogeneous language and cloud native.

As a Relational Database Middleware, rather than implementing a new database, Sharding-Sphere is aimed at making the most of original capacities of database (like compute and storage) and making it simpler and more efficient on distribute environment.
It can cooperate with NoSQL and NewSQL, which keep staying in the cutting edge of technology and recommendable. Meanwhile, Sharding-Sphere focuses on technologies remaining stable and changeless so as to grasp the intrinsic quality.
Relational Database is still the cornerstone of core business in enterprises and possesses huge market share. It is difficult to estimate trendy in future and now we focus on how to enhance the capacity of Sharding-Sphere with Relational Database.
Aiming at reasonably making full use of the computation and storage capacity of database in distributed system, Sharding-Sphere defines itself as a middleware, rather than a totally new type of database.
As the cornerstone of many enterprises, relational database still takes a huge market share.
Therefore, at current stage, we prefer to focus on its increment instead of a total overturn.

![Sharding-Sphere Scope](http://ovfotjrsi.bkt.clouddn.com/sharding-sphere-scope_en.png)
![Sharding-Sphere Scope](http://ovfotjrsi.bkt.clouddn.com/sharding-sphere-scope_en_v2.png)

### Sharding-JDBC

[![Maven Status](https://maven-badges.herokuapp.com/maven-central/io.shardingsphere/sharding-jdbc/badge.svg)](https://mvnrepository.com/artifact/io.shardingsphere/sharding-jdbc)

Sharding-JDBC is positioned as a lightweight Java framework that provides Sharding service in the JDBC layer.
It directly connects to databases to provide services in the form of jar, with no additional deployment and dependencies, and it can be understood as an enhanced version of the JDBC driver, fully compliant with JDBC and various ORM frameworks.
Sharding-JDBC defines itself as a lightweight Java framework that provides extra service at Java JDBC layer.
With client end connecting directly to the database, it provides service in the form of jar and requires no extra deployment and dependence.
It can be considered as an enhanced JDBC driver, which is fully compatible with JDBC and all kinds of ORM frameworks.

* It is compliant with any Java-based ORM framework, such as JPA, Hibernate, Mybatis and Spring JDBC Template; or you can use JDBC directly.
* It is suitable for connection-pool based on any third party, such as DBCP, C3P0, BoneCP, Druid, HikariCP and so on.
* It supports databases implementing JDBC specification. Supporting MySQL, Oracle, SQLServer and PostgreSQL right now.
* Applicable in any ORM framework based on Java, such as JPA, Hibernate, Mybatis, Spring JDBC Template or direct use of JDBC.
* Based on any third-party database connection pool, such as DBCP, C3P0, BoneCP, Druid, HikariCP.
* Support any kind of database that conforms to JDBC standard: MySQL,Oracle,SQLServer and PostgreSQL for now.

![Sharding-JDBC Architecture](http://ovfotjrsi.bkt.clouddn.com/sharding-jdbc-brief.png)

Expand All @@ -47,38 +49,41 @@ It directly connects to databases to provide services in the form of jar, with n
[![Download](https://img.shields.io/badge/release-download-orange.svg)](https://github.com/sharding-sphere/sharding-sphere-doc/raw/master/dist/sharding-proxy-3.0.0.M4.tar.gz)
[![Docker Pulls](https://img.shields.io/docker/pulls/shardingsphere/sharding-proxy.svg)](https://store.docker.com/community/images/shardingsphere/sharding-proxy)

Sharding-Proxy is positioned as a transparent database proxy, providing a database server that encapsulates database binary protocol to support heterogeneous languages.
At present, we provide Sharding-Proxy for MySQL, and you can use any clients compatible with MySQL protocol (such as MySQL Command Client, MySQL Workbench, etc.) to manipulate data.
Sharding-Proxy defines itself as a transparent database proxy, providing a database server that encapsulates database binary protocol to support heterogeneous languages.
Friendlier to DBA, the MySQL version provided now can use any kind of client access (such as MySQL Command Client, MySQL Workbench, etc.) that is compatible of MySQL protocol to operate data.

* It is transparent to application, which can be used as MySQL.
* It is suitable for any clients compatible with MySQL protocol.
* Totally transparent to applications, it can be used directly as MySQL.
* Applicable to any kind of compatible of client end that is compatible of MySQL protocol.

![Sharding-Proxy Architecture](http://ovfotjrsi.bkt.clouddn.com/sharding-proxy-brief_v2.png)

### Sharding-Sidecar(TBD)

It is positioned as cloud native database agent of Kubernetes or Mesos environment, in charge of all database access in the form of DaemonSet.
Through a centre-less and zero-cost solution, it provides mesh layer interacting with database. We call it `Database Mesh`.
Sharding-Sidecar (TBD) defines itself as a cloud native database agent of Kubernetes or Mesos environment, in charge of all the access to database in the form of DaemonSet.
Through a decentralized and zero-cost solution, it provides a mesh layer interacting with database, i.e., Database Mesh, also referred to as database grid.

Database Mesh emphasizes how to connect distributed data-access-layer with databases. It pays more attention to interaction, which means the messy interaction between applications and databases will be effectively orchestrated.
By using Database Mesh, applications and databases will form a large grid system, where they just need to be put into the right position accordingly. They are all governed by mesh layer.
Database Mesh emphasizes on how to connect distributed database access application with the database.
Focusing on interaction, it effectively organizes the interaction between messy applications and database.
The application and database that use Database Mesh to visit database will form a large grid system, where they just need to be put into the right position accordingly.
They are all governed by mesh layer.

![Sharding-Sidecar Architecture](http://ovfotjrsi.bkt.clouddn.com/sharding-sidecar-brief_v2.png)

| | *Sharding-JDBC* | *Sharding-Proxy* | *Sharding-Sidecar* |
| ---------------------- | --------------- | ---------------- | ------------------ |
| Database | Any | MySQL | MySQL |
| Connections Cost | More | Less | More |
| Heterogeneous Language | Java Only | Any | Any |
| Performance | Low loss | High loss | Low loss |
| Centre-less | Yes | No | No |
| Static Entry | No | Yes | No |
| | *Sharding-JDBC* | *Sharding-Proxy* | *Sharding-Sidecar* |
| ----------------------- | --------------- | -------------------- | ------------------ |
| Database | Any | MySQL | MySQL |
| Connections Cost Number | High | Low | High |
| Heterogeneous Language | Java Only | Any | Any |
| Performance | Low loss | Relatively High loss | Low loss |
| Decentralization | Yes | No | No |
| Static Entry | No | Yes | No |


## Features

### Data Sharding

* Both databases and tables sharding
* Database sharding & Table sharding
* Read-write splitting
* Distributed primary key

Expand All @@ -89,12 +94,12 @@ By using Database Mesh, applications and databases will form a large grid system

### Database Orchestration

* Configuration center
* Circuit breaker & disable
* Dynamic configuration
* Fusing & Disabling
* Open tracing
* Data replica (Planing)
* Elastic scaling (Planing)
* Multiple data replication (Planing)
* Elastic scaling out (Planing)

## Roadmap

![Roadmap](http://ovfotjrsi.bkt.clouddn.com/roadmap_en_v2.png)
![Roadmap](http://ovfotjrsi.bkt.clouddn.com/roadmap_en_v3.png)
13 changes: 8 additions & 5 deletions README_ZH.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,10 +19,12 @@

## 概述

Sharding-Sphere是一套开源的分布式数据库中间件解决方案组成的生态圈,它由Sharding-JDBC、Sharding-Proxy和Sharding-Sidecar(计划中)这3款相互独立的产品组成。他们均提供标准化的数据分片、分布式事务和数据库治理功能,可适用于如Java同构、异构语言、容器、云原生等各种多样化的应用场景。
Sharding-Sphere是一套开源的分布式数据库中间件解决方案组成的生态圈,它由Sharding-JDBC、Sharding-Proxy和Sharding-Sidecar(计划中)这3款相互独立的产品组成。
他们均提供标准化的数据分片、分布式事务和数据库治理功能,可适用于如Java同构、异构语言、容器、云原生等各种多样化的应用场景。

Sharding-Sphere定位为关系型数据库中间件,旨在充分合理地在分布式的场景下利用关系型数据库的计算和存储能力,而并非实现一个全新的关系型数据库。
它与NoSQL和NewSQL是并存而非互斥的关系。NoSQL和NewSQL作为新技术探索的前沿,放眼未来,拥抱变化,是非常值得推荐的。反之,也可以用另一种思路看待问题,放眼未来,关注不变的东西,进而抓住事物本质。关系型数据库当今依然占有巨大市场,是各个公司核心业务的基石,未来也难于撼动,我们目前阶段更加关注在原有基础上的增量,而非颠覆。
它与NoSQL和NewSQL是并存而非互斥的关系。NoSQL和NewSQL作为新技术探索的前沿,放眼未来,拥抱变化,是非常值得推荐的。反之,也可以用另一种思路看待问题,放眼未来,关注不变的东西,进而抓住事物本质。
关系型数据库当今依然占有巨大市场,是各个公司核心业务的基石,未来也难于撼动,我们目前阶段更加关注在原有基础上的增量,而非颠覆。

![Sharding-Sphere Scope](http://ovfotjrsi.bkt.clouddn.com/sharding-sphere-scope_cn.png)

Expand Down Expand Up @@ -55,9 +57,10 @@ Sharding-Sphere定位为关系型数据库中间件,旨在充分合理地在
### Sharding-Sidecar(TBD)

定位为Kubernetes或Mesos的云原生数据库代理,以DaemonSet的形式代理所有对数据库的访问。
通过无中心、零侵入的方案提供与数据库交互的的啮合层,即Database Mesh,又可称数据网格
通过无中心、零侵入的方案提供与数据库交互的的啮合层,即Database Mesh,又可称数据库网格

Database Mesh的关注重点在于如何将分布式的数据访问应用与数据库有机串联起来,它更加关注的是交互,是将杂乱无章的应用与数据库之间的交互有效的梳理。使用Database Mesh,访问数据库的应用和数据库终将形成一个巨大的网格体系,应用和数据库只需在网格体系中对号入座即可,它们都是被啮合层所治理的对象。
Database Mesh的关注重点在于如何将分布式的数据访问应用与数据库有机串联起来,它更加关注的是交互,是将杂乱无章的应用与数据库之间的交互有效的梳理。
使用Database Mesh,访问数据库的应用和数据库终将形成一个巨大的网格体系,应用和数据库只需在网格体系中对号入座即可,它们都是被啮合层所治理的对象。

![Sharding-Sidecar Architecture](http://ovfotjrsi.bkt.clouddn.com/sharding-sidecar-brief_v2.png)

Expand All @@ -80,7 +83,7 @@ Database Mesh的关注重点在于如何将分布式的数据访问应用与数

### 分布式事务(Doing)

* XA分布式事务
* XA强一致事务
* 柔性事务

### 数据库治理
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -22,14 +22,14 @@
import lombok.Setter;

/**
* Proxy authority.
* Authentication.
*
* @author panjuan
*/
@NoArgsConstructor
@Getter
@Setter
public final class ProxyAuthority {
public final class Authentication {

private String username;

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@

package io.shardingsphere.core.yaml.other;

import io.shardingsphere.core.rule.ProxyAuthority;
import io.shardingsphere.core.rule.Authentication;
import lombok.Getter;
import lombok.Setter;

Expand All @@ -32,7 +32,7 @@
@Setter
public final class YamlServerConfiguration {

private ProxyAuthority proxyAuthority = new ProxyAuthority();
private Authentication authentication = new Authentication();

private Properties props = new Properties();
}
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,7 @@ public MasterSlaveStatement(final MasterSlaveConnection connection, final int re

@Override
public ResultSet executeQuery(final String sql) throws SQLException {
clearPrevious();
Collection<String> dataSourceNames = masterSlaveRouter.route(sql);
Preconditions.checkState(1 == dataSourceNames.size(), "Cannot support executeQuery for DML or DDL");
Statement statement = connection.getConnection(dataSourceNames.iterator().next()).createStatement(resultSetType, resultSetConcurrency, resultSetHoldability);
Expand All @@ -80,6 +81,7 @@ public ResultSet executeQuery(final String sql) throws SQLException {

@Override
public int executeUpdate(final String sql) throws SQLException {
clearPrevious();
int result = 0;
for (String each : masterSlaveRouter.route(sql)) {
Statement statement = connection.getConnection(each).createStatement(resultSetType, resultSetConcurrency, resultSetHoldability);
Expand All @@ -91,6 +93,7 @@ public int executeUpdate(final String sql) throws SQLException {

@Override
public int executeUpdate(final String sql, final int autoGeneratedKeys) throws SQLException {
clearPrevious();
int result = 0;
for (String each : masterSlaveRouter.route(sql)) {
Statement statement = connection.getConnection(each).createStatement(resultSetType, resultSetConcurrency, resultSetHoldability);
Expand All @@ -102,6 +105,7 @@ public int executeUpdate(final String sql, final int autoGeneratedKeys) throws S

@Override
public int executeUpdate(final String sql, final int[] columnIndexes) throws SQLException {
clearPrevious();
int result = 0;
for (String each : masterSlaveRouter.route(sql)) {
Statement statement = connection.getConnection(each).createStatement(resultSetType, resultSetConcurrency, resultSetHoldability);
Expand All @@ -113,6 +117,7 @@ public int executeUpdate(final String sql, final int[] columnIndexes) throws SQL

@Override
public int executeUpdate(final String sql, final String[] columnNames) throws SQLException {
clearPrevious();
int result = 0;
for (String each : masterSlaveRouter.route(sql)) {
Statement statement = connection.getConnection(each).createStatement(resultSetType, resultSetConcurrency, resultSetHoldability);
Expand All @@ -124,6 +129,7 @@ public int executeUpdate(final String sql, final String[] columnNames) throws SQ

@Override
public boolean execute(final String sql) throws SQLException {
clearPrevious();
boolean result = false;
for (String each : masterSlaveRouter.route(sql)) {
Statement statement = connection.getConnection(each).createStatement(resultSetType, resultSetConcurrency, resultSetHoldability);
Expand All @@ -135,6 +141,7 @@ public boolean execute(final String sql) throws SQLException {

@Override
public boolean execute(final String sql, final int autoGeneratedKeys) throws SQLException {
clearPrevious();
boolean result = false;
for (String each : masterSlaveRouter.route(sql)) {
Statement statement = connection.getConnection(each).createStatement(resultSetType, resultSetConcurrency, resultSetHoldability);
Expand All @@ -146,6 +153,7 @@ public boolean execute(final String sql, final int autoGeneratedKeys) throws SQL

@Override
public boolean execute(final String sql, final int[] columnIndexes) throws SQLException {
clearPrevious();
boolean result = false;
for (String each : masterSlaveRouter.route(sql)) {
Statement statement = connection.getConnection(each).createStatement(resultSetType, resultSetConcurrency, resultSetHoldability);
Expand All @@ -157,6 +165,7 @@ public boolean execute(final String sql, final int[] columnIndexes) throws SQLEx

@Override
public boolean execute(final String sql, final String[] columnNames) throws SQLException {
clearPrevious();
boolean result = false;
for (String each : masterSlaveRouter.route(sql)) {
Statement statement = connection.getConnection(each).createStatement(resultSetType, resultSetConcurrency, resultSetHoldability);
Expand All @@ -177,4 +186,11 @@ public ResultSet getResultSet() throws SQLException {
Preconditions.checkState(1 == routedStatements.size());
return routedStatements.iterator().next().getResultSet();
}

private void clearPrevious() throws SQLException {
for (Statement each : routedStatements) {
each.close();
}
routedStatements.clear();
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,10 @@
<assertion expected-data-file="insert_for_order_1_2.xml" />
</dml-test-case>

<dml-test-case sql-case-id="insert_on_duplicate_key_update">
<assertion parameters="1000:int, 10:int, update:String" expected-data-file="update.xml" />
</dml-test-case>

<!-- TODO fixme -->
<!--<dml-test-case sql-case-id="insert_with_all_placeholders_for_table_identifier">-->
<!--<assertion parameters="1:int, 1:int, insert:String" expected-data-file="insert_for_order_1.xml" />-->
Expand Down Expand Up @@ -70,14 +74,13 @@
<!--<assertion parameters="1:int, 1:int" expected-data-file="insert_with_generate_key_column.xml" />-->
</dml-test-case>

<!-- TODO parse generator context for insert set -->
<!--<dml-test-case sql-case-id="insert_without_generate_key_column">-->
<!--<assertion parameters="1:int, 1:int" expected-data-file="insert_without_generate_key_column.xml" />-->
<!--</dml-test-case>-->
<!---->
<!--<dml-test-case sql-case-id="insert_set_without_generate_key_column">-->
<!--<assertion parameters="1:int, 1:int" expected-data-file="insert_without_generate_key_column.xml" />-->
<!--</dml-test-case>-->
<dml-test-case sql-case-id="insert_without_generate_key_column">
<assertion parameters="1:int, 1:int" expected-data-file="insert_without_generate_key_column.xml" />
</dml-test-case>

<dml-test-case sql-case-id="insert_set_without_generate_key_column">
<assertion parameters="1:int, 1:int" expected-data-file="insert_without_generate_key_column.xml" />
</dml-test-case>

<dml-test-case sql-case-id="insert_with_batch">
<assertion parameters="1:int, 1:int, insert:String, 2:int, 2:int, insert:String" expected-data-file="batch_insert_for_order.xml" />
Expand Down
Loading

0 comments on commit d86a31d

Please sign in to comment.