TiSpark is a thin layer built for running Apache Spark on top of TiDB/TiKV/TiFlash to answer complex OLAP queries. It enjoys the merits of both the Spark platform and the distributed clusters of TiKV/TiFlash while seamlessly integrated to TiDB.
The figure below show the architecture of TiSpark.
- TiSpark integrates well with the Spark Catalyst Engine. It provides precise control of computing, which allows Spark to read data from TiKV efficiently. It also supports index seek, which significantly improves the performance of the point query execution.
- It utilizes several strategies to push down computing to reduce the size of dataset handling by Spark SQL, which accelerates query execution. It also uses the TiDB built-in statistical information for the query plan optimization.
- From the perspective of data integration, TiSpark + TiDB provides a solution that performs both transaction and analysis directly on the same platform without building and maintaining any ETLs. It simplifies the system architecture and reduces the cost of maintenance.
- In addition, you can deploy and utilize the tools from the Spark ecosystem for further data processing and manipulation on TiDB. For example, using TiSpark for data analysis and ETL, retrieving data from TiKV as a data source for machine learning, generating reports from the scheduling system and so on.
TiSpark relies on the availability of TiKV clusters and PDs. You also need to set up and use the Spark clustering platform.
Most of the TiSpark logic is inside a thin layer, namely, the tikv-client library.
We will not provide the mysql-connector-java dependency because of the limit of the GPL license.
The following versions of TiSpark's jar will no longer include mysql-connector-java.
- TiSpark > 3.0.1
- TiSpark > 2.5.1 for TiSpark 2.5.x
- TiSpark > 2.4.3 for TiSpark 2.4.x
Now, TiSpark needs mysql-connector-java for writing and auth. Please import mysql-connector-java manually when you need to write or auth.
-
you can import it by putting the jar into spark jars file
-
you can also import it when you submit spark job like
spark-submit --jars tispark-assembly-3.0_2.12-3.1.0-SNAPSHOT.jar,mysql-connector-java-8.0.29.jar
| Feature Support | TiSpark 2.4.x | TiSpark 2.5.x | TiSpark 3.0.x | TiSpark master |
|---|---|---|---|---|
| SQL select without tidb_catalog | ✔ | ✔ | ||
| SQL select with tidb_catalog | ✔ | ✔ | ✔ | |
| SQL delete from with tidb_catalog | ✔ | ✔ | ||
| DataFrame append | ✔ | ✔ | ✔ | ✔ |
| DataFrame reads | ✔ | ✔ | ✔ | ✔ |
see here for more detail.
-
TiDB starts to support
viewsincetidb-3.0. TiSpark currently does not supportview. Users are not be able to observe or access data throughviewwith TiSpark. -
Spark config
spark.sql.runSQLOnFilesshould not be set tofalse, or you may gotError in query: Table or view not founderror. -
Using the style of "{db}.{table}.{colname}" in the condition is not supported, e.g.
select * from t where db.t.col1 = 1. -
Null in aggregrationis not supported, e.g.select sum(null) from t group by col1. -
The dependency
tispark-assemblyshould not be packaged intoJAR of JARSfile (for example, build with spring-boot-maven-plugin), or you will getClassNotFoundException. You can solve it by addingspark-wrapper-spark-versionin your dependency or constructing another forms of jar file. -
TiSpark doesn't support GBK character set.
-
TiSpark doesn't support the whole collations rule. Currently, TiSpark only supports the following collations: utf8_bin, utf8_general_ci, utf8_unicode_ci, utf8mb4_bin, utf8mb4_general_ci and utf8mb4_unicode_ci.
-
If
spark.sql.ansi.enabledis false an overflow of sum(bigint) will not cause an error but “wrap” the result, or you can cast bigint to decimal to avoid the overflow. -
TiSpark supports retrieving data from table with
Expression Index, but theExpression Indexwill not be used by the planner of TiSpark.
For English users, go to TiDB internals.
For Chinese users, go to AskTUG.
TiSpark is under the Apache 2.0 license. See the LICENSE file for details.