Skip to content

Commit

Permalink
Fix map single table (#319)
Browse files Browse the repository at this point in the history
  • Loading branch information
ilovesoup authored and Novemser committed Apr 25, 2018
1 parent 649ca6b commit 8006a31
Show file tree
Hide file tree
Showing 2 changed files with 26 additions and 3 deletions.
23 changes: 22 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,12 +31,33 @@ Uses as below
import org.apache.spark.sql.TiContext
val ti = new TiContext(spark)
// Mapping all TiDB tables from database tpch as Spark SQL tables
// Map all TiDB tables from database tpch as Spark SQL tables
ti.tidbMapDatabase("tpch")
spark.sql("select count(*) from lineitem").show
```

## Metadata loading
If you are using spark-shell, you need to manually load schema information as decribed above.

If you have too many tables, you might choose to disable histogram preparison and loading will be faster.

```
ti.tidbMapDatabase("tpch", autoLoadStatistics = true)
```

If you have two tables with same name in different databases, you might choose to append database name as prefix for table name:

```
ti.tidbMapDatabase("tpch", dbNameAsPrefix = true)
```

If you have too many tables and use only some of them, to speed up meta loading process, you might manually load only tables you use:

```
ti.tidbTable("tpch", "lineitem")
```

## Current Version
```
ti.version
Expand Down
6 changes: 4 additions & 2 deletions core/src/main/scala/org/apache/spark/sql/TiContext.scala
Original file line number Diff line number Diff line change
Expand Up @@ -129,13 +129,15 @@ class TiContext(val session: SparkSession) extends Serializable with Logging {
}
}

def tidbTable(dbName: String, tableName: String): DataFrame = {
// tidbMapTable does not do any check any meta information
// it just register table for later use
def tidbMapTable(dbName: String, tableName: String): Unit = {
val tiRelation = new TiDBRelation(
tiSession,
new TiTableReference(dbName, tableName),
meta
)(sqlContext)
sqlContext.baseRelationToDataFrame(tiRelation)
sqlContext.baseRelationToDataFrame(tiRelation).createTempView(tableName)
}

def tidbMapDatabase(dbName: String,
Expand Down

0 comments on commit 8006a31

Please sign in to comment.