Use tidb as metastore

config/hive-site.xml.template

@@ -0,0 +1,43 @@
+<?xml version="1.0"?>
+<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
+<!--
+   Licensed to the Apache Software Foundation (ASF) under one or more
+   contributor license agreements.  See the NOTICE file distributed with
+   this work for additional information regarding copyright ownership.
+   The ASF licenses this file to You under the Apache License, Version 2.0
+   (the "License"); you may not use this file except in compliance with
+   the License.  You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+-->
+
+<configuration>
+<property>
+  <name>javax.jdo.option.ConnectionURL</name>
+  <value>jdbc:mysql://localhost:4000/metastore_db?createDatabaseIfNotExist=true&amp;useSSL=false</value>
+  <description>JDBC connect string for a JDBC metastore</description>
+</property>
+
+<property>
+  <name>javax.jdo.option.ConnectionDriverName</name>
+  <value>com.mysql.jdbc.Driver</value>
+  <description>Driver class name for a JDBC metastore</description>
+</property>
+
+<property>
+  <name>javax.jdo.option.ConnectionUserName</name>
+  <value>hive</value>
+  <description>username to use against metastore database</description>
+</property>
+
+<property>
+  <name>javax.jdo.option.ConnectionPassword</name>
+  <value>mine</value>
+</property>
+</configuration>

docs/how_to_use_tidb_as_metastore_db.md

@@ -0,0 +1,53 @@
+# Setting TiDB as metastore db
+
+From time to time, users may need run multiple `spark-shell`s at same directory which often leads to some 
+exceptions. Exceptions caused by lock conflicts: you already have a spark-shell running which blocks you run another spark-shell
+at same directory. The way to address this need is setting tidb up as metastore db. 
+
+## Setup TiDB
+
+First you need a TiDB cluster(before 2.1 release), and then use a mysql client log into TiDB cluster. 
+
+You will need to create a TiDB user with its password, e.g., `hive` with password `mine`, for Spark to access the metastore.
+
+```$xslt
+CREATE USER 'hive'@'%' IDENTIFIED BY 'mine';
+GRANT ALL PRIVILEGES ON metastore_db.* TO 'hive'@'%';
+FLUSH PRIVILEGES;
+```
+
+Above SQLs help you create a user and grant access privileges to tables under `metastore_db`. 
+
+### When you rely on spark itself to initialize metastore
+
+This is actually very dangerous and not recommended. If you rely on spark itself to initialize metastore, 
+please do following:
+ 1. Make sure there is no existing metastore. If so, please use official spark schema tools to upgrade or migrate.
+ 2. Fill in root account in hive-site.xml. Let spark use root account to create metastore tables.
+ 3. Then switch back to a normal account without any create table and alter table privileges.
+
+This preventing unexpected schema corruption when code changes.
+
+### Why only TiDB before 2.1 release works?
+
+On Dec 10, 2018, a [PR](https://github.com/pingcap/tidb/pull/8625) got merged into TiDB's master.
+The intention of this PR is to restrict the use of setting transaction isolation level such as serialize. 
+After this change, setting transaction isolation level to serialize
+will be an error rather than the noop in the past.
+
+When hive initializes its metastore client; it will explicitly set transaction isolation level to 
+serialize and cannot be adjusted by any configuration. This leads to restrict the specific version
+of TiDB when you want to use tidb as a backend database to store metastore.
+
+That is why we need choose a specific version of TiDB serving as an backend database to store
+metastore.
+
+
+## Adding hive-site.xml configuration to Spark
+
+Then you can find a sample conf file [hive-site.xml.template](../config/hive-site.xml.template) and 
+adjust some settings. You also need put the file into `SPARK_HOME/conf`.
+
+After you finish these two steps, you are able to use tidb to store meta info of Spark.
+
+