From 0c1239e3fc1aafd3c72292664ee6a4658f111579 Mon Sep 17 00:00:00 2001 From: Ti Chi Robot Date: Mon, 4 Sep 2023 11:47:13 +0800 Subject: [PATCH] develop: refactor MySQL Connector/Python dev guide (#14632) (#14708) --- develop/dev-guide-choose-driver-or-orm.md | 2 +- develop/dev-guide-insert-data.md | 2 +- ...mple-application-python-mysql-connector.md | 376 +++++++++--------- 3 files changed, 186 insertions(+), 194 deletions(-) diff --git a/develop/dev-guide-choose-driver-or-orm.md b/develop/dev-guide-choose-driver-or-orm.md index 469800c7237a5..b3a491a4324f5 100644 --- a/develop/dev-guide-choose-driver-or-orm.md +++ b/develop/dev-guide-choose-driver-or-orm.md @@ -265,7 +265,7 @@ Support level: **Compatible** You can follow the [MySQL Connector/Python documentation](https://dev.mysql.com/doc/connector-python/en/connector-python-installation-binary.html) to download and configure the driver. It is recommended to use Connector/Python 8.0.31 or later versions. -For an example of using MySQL Connector/Python to build a TiDB application, see [Build a simple CRUD app with TiDB and MySQL Connector/Python](/develop/dev-guide-sample-application-python-mysql-connector.md#step-2-get-the-code). +For an example of using MySQL Connector/Python to build a TiDB application, see [Connect to TiDB with MySQL Connector/Python](/develop/dev-guide-sample-application-python-mysql-connector.md). diff --git a/develop/dev-guide-insert-data.md b/develop/dev-guide-insert-data.md index af1772a3dff1f..98a89eee939ad 100644 --- a/develop/dev-guide-insert-data.md +++ b/develop/dev-guide-insert-data.md @@ -220,7 +220,7 @@ For complete examples in Python, see: - [Connect to TiDB with PyMySQL](/develop/dev-guide-sample-application-python-pymysql.md) - [Connect to TiDB with mysqlclient](https://github.com/tidb-samples/tidb-python-mysqlclient-quickstart) -- [Use MySQL Connector/Python to build a simple CRUD app with TiDB and Python](/develop/dev-guide-sample-application-python-mysql-connector.md#step-2-get-the-code) +- [Connect to TiDB with MySQL Connector/Python](/develop/dev-guide-sample-application-python-mysql-connector.md) - [Use SQLAlchemy to build a simple CRUD app with TiDB and Python](/develop/dev-guide-sample-application-python-sqlalchemy.md#step-2-get-the-code) - [Use peewee to build a simple CRUD app with TiDB and Python](/develop/dev-guide-sample-application-python-peewee.md#step-2-get-the-code) diff --git a/develop/dev-guide-sample-application-python-mysql-connector.md b/develop/dev-guide-sample-application-python-mysql-connector.md index e23668117308b..b6c708a9e4272 100644 --- a/develop/dev-guide-sample-application-python-mysql-connector.md +++ b/develop/dev-guide-sample-application-python-mysql-connector.md @@ -1,291 +1,283 @@ --- -title: Build a Simple CRUD App with TiDB and MySQL Connector/Python -summary: Learn how to build a simple CRUD application with TiDB and MySQL Connector/Python. +title: Connect to TiDB with MySQL Connector/Python +summary: Learn how to connect to TiDB using MySQL Connector/Python. This tutorial gives Python sample code snippets that work with TiDB using MySQL Connector/Python. aliases: ['/tidb/v6.5/dev-guide-sample-application-python','/tidb/stable/dev-guide-sample-application-python','/tidbcloud/dev-guide-sample-application-python','/tidb/v6.5/dev-guide-outdated-for-python-mysql-connector'] --- - - +# Connect to TiDB with MySQL Connector/Python -# Build a Simple CRUD App with TiDB and MySQL Connector/Python +TiDB is a MySQL-compatible database, and [MySQL Connector/Python](https://dev.mysql.com/doc/connector-python/en/) is the official MySQL driver for Python. -[MySQL Connector/Python](https://dev.mysql.com/doc/connector-python/en/) is a popular open-source driver for Python. +In this tutorial, you can learn how to use TiDB and MySQL Connector/Python to accomplish the following tasks: -This document describes how to use TiDB and MySQL Connector/Python to build a simple CRUD application. +- Set up your environment. +- Connect to your TiDB cluster using MySQL Connector/Python. +- Build and run your application. Optionally, you can find sample code snippets for basic CRUD operations. > **Note:** > -> It is recommended to use Python 3.10 or a later Python version. +> This tutorial works with TiDB Serverless, TiDB Dedicated, and TiDB Self-Hosted clusters. -## Step 1. Launch your TiDB cluster +## Prerequisites + +To complete this tutorial, you need: + +- [Python 3.8 or higher](https://www.python.org/downloads/). +- [Git](https://git-scm.com/downloads). +- A TiDB cluster. -The following introduces how to start a TiDB cluster. +**If you don't have a TiDB cluster, you can create one as follows:** -**Use a TiDB Serverless cluster** +- (Recommended) Follow [Creating a TiDB Serverless cluster](/develop/dev-guide-build-cluster-in-cloud.md) to create your own TiDB Cloud cluster. +- Follow [Deploy a local test TiDB cluster](/quick-start-with-tidb.md#deploy-a-local-test-cluster) or [Deploy a production TiDB cluster](/production-deployment-using-tiup.md) to create a local cluster. -For detailed steps, see [Create a TiDB Serverless cluster](/develop/dev-guide-build-cluster-in-cloud.md#step-1-create-a-tidb-serverless-cluster). + + -**Use a local cluster** +**If you don't have a TiDB cluster, you can create one as follows:** -For detailed steps, see [Deploy a local test cluster](/quick-start-with-tidb.md#deploy-a-local-test-cluster) or [Deploy a TiDB cluster using TiUP](/production-deployment-using-tiup.md). +- (Recommended) Follow [Creating a TiDB Serverless cluster](/develop/dev-guide-build-cluster-in-cloud.md) to create your own TiDB Cloud cluster. +- Follow [Deploy a local test TiDB cluster](https://docs.pingcap.com/tidb/stable/quick-start-with-tidb#deploy-a-local-test-cluster) or [Deploy a production TiDB cluster](https://docs.pingcap.com/tidb/stable/production-deployment-using-tiup) to create a local cluster. - +## Run the sample app to connect to TiDB -See [Create a TiDB Serverless cluster](/develop/dev-guide-build-cluster-in-cloud.md#step-1-create-a-tidb-serverless-cluster). +This section demonstrates how to run the sample application code and connect to TiDB. - +### Step 1: Clone the sample app repository -## Step 2. Get the code +Run the following commands in your terminal window to clone the sample code repository: ```shell -git clone https://github.com/pingcap-inc/tidb-example-python.git +git clone https://github.com/tidb-samples/tidb-python-mysqlconnector-quickstart.git +cd tidb-python-mysqlconnector-quickstart ``` -The following uses MySQL Connector/Python 8.0.31 as an example. Drivers for Python are more convenient to use than other languages, but they do not shield the underlying implementation and require manual management of transactions. If there are not a lot of scenarios where SQL is required, it is recommended to use ORM, which can help reduce the coupling of your program. - -```python -import uuid -from typing import List +### Step 2: Install dependencies -from mysql.connector import connect, MySQLConnection -from mysql.connector.cursor import MySQLCursor +Run the following command to install the required packages (including mysql-connector-python) for the sample app: +```shell +pip install -r requirements.txt +``` -def get_connection(autocommit: bool = True) -> MySQLConnection: - connection = connect(host='127.0.0.1', - port=4000, - user='root', - password='', - database='test') - connection.autocommit = autocommit - return connection +### Step 3: Configure connection information +Connect to your TiDB cluster depending on the TiDB deployment option you've selected. -def create_player(cursor: MySQLCursor, player: tuple) -> None: - cursor.execute("INSERT INTO player (id, coins, goods) VALUES (%s, %s, %s)", player) + +
+1. Navigate to the [**Clusters**](https://tidbcloud.com/console/clusters) page, and then click the name of your target cluster to go to its overview page. -def get_player(cursor: MySQLCursor, player_id: str) -> tuple: - cursor.execute("SELECT id, coins, goods FROM player WHERE id = %s", (player_id,)) - return cursor.fetchone() +2. Click **Connect** in the upper-right corner. A connection dialog is displayed. +3. Ensure the configurations in the connection dialog match your operating environment. -def get_players_with_limit(cursor: MySQLCursor, limit: int) -> List[tuple]: - cursor.execute("SELECT id, coins, goods FROM player LIMIT %s", (limit,)) - return cursor.fetchall() + - **Endpoint Type** is set to `Public` + - **Connect With** is set to `General` + - **Operating System** matches your environment. + > **Tip:** + > + > If your program is running in Windows Subsystem for Linux (WSL), switch to the corresponding Linux distribution. -def random_player(amount: int) -> List[tuple]: - players = [] - for _ in range(amount): - players.append((str(uuid.uuid4()), 10000, 10000)) +4. Click **Create password** to create a random password. - return players + > **Tip:** + > + > If you have created a password before, you can either use the original password or click **Reset password** to generate a new one. +5. Run the following command to copy `.env.example` and rename it to `.env`: -def bulk_create_player(cursor: MySQLCursor, players: List[tuple]) -> None: - cursor.executemany("INSERT INTO player (id, coins, goods) VALUES (%s, %s, %s)", players) + ```shell + cp .env.example .env + ``` +6. Copy and paste the corresponding connection string into the `.env` file. The example result is as follows: -def get_count(cursor: MySQLCursor) -> int: - cursor.execute("SELECT count(*) FROM player") - return cursor.fetchone()[0] + ```dotenv + TIDB_HOST='{host}' # e.g. gateway01.ap-northeast-1.prod.aws.tidbcloud.com + TIDB_PORT='4000' + TIDB_USER='{user}' # e.g. xxxxxx.root + TIDB_PASSWORD='{password}' + TIDB_DB_NAME='test' + CA_PATH='{ssl_ca}' # e.g. /etc/ssl/certs/ca-certificates.crt (Debian / Ubuntu / Arch) + ``` + Be sure to replace the placeholders `{}` with the connection parameters obtained from the connection dialog. -def trade_check(cursor: MySQLCursor, sell_id: str, buy_id: str, amount: int, price: int) -> bool: - get_player_with_lock_sql = "SELECT coins, goods FROM player WHERE id = %s FOR UPDATE" +7. Save the `.env` file. - # sell player goods check - cursor.execute(get_player_with_lock_sql, (sell_id,)) - _, sell_goods = cursor.fetchone() - if sell_goods < amount: - print(f'sell player {sell_id} goods not enough') - return False +
+
- # buy player coins check - cursor.execute(get_player_with_lock_sql, (buy_id,)) - buy_coins, _ = cursor.fetchone() - if buy_coins < price: - print(f'buy player {buy_id} coins not enough') - return False +1. Navigate to the [**Clusters**](https://tidbcloud.com/console/clusters) page, and then click the name of your target cluster to go to its overview page. +2. Click **Connect** in the upper-right corner. A connection dialog is displayed. -def trade_update(cursor: MySQLCursor, sell_id: str, buy_id: str, amount: int, price: int) -> None: - update_player_sql = "UPDATE player set goods = goods + %s, coins = coins + %s WHERE id = %s" +3. Click **Allow Access from Anywhere** and then click **Download TiDB cluster CA** to download the CA certificate. - # deduct the goods of seller, and raise his/her the coins - cursor.execute(update_player_sql, (-amount, price, sell_id)) - # deduct the coins of buyer, and raise his/her the goods - cursor.execute(update_player_sql, (amount, -price, buy_id)) + For more details about how to obtain the connection string, refer to [TiDB Dedicated standard connection](https://docs.pingcap.com/tidbcloud/connect-via-standard-connection). +4. Run the following command to copy `.env.example` and rename it to `.env`: -def trade(connection: MySQLConnection, sell_id: str, buy_id: str, amount: int, price: int) -> None: - with connection.cursor() as cursor: - if trade_check(cursor, sell_id, buy_id, amount, price) is False: - connection.rollback() - return + ```shell + cp .env.example .env + ``` - try: - trade_update(cursor, sell_id, buy_id, amount, price) - except Exception as err: - connection.rollback() - print(f'something went wrong: {err}') - else: - connection.commit() - print("trade success") +5. Copy and paste the corresponding connection string into the `.env` file. The example result is as follows: + ```dotenv + TIDB_HOST='{host}' # e.g. tidb.xxxx.clusters.tidb-cloud.com + TIDB_PORT='4000' + TIDB_USER='{user}' # e.g. root + TIDB_PASSWORD='{password}' + TIDB_DB_NAME='test' + CA_PATH='{your-downloaded-ca-path}' + ``` -def simple_example() -> None: - with get_connection(autocommit=True) as connection: - with connection.cursor() as cur: - # create a player, who has a coin and a goods. - create_player(cur, ("test", 1, 1)) + Be sure to replace the placeholders `{}` with the connection parameters obtained from the connection dialog, and configure `CA_PATH` with the certificate path downloaded in the previous step. - # get this player, and print it. - test_player = get_player(cur, "test") - print(f'id:{test_player[0]}, coins:{test_player[1]}, goods:{test_player[2]}') +6. Save the `.env` file. - # create players with bulk inserts. - # insert 1919 players totally, with 114 players per batch. - # each player has a random UUID - player_list = random_player(1919) - for idx in range(0, len(player_list), 114): - bulk_create_player(cur, player_list[idx:idx + 114]) +
+
- # print the number of players - count = get_count(cur) - print(f'number of players: {count}') +1. Run the following command to copy `.env.example` and rename it to `.env`: - # print 3 players. - three_players = get_players_with_limit(cur, 3) - for player in three_players: - print(f'id:{player[0]}, coins:{player[1]}, goods:{player[2]}') + ```shell + cp .env.example .env + ``` +2. Copy and paste the corresponding connection string into the `.env` file. The example result is as follows: -def trade_example() -> None: - with get_connection(autocommit=False) as conn: - with conn.cursor() as cur: - # create two players - # player 1: id is "1", has only 100 coins. - # player 2: id is "2", has 114514 coins, and 20 goods. - create_player(cur, ("1", 100, 0)) - create_player(cur, ("2", 114514, 20)) - conn.commit() + ```dotenv + TIDB_HOST='{tidb_server_host}' + TIDB_PORT='4000' + TIDB_USER='root' + TIDB_PASSWORD='{password}' + TIDB_DB_NAME='test' + ``` - # player 1 wants to buy 10 goods from player 2. - # it will cost 500 coins, but player 1 cannot afford it. - # so this trade will fail, and nobody will lose their coins or goods - trade(conn, sell_id="2", buy_id="1", amount=10, price=500) + Be sure to replace the placeholders `{}` with the connection parameters, and remove the `CA_PATH` line. If you are running TiDB locally, the default host address is `127.0.0.1`, and the password is empty. - # then player 1 has to reduce the incoming quantity to 2. - # this trade will be successful - trade(conn, sell_id="2", buy_id="1", amount=2, price=100) +3. Save the `.env` file. - # let's take a look for player 1 and player 2 currently - with conn.cursor() as cur: - _, player1_coin, player1_goods = get_player(cur, "1") - print(f'id:1, coins:{player1_coin}, goods:{player1_goods}') - _, player2_coin, player2_goods = get_player(cur, "2") - print(f'id:2, coins:{player2_coin}, goods:{player2_goods}') - - -simple_example() -trade_example() -``` +
+ -The driver has a lower level of encapsulation than ORM, so there are a lot of SQL statements in the program. Unlike ORM, there is no data object in drivers, so the `Player` queried by the driver is represented as a tuple. +### Step 4: Run the code and check the result -For more information about how to use MySQL Connector/Python, refer to [MySQL Connector/Python documentation](https://dev.mysql.com/doc/connector-python/en/). +1. Execute the following command to run the sample code: -## Step 3. Run the code + ```shell + python mysql_connector_example.py + ``` -The following content introduces how to run the code step by step. +2. Check the [Expected-Output.txt](https://github.com/tidb-samples/tidb-python-mysqlconnector-quickstart/blob/main/Expected-Output.txt) to see if the output matches. -### Step 3.1 Initialize table +## Sample code snippets -Before running the code, you need to initialize the table manually. If you are using a local TiDB cluster, you can run the following command: +You can refer to the following sample code snippets to complete your own application development. - +For complete sample code and how to run it, check out the [tidb-samples/tidb-python-mysqlconnector-quickstart](https://github.com/tidb-samples/tidb-python-mysqlconnector-quickstart) repository. -
+### Connect to TiDB -```shell -mysql --host 127.0.0.1 --port 4000 -u root < player_init.sql +```python +def get_connection(autocommit: bool = True) -> MySQLConnection: + config = Config() + db_conf = { + "host": ${tidb_host}, + "port": ${tidb_port}, + "user": ${tidb_user}, + "password": ${tidb_password}, + "database": ${tidb_db_name}, + "autocommit": autocommit, + "use_pure": True, + } + + if ${ca_path}: + db_conf["ssl_verify_cert"] = True + db_conf["ssl_verify_identity"] = True + db_conf["ssl_ca"] = ${ca_path} + return mysql.connector.connect(**db_conf) ``` -
+When using this function, you need to replace `${tidb_host}`, `${tidb_port}`, `${tidb_user}`, `${tidb_password}`, `${tidb_db_name}` and `${ca_path}` with the actual values of your TiDB cluster. -
+### Insert data -```shell -mycli --host 127.0.0.1 --port 4000 -u root --no-warn < player_init.sql +```python +with get_connection(autocommit=True) as conn: + with conn.cursor() as cur: + player = ("1", 1, 1) + cursor.execute("INSERT INTO players (id, coins, goods) VALUES (%s, %s, %s)", player) ``` -
+For more information, refer to [Insert data](/develop/dev-guide-insert-data.md). - - -If you are not using a local cluster, or have not installed a MySQL client, connect to your cluster using your preferred method (such as Navicat, DBeaver, or other GUI tools) and run the SQL statements in the `player_init.sql` file. +### Query data -### Step 3.2 Modify parameters for TiDB Cloud +```python +with get_connection(autocommit=True) as conn: + with conn.cursor() as cur: + cur.execute("SELECT count(*) FROM players") + print(cur.fetchone()[0]) +``` -If you are using a TiDB Serverless cluster, you need to provide your CA root path and replace `` in the following examples with your CA path. To get the CA root path on your system, refer to [Root certificate management](https://docs.pingcap.com/tidbcloud/secure-connections-to-serverless-clusters#root-certificate-management). +For more information, refer to [Query data](/develop/dev-guide-get-data-from-single-table.md). -If you are using a TiDB Serverless cluster, change the `get_connection` function in `mysql_connector_python_example.py`: +### Update data ```python -def get_connection(autocommit: bool = True) -> MySQLConnection: - connection = connect(host='127.0.0.1', - port=4000, - user='root', - password='', - database='test') - connection.autocommit = autocommit - return connection +with get_connection(autocommit=True) as conn: + with conn.cursor() as cur: + player_id, amount, price="1", 10, 500 + cursor.execute( + "UPDATE players SET goods = goods + %s, coins = coins + %s WHERE id = %s", + (-amount, price, player_id), + ) ``` -Suppose that the password you set is `123456`, and the connection parameters you get from the cluster details page are the following: - -- Endpoint: `xxx.tidbcloud.com` -- Port: `4000` -- User: `2aEp24QWEDLqRFs.root` +For more information, refer to [Update data](/develop/dev-guide-update-data.md). -In this case, you can modify the `get_connection` as follows: +### Delete data ```python -def get_connection(autocommit: bool = True) -> MySQLConnection: - connection = connect( - host="xxx.tidbcloud.com", - port=4000, - user="2aEp24QWEDLqRFs.root", - password="123456", - database="test", - autocommit=autocommit, - ssl_ca='', - ssl_verify_identity=True - ) - connection.autocommit = autocommit - return connection +with get_connection(autocommit=True) as conn: + with conn.cursor() as cur: + player_id = "1" + cursor.execute("DELETE FROM players WHERE id = %s", (player_id,)) ``` -### Step 3.3 Run the code +For more information, refer to [Delete data](/develop/dev-guide-delete-data.md). -Before running the code, use the following command to install dependencies: +## Useful notes -```bash -pip3 install -r requirement.txt -``` +### Using driver or ORM framework? -If you need to run the script multiple times, follow the [Table initialization](#step-31-initialize-table) section to initialize the table again before each run. +The Python driver provides low-level access to the database, but it requires the developers to: -```bash -python3 mysql_connector_python_example.py -``` +- Manually establish and release database connections. +- Manually manage database transactions. +- Manually map data rows (represented as a tuple or dict in `mysql-connector-python`) to data objects. + +Unless you need to write complex SQL statements, it is recommended to use [ORM](https://en.wikipedia.org/w/index.php?title=Object-relational_mapping) framework for development, such as [SQLAlchemy](/develop/dev-guide-sample-application-python-sqlalchemy.md), [Peewee](/develop/dev-guide-sample-application-python-peewee.md), and Django ORM. It can help you: + +- Reduce [boilerplate code](https://en.wikipedia.org/wiki/Boilerplate_code) for managing connections and transactions. +- Manipulate data with data objects instead of a number of SQL statements. + +## Next steps + +- Learn more usage of mysql-connector-python from [the documentation of MySQL Connector/Python](https://dev.mysql.com/doc/connector-python/en/). +- Learn the best practices for TiDB application development with the chapters in the [Developer guide](/develop/dev-guide-overview.md), such as [Insert data](/develop/dev-guide-insert-data.md), [Update data](/develop/dev-guide-update-data.md), [Delete data](/develop/dev-guide-delete-data.md), [Single table reading](/develop/dev-guide-get-data-from-single-table.md), [Transactions](/develop/dev-guide-transaction-overview.md), and [SQL performance optimization](/develop/dev-guide-optimize-sql-overview.md). +- Learn through the professional [TiDB developer courses](https://www.pingcap.com/education/) and earn [TiDB certifications](https://www.pingcap.com/education/certification/) after passing the exam. -## Step 4. Expected output +## Need help? -[MySQL Connector/Python Expected Output](https://github.com/pingcap-inc/tidb-example-python/blob/main/Expected-Output.md#mysql-connector-python) \ No newline at end of file +Ask questions on the [Discord](https://discord.gg/vYU9h56kAX), or [create a support ticket](https://support.pingcap.com/).