diff --git a/accelerated-table-creation.md b/accelerated-table-creation.md
index f97eec9d5cfd0..ec3ee720f825c 100644
--- a/accelerated-table-creation.md
+++ b/accelerated-table-creation.md
@@ -8,9 +8,9 @@ aliases: ['/tidb/dev/ddl-v2/']
TiDB v7.6.0 introduces the system variable [`tidb_ddl_version`](https://docs.pingcap.com/tidb/v7.6/system-variables#tidb_enable_fast_create_table-new-in-v800) to support accelerating table creation, which improves the efficiency of bulk table creation. Starting from v8.0.0, this system variable is renamed to [`tidb_enable_fast_create_table`](/system-variables.md#tidb_enable_fast_create_table-new-in-v800).
-TiDB uses the online asynchronous schema change algorithm to change the metadata. All DDL jobs are submitted to the `mysql.tidb_ddl_job` table, and the owner node pulls the DDL job to execute. After executing each phase of the online DDL algorithm, the DDL job is marked as completed and moved to the `mysql.tidb_ddl_history` table. Therefore, DDL statements can only be executed on the owner node and cannot be linearly extended.
+When accelerated table creation is enabled via [`tidb_enable_fast_create_table`](/system-variables.md#tidb_enable_fast_create_table-new-in-v800), table creation statements with the same schema committed to the same TiDB node at the same time are merged into batch table creation statements to improve table creation performance. Therefore, to improve the table creation performance, try to connect to the same TiDB node, create tables with the same schema concurrently, and increase the concurrency appropriately.
-However, for some DDL statements, it is not necessary to strictly follow the online DDL algorithm. For example, the `CREATE TABLE` statement only has two states for the job: `none` and `public`. Therefore, TiDB can simplify the execution process of DDL, and executes the `CREATE TABLE` statement on a non-owner node to accelerate table creation.
+The merged batch table creation statements are executed within the same transaction, so if one statement of them fails, all of them will fail.
> **Warning:**
>
@@ -39,19 +39,3 @@ To disable performance optimization for creating tables, set the value of this v
```sql
SET GLOBAL tidb_enable_fast_create_table = OFF;
```
-
-## Implementation principle
-
-The detailed implementation principle of performance optimization for table creation is as follows:
-
-1. Create a `CREATE TABLE` Job.
-
- The corresponding DDL Job is generated by parsing the `CREATE TABLE` statement.
-
-2. Execute the `CREATE TABLE` job.
-
- The TiDB node that receives the `CREATE TABLE` statement executes it directly, and then persists the table structure to TiKV. At the same time, the `CREATE TABLE` job is marked as completed and inserted into the `mysql.tidb_ddl_history` table.
-
-3. Synchronize the table information.
-
- TiDB notifies other nodes to synchronize the newly created table structure.
diff --git a/auto-random.md b/auto-random.md
index 0545fec78139e..b7b0f78b04420 100644
--- a/auto-random.md
+++ b/auto-random.md
@@ -47,7 +47,7 @@ When you execute an `INSERT` statement:
- If you do not explicitly specify the value of the `AUTO_RANDOM` column, TiDB generates a random value and inserts it into the table.
```sql
-tidb> CREATE TABLE t (a BIGINT PRIMARY KEY AUTO_RANDOM, b VARCHAR(255));
+tidb> CREATE TABLE t (a BIGINT PRIMARY KEY AUTO_RANDOM, b VARCHAR(255)) /*T! PRE_SPLIT_REGIONS=2 */ ;
Query OK, 0 rows affected, 1 warning (0.01 sec)
tidb> INSERT INTO t(a, b) VALUES (1, 'string');
@@ -76,6 +76,29 @@ tidb> SELECT * FROM t;
| 4899916394579099651 | string3 |
+---------------------+---------+
3 rows in set (0.00 sec)
+
+tidb> SHOW CREATE TABLE t;
++-------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+| Table | Create Table |
++-------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+| t | CREATE TABLE `t` (
+ `a` bigint(20) NOT NULL /*T![auto_rand] AUTO_RANDOM(5) */,
+ `b` varchar(255) DEFAULT NULL,
+ PRIMARY KEY (`a`) /*T![clustered_index] CLUSTERED */
+) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin /*T! PRE_SPLIT_REGIONS=2 */ |
++-------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+1 row in set (0.00 sec)
+
+tidb> SHOW TABLE t REGIONS;
++-----------+-----------------------------+-----------------------------+-----------+-----------------+---------------------+------------+---------------+------------+----------------------+------------------+------------------------+------------------+
+| REGION_ID | START_KEY | END_KEY | LEADER_ID | LEADER_STORE_ID | PEERS | SCATTERING | WRITTEN_BYTES | READ_BYTES | APPROXIMATE_SIZE(MB) | APPROXIMATE_KEYS | SCHEDULING_CONSTRAINTS | SCHEDULING_STATE |
++-----------+-----------------------------+-----------------------------+-----------+-----------------+---------------------+------------+---------------+------------+----------------------+------------------+------------------------+------------------+
+| 62798 | t_158_ | t_158_r_2305843009213693952 | 62810 | 28 | 62811, 62812, 62810 | 0 | 151 | 0 | 1 | 0 | | |
+| 62802 | t_158_r_2305843009213693952 | t_158_r_4611686018427387904 | 62803 | 1 | 62803, 62804, 62805 | 0 | 39 | 0 | 1 | 0 | | |
+| 62806 | t_158_r_4611686018427387904 | t_158_r_6917529027641081856 | 62813 | 4 | 62813, 62814, 62815 | 0 | 160 | 0 | 1 | 0 | | |
+| 9289 | t_158_r_6917529027641081856 | 78000000 | 48268 | 1 | 48268, 58951, 62791 | 0 | 10628 | 43639 | 2 | 7999 | | |
++-----------+-----------------------------+-----------------------------+-----------+-----------------+---------------------+------------+---------------+------------+----------------------+------------------+------------------------+------------------+
+4 rows in set (0.00 sec)
```
The `AUTO_RANDOM(S, R)` column value automatically assigned by TiDB has a total of 64 bits:
@@ -101,6 +124,7 @@ The structure of an `AUTO_RANDOM` value without a signed bit is as follows:
- The content of the shard bits is obtained by calculating the hash value of the starting time of the current transaction. To use a different length of shard bits (such as 10), you can specify `AUTO_RANDOM(10)` when creating the table.
- The value of the auto-increment bits is stored in the storage engine and allocated sequentially. Each time a new value is allocated, the value is incremented by 1. The auto-increment bits ensure that the values of `AUTO_RANDOM` are unique globally. When the auto-increment bits are exhausted, an error `Failed to read auto-increment value from storage engine` is reported when the value is allocated again.
- Value range: the maximum number of bits for the final generated value = shard bits + auto-increment bits. The range of a signed column is `[-(2^(R-1))+1, (2^(R-1))-1]`, and the range of an unsigned column is `[0, (2^R)-1]`.
+- You can use `AUTO_RANDOM` with `PRE_SPLIT_REGIONS`. When a table is created successfully, `PRE_SPLIT_REGIONS` pre-splits data in the table into the number of Regions as specified by `2^(PRE_SPLIT_REGIONS)`.
> **Note:**
>
diff --git a/benchmark/benchmark-tidb-using-ch.md b/benchmark/benchmark-tidb-using-ch.md
index a585a5f7d336b..78f5c872cc826 100644
--- a/benchmark/benchmark-tidb-using-ch.md
+++ b/benchmark/benchmark-tidb-using-ch.md
@@ -7,7 +7,7 @@ summary: Learn how to run CH-benCHmark test on TiDB.
This document describes how to test TiDB using CH-benCHmark.
-CH-benCHmark is a mixed workload containing both [TPC-C](http://www.tpc.org/tpcc/) and [TPC-H](http://www.tpc.org/tpch/) tests. It is the most common workload to test HTAP systems. For more information, see [The mixed workload CH-benCHmark](https://research.tableau.com/sites/default/files/a8-cole.pdf).
+CH-benCHmark is a mixed workload containing both [TPC-C](http://www.tpc.org/tpcc/) and [TPC-H](http://www.tpc.org/tpch/) tests. It is the most common workload to test HTAP systems. For more information, see [The mixed workload CH-benCHmark](https://dl.acm.org/doi/10.1145/1988842.1988850).
Before running the CH-benCHmark test, you need to deploy [TiFlash](/tiflash/tiflash-overview.md) first, which is a TiDB's HTAP component. After you deploy TiFlash and [create TiFlash replicas](#create-tiflash-replicas), TiKV will replicate the latest data of TPC-C online transactions to TiFlash in real time, and the TiDB optimizer will automatically push down OLAP queries from TPC-H workload to the MPP engine of TiFlash for efficient execution.
@@ -85,9 +85,10 @@ In the result of the above statement:
## Collect statistics
-To ensure that the TiDB optimizer can generate the optimal execution plan, execute the following SQL statements to collect statistics in advance.
+To ensure that the TiDB optimizer can generate the optimal execution plan, execute the following SQL statements to collect statistics in advance. **Be sure to set [`tidb_analyze_column_options`](/system-variables.md#tidb_analyze_column_options-new-in-v830) to `ALL`, otherwise collecting statistics can result in a significant drop in query performance.**
```
+set global tidb_analyze_column_options='ALL';
analyze table customer;
analyze table district;
analyze table history;
@@ -166,4 +167,4 @@ tpmC: 93826.9, efficiency: 729.6%
[Summary] Q7 - Count: 11, Sum(ms): 158928.2, Avg(ms): 14446.3
```
-After the test is finished, you can execute the `tiup bench tpcc -H 172.16.5.140 -P 4000 -D tpcc --warehouses 1000 check` command to validate the data correctness.
\ No newline at end of file
+After the test is finished, you can execute the `tiup bench tpcc -H 172.16.5.140 -P 4000 -D tpcc --warehouses 1000 check` command to validate the data correctness.
diff --git a/best-practices/java-app-best-practices.md b/best-practices/java-app-best-practices.md
index f593ea6e02827..24324f44c132f 100644
--- a/best-practices/java-app-best-practices.md
+++ b/best-practices/java-app-best-practices.md
@@ -62,17 +62,21 @@ For batch inserts, you can use the [`addBatch`/`executeBatch` API](https://www.t
In most scenarios, to improve execution efficiency, JDBC obtains query results in advance and save them in client memory by default. But when the query returns a super large result set, the client often wants the database server to reduce the number of records returned at a time, and waits until the client's memory is ready and it requests for the next batch.
-Usually, there are two kinds of processing methods in JDBC:
+The following two processing methods are usually used in JDBC:
-- [Set `FetchSize` to `Integer.MIN_VALUE`](https://dev.mysql.com/doc/connector-j/en/connector-j-reference-implementation-notes.html#ResultSet) to ensure that the client does not cache. The client will read the execution result from the network connection through `StreamingResult`.
+- The first method: [Set `FetchSize` to `Integer.MIN_VALUE`](https://dev.mysql.com/doc/connector-j/en/connector-j-reference-implementation-notes.html#ResultSet) to ensure that the client does not cache. The client will read the execution result from the network connection through `StreamingResult`.
When the client uses the streaming read method, it needs to finish reading or close `resultset` before continuing to use the statement to make a query. Otherwise, the error `No statements may be issued when any streaming result sets are open and in use on a given connection. Ensure that you have called .close() on any active streaming result sets before attempting more queries.` is returned.
To avoid such an error in queries before the client finishes reading or closes `resultset`, you can add the `clobberStreamingResults=true` parameter in the URL. Then, `resultset` is automatically closed but the result set to be read in the previous streaming query is lost.
-- To use Cursor Fetch, first [set `FetchSize`](http://makejavafaster.blogspot.com/2015/06/jdbc-fetch-size-performance.html) as a positive integer and configure `useCursorFetch=true` in the JDBC URL.
+- The second method: Use Cursor Fetch by first [setting `FetchSize`](http://makejavafaster.blogspot.com/2015/06/jdbc-fetch-size-performance.html) as a positive integer and then configuring `useCursorFetch = true` in the JDBC URL.
-TiDB supports both methods, but it is preferred that you use the first method, because it is a simpler implementation and has a better execution efficiency.
+TiDB supports both methods, but it is recommended that you use the first method that sets `FetchSize` to `Integer.MIN_VALUE`, because it is a simpler implementation and has better execution efficiency.
+
+For the second method, TiDB first loads all data to the TiDB node, and then returns data to the client according to the `FetchSize`. Therefore, it usually consumes more memory than the first method. If [`tidb_enable_tmp_storage_on_oom`](/system-variables.md#tidb_enable_tmp_storage_on_oom) is set to `ON`, TiDB might temporarily write the result to the hard disk.
+
+If the [`tidb_enable_lazy_cursor_fetch`](/system-variables.md#tidb_enable_lazy_cursor_fetch-new-in-v830) system variable is set to `ON`, TiDB tries to read part of the data only when the client fetches it, which uses less memory. For more details and limitations, read the [complete descriptions for the `tidb_enable_lazy_cursor_fetch` system variable](/system-variables.md#tidb_enable_lazy_cursor_fetch-new-in-v830).
### MySQL JDBC parameters
diff --git a/br/br-incremental-guide.md b/br/br-incremental-guide.md
index 56f159b5ef102..e7b4815cf75f9 100644
--- a/br/br-incremental-guide.md
+++ b/br/br-incremental-guide.md
@@ -17,6 +17,12 @@ Because restoring the incremental backup relies on the snapshot of the database
The incremental backup does not support batch renaming of tables. If batch renaming of tables occurs during the incremental backup process, the data restore might fail. It is recommended to perform a full backup after batch renaming tables, and use the latest full backup to replace the incremental data during restore.
+Starting from v8.3.0, the `--allow-pitr-from-incremental` configuration parameter is introduced to control whether incremental backups and subsequent log backups are compatible. The default value is `true`, which means that incremental backups are compatible with subsequent log backups.
+
+- When you keep the default value `true`, the DDLs that need to be replayed are strictly checked before the incremental restore begins. This mode does not yet support `ADD INDEX`, `MODIFY COLUMN`, or `REORG PARTITION`. If you want to use incremental backups together with log backups, make sure that none of the preceding DDLs exist during the incremental backup process. Otherwise, these three DDLs cannot be replayed correctly.
+
+- If you want to use incremental restores without log backups during the whole recovery process, you can set `--allow-pitr-from-incremental` to `false` to skip the checks in the incremental recovery phase.
+
## Back up incremental data
To back up incremental data, run the `tiup br backup` command with **the last backup timestamp** `--lastbackupts` specified. In this way, br command-line tool automatically backs up incremental data generated between `lastbackupts` and the current time. To get `--lastbackupts`, run the `validate` command. The following is an example:
diff --git a/br/br-pitr-manual.md b/br/br-pitr-manual.md
index d4c9076ea1949..5a8e128d5d627 100644
--- a/br/br-pitr-manual.md
+++ b/br/br-pitr-manual.md
@@ -341,6 +341,10 @@ Expected output:
## Restore to a specified point in time (PITR)
+> **Note:**
+>
+> If you specify `--full-backup-storage` as the incremental backup address for `restore point`, for restores of this backup and any previous incremental backups, you need to set the parameter `--allow-pitr-from-incremental` to `true` to make the incremental backups compatible with the subsequent log backups.
+
You can run the `tiup br restore point` command to perform a PITR on a new cluster or just restore the log backup data.
Run `tiup br restore point --help` to see the help information:
diff --git a/develop/dev-guide-connection-parameters.md b/develop/dev-guide-connection-parameters.md
index af208398374c1..da75d89cd63f2 100644
--- a/develop/dev-guide-connection-parameters.md
+++ b/develop/dev-guide-connection-parameters.md
@@ -143,17 +143,21 @@ For batch inserts, you can use the [`addBatch`/`executeBatch` API](https://www.t
In most scenarios, to improve execution efficiency, JDBC obtains query results in advance and saves them in client memory by default. But when the query returns a super large result set, the client often wants the database server to reduce the number of records returned at a time and waits until the client's memory is ready and it requests for the next batch.
-Usually, there are two kinds of processing methods in JDBC:
+The following two processing methods are usually used in JDBC:
-- [Set **FetchSize** to `Integer.MIN_VALUE`](https://dev.mysql.com/doc/connector-j/en/connector-j-reference-implementation-notes.html#ResultSet) to ensure that the client does not cache. The client will read the execution result from the network connection through `StreamingResult`.
+- The first method: [Set **FetchSize** to `Integer.MIN_VALUE`](https://dev.mysql.com/doc/connector-j/en/connector-j-reference-implementation-notes.html#ResultSet) to ensure that the client does not cache. The client will read the execution result from the network connection through `StreamingResult`.
When the client uses the streaming read method, it needs to finish reading or close `resultset` before continuing to use the statement to make a query. Otherwise, the error `No statements may be issued when any streaming result sets are open and in use on a given connection. Ensure that you have called .close() on any active streaming result sets before attempting more queries.` is returned.
To avoid such an error in queries before the client finishes reading or closes `resultset`, you can add the `clobberStreamingResults=true` parameter in the URL. Then, `resultset` is automatically closed but the result set to be read in the previous streaming query is lost.
-- To use Cursor Fetch, first [set `FetchSize`](http://makejavafaster.blogspot.com/2015/06/jdbc-fetch-size-performance.html) as a positive integer and configure `useCursorFetch=true` in the JDBC URL.
+- The second method: Use Cursor Fetch by first [setting `FetchSize`](http://makejavafaster.blogspot.com/2015/06/jdbc-fetch-size-performance.html) as a positive integer and then configuring `useCursorFetch = true` in the JDBC URL.
-TiDB supports both methods, but it is preferred that you use the first method, because it is a simpler implementation and has a better execution efficiency.
+TiDB supports both methods, but it is recommended that you use the first method that sets `FetchSize` to `Integer.MIN_VALUE`, because it is a simpler implementation and has better execution efficiency.
+
+For the second method, TiDB first loads all data to the TiDB node, and then returns data to the client according to the `FetchSize`. Therefore, it usually consumes more memory than the first method. If [`tidb_enable_tmp_storage_on_oom`](/system-variables.md#tidb_enable_tmp_storage_on_oom) is set to `ON`, TiDB might temporarily write the result to the hard disk.
+
+If the [`tidb_enable_lazy_cursor_fetch`](/system-variables.md#tidb_enable_lazy_cursor_fetch-new-in-v830) system variable is set to `ON`, TiDB tries to read part of the data only when the client fetches it, which uses less memory. For more details and limitations, read the [complete descriptions for the `tidb_enable_lazy_cursor_fetch` system variable](/system-variables.md#tidb_enable_lazy_cursor_fetch-new-in-v830).
### MySQL JDBC parameters
diff --git a/glossary.md b/glossary.md
index 39b96d20f7288..5b404be6358cd 100644
--- a/glossary.md
+++ b/glossary.md
@@ -173,7 +173,3 @@ Top SQL helps locate SQL queries that contribute to a high load of a TiDB or TiK
### TSO
Because TiKV is a distributed storage system, it requires a global timing service, Timestamp Oracle (TSO), to assign a monotonically increasing timestamp. In TiKV, such a feature is provided by PD, and in Google [Spanner](http://static.googleusercontent.com/media/research.google.com/en//archive/spanner-osdi2012.pdf), this feature is provided by multiple atomic clocks and GPS.
-
-### TTL
-
-[Time to live (TTL)](/time-to-live.md) is a feature that allows you to manage TiDB data lifetime at the row level. For a table with the TTL attribute, TiDB automatically checks data lifetime and deletes expired data at the row level.
diff --git a/log-redaction.md b/log-redaction.md
index 04c8ff7965de9..61108e93af67a 100644
--- a/log-redaction.md
+++ b/log-redaction.md
@@ -9,14 +9,12 @@ When TiDB provides detailed log information, it might print sensitive data (for
## Log redaction in TiDB side
-To enable log redaction in the TiDB side, set the value of [`global.tidb_redact_log`](/system-variables.md#tidb_redact_log) to `1`. This configuration value defaults to `0`, which means that log redaction is disabled.
+To enable log redaction in the TiDB side, set the value of [`global.tidb_redact_log`](/system-variables.md#tidb_redact_log) to `ON` or `MARKER`. This configuration value defaults to `OFF`, which means that log redaction is disabled.
You can use the `set` syntax to set the global variable `tidb_redact_log`:
-{{< copyable "sql" >}}
-
```sql
-set @@global.tidb_redact_log=1;
+set @@global.tidb_redact_log = ON;
```
After the setting, all logs generated in new sessions are redacted:
@@ -32,19 +30,43 @@ ERROR 1062 (23000): Duplicate entry '1' for key 't.a'
The error log for the `INSERT` statement above is printed as follows:
```
-[2020/10/20 11:45:49.539 +08:00] [INFO] [conn.go:800] ["command dispatched failed"] [conn=5] [connInfo="id:5, addr:127.0.0.1:57222 status:10, collation:utf8_general_ci, user:root"] [command=Query] [status="inTxn:0, autocommit:1"] [sql="insert into t values ( ? ) , ( ? )"] [txn_mode=OPTIMISTIC] [err="[kv:1062]Duplicate entry '?' for key 't.a'"]
+[2024/07/02 11:35:32.686 +08:00] [INFO] [conn.go:1146] ["command dispatched failed"] [conn=1482686470] [session_alias=] [connInfo="id:1482686470, addr:127.0.0.1:52258 status:10, collation:utf8mb4_0900_ai_ci, user:root"] [command=Query] [status="inTxn:0, autocommit:1"] [sql="insert into `t` values ( ... )"] [txn_mode=PESSIMISTIC] [timestamp=450859193514065921] [err="[kv:1062]Duplicate entry '?' for key 't.a'"]
+```
+
+From the preceding error log, you can see that when the value of `tidb_redact_log` is set to `ON`, sensitive information is replaced by the `?` mark in the TiDB log to avoid data security risks.
+
+In addition, TiDB provides the `MARKER` option. When the value of `tidb_redact_log` is set to `MARKER`, TiDB marks sensitive information in the log with `‹›` instead of replacing it directly, so you can customize the redaction rules.
+
+```sql
+set @@global.tidb_redact_log = MARKER;
+```
+
+After the preceding configuration, the sensitive information is marked rather than replaced in all logs generated by new sessions:
+
+```sql
+create table t (a int, unique key (a));
+Query OK, 0 rows affected (0.07 sec)
+
+insert into t values (1),(1);
+ERROR 1062 (23000): Duplicate entry '‹1›' for key 't.a'
+```
+
+The error log is as follows:
+
+```
+[2024/07/02 11:35:01.426 +08:00] [INFO] [conn.go:1146] ["command dispatched failed"] [conn=1482686470] [session_alias=] [connInfo="id:1482686470, addr:127.0.0.1:52258 status:10, collation:utf8mb4_0900_ai_ci, user:root"] [command=Query] [status="inTxn:0, autocommit:1"] [sql="insert into `t` values ( ‹1› ) , ( ‹1› )"] [txn_mode=PESSIMISTIC] [timestamp=450859185309483010] [err="[kv:1062]Duplicate entry '‹1›' for key 't.a'"]
```
-From the error log above, you can see that all sensitive information is shielded using `?` after `tidb_redact_log` is enabled. In this way, data security risks are avoided.
+As you can see from the preceding error log, after you set `tidb_redact_log` to `MARKER`, TiDB marks sensitive information using `‹ ›` in the log. You can customize redaction rules to handle sensitive information in the log as needed.
## Log redaction in TiKV side
-To enable log redaction in the TiKV side, set the value of [`security.redact-info-log`](/tikv-configuration-file.md#redact-info-log-new-in-v408) to `true`. This configuration value defaults to `false`, which means that log redaction is disabled.
+To enable log redaction in the TiKV side, set the value of [`security.redact-info-log`](/tikv-configuration-file.md#redact-info-log-new-in-v408) to `true` or `"marker"`. This configuration value defaults to `false`, which means that log redaction is disabled.
## Log redaction in PD side
-To enable log redaction in the PD side, set the value of [`security.redact-info-log`](/pd-configuration-file.md#redact-info-log-new-in-v50) to `true`. This configuration value defaults to `false`, which means that log redaction is disabled.
+To enable log redaction in the PD side, set the value of [`security.redact-info-log`](/pd-configuration-file.md#redact-info-log-new-in-v50) to `true` or `"marker"`. This configuration value defaults to `false`, which means that log redaction is disabled.
## Log redaction in TiFlash side
-To enable log redaction in the TiFlash side, set both the [`security.redact_info_log`](/tiflash/tiflash-configuration.md#configure-the-tiflashtoml-file) value in tiflash-server and the [`security.redact-info-log`](/tiflash/tiflash-configuration.md#configure-the-tiflash-learnertoml-file) value in tiflash-learner to `true`. Both configuration values default to `false`, which means that log redaction is disabled.
+To enable log redaction in the TiFlash side, set both the [`security.redact_info_log`](/tiflash/tiflash-configuration.md#configure-the-tiflashtoml-file) value in tiflash-server and the [`security.redact-info-log`](/tiflash/tiflash-configuration.md#configure-the-tiflash-learnertoml-file) value in tiflash-learner to `true` or `"marker"`. Both configuration values default to `false`, which means that log redaction is disabled.
diff --git a/pd-configuration-file.md b/pd-configuration-file.md
index f727ce33ba9b5..eebe1e5420430 100644
--- a/pd-configuration-file.md
+++ b/pd-configuration-file.md
@@ -198,8 +198,9 @@ Configuration items related to security
### `redact-info-log` New in v5.0
+ Controls whether to enable log redaction in the PD log
-+ When you set the configuration value to `true`, user data is redacted in the PD log.
++ Optional value: `false`, `true`, `"marker"`
+ Default value: `false`
++ For details on how to use it, see [Log redaction in PD side](/log-redaction.md#log-redaction-in-pd-side).
## `log`
diff --git a/pessimistic-transaction.md b/pessimistic-transaction.md
index b759d2af95854..b2bfc808f7f7c 100644
--- a/pessimistic-transaction.md
+++ b/pessimistic-transaction.md
@@ -111,7 +111,9 @@ Pessimistic transactions in TiDB behave similarly to those in MySQL. See the min
2. TiDB does not support `SELECT LOCK IN SHARE MODE`.
- When `SELECT LOCK IN SHARE MODE` is executed, it has the same effect as that without the lock, so the read or write operation of other transactions is not blocked.
+ TiDB does not support the `SELECT LOCK IN SHARE MODE` syntax by default. You can enable [`tidb_enable_noop_functions`](/system-variables.md#tidb_enable_noop_functions-new-in-v40) to make TiDB compatible with the `SELECT LOCK IN SHARE MODE` syntax. Executing `SELECT LOCK IN SHARE MODE` has the same effect as that without the lock, so it does not block read or write operations of other transactions.
+
+ Starting from v8.3.0, TiDB supports using the [`tidb_enable_shared_lock_promotion`](/system-variables.md#tidb_enable_shared_lock_promotion-new-in-v830) system variable to enable the `SELECT LOCK IN SHARE MODE` statement to add locks. However, note that the lock added at this time is not a true shared lock, but an exclusive lock, which is consistent with `SELECT FOR UPDATE`. If you want to block writes to prevent data from being modified by write transactions in parallel during reads while keeping TiDB compatible with the `SELECT LOCK IN SHARE MODE` syntax, you can enable this variable. Enabling this variable takes effect on the `SELECT LOCK IN SHARE MODE` statement, regardless of whether [`tidb_enable_noop_functions`](/system-variables.md#tidb_enable_noop_functions-new-in-v40) is enabled or not.
3. DDL may result in failure of the pessimistic transaction commit.
diff --git a/scripts/release_notes_update_pr_author_info_add_dup.py b/scripts/release_notes_update_pr_author_info_add_dup.py
index 73d9f862c6ed1..73867d1e596d9 100644
--- a/scripts/release_notes_update_pr_author_info_add_dup.py
+++ b/scripts/release_notes_update_pr_author_info_add_dup.py
@@ -70,7 +70,7 @@ def store_exst_rn(ext_path, version):
else:
return 0
-def get_pr_info_from_github(cp_pr_link,cp_pr_title, current_pr_author):
+def get_pr_info_from_github(row_number, cp_pr_link,cp_pr_title, current_pr_author):
g = Github(access_token, timeout=30)# Create a Github object with the access token
target_pr_number_existence = 1
@@ -103,9 +103,10 @@ def get_pr_info_from_github(cp_pr_link,cp_pr_title, current_pr_author):
pr_obj = repo_obj.get_pull(int(target_pr_number))# Get the pull request object
pr_author = pr_obj.user.login # Get the author of the pull request
except:
- print("Failed to get the original PR information for this PR: " + cp_pr_link)
+ print(f"Row {row_number}: failed to find the non-bot author for this PR ({cp_pr_link}) created by {current_pr_author}.\n")
else:
pr_author = current_pr_author # Use the current author if the cherry-pick PR cannot be found
+ print(f"Row {row_number}: failed to find the non-bot author for this PR ({cp_pr_link}) created by {current_pr_author}.\n")
return(pr_author)
@@ -135,14 +136,24 @@ def update_pr_author_and_release_notes(excel_path):
# If pr_author is ti-chi-bot or ti-srebot
current_pr_author = row[pr_author_index]
current_formated_rn= row[pr_formated_rn_index]
- pr_response = requests.get(row[pr_link_index])
- if (current_pr_author in ['ti-chi-bot', 'ti-srebot']) and (pr_response.status_code == 200):
- print ("Replacing the author info for row " + str(row_index) + ".")
- actual_pr_author = get_pr_info_from_github(row[pr_link_index], row[pr_title_index], current_pr_author) # Get the PR author according to the cherry-pick PR
- pr_author_cell = sheet.cell(row=row_index, column=pr_author_index+1, value = actual_pr_author)#Fill in the pr_author_cell
- updated_formated_rn = current_formated_rn.replace("[{}](https://github.com/{}".format(current_pr_author, current_pr_author),"[{}](https://github.com/{}".format(actual_pr_author, actual_pr_author))
- formated_release_note_cell = sheet.cell(row=row_index, column=pr_formated_rn_index+1, value = updated_formated_rn) # Fill in the formated_release_note_cell
- current_pr_author = actual_pr_author
+
+ if (current_pr_author in ['ti-chi-bot', 'ti-srebot']):
+ try:
+ actual_pr_author = get_pr_info_from_github(str(row_index), row[pr_link_index], row[pr_title_index], current_pr_author) # Get the PR author according to the cherry-pick PR
+ if actual_pr_author != current_pr_author:
+ print ("Replacing the author info for row " + str(row_index) + ".")
+ pr_author_cell = sheet.cell(row=row_index, column=pr_author_index+1, value = actual_pr_author)#Fill in the pr_author_cell
+ updated_formated_rn = current_formated_rn.replace("[{}](https://github.com/{}".format(current_pr_author, current_pr_author),"[{}](https://github.com/{}".format(actual_pr_author, actual_pr_author))
+ formated_release_note_cell = sheet.cell(row=row_index, column=pr_formated_rn_index+1, value = updated_formated_rn) # Fill in the formated_release_note_cell
+ current_pr_author = actual_pr_author
+ else: # Do nothing if non-bot author is not found.
+ pass
+ except:
+ pr_response = requests.get(row[pr_link_index])
+ if pr_response.status_code != 200:
+ print (f"\nRow {str(row_index)}: failed to find the non-bot author for this PR ({row[pr_link_index]}) because this link cannot be accessed now.")
+ else:
+ print (f"\nRow {str(row_index)}: failed to find the non-bot author for this PR ({row[pr_link_index]}).")
else:
pass
@@ -232,12 +243,12 @@ def create_release_file(version, dup_notes_levels, dup_notes):
file.seek(0)
file.write(content)
file.truncate()
- print(f'The v{version} release note is now created in the following directory: \n {release_file}')
+ print(f'\nThe v{version} release note is now created in the following directory: \n {release_file}')
if __name__ == '__main__':
note_pairs = store_exst_rn(ext_path, version)
dup_notes, dup_notes_levels = update_pr_author_and_release_notes(release_note_excel)
- print ("The bot author info in the excel is now replaced with the actual authors.")
+ print ("\nThe bot author info in the excel is now replaced with the actual authors.")
version_parts = version.split('.')
if len(version_parts) >= 2:
create_release_file(version, dup_notes_levels, dup_notes)
\ No newline at end of file
diff --git a/sql-plan-management.md b/sql-plan-management.md
index 956c283f91bf6..e7d349ec7827c 100644
--- a/sql-plan-management.md
+++ b/sql-plan-management.md
@@ -231,25 +231,25 @@ The original SQL statement and the bound statement must have the same text after
#### Create a binding according to a historical execution plan
-To make the execution plan of a SQL statement fixed to a historical execution plan, you can use `plan_digest` to bind that historical execution plan to the SQL statement, which is more convenient than binding it according to a SQL statement.
+To make the execution plan of a SQL statement fixed to a historical execution plan, you can use Plan Digest to bind that historical execution plan to the SQL statement, which is more convenient than binding it according to a SQL statement. In addition, you can bind the execution plan for multiple SQL statements at once. For more details and examples, see [`CREATE [GLOBAL|SESSION] BINDING`](/sql-statements/sql-statement-create-binding.md).
When using this feature, note the following:
- The feature generates hints according to historical execution plans and uses the generated hints for binding. Because historical execution plans are stored in [Statement Summary Tables](/statement-summary-tables.md), before using this feature, you need to enable the [`tidb_enable_stmt_summary`](/system-variables.md#tidb_enable_stmt_summary-new-in-v304) system variable first.
- For TiFlash queries, Join queries with three or more tables, and queries that contain subqueries, the auto-generated hints are not adequate, which might result in the plan not being fully bound. In such cases, a warning will occur when creating a binding.
-- If a historical execution plan is for a SQL statement with hints, the hints will be added to the binding. For example, after executing `SELECT /*+ max_execution_time(1000) */ * FROM t`, the binding created with its `plan_digest` will include `max_execution_time(1000)`.
+- If a historical execution plan is for a SQL statement with hints, the hints will be added to the binding. For example, after executing `SELECT /*+ max_execution_time(1000) */ * FROM t`, the binding created with its Plan Digest will include `max_execution_time(1000)`.
The SQL statement of this binding method is as follows:
```sql
-CREATE [GLOBAL | SESSION] BINDING FROM HISTORY USING PLAN DIGEST 'plan_digest';
+CREATE [GLOBAL | SESSION] BINDING FROM HISTORY USING PLAN DIGEST StringLiteralOrUserVariableList;
```
-This statement binds an execution plan to a SQL statement using `plan_digest`. The default scope is SESSION. The applicable SQL statements, priorities, scopes, and effective conditions of the created bindings are the same as that of [bindings created according to SQL statements](#create-a-binding-according-to-a-sql-statement).
+The preceding statement binds an execution plan to a SQL statement using Plan Digest. The default scope is SESSION. The applicable SQL statements, priorities, scopes, and effective conditions of the created bindings are the same as that of [bindings created according to SQL statements](#create-a-binding-according-to-a-sql-statement).
-To use this binding method, you need to first get the `plan_digest` corresponding to the target historical execution plan in `statements_summary`, and then create a binding using the `plan_digest`. The detailed steps are as follows:
+To use this binding method, you need to first get the Plan Digest corresponding to the target historical execution plan in `statements_summary`, and then create a binding using the Plan Digest. The detailed steps are as follows:
-1. Get the `plan_digest` corresponding to the target execution plan in `statements_summary`.
+1. Get the Plan Digest corresponding to the target execution plan in `statements_summary`.
For example:
@@ -274,9 +274,9 @@ To use this binding method, you need to first get the `plan_digest` correspondin
BINARY_PLAN: 6QOYCuQDCg1UYWJsZVJlYWRlcl83Ev8BCgtTZWxlY3Rpb25fNhKOAQoPBSJQRnVsbFNjYW5fNSEBAAAAOA0/QSkAAQHwW4jDQDgCQAJKCwoJCgR0ZXN0EgF0Uh5rZWVwIG9yZGVyOmZhbHNlLCBzdGF0czpwc2V1ZG9qInRpa3ZfdGFzazp7dGltZTo1NjAuOMK1cywgbG9vcHM6MH1w////CQMEAXgJCBD///8BIQFzCDhVQw19BAAkBX0QUg9lcSgBfCAudC5hLCAxKWrmYQAYHOi0gc6hBB1hJAFAAVIQZGF0YTo9GgRaFAW4HDQuMDVtcywgCbYcMWKEAWNvcF8F2agge251bTogMSwgbWF4OiA1OTguNsK1cywgcHJvY19rZXlzOiAwLCBycGNfBSkAMgkMBVcQIDYwOS4pEPBDY29wcl9jYWNoZV9oaXRfcmF0aW86IDAuMDAsIGRpc3RzcWxfY29uY3VycmVuY3k6IDE1fXCwAXj///////////8BGAE=
```
- In this example, you can see that the execution plan corresponding to `plan_digest` is `4e3159169cc63c14b139a4e7d72eae1759875c9a9581f94bb2079aae961189cb`.
+ In this example, you can see that the execution plan corresponding to Plan Digest is `4e3159169cc63c14b139a4e7d72eae1759875c9a9581f94bb2079aae961189cb`.
-2. Use `plan_digest` to create a binding:
+2. Use Plan Digest to create a binding:
```sql
CREATE BINDING FROM HISTORY USING PLAN DIGEST '4e3159169cc63c14b139a4e7d72eae1759875c9a9581f94bb2079aae961189cb';
@@ -317,7 +317,7 @@ SELECT @@LAST_PLAN_FROM_BINDING;
### Remove a binding
-You can remove a binding according to a SQL statement or `sql_digest`.
+You can remove a binding according to a SQL statement or SQL Digest.
#### Remove a binding according to a SQL statement
@@ -343,15 +343,15 @@ explain SELECT * FROM t1,t2 WHERE t1.id = t2.id;
In the example above, the dropped binding in the SESSION scope shields the corresponding binding in the GLOBAL scope. The optimizer does not add the `sm_join(t1, t2)` hint to the statement. The top node of the execution plan in the `explain` result is not fixed to MergeJoin by this hint. Instead, the top node is independently selected by the optimizer according to the cost estimation.
-#### Remove a binding according to `sql_digest`
+#### Remove a binding according to SQL Digest
-In addition to removing a binding according to a SQL statement, you can also remove a binding according to `sql_digest`.
+In addition to removing a binding according to a SQL statement, you can also remove a binding according to SQL Digest. For more details and examples, see [`DROP [GLOBAL|SESSION] BINDING`](/sql-statements/sql-statement-drop-binding.md).
```sql
-DROP [GLOBAL | SESSION] BINDING FOR SQL DIGEST 'sql_digest';
+DROP [GLOBAL | SESSION] BINDING FOR SQL DIGEST StringLiteralOrUserVariableList;
```
-This statement removes an execution plan binding corresponding to `sql_digest` at the GLOBAL or SESSION level. The default scope is SESSION. You can get the `sql_digest` by [viewing bindings](#view-bindings).
+This statement removes an execution plan binding corresponding to SQL Digest at the GLOBAL or SESSION level. The default scope is SESSION. You can get the SQL Digest by [viewing bindings](#view-bindings).
> **Note:**
>
diff --git a/sql-statements/sql-statement-admin-bdr-role.md b/sql-statements/sql-statement-admin-bdr-role.md
index 4781794542dfa..d13e6400ee108 100644
--- a/sql-statements/sql-statement-admin-bdr-role.md
+++ b/sql-statements/sql-statement-admin-bdr-role.md
@@ -9,10 +9,6 @@ summary: An overview of the usage of ADMIN [SET|SHOW|UNSET] BDR ROLE for the TiD
- Use `ADMIN SHOW BDR ROLE` to show the BDR role of the cluster.
- Use `ADMIN UNSET BDR ROLE` to unset the BDR role of the cluster.
-> **Warning:**
->
-> This feature is experimental. It is not recommended that you use it in the production environment. This feature might be changed or removed without prior notice. If you find a bug, you can report an [issue](https://github.com/pingcap/tidb/issues) on GitHub.
-
## Synopsis
```ebnf+diagram
diff --git a/sql-statements/sql-statement-create-binding.md b/sql-statements/sql-statement-create-binding.md
index fcf3dc8ce70d8..7788679e68967 100644
--- a/sql-statements/sql-statement-create-binding.md
+++ b/sql-statements/sql-statement-create-binding.md
@@ -17,13 +17,19 @@ The bound SQL statement is parameterized and stored in the system table. When a
```ebnf+diagram
CreateBindingStmt ::=
'CREATE' GlobalScope 'BINDING' ( 'FOR' BindableStmt 'USING' BindableStmt
-| 'FROM' 'HISTORY' 'USING' 'PLAN' 'DIGEST' PlanDigest )
+| 'FROM' 'HISTORY' 'USING' 'PLAN' 'DIGEST' StringLiteralOrUserVariableList )
GlobalScope ::=
( 'GLOBAL' | 'SESSION' )?
BindableStmt ::=
( SelectStmt | UpdateStmt | InsertIntoStmt | ReplaceIntoStmt | DeleteStmt )
+
+StringLiteralOrUserVariableList ::=
+ ( StringLitOrUserVariable | StringLiteralOrUserVariableList ',' StringLitOrUserVariable )
+
+StringLiteralOrUserVariable ::=
+ ( stringLiteral | UserVariable )
```
****
@@ -32,6 +38,11 @@ BindableStmt ::=
You can create a binding according to a SQL statement or a historical execution plan.
+When you create a binding according to a historical execution plan, you need to specify the corresponding Plan Digest:
+
+- You can use either the string literal or user variable of the string type to specify the Plan Digest.
+- You can specify multiple Plan Digests to create bindings for multiple statements at the same time. In this case, you can specify multiple strings, and include multiple digests in each string. Note that the strings or digests need to be separated by commas.
+
The following example shows how to create a binding according to a SQL statement.
{{< copyable "sql" >}}
@@ -139,34 +150,165 @@ mysql> EXPLAIN ANALYZE SELECT * FROM t1 WHERE b = 123;
The following example shows how to create a binding according to a historical execution plan.
```sql
-mysql> CREATE TABLE t(id INT PRIMARY KEY , a INT, KEY(a));
-Query OK, 0 rows affected (0.06 sec)
+USE test;
+CREATE TABLE t1(a INT, b INT, c INT, INDEX ia(a));
+CREATE TABLE t2(a INT, b INT, c INT, INDEX ia(a));
+INSERT INTO t1 SELECT * FROM t2 WHERE a = 1;
+SELECT @@LAST_PLAN_FROM_BINDING;
+UPDATE /*+ INL_JOIN(t2) */ t1, t2 SET t1.a = 1 WHERE t1.b = t2.a;
+SELECT @@LAST_PLAN_FROM_BINDING;
+DELETE /*+ HASH_JOIN(t1) */ t1 FROM t1 JOIN t2 WHERE t1.b = t2.a;
+SELECT @@LAST_PLAN_FROM_BINDING;
+SELECT * FROM t1 WHERE t1.a IN (SELECT a FROM t2);
+SELECT @@LAST_PLAN_FROM_BINDING;
+```
+
+Method 1:
+
+```sql
+SELECT query_sample_text, stmt_type, table_names, plan_digest FROM information_schema.statements_summary_history WHERE table_names LIKE '%test.t1%' AND stmt_type != 'CreateTable';
+CREATE GLOBAL BINDING FROM HISTORY USING PLAN DIGEST 'e72819cf99932f63a548156dbf433adda60e10337e89dcaa8638b4caf16f64d8,c291edc36b2482738d3389d335f37efc76290be2930330fe5034c5f4c42eeb36,8dc146249484f4a6ab219bfe9effa6b7a18aeed3764d49b610da61ac347ab914,73b2dec866595688ea416675f88ccb3456eb8e7443a79cd816695b688e07ac6b';
+```
+
+Method 2:
+
+```sql
+SELECT @digests:=GROUP_CONCAT(plan_digest) FROM information_schema.statements_summary_history WHERE table_names LIKE '%test.t1%' AND stmt_type != 'CreateTable';
+CREATE GLOBAL BINDING FROM HISTORY USING PLAN DIGEST @digests;
+```
+
+```sql
+SHOW GLOBAL BINDINGS;
+INSERT INTO t1 SELECT * FROM t2 WHERE a = 1;
+SELECT @@LAST_PLAN_FROM_BINDING;
+UPDATE t1, t2 SET t1.a = 1 WHERE t1.b = t2.a;
+SELECT @@LAST_PLAN_FROM_BINDING;
+DELETE t1 FROM t1 JOIN t2 WHERE t1.b = t2.a;
+SELECT @@LAST_PLAN_FROM_BINDING;
+SELECT * FROM t1 WHERE t1.a IN (SELECT a FROM t2);
+SELECT @@LAST_PLAN_FROM_BINDING;
+```
+
+```sql
+> CREATE TABLE t1(a INT, b INT, c INT, INDEX ia(a));
+Query OK, 0 rows affected (0.048 sec)
+
+> CREATE TABLE t2(a INT, b INT, c INT, INDEX ia(a));
+Query OK, 0 rows affected (0.035 sec)
+
+> INSERT INTO t1 SELECT * FROM t2 WHERE a = 1;
+Query OK, 0 rows affected (0.002 sec)
+Records: 0 Duplicates: 0 Warnings: 0
+
+> SELECT @@LAST_PLAN_FROM_BINDING;
++--------------------------+
+| @@LAST_PLAN_FROM_BINDING |
++--------------------------+
+| 0 |
++--------------------------+
+1 row in set (0.001 sec)
+
+> UPDATE /*+ INL_JOIN(t2) */ t1, t2 SET t1.a = 1 WHERE t1.b = t2.a;
+Query OK, 0 rows affected (0.005 sec)
+Rows matched: 0 Changed: 0 Warnings: 0
+
+> SELECT @@LAST_PLAN_FROM_BINDING;
++--------------------------+
+| @@LAST_PLAN_FROM_BINDING |
++--------------------------+
+| 0 |
++--------------------------+
+1 row in set (0.000 sec)
-mysql> SELECT /*+ IGNORE_INDEX(t, a) */ * FROM t WHERE a = 1;
-Empty set (0.01 sec)
+> DELETE /*+ HASH_JOIN(t1) */ t1 FROM t1 JOIN t2 WHERE t1.b = t2.a;
+Query OK, 0 rows affected (0.003 sec)
-mysql> SELECT plan_digest FROM INFORMATION_SCHEMA.STATEMENTS_SUMMARY WHERE QUERY_SAMPLE_TEXT = 'SELECT /*+ IGNORE_INDEX(t, a) */ * FROM t WHERE a = 1';
-+------------------------------------------------------------------+
-| plan_digest |
-+------------------------------------------------------------------+
-| 4e3159169cc63c14b139a4e7d72eae1759875c9a9581f94bb2079aae961189cb |
-+------------------------------------------------------------------+
-1 row in set (0.01 sec)
+> SELECT @@LAST_PLAN_FROM_BINDING;
++--------------------------+
+| @@LAST_PLAN_FROM_BINDING |
++--------------------------+
+| 0 |
++--------------------------+
+1 row in set (0.000 sec)
-mysql> CREATE BINDING FROM HISTORY USING PLAN DIGEST '4e3159169cc63c14b139a4e7d72eae1759875c9a9581f94bb2079aae961189cb';
-Query OK, 0 rows affected (0.02 sec)
+> SELECT * FROM t1 WHERE t1.a IN (SELECT a FROM t2);
+Empty set (0.002 sec)
-mysql> SELECT * FROM t WHERE a = 1;
-Empty set (0.01 sec)
+> SELECT @@LAST_PLAN_FROM_BINDING;
++--------------------------+
+| @@LAST_PLAN_FROM_BINDING |
++--------------------------+
+| 0 |
++--------------------------+
+1 row in set (0.001 sec)
+
+> SELECT @digests:=GROUP_CONCAT(plan_digest) FROM information_schema.statements_summary_history WHERE table_names LIKE '%test.t1%' AND stmt_type != 'CreateTable';
++---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+| @digests:=GROUP_CONCAT(plan_digest) |
++---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+| 73b2dec866595688ea416675f88ccb3456eb8e7443a79cd816695b688e07ac6b,8dc146249484f4a6ab219bfe9effa6b7a18aeed3764d49b610da61ac347ab914,c291edc36b2482738d3389d335f37efc76290be2930330fe5034c5f4c42eeb36,e72819cf99932f63a548156dbf433adda60e10337e89dcaa8638b4caf16f64d8 |
++---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+1 row in set (0.001 sec)
+
+> CREATE GLOBAL BINDING FROM HISTORY USING PLAN DIGEST @digests;
+Query OK, 0 rows affected (0.060 sec)
+
+> SHOW GLOBAL BINDINGS;
++----------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------+---------+-------------------------+-------------------------+---------+-----------------+---------+------------------------------------------------------------------+------------------------------------------------------------------+
+| Original_sql | Bind_sql | Default_db | Status | Create_time | Update_time | Charset | Collation | Source | Sql_digest | Plan_digest |
++----------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------+---------+-------------------------+-------------------------+---------+-----------------+---------+------------------------------------------------------------------+------------------------------------------------------------------+
+| insert into `test` . `t1` select * from `test` . `t2` where `a` = ? | INSERT INTO `test`.`t1` SELECT /*+ use_index(@`sel_1` `test`.`t2` `ia`) no_order_index(@`sel_1` `test`.`t2` `ia`)*/ * FROM `test`.`t2` WHERE `a` = 1 | test | enabled | 2024-08-11 05:27:19.669 | 2024-08-11 05:27:19.669 | utf8 | utf8_general_ci | history | bd23e6af17e7b77b25383e50e258f0dece18583d19772f08caacb2021945a300 | e72819cf99932f63a548156dbf433adda60e10337e89dcaa8638b4caf16f64d8 |
+| update ( `test` . `t1` ) join `test` . `t2` set `t1` . `a` = ? where `t1` . `b` = `t2` . `a` | UPDATE /*+ inl_join(`test`.`t2`) use_index(@`upd_1` `test`.`t1` ) use_index(@`upd_1` `test`.`t2` `ia`) no_order_index(@`upd_1` `test`.`t2` `ia`)*/ (`test`.`t1`) JOIN `test`.`t2` SET `t1`.`a`=1 WHERE `t1`.`b` = `t2`.`a` | test | enabled | 2024-08-11 05:27:19.667 | 2024-08-11 05:27:19.667 | utf8 | utf8_general_ci | history | 987e91af17eb40e36fecfc0634cce0b6a736de02bb009091810f932804fc02e9 | c291edc36b2482738d3389d335f37efc76290be2930330fe5034c5f4c42eeb36 |
+| delete `test` . `t1` from `test` . `t1` join `test` . `t2` where `t1` . `b` = `t2` . `a` | DELETE /*+ hash_join_build(`test`.`t2`) use_index(@`del_1` `test`.`t1` ) use_index(@`del_1` `test`.`t2` )*/ `test`.`t1` FROM `test`.`t1` JOIN `test`.`t2` WHERE `t1`.`b` = `t2`.`a` | test | enabled | 2024-08-11 05:27:19.664 | 2024-08-11 05:27:19.664 | utf8 | utf8_general_ci | history | 70ef3d442d95c51020a76c7c86a3ab674258606d4dd24bbd16ac6f69d87a4316 | 8dc146249484f4a6ab219bfe9effa6b7a18aeed3764d49b610da61ac347ab914 |
+| select * from `test` . `t1` where `t1` . `a` in ( select `a` from `test` . `t2` ) | SELECT /*+ use_index(@`sel_1` `test`.`t1` ) stream_agg(@`sel_2`) use_index(@`sel_2` `test`.`t2` `ia`) order_index(@`sel_2` `test`.`t2` `ia`) agg_to_cop(@`sel_2`)*/ * FROM `test`.`t1` WHERE `t1`.`a` IN (SELECT `a` FROM `test`.`t2`) | test | enabled | 2024-08-11 05:27:19.649 | 2024-08-11 05:27:19.649 | utf8 | utf8_general_ci | history | b58508a5e29d7889adf98cad50343d7a575fd32ad55dbdaa88e14ecde54f3d93 | 73b2dec866595688ea416675f88ccb3456eb8e7443a79cd816695b688e07ac6b |
++----------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------+---------+-------------------------+-------------------------+---------+-----------------+---------+------------------------------------------------------------------+------------------------------------------------------------------+
+4 rows in set (0.001 sec)
+
+> INSERT INTO t1 SELECT * FROM t2 WHERE a = 1;
+Query OK, 0 rows affected (0.002 sec)
+Records: 0 Duplicates: 0 Warnings: 0
+
+> SELECT @@LAST_PLAN_FROM_BINDING;
++--------------------------+
+| @@LAST_PLAN_FROM_BINDING |
++--------------------------+
+| 1 |
++--------------------------+
+1 row in set (0.000 sec)
-mysql> SELECT @@LAST_PLAN_FROM_BINDING;
+> UPDATE t1, t2 SET t1.a = 1 WHERE t1.b = t2.a;
+Query OK, 0 rows affected (0.002 sec)
+Rows matched: 0 Changed: 0 Warnings: 0
+
+> SELECT @@LAST_PLAN_FROM_BINDING;
++--------------------------+
+| @@LAST_PLAN_FROM_BINDING |
++--------------------------+
+| 1 |
++--------------------------+
+1 row in set (0.000 sec)
+
+> DELETE t1 FROM t1 JOIN t2 WHERE t1.b = t2.a;
+Query OK, 0 rows affected (0.002 sec)
+
+> SELECT @@LAST_PLAN_FROM_BINDING;
+--------------------------+
| @@LAST_PLAN_FROM_BINDING |
+--------------------------+
| 1 |
+--------------------------+
-1 row in set (0.01 sec)
+1 row in set (0.000 sec)
+
+> SELECT * FROM t1 WHERE t1.a IN (SELECT a FROM t2);
+Empty set (0.002 sec)
+> SELECT @@LAST_PLAN_FROM_BINDING;
++--------------------------+
+| @@LAST_PLAN_FROM_BINDING |
++--------------------------+
+| 1 |
++--------------------------+
+1 row in set (0.002 sec)
```
## MySQL compatibility
diff --git a/sql-statements/sql-statement-drop-binding.md b/sql-statements/sql-statement-drop-binding.md
index 6413ccf5fc4d2..25ddb098a6cdf 100644
--- a/sql-statements/sql-statement-drop-binding.md
+++ b/sql-statements/sql-statement-drop-binding.md
@@ -15,18 +15,29 @@ A `BINDING` can be on either a `GLOBAL` or `SESSION` basis. The default is `SESS
```ebnf+diagram
DropBindingStmt ::=
'DROP' GlobalScope 'BINDING' 'FOR' ( BindableStmt ( 'USING' BindableStmt )?
-| 'SQL' 'DIGEST' SqlDigest)
+| 'SQL' 'DIGEST' StringLiteralOrUserVariableList )
GlobalScope ::=
( 'GLOBAL' | 'SESSION' )?
BindableStmt ::=
( SelectStmt | UpdateStmt | InsertIntoStmt | ReplaceIntoStmt | DeleteStmt )
+
+StringLiteralOrUserVariableList ::=
+ ( StringLitOrUserVariable | StringLiteralOrUserVariableList ',' StringLitOrUserVariable )
+
+StringLiteralOrUserVariable ::=
+ ( stringLiteral | UserVariable )
```
## Examples
-You can remove a binding according to a SQL statement or `sql_digest`.
+You can remove a binding according to a SQL statement or SQL Digest.
+
+When you remove a binding according to SQL Digest, you need to specify the corresponding SQL Digest:
+
+- You can use either the string literal or user variable of the string type to specify the Plan Digest.
+- You can specify multiple string values, and include multiple digests in each string. Note that the strings or digests need to be separated by commas.
The following example shows how to remove a binding according to a SQL statement.
@@ -135,63 +146,63 @@ mysql> SHOW SESSION BINDINGS\G
Empty set (0.00 sec)
```
-The following example shows how to remove a binding according to `sql_digest`.
+The following example shows how to remove a binding according to SQL Digest.
```sql
-mysql> CREATE TABLE t(id INT PRIMARY KEY , a INT, KEY(a));
-Query OK, 0 rows affected (0.06 sec)
-
-mysql> SELECT /*+ IGNORE_INDEX(t, a) */ * FROM t WHERE a = 1;
-Empty set (0.01 sec)
-
-mysql> SELECT plan_digest FROM INFORMATION_SCHEMA.STATEMENTS_SUMMARY WHERE QUERY_SAMPLE_TEXT = 'SELECT /*+ IGNORE_INDEX(t, a) */ * FROM t WHERE a = 1';
-+------------------------------------------------------------------+
-| plan_digest |
-+------------------------------------------------------------------+
-| 4e3159169cc63c14b139a4e7d72eae1759875c9a9581f94bb2079aae961189cb |
-+------------------------------------------------------------------+
-1 row in set (0.01 sec)
-
-mysql> CREATE BINDING FROM HISTORY USING PLAN DIGEST '4e3159169cc63c14b139a4e7d72eae1759875c9a9581f94bb2079aae961189cb';
-Query OK, 0 rows affected (0.02 sec)
-
-mysql> SELECT * FROM t WHERE a = 1;
-Empty set (0.01 sec)
-
-mysql> SELECT @@LAST_PLAN_FROM_BINDING;
-+--------------------------+
-| @@LAST_PLAN_FROM_BINDING |
-+--------------------------+
-| 1 |
-+--------------------------+
-1 row in set (0.01 sec)
-
-mysql> SHOW BINDINGS\G
-*************************** 1. row ***************************
-Original_sql: select * from `test` . `t` where `a` = ?
- Bind_sql: SELECT /*+ use_index(@`sel_1` `test`.`t` ) ignore_index(`t` `a`)*/ * FROM `test`.`t` WHERE `a` = 1
- Default_db: test
- Status: enabled
- Create_time: 2022-12-14 15:26:22.277
- Update_time: 2022-12-14 15:26:22.277
- Charset: utf8mb4
- Collation: utf8mb4_general_ci
- Source: history
- Sql_digest: 6909a1bbce5f64ade0a532d7058dd77b6ad5d5068aee22a531304280de48349f
- Plan_digest: 4e3159169cc63c14b139a4e7d72eae1759875c9a9581f94bb2079aae961189cb
-1 row in set (0.02 sec)
+CREATE TABLE t1(a INT, b INT, c INT, INDEX ia(a));
+CREATE TABLE t2(a INT, b INT, c INT, INDEX ia(a));
+CREATE GLOBAL BINDING FOR SELECT * FROM t1 WHERE a > 1 USING SELECT * FROM t1 USE INDEX (ia) WHERE a > 1;
+CREATE GLOBAL BINDING FOR SELECT * FROM t2 WHERE a < 1 USING SELECT * FROM t2 USE INDEX (ia) WHERE a < 1;
+CREATE GLOBAL BINDING FOR SELECT * FROM t1 JOIN t2 ON t1.b = t2.a USING SELECT /*+ HASH_JOIN(t1) */ * FROM t1 JOIN t2 ON t1.b = t2.a;
+SHOW GLOBAL BINDINGS;
+```
-ERROR:
-No query specified
+Method 1:
-mysql> DROP BINDING FOR SQL DIGEST '6909a1bbce5f64ade0a532d7058dd77b6ad5d5068aee22a531304280de48349f';
-Query OK, 0 rows affected (0.00 sec)
+```sql
+DROP GLOBAL BINDING FOR SQL DIGEST '31026623c8f22264fe0dfc26f29c69c5c457d6b85960c578ebcf17a967ed7893', '0f38b2e769927ae37981c66f0988c6299b602e03f029e38aa071e656fc321593', '3c8dfc451b0e36afd904cefca5137e68fb051f02964e1958ed60afdadc25f57e';
+SHOW GLOBAL BINDINGS;
+```
+
+Method 2:
+
+```sql
+SET @digests='31026623c8f22264fe0dfc26f29c69c5c457d6b85960c578ebcf17a967ed7893, 0f38b2e769927ae37981c66f0988c6299b602e03f029e38aa071e656fc321593, 3c8dfc451b0e36afd904cefca5137e68fb051f02964e1958ed60afdadc25f57e';
+DROP GLOBAL BINDING FOR SQL DIGEST @digests;
+SHOW GLOBAL BINDINGS;
+```
+
+```sql
+> CREATE TABLE t1(a INT, b INT, c INT, INDEX ia(a));
+Query OK, 0 rows affected (0.044 sec)
+
+> CREATE TABLE t2(a INT, b INT, c INT, INDEX ia(a));
+Query OK, 0 rows affected (0.035 sec)
+
+> CREATE GLOBAL BINDING FOR SELECT * FROM t1 WHERE a > 1 USING SELECT * FROM t1 USE INDEX (ia) WHERE a > 1;
+Query OK, 0 rows affected (0.011 sec)
+
+> CREATE GLOBAL BINDING FOR SELECT * FROM t2 WHERE a < 1 USING SELECT * FROM t2 USE INDEX (ia) WHERE a < 1;
+Query OK, 0 rows affected (0.013 sec)
+
+> CREATE GLOBAL BINDING FOR SELECT * FROM t1 JOIN t2 ON t1.b = t2.a USING SELECT /*+ HASH_JOIN(t1) */ * FROM t1 JOIN t2 ON t1.b = t2.a;
+Query OK, 0 rows affected (0.012 sec)
+
+> SHOW GLOBAL BINDINGS;
++---------------------------------------------------------------------------+-----------------------------------------------------------------------------------------+------------+---------+-------------------------+-------------------------+---------+-----------------+--------+------------------------------------------------------------------+-------------+
+| Original_sql | Bind_sql | Default_db | Status | Create_time | Update_time | Charset | Collation | Source | Sql_digest | Plan_digest |
++---------------------------------------------------------------------------+-----------------------------------------------------------------------------------------+------------+---------+-------------------------+-------------------------+---------+-----------------+--------+------------------------------------------------------------------+-------------+
+| select * from `test` . `t1` join `test` . `t2` on `t1` . `b` = `t2` . `a` | SELECT /*+ HASH_JOIN(`t1`)*/ * FROM `test`.`t1` JOIN `test`.`t2` ON `t1`.`b` = `t2`.`a` | test | enabled | 2024-08-11 04:06:49.953 | 2024-08-11 04:06:49.953 | utf8 | utf8_general_ci | manual | 31026623c8f22264fe0dfc26f29c69c5c457d6b85960c578ebcf17a967ed7893 | |
+| select * from `test` . `t2` where `a` < ? | SELECT * FROM `test`.`t2` USE INDEX (`ia`) WHERE `a` < 1 | test | enabled | 2024-08-11 04:06:49.937 | 2024-08-11 04:06:49.937 | utf8 | utf8_general_ci | manual | 0f38b2e769927ae37981c66f0988c6299b602e03f029e38aa071e656fc321593 | |
+| select * from `test` . `t1` where `a` > ? | SELECT * FROM `test`.`t1` USE INDEX (`ia`) WHERE `a` > 1 | test | enabled | 2024-08-11 04:06:49.924 | 2024-08-11 04:06:49.924 | utf8 | utf8_general_ci | manual | 3c8dfc451b0e36afd904cefca5137e68fb051f02964e1958ed60afdadc25f57e | |
++---------------------------------------------------------------------------+-----------------------------------------------------------------------------------------+------------+---------+-------------------------+-------------------------+---------+-----------------+--------+------------------------------------------------------------------+-------------+
+3 rows in set (0.001 sec)
-mysql> SHOW BINDINGS\G
-Empty set (0.01 sec)
+> DROP GLOBAL BINDING FOR SQL DIGEST '31026623c8f22264fe0dfc26f29c69c5c457d6b85960c578ebcf17a967ed7893', '0f38b2e769927ae37981c66f0988c6299b602e03f029e38aa071e656fc321593', '3c8dfc451b0e36afd904cefca5137e68fb051f02964e1958ed60afdadc25f57e';
+Query OK, 3 rows affected (0.019 sec)
-ERROR:
-No query specified
+> SHOW GLOBAL BINDINGS;
+Empty set (0.002 sec)
```
## MySQL compatibility
diff --git a/sql-statements/sql-statement-split-region.md b/sql-statements/sql-statement-split-region.md
index c5a10cdacb017..9cad8f7116967 100644
--- a/sql-statements/sql-statement-split-region.md
+++ b/sql-statements/sql-statement-split-region.md
@@ -355,11 +355,11 @@ You can specify the partition to be split.
## pre_split_regions
-To have evenly split Regions when a table is created, it is recommended you use `SHARD_ROW_ID_BITS` together with `PRE_SPLIT_REGIONS`. When a table is created successfully, `PRE_SPLIT_REGIONS` pre-spilts tables into the number of Regions as specified by `2^(PRE_SPLIT_REGIONS)`.
+When creating a table with the `AUTO_RANDOM` or `SHARD_ROW_ID_BITS` attribute, you can also specify the `PRE_SPLIT_REGIONS` option if you want to evenly pre-split the table into Regions immediately after the table is created. The number of pre-split Regions for a table is `2^(PRE_SPLIT_REGIONS)`.
> **Note:**
>
-> The value of `PRE_SPLIT_REGIONS` must be less than or equal to that of `SHARD_ROW_ID_BITS`.
+> The value of `PRE_SPLIT_REGIONS` must be less than or equal to that of `SHARD_ROW_ID_BITS` or `AUTO_RANDOM`.
The `tidb_scatter_region` global variable affects the behavior of `PRE_SPLIT_REGIONS`. This variable controls whether to wait for Regions to be pre-split and scattered before returning results after the table creation. If there are intensive writes after creating the table, you need to set the value of this variable to `1`, then TiDB will not return the results to the client until all the Regions are split and scattered. Otherwise, TiDB writes the data before the scattering is completed, which will have a significant impact on write performance.
@@ -384,7 +384,7 @@ region4: [ 3<<61 , +inf )
> **Note:**
>
-> The Region split by the Split Region statement is controlled by the [Region merge](/best-practices/pd-scheduling-best-practices.md#region-merge) scheduler in PD. To avoid PD re-merging the newly split Region soon after, you need to [dynamically modify](/pd-control.md) configuration items related to the Region merge feature.
+> The Region split by the Split Region statement is controlled by the [Region merge](/best-practices/pd-scheduling-best-practices.md#region-merge) scheduler in PD. To avoid PD re-merging the newly split Region soon after, you need to use [table attributes](/table-attributes.md) or [dynamically modify](/pd-control.md) configuration items related to the Region merge feature.
diff --git a/statistics.md b/statistics.md
index b94055b38bfbe..16f2bd35e0250 100644
--- a/statistics.md
+++ b/statistics.md
@@ -123,13 +123,13 @@ When `IndexNameList` is empty, this syntax collects statistics on all indexes in
> **Note:**
>
-> To ensure that the statistical information before and after the collection is consistent, when `tidb_analyze_version` is `2`, this syntax collects statistics on the entire table (including all columns and indexes), instead of only on indexes.
+> To ensure that the statistical information before and after the collection is consistent, when `tidb_analyze_version` is `2`, this syntax collects statistics on the indexed columns and all indexes.
### Collect statistics on some columns
-In most cases, the optimizer only uses statistics on columns in the `WHERE`, `JOIN`, `ORDER BY`, and `GROUP BY` statements. These columns can be referred to as `PREDICATE COLUMNS`.
+When TiDB executes SQL statements, the optimizer uses statistics for only a subset of columns in most cases. For example, columns that appear in the `WHERE`, `JOIN`, `ORDER BY`, and `GROUP BY` clauses. These columns are referred to as predicate columns.
-If a table has many columns, collecting statistics on all the columns can cause a large overhead. To reduce the overhead, you can collect statistics on only specific columns (that you choose) or `PREDICATE COLUMNS` to be used by the optimizer. To persist the column list of any subset of columns for reuse in future, see [Persist column configurations](#persist-column-configurations).
+If a table has many columns, collecting statistics on all the columns can cause a large overhead. To reduce the overhead, you can collect statistics for only specific columns (that you choose) or `PREDICATE COLUMNS` to be used by the optimizer. To persist the column list of any subset of columns for reuse in future, see [Persist column configurations](#persist-column-configurations).
> **Note:**
>
@@ -144,38 +144,30 @@ If a table has many columns, collecting statistics on all the columns can cause
In the syntax, `ColumnNameList` specifies the name list of the target columns. If you need to specify more than one column, use comma `,` to separate the column names. For example, `ANALYZE table t columns a, b`. Besides collecting statistics on the specific columns in a specific table, this syntax collects statistics on the indexed columns and all indexes in that table at the same time.
-- To collect statistics on `PREDICATE COLUMNS`, do the following:
+- To collect statistics on `PREDICATE COLUMNS`, use the following syntax:
- > **Warning:**
- >
- > Currently, collecting statistics on `PREDICATE COLUMNS` is an experimental feature. It is not recommended that you use it in production environments.
-
- 1. Set the value of the [`tidb_enable_column_tracking`](/system-variables.md#tidb_enable_column_tracking-new-in-v540) system variable to `ON` to enable TiDB to collect `PREDICATE COLUMNS`.
-
-
-
- After the setting, TiDB writes the `PREDICATE COLUMNS` information to the [`mysql.column_stats_usage`](/mysql-schema/mysql-schema.md#statistics-system-tables) system table every 100 * [`stats-lease`](/tidb-configuration-file.md#stats-lease).
+ ```sql
+ ANALYZE TABLE TableName PREDICATE COLUMNS [WITH NUM BUCKETS|TOPN|CMSKETCH DEPTH|CMSKETCH WIDTH]|[WITH NUM SAMPLES|WITH FLOATNUM SAMPLERATE];
+ ```
-
+
-
+ TiDB always writes the `PREDICATE COLUMNS` information to the [`mysql.column_stats_usage`](/mysql-schema/mysql-schema.md#statistics-system-tables) system table every 100 * [`stats-lease`](/tidb-configuration-file.md#stats-lease).
- After the setting, TiDB writes the `PREDICATE COLUMNS` information to the [`mysql.column_stats_usage`](/mysql-schema/mysql-schema.md#statistics-system-tables) system table every 300 seconds.
+
-
+
- 2. After the query pattern of your business is relatively stable, collect statistics on `PREDICATE COLUMNS` by using the following syntax:
+ TiDB always writes the `PREDICATE COLUMNS` information to the [`mysql.column_stats_usage`](/mysql-schema/mysql-schema.md#statistics-system-tables) system table every 300 seconds.
- ```sql
- ANALYZE TABLE TableName PREDICATE COLUMNS [WITH NUM BUCKETS|TOPN|CMSKETCH DEPTH|CMSKETCH WIDTH]|[WITH NUM SAMPLES|WITH FLOATNUM SAMPLERATE];
- ```
+
- Besides collecting statistics on `PREDICATE COLUMNS` in a specific table, this syntax collects statistics on indexed columns and all indexes in that table at the same time.
+ In addition to collecting statistics on `PREDICATE COLUMNS` in a specific table, this syntax collects statistics on indexed columns and all indexes in that table at the same time.
- > **Note:**
- >
- > - If the [`mysql.column_stats_usage`](/mysql-schema/mysql-schema.md#statistics-system-tables) system table does not contain any `PREDICATE COLUMNS` recorded for that table, the preceding syntax collects statistics on all columns and all indexes in that table.
- > - Any columns excluded from collection (either by manually listing columns or using `PREDICATE COLUMNS`) will not have their statistics overwritten. When executing a new type of SQL query, the optimizer will use the old statistics for such columns if it exists or pseudo column statistics if columns never had statistics collected. The next ANALYZE using `PREDICATE COLUMNS` will collect the statistics on those columns.
+ > **Note:**
+ >
+ > - If the [`mysql.column_stats_usage`](/mysql-schema/mysql-schema.md#statistics-system-tables) system table does not contain any `PREDICATE COLUMNS` recorded for that table, the preceding syntax collects statistics on indexed columns and all indexes in that table.
+ > - Any columns excluded from collection (either by manually listing columns or using `PREDICATE COLUMNS`) will not have their statistics overwritten. When executing a new type of SQL query, the optimizer will use the old statistics for such columns if it exists, or pseudo column statistics if columns never had statistics collected. The next ANALYZE using `PREDICATE COLUMNS` will collect the statistics on those columns.
- To collect statistics on all columns and indexes, use the following syntax:
@@ -329,9 +321,6 @@ To locate `PREDICATE COLUMNS` and columns on which statistics have been collecte
In the following example, after executing `ANALYZE TABLE t PREDICATE COLUMNS;`, TiDB collects statistics on columns `b`, `c`, and `d`, where column `b` is a `PREDICATE COLUMN` and columns `c` and `d` are index columns.
```sql
-SET GLOBAL tidb_enable_column_tracking = ON;
-Query OK, 0 rows affected (0.00 sec)
-
CREATE TABLE t (a INT, b INT, c INT, d INT, INDEX idx_c_d(c, d));
Query OK, 0 rows affected (0.00 sec)
diff --git a/system-variables.md b/system-variables.md
index ada2df089b91e..9fcb7740059f5 100644
--- a/system-variables.md
+++ b/system-variables.md
@@ -1112,6 +1112,22 @@ MPP is a distributed computing framework provided by the TiFlash engine, which a
- Default value: `OFF`
- This variable is used to set whether the `AUTO_INCREMENT` property of a column is allowed to be removed by executing `ALTER TABLE MODIFY` or `ALTER TABLE CHANGE` statements. It is not allowed by default.
+### tidb_analyze_column_options New in v8.3.0
+
+> **Note:**
+>
+> - This variable only works when [`tidb_analyze_version`](#tidb_analyze_version-new-in-v510) is set to `2`.
+> - If you upgrade your TiDB cluster from a version earlier than v8.3.0 to v8.3.0 or later, this variable is set to `ALL` by default to keep the original behavior.
+> - Starting from v8.3.0, for a newly deployed TiDB cluster, this variable is set to `PREDICATE` by default.
+
+- Scope: GLOBAL
+- Persists to cluster: Yes
+- Applies to hint [SET_VAR](/optimizer-hints.md#set_varvar_namevar_value): No
+- Type: Enumeration
+- Default value: `PREDICATE`
+- Value options:`ALL`, `PREDICATE`
+- This variable controls the behavior of the `ANALYZE TABLE` statement. Setting it to `PREDICATE` means only collecting statistics for [predicate columns](/statistics.md#collect-statistics-on-some-columns); setting it to `ALL` means collecting statistics for all columns. In scenarios where OLAP queries are used, it is recommended to set it to `ALL`, otherwise collecting statistics can result in a significant drop in query performance.
+
### tidb_analyze_distsql_scan_concurrency New in v7.6.0
- Scope: SESSION | GLOBAL
@@ -1670,7 +1686,7 @@ mysql> SELECT job_info FROM mysql.analyze_jobs ORDER BY end_time DESC LIMIT 1;
>
> This variable is read-only for [TiDB Serverless](https://docs.pingcap.com/tidbcloud/select-cluster-tier#tidb-serverless).
-- Scope: GLOBAL
+- Scope: SESSION | GLOBAL
- Persists to cluster: Yes
- Applies to hint [SET_VAR](/optimizer-hints.md#set_varvar_namevar_value): No
- Type: Integer
@@ -1701,7 +1717,7 @@ mysql> SELECT job_info FROM mysql.analyze_jobs ORDER BY end_time DESC LIMIT 1;
>
> This variable is read-only for [TiDB Serverless](https://docs.pingcap.com/tidbcloud/select-cluster-tier#tidb-serverless).
-- Scope: GLOBAL
+- Scope: SESSION | GLOBAL
- Persists to cluster: Yes
- Applies to hint [SET_VAR](/optimizer-hints.md#set_varvar_namevar_value): No
- Type: Integer
@@ -2003,7 +2019,7 @@ mysql> SELECT job_info FROM mysql.analyze_jobs ORDER BY end_time DESC LIMIT 1;
> **Warning:**
>
-> Currently, collecting statistics on `PREDICATE COLUMNS` is an experimental feature. It is not recommended that you use it in production environments.
+> Starting from v8.3.0, this variable is deprecated. TiDB tracks predicate columns by default. For more information, see [`tidb_analyze_column_options`](#tidb_analyze_column_options-new-in-v830).
- Scope: GLOBAL
- Persists to cluster: Yes
@@ -2139,6 +2155,50 @@ mysql> SELECT job_info FROM mysql.analyze_jobs ORDER BY end_time DESC LIMIT 1;
- Possible values: `OFF`, `ON`
- This variable controls whether to support creating `Global indexes` for partitioned tables. `Global index` is currently in the development stage. **It is not recommended to modify the value of this system variable**.
+### tidb_enable_lazy_cursor_fetch New in v8.3.0
+
+> **Warning:**
+>
+> The feature controlled by this variable is experimental. It is not recommended that you use it in the production environment. This feature might be changed or removed without prior notice. If you find a bug, you can report an [issue](https://github.com/pingcap/tidb/issues) on GitHub.
+
+
+
+- Scope: GLOBAL
+- Persists to cluster: Yes
+- Applies to hint [SET_VAR](/optimizer-hints.md#set_varvar_namevar_value): No
+- Type: Boolean
+- Default value: `OFF`
+- Possible values: `OFF`, `ON`
+- This variable controls the behavior of the [Cursor Fetch](/develop/dev-guide-connection-parameters.md#use-streamingresult-to-get-the-execution-result) feature.
+ - When Cursor Fetch is enabled and this variable is set to `OFF`, TiDB reads all the data at the start of statement execution, stores the data into TiDB's memory, and returns it to the client based on the client's specified `FetchSize` for subsequent client reads. If the result set is too large, TiDB might temporarily write the result to the hard disk.
+ - When Cursor Fetch is enabled and this variable is set to `ON`, TiDB does not read all the data into the TiDB node at once, but reads data into the TiDB node incrementally as the client fetches it.
+- The feature controlled by this variable has the following limitations:
+ - It does not support statements within explicit transactions.
+ - It only supports execution plans that contain and only contain `TableReader`, `IndexReader`, `IndexLookUp`, `Projection`, and `Selection` operators.
+ - For statements using Lazy Cursor Fetch, execution information does not appear in the [statements summary](/statement-summary-tables.md) and [slow query log](/identify-slow-queries.md).
+- For unsupported scenarios, its behavior is the same as when setting this variable to `OFF`.
+
+
+
+
+
+- Scope: GLOBAL
+- Persists to cluster: Yes
+- Applies to hint [SET_VAR](/optimizer-hints.md#set_varvar_namevar_value): No
+- Type: Boolean
+- Default value: `OFF`
+- Possible values: `OFF`, `ON`
+- This variable controls the behavior of the [Cursor Fetch](/develop/dev-guide-connection-parameters.md#use-streamingresult-to-get-the-execution-result) feature.
+ - When Cursor Fetch is enabled and this variable is set to `OFF`, TiDB reads all the data at the start of statement execution, stores the data into TiDB's memory, and returns it to the client based on the client's specified `FetchSize` for subsequent client reads. If the result set is too large, TiDB might temporarily write the result to the hard disk.
+ - When Cursor Fetch is enabled and this variable is set to `ON`, TiDB does not read all the data into the TiDB node at once, but reads data into the TiDB node incrementally as the client fetches it.
+- The feature controlled by this variable has the following limitations:
+ - It does not support statements within explicit transactions.
+ - It only supports execution plans that contain and only contain `TableReader`, `IndexReader`, `IndexLookUp`, `Projection`, and `Selection` operators.
+ - For statements using Lazy Cursor Fetch, execution information does not appear in the [statements summary](/statement-summary-tables.md) and [slow query log](https://docs.pingcap.com/tidb/stable/identify-slow-queries).
+- For unsupported scenarios, its behavior is the same as when setting this variable to `OFF`.
+
+
+
### tidb_enable_non_prepared_plan_cache
- Scope: SESSION | GLOBAL
@@ -2567,6 +2627,16 @@ mysql> SELECT job_info FROM mysql.analyze_jobs ORDER BY end_time DESC LIMIT 1;
- Value options: `OFF`, `ON`
- This variable controls whether TiDB enables chunk objects cache. If the value is `ON`, TiDB prefers to use the cached chunk object and only requests from the system if the requested object is not in the cache. If the value is `OFF`, TiDB requests chunk objects from the system directly.
+### tidb_enable_shared_lock_promotion New in v8.3.0
+
+- Scope: SESSION | GLOBAL
+- Persists to cluster: Yes
+- Applies to hint [SET_VAR](/optimizer-hints.md#set_varvar_namevar_value): No
+- Type: Boolean
+- Default value: `OFF`
+- This variable controls whether to enable the feature of upgrading shared locks to exclusive locks. TiDB does not support `SELECT LOCK IN SHARE MODE` by default. When the variable value is `ON`, TiDB tries to upgrade the `SELECT LOCK IN SHARE MODE` statement to `SELECT FOR UPDATE` and add a pessimistic lock. The default value of this variable is `OFF`, which means that the feature of upgrading shared locks to exclusive locks is disabled.
+- Enabling this variable takes effect on the `SELECT LOCK IN SHARE MODE` statement, regardless of whether [`tidb_enable_noop_functions`](/system-variables.md#tidb_enable_noop_functions-new-in-v40) is enabled or not.
+
### tidb_enable_slow_log
> **Note:**
@@ -3464,12 +3534,13 @@ For a system upgraded to v5.0 from an earlier version, if you have not modified
### tidb_low_resolution_tso
-- Scope: SESSION
+- Scope: SESSION | GLOBAL
- Applies to hint [SET_VAR](/optimizer-hints.md#set_varvar_namevar_value): No
- Type: Boolean
- Default value: `OFF`
- This variable is used to set whether to enable the low-precision TSO feature. After this feature is enabled, TiDB uses the cached timestamp to read data. The cached timestamp is updated every 2 seconds by default. Starting from v8.0.0, you can configure the update interval by [`tidb_low_resolution_tso_update_interval`](#tidb_low_resolution_tso_update_interval-new-in-v800).
- The main applicable scenario is to reduce the overhead of acquiring TSO for small read-only transactions when reading old data is acceptable.
+- Starting from v8.3.0, this variable supports the GLOBAL scope.
### `tidb_low_resolution_tso_update_interval` New in v8.0.0
diff --git a/ticdc/ticdc-bidirectional-replication.md b/ticdc/ticdc-bidirectional-replication.md
index 607f23dda2aae..09d5252fc00fa 100644
--- a/ticdc/ticdc-bidirectional-replication.md
+++ b/ticdc/ticdc-bidirectional-replication.md
@@ -89,14 +89,12 @@ Non-replicable DDLs include:
To solve the problem of replicable DDLs and non-replicable DDLs, TiDB introduces the following BDR roles:
-- `PRIMARY`: you can execute replicable DDLs, but cannot execute non-replicable DDLs. Replicable DDLs will be replicated to the downstream by TiCDC.
-- `SECONDARY`: you cannot execute replicable DDLs or non-replicable DDLs, but can execute the DDLs replicated by TiCDC.
+- `PRIMARY`: You can execute replicable DDLs, but not non-replicable DDLs. Replicable DDLs executed in a PRIMARY cluster will be replicated to the downstream by TiCDC.
+- `SECONDARY`: You cannot execute replicable DDLs or non-replicable DDLs. However, DDLs executed in a PRIMARY cluster can be replicated to a SECONDARY cluster by TiCDC.
-When no BDR role is set, you can execute any DDL. But after you set `bdr_mode=true` on TiCDC, the executed DDL will not be replicated by TiCDC.
+When no BDR role is set, you can execute any DDL. However, the changefeed in BDR mode does not replicate any DDL on that cluster.
-> **Warning:**
->
-> This feature is experimental. It is not recommended that you use it in the production environment. This feature might be changed or removed without prior notice. If you find a bug, you can report an [issue](https://github.com/pingcap/tidb/issues) on GitHub.
+In short, in BDR mode, TiCDC only replicates replicable DDLs in the PRIMARY cluster to the downstream.
### Replication scenarios of replicable DDLs
diff --git a/ticdc/ticdc-canal-json.md b/ticdc/ticdc-canal-json.md
index c7c076298396e..c7ee505c3d922 100644
--- a/ticdc/ticdc-canal-json.md
+++ b/ticdc/ticdc-canal-json.md
@@ -64,7 +64,7 @@ TiCDC encodes a DDL Event into the following Canal-JSON format.
"data": null,
"old": null,
"_tidb": { // TiDB extension field
- "commitTs": 163963309467037594
+ "commitTs": 429918007904436226 // A TiDB TSO timestamp
}
}
```
@@ -133,7 +133,7 @@ TiCDC encodes a row of DML data change event as follows:
],
"old": null,
"_tidb": { // TiDB extension field
- "commitTs": 163963314122145239
+ "commitTs": 429918007904436226 // A TiDB TSO timestamp
}
}
```
@@ -162,7 +162,7 @@ The following is an example of the WATERMARK Event.
"data": null,
"old": null,
"_tidb": { // TiDB extension field
- "watermarkTs": 429918007904436226
+ "watermarkTs": 429918007904436226 // A TiDB TSO timestamp
}
}
```
diff --git a/ticdc/ticdc-sink-to-kafka.md b/ticdc/ticdc-sink-to-kafka.md
index bbdebf2ca8d83..691d14a29c672 100644
--- a/ticdc/ticdc-sink-to-kafka.md
+++ b/ticdc/ticdc-sink-to-kafka.md
@@ -450,7 +450,7 @@ The message format with handle keys only is as follows:
],
"old": null,
"_tidb": { // TiDB extension fields
- "commitTs": 163963314122145239,
+ "commitTs": 429918007904436226, // A TiDB TSO timestamp
"onlyHandleKey": true
}
}
@@ -516,7 +516,7 @@ The Kafka consumer receives a message that contains the address of the large mes
],
"old": null,
"_tidb": { // TiDB extension fields
- "commitTs": 163963314122145239,
+ "commitTs": 429918007904436226, // A TiDB TSO timestamp
"claimCheckLocation": "s3:/claim-check-bucket/${uuid}.json"
}
}
diff --git a/tidb-binlog/bidirectional-replication-between-tidb-clusters.md b/tidb-binlog/bidirectional-replication-between-tidb-clusters.md
index a851544e56783..66f30b3b75356 100644
--- a/tidb-binlog/bidirectional-replication-between-tidb-clusters.md
+++ b/tidb-binlog/bidirectional-replication-between-tidb-clusters.md
@@ -8,7 +8,6 @@ aliases: ['/docs/dev/tidb-binlog/bidirectional-replication-between-tidb-clusters
> **Warning:**
>
-> - Currently, bidirectional replication is still an experimental feature. It is **NOT** recommended to use it in the production environment.
> - Starting from v7.5.0, [TiDB Binlog](/tidb-binlog/tidb-binlog-overview.md) replication is deprecated. Starting from v8.3.0, TiDB Binlog is fully deprecated, with removal planned for a future release. Use [TiCDC](/ticdc/ticdc-overview.md) instead for incremental data replication. Use [Point-in-Time Recovery (PITR)](/br/br-pitr-guide.md) for the PITR scenario needs.
> - TiDB Binlog is not compatible with some features introduced in TiDB v5.0 and they cannot be used together. For details, see [Notes](/tidb-binlog/tidb-binlog-overview.md#notes).
diff --git a/tiflash/tiflash-configuration.md b/tiflash/tiflash-configuration.md
index e8c383b6e70d3..78f036d1d2d63 100644
--- a/tiflash/tiflash-configuration.md
+++ b/tiflash/tiflash-configuration.md
@@ -256,10 +256,11 @@ delta_index_cache_size = 0
## Security settings take effect starting from v4.0.5.
[security]
- ## New in v5.0. This configuration item enables or disables log redaction. Value options: true, false, marker. The marker option is introduced in v8.2.0.
- ## The default value is false, which means that log redaction is disabled.
- ## If the configuration item is set to true, all user data in the log is replaced by `?`.
- ## If the configuration item is set to marker, all user data in the log is wrapped in `‹ ›`. If user data contains `‹` or `›`, `‹` is escaped as `‹‹`, and `›` is escaped as `››`. Based on the marked logs, you can decide whether to desensitize the marked information when the logs are displayed.
+ ## New in v5.0. This configuration item enables or disables log redaction. Value options: `true`, `false`, `"on"`, `"off"`, and `"marker"`. The `"on"`, `"off"`, and `"marker"` options are introduced in v8.2.0.
+ ## If the configuration item is set to `false` or `"off"`, log redaction is disabled.
+ ## If the configuration item is set to `true` or `"on"`, all user data in the log is replaced by `?`.
+ ## If the configuration item is set to `"marker"`, all user data in the log is wrapped in `‹ ›`. If user data contains `‹` or `›`, `‹` is escaped as `‹‹`, and `›` is escaped as `››`. Based on the marked logs, you can decide whether to desensitize the marked information when the logs are displayed.
+ ## The default value is `false`.
## Note that you also need to set security.redact-info-log for tiflash-learner's logging in tiflash-learner.toml.
# redact_info_log = false
@@ -307,9 +308,11 @@ The parameters in `tiflash-learner.toml` are basically the same as those in TiKV
snap-handle-pool-size = 2
[security]
- ## New in v5.0. This configuration item enables or disables log redaction. Value options: true, false.
- ## The default value is false, which means that log redaction is disabled.
- ## If the configuration item is set to true, all user data in the log is replaced by `?`.
+ ## New in v5.0. This configuration item enables or disables log redaction. Value options: `true`, `false`, `"on"`, `"off"`, and `"marker"`. The `"on"`, `"off"`, and `"marker"` options are introduced in v8.3.0.
+ ## If the configuration item is set to `false` or `"off"`, log redaction is disabled.
+ ## If the configuration item is set to `true` or `"on"`, all user data in the log is replaced by `?`.
+ ## If the configuration item is set to `"marker"`, all user data in the log is wrapped in `‹ ›`. If user data contains `‹` or `›`, `‹` is escaped as `‹‹`, and `›` is escaped as `››`. Based on the marked logs, you can decide whether to desensitize the marked information when the logs are displayed.
+ ## The default value is `false`.
redact-info-log = false
[security.encryption]
diff --git a/tikv-configuration-file.md b/tikv-configuration-file.md
index 8c96e8528b37c..f07fdd1ca6c08 100644
--- a/tikv-configuration-file.md
+++ b/tikv-configuration-file.md
@@ -2041,8 +2041,12 @@ Configuration items related to security.
### `redact-info-log` New in v4.0.8
-+ This configuration item enables or disables log redaction. If the configuration value is set to `true`, all user data in the log will be replaced by `?`.
++ This configuration item enables or disables log redaction. Value options: `true`, `false`, `"on"`, `"off"`, and `"marker"`. The `"on"`, `"off"`, and `"marker"` options are introduced in v8.3.0.
++ If the configuration item is set to `false` or `"off"`, log redaction is disabled.
++ If the configuration item is set to `true` or `"on"`, all user data in the log is replaced by `?`.
++ If the configuration item is set to `"marker"`, all user data in the log is wrapped in `‹ ›`. If user data contains `‹` or `›`, `‹` is escaped as `‹‹`, and `›` is escaped as `››`. Based on the marked logs, you can decide whether to desensitize the marked information when the logs are displayed.
+ Default value: `false`
++ For details on how to use it, see [Log redaction in TiKV side](/log-redaction.md#log-redaction-in-tikv-side).
## security.encryption
diff --git a/tiup/tiup-bench.md b/tiup/tiup-bench.md
index c69b483af73eb..a7d631ad43557 100644
--- a/tiup/tiup-bench.md
+++ b/tiup/tiup-bench.md
@@ -152,7 +152,17 @@ Flags:
tiup bench tpch --sf=1 prepare
```
-2. Run the TPC-H test by executing one of the following commands:
+2. Collect statistics:
+
+ For the OLAP scenarios, to ensure that the TiDB optimizer can generate the optimal execution plan, execute the following SQL statements to collect statistics in advance. **Be sure to set [`tidb_analyze_column_options`](/system-variables.md#tidb_analyze_column_options-new-in-v830) to `ALL`, otherwise collecting statistics can result in a significant drop in query performance.**
+
+ {{< copyable "shell-regular" >}}
+
+ ```sql
+ set global tidb_analyze_column_options='ALL';
+ ```
+
+3. Run the TPC-H test by executing one of the following commands:
- If you check the result, run this command:
@@ -170,7 +180,7 @@ Flags:
tiup bench tpch --count=22 --sf=1 run
```
-3. Clean up data:
+4. Clean up data:
{{< copyable "shell-regular" >}}
diff --git a/upgrade-tidb-using-tiup.md b/upgrade-tidb-using-tiup.md
index 61ec377a1903d..19c7a4f930ad5 100644
--- a/upgrade-tidb-using-tiup.md
+++ b/upgrade-tidb-using-tiup.md
@@ -21,6 +21,7 @@ This document is targeted for the following upgrade paths:
> 3. **DO NOT** upgrade a TiDB cluster when a DDL statement is being executed in the cluster (usually for the time-consuming DDL statements such as `ADD INDEX` and the column type changes). Before the upgrade, it is recommended to use the [`ADMIN SHOW DDL`](/sql-statements/sql-statement-admin-show-ddl.md) command to check whether the TiDB cluster has an ongoing DDL job. If the cluster has a DDL job, to upgrade the cluster, wait until the DDL execution is finished or use the [`ADMIN CANCEL DDL`](/sql-statements/sql-statement-admin-cancel-ddl.md) command to cancel the DDL job before you upgrade the cluster.
> 4. If the TiDB version before upgrade is 7.1.0 or later, you can ignore the preceding warnings 2 and 3. For more information, see [limitations on using TiDB smooth upgrade](/smooth-upgrade-tidb.md#limitations).
> 5. Be sure to read [limitations on user operations](/smooth-upgrade-tidb.md#limitations-on-user-operations) before upgrading your TiDB cluster using TiUP.
+> 6. If the version of your current TiDB cluster is TiDB v7.6.0 to v8.2.0, the upgrade target version is v8.3.0 or later, and accelerated table creation is enabled via [`tidb_enable_fast_create_table`](/system-variables.md#tidb_enable_fast_create_table-new-in-v800), you need to first disable the accelerated table creation feature, and then enable it as needed after the upgrade is completed. Otherwise, some metadata KVs added by this feature remain in the cluster. Starting from v8.3.0, this feature is optimized. Upgrading to a later TiDB version no longer generates and retains this type of metadata KVs.
> **Note:**
>