Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add docs for global memory control #10908

Merged
merged 13 commits into from
Nov 7, 2022
2 changes: 2 additions & 0 deletions TOC.md
Original file line number Diff line number Diff line change
Expand Up @@ -782,6 +782,8 @@
- [`INSPECTION_RULES`](/information-schema/information-schema-inspection-rules.md)
- [`INSPECTION_SUMMARY`](/information-schema/information-schema-inspection-summary.md)
- [`KEY_COLUMN_USAGE`](/information-schema/information-schema-key-column-usage.md)
- [`MEMORY_USAGE`](/information-schema/information-schema-memory-usage.md)
- [`MEMORY_USAGE_OPS_HISTORY`](/information-schema/information-schema-memory-usage-ops-history.md)
- [`METRICS_SUMMARY`](/information-schema/information-schema-metrics-summary.md)
- [`METRICS_TABLES`](/information-schema/information-schema-metrics-tables.md)
- [`PARTITIONS`](/information-schema/information-schema-partitions.md)
Expand Down
28 changes: 21 additions & 7 deletions configure-memory-usage.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,23 +38,37 @@ SET tidb_mem_quota_query = 8 << 10;

## Configure the memory usage threshold of a tidb-server instance

In the TiDB configuration file, you can set the memory usage threshold of a tidb-server instance by configuring [`server-memory-quota`](/tidb-configuration-file.md#server-memory-quota-new-in-v409).
You can set the memory usage threshold of a tidb-server instance by configuring the system variable [`tidb_server_memory_limit`](/system-variables.md#tidb_server_memory_limit-new-in-v640).

The following example sets the total memory usage of a tidb-server instance to 32 GB:

{{< copyable "" >}}

```toml
[performance]
server-memory-quota = 34359738368
```sql
SET GLOBAL tidb_server_memory_limit = "32GB";
```

In this configuration, when the memory usage of a tidb-server instance reaches 32 GB, the instance starts to kill running SQL statements randomly until the memory usage drops below 32 GB. SQL operations that are forced to terminate return an `Out Of Global Memory Limit!` error message to the client.
After setting this variable, when the memory usage of a tidb-server instance reaches 32 GB, TiDB will terminate the SQL operation with the largest memory usage among all running SQL operations in order, until the memory usage of the instance drops below 32 GB. The forcibly terminated SQL operation will return the `Out Of Memory Quota!` error to the client.
TomShawn marked this conversation as resolved.
Show resolved Hide resolved

Currently, the memory limit set by `tidb_server_memory_limit` **DOES NOT** terminate the following SQL operations:

- DDL operations
- INSERT, UPDATE, and DELETE operations
- SQL operations that contain window functions and common table expressions

> **Warning:**
>
> + `server-memory-quota` is still an experimental feature. It is **NOT** recommended that you use it in a production environment.
> + The default value of `server-memory-quota` is `0`, which means no memory limit.
> + During the startup process, TiDB does not guarantee that the [`tidb_server_memory_limit`](/system-variables.md#tidb_server_memory_limit-new-in-v640) limit is enforced. If the free memory of the operating system is insufficient, TiDB might still encounter OOM. You need to ensure that the TiDB instance has enough available memory.
> + In the process of memory control, the total memory usage of TiDB might slightly exceed the limit set by `tidb_server_memory_limit`.
> + Since v6.4.0, the `server-memory-quota` configuration item is deprecated. To ensure compatibility, in v6.4.0 or clusters of later versions, `server-memory-quota` overrides `tidb_server_memory_limit` in a single instance. If `server-memory-quota` is not configured in the cluster before the cluster is upgraded to v6.4.0 or later, the memory usage limit of the tidb-server instance will be set by `tidb_server_memory_limit`.
TomShawn marked this conversation as resolved.
Show resolved Hide resolved

When the memory usage of a tidb-server instance reaches a certain proportion of the total memory (the proportion is controlled by the system variable [`tidb_server_memory_limit_gc_trigger`](/system-variables.md#tidb_server_memory_limit_gc_trigger-new-in-v640)), tidb-server will try to trigger a Golang GC to relieve memory stress. To avoid frequent GCs that cause performance issues due to the instance memory fluctuating around the threshold, this GC method will trigger GC at most once every minute.

## View the memory usage of the current tidb-server instance using the INFORMATION_SCHEMA system table

To view the memory usage of the current instance or cluster, you can query the system table [`INFORMATION_SCHEMA.(CLUSTER_)MEMORY_USAGE`](/information-schema/information-schema-memory-usage.md).

To view the memory-related operations and execution basis of the current instance or cluster, you can query the system table [`INFORMATION_SCHEMA.(CLUSTER_)MEMORY_USAGE_OPS_HISTORY`](/information-schema/information-schema-memory-usage-ops-history.md). For each instance, this table retains the last 50 records.
TomShawn marked this conversation as resolved.
Show resolved Hide resolved

## Trigger the alarm of excessive memory usage

Expand Down
63 changes: 63 additions & 0 deletions information-schema/information-schema-memory-usage-ops-history.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
---
title: MEMORY_USAGE_OPS_HISTORY
summary: Learn the `MEMORY_USAGE_OPS_HISTORY` information_schema system table.
---

# MEMORY_USAGE_OPS_HISTORY

The `MEMORY_USAGE_OPS_HISTORY` table describes the history of memory-related operations and the execution basis of the current TiDB instance.

```sql
USE information_schema;
DESC memory_usage_ops_history;
```

```sql
+----------------+---------------------+------+------+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+----------------+---------------------+------+------+---------+-------+
| TIME | datetime | NO | | NULL | |
| OPS | varchar(20) | NO | | NULL | |
| MEMORY_LIMIT | bigint(21) | NO | | NULL | |
| MEMORY_CURRENT | bigint(21) | NO | | NULL | |
| PROCESSID | bigint(21) unsigned | YES | | NULL | |
| MEM | bigint(21) unsigned | YES | | NULL | |
| DISK | bigint(21) unsigned | YES | | NULL | |
| CLIENT | varchar(64) | YES | | NULL | |
| DB | varchar(64) | YES | | NULL | |
| USER | varchar(16) | YES | | NULL | |
| SQL_DIGEST | varchar(64) | YES | | NULL | |
| SQL_TEXT | varchar(256) | YES | | NULL | |
+----------------+---------------------+------+------+---------+-------+
12 rows in set (0.000 sec)
```

{{< copyable "sql" >}}

```sql
SELECT * FROM information_schema.memory_usage_ops_history;
```

```sql
+---------------------+-------------+--------------+----------------+---------------------+------------+------+-----------------+------+------+------------------------------------------------------------------+----------------------------------------------------------------------+
| TIME | OPS | MEMORY_LIMIT | MEMORY_CURRENT | PROCESSID | MEM | DISK | CLIENT | DB | USER | SQL_DIGEST | SQL_TEXT |
+---------------------+-------------+--------------+----------------+---------------------+------------+------+-----------------+------+------+------------------------------------------------------------------+----------------------------------------------------------------------+
| 2022-10-17 22:46:25 | SessionKill | 10737418240 | 10880237568 | 6718275530455515543 | 7905028235 | 0 | 127.0.0.1:34394 | test | root | 146b3d812852663a20635fbcf02be01688f52c8d433dafec0d496a14f0b59df6 | desc analyze select * from t t1 join t t2 on t1.a=t2.a order by t1.a |
+---------------------+-------------+--------------+----------------+---------------------+------------+------+-----------------+------+------+------------------------------------------------------------------+----------------------------------------------------------------------+
2 rows in set (0.002 sec)
```

The columns in the `MEMORY_USAGE_OPS_HISTORY` table are described as follows:

* `TIME`: The timestamp when the session is terminated.
* `OPS`: "SessionKill"
* `MEMORY_LIMIT`: The memory usage limit of TiDB at the time of termination, in bytes. Its value is the same as that of the system variable `tidb_server_memory_limit`](/system-variables.md#tidb_server_memory_limit-new-in-v640).
* `MEMORY_CURRENT`: The current memory usage of TiDB, in bytes.
* `PROCESSID`: The connection ID of the terminated session.
* `MEM`: The memory usage of the terminated session, in bytes.
* `DISK`: The disk usage of the terminated session, in bytes.
* `CLIENT`: The connection address of the terminated session.
TomShawn marked this conversation as resolved.
Show resolved Hide resolved
* `DB`: The name of the database connected to the terminated session.
* `USER`: The user name of the terminated session.
* `SQL_DIGEST`: The digest of the SQL statement being executed in the terminated session.
* `SQL_TEXT`: The SQL statement being executed in the terminated session.
61 changes: 61 additions & 0 deletions information-schema/information-schema-memory-usage.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
---
title: MEMORY_USAGE
summary: Learn the `MEMORY_USAGE` information_schema system table.
---

# MEMORY_USAGE

The `MEMORY_USAGE` table describes the current memory usage of the current TiDB instance.

```sql
USE information_schema;
DESC memory_usage;
```

```sql
+--------------------+-------------+------+------+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+--------------------+-------------+------+------+---------+-------+
| MEMORY_TOTAL | bigint(21) | NO | | NULL | |
| MEMORY_LIMIT | bigint(21) | NO | | NULL | |
| MEMORY_CURRENT | bigint(21) | NO | | NULL | |
| MEMORY_MAX_USED | bigint(21) | NO | | NULL | |
| CURRENT_OPS | varchar(50) | YES | | NULL | |
| SESSION_KILL_LAST | datetime | YES | | NULL | |
| SESSION_KILL_TOTAL | bigint(21) | NO | | NULL | |
| GC_LAST | datetime | YES | | NULL | |
| GC_TOTAL | bigint(21) | NO | | NULL | |
| DISK_USAGE | bigint(21) | NO | | NULL | |
| QUERY_FORCE_DISK | bigint(21) | NO | | NULL | |
+--------------------+-------------+------+------+---------+-------+
11 rows in set (0.000 sec)
```

{{< copyable "sql" >}}

```sql
SELECT * FROM information_schema.memory_usage;
```

```sql
+--------------+--------------+----------------+-----------------+-------------+---------------------+--------------------+---------------------+----------+------------+------------------+
| MEMORY_TOTAL | MEMORY_LIMIT | MEMORY_CURRENT | MEMORY_MAX_USED | CURRENT_OPS | SESSION_KILL_LAST | SESSION_KILL_TOTAL | GC_LAST | GC_TOTAL | DISK_USAGE | QUERY_FORCE_DISK |
+--------------+--------------+----------------+-----------------+-------------+---------------------+--------------------+---------------------+----------+------------+------------------+
| 33674170368 | 10737418240 | 5097644032 | 10826604544 | NULL | 2022-10-17 22:47:47 | 1 | 2022-10-17 22:47:47 | 20 | 0 | 0 |
+--------------+--------------+----------------+-----------------+-------------+---------------------+--------------------+---------------------+----------+------------+------------------+
2 rows in set (0.002 sec)
```

The columns in the `MEMORY_USAGE` table are described as follows:

* MEMORY_TOTAL: The total available memory of TiDB, in bytes.
* MEMORY_LIMIT: The memory usage limit of TiDB, in bytes. The value is the same as the value of the [`tidb_server_memory_limit`](/system-variables.md#tidb_server_memory_limit-new-in-v640) system variable.
TomShawn marked this conversation as resolved.
Show resolved Hide resolved
* MEMORY_CURRENT: The current memory usage of TiDB, in bytes.
* MEMORY_MAX_USED: The maximum memory usage of TiDB from the time it is started to the current time, in bytes.
* CURRENT_OPS: "shrinking" | null. "shrinking" means that TiDB is performing operations that shrink memory usage.
* SESSION_KILL_LAST: The timestamp of the last time a session is terminated.
* SESSION_KILL_TOTAL: The number of times sessions are terminated, from the time TiDB is started to the current time.
* GC_LAST: The timestamp of the last time Golang GC is triggered by memory usage.
* GC_TOTAL: The number of times Golang GC is triggered by memory usage, from the time TiDB is started to the current time.
* DISK_USAGE: The disk usage of the current data write operation, in bytes.
TomShawn marked this conversation as resolved.
Show resolved Hide resolved
* QUERY_FORCE_DISK: The number of times data is written to disk, from the time TiDB is started to the current time.
TomShawn marked this conversation as resolved.
Show resolved Hide resolved
4 changes: 4 additions & 0 deletions information-schema/information-schema.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,8 @@ Many `INFORMATION_SCHEMA` tables have a corresponding `SHOW` command. The benefi
| [`CLUSTER_INFO`](/information-schema/information-schema-cluster-info.md) | Provides details on the current cluster topology. |
| [`CLUSTER_LOAD`](https://docs.pingcap.com/tidb/stable/information-schema-cluster-load) | Provides current load information for TiDB servers in the cluster. This table is not applicable to TiDB Cloud. |
| [`CLUSTER_LOG`](https://docs.pingcap.com/tidb/stable/information-schema-cluster-log) | Provides a log for the entire TiDB cluster. This table is not applicable to TiDB Cloud. |
| `CLUSTER_MEMORY_USAGE` | Provides a cluster-level view of the `MEMORY_USAGE` table. |
| `CLUSTER_MEMORY_USAGE_OPS_HISTORY` | Provides a cluster-level view of the `MEMORY_USAGE_OPS_HISTORY` table. |
| `CLUSTER_PROCESSLIST` | Provides a cluster-level view of the `PROCESSLIST` table. |
| `CLUSTER_SLOW_QUERY` | Provides a cluster-level view of the `SLOW_QUERY` table. |
| `CLUSTER_STATEMENTS_SUMMARY` | Provides a cluster-level view of the `STATEMENTS_SUMMARY` table. |
Expand All @@ -74,6 +76,8 @@ Many `INFORMATION_SCHEMA` tables have a corresponding `SHOW` command. The benefi
| [`INSPECTION_RESULT`](https://docs.pingcap.com/tidb/stable/information-schema-inspection-result) | Triggers internal diagnostics checks. This table is not applicable to TiDB Cloud. |
| [`INSPECTION_RULES`](https://docs.pingcap.com/tidb/stable/information-schema-inspection-rules) | A list of internal diagnostic checks performed. This table is not applicable to TiDB Cloud. |
| [`INSPECTION_SUMMARY`](https://docs.pingcap.com/tidb/stable/information-schema-inspection-summary) | A summarized report of important monitoring metrics. This table is not applicable to TiDB Cloud. |
| [`MEMORY_USAGE`](/information-schema/information-schema-memory-usage.md) | The memory usage of the current TiDB instance. |
| [`MEMORY_USAGE_OPS_HISTORY`](/information-schema/information-schema-memory-usage-ops-history.md) | The history of memory-related operations and the execution basis of the current TiDB instance. |
| [`METRICS_SUMMARY`](https://docs.pingcap.com/tidb/stable/information-schema-metrics-summary) | A summary of metrics extracted from Prometheus. This table is not applicable to TiDB Cloud. |
| `METRICS_SUMMARY_BY_LABEL` | See `METRICS_SUMMARY` table. |
| [`METRICS_TABLES`](https://docs.pingcap.com/tidb/stable/information-schema-metrics-tables) | Provides the PromQL definitions for tables in `METRICS_SCHEMA`. This table is not applicable to TiDB Cloud. |
Expand Down
28 changes: 28 additions & 0 deletions system-variables.md
Original file line number Diff line number Diff line change
Expand Up @@ -3100,6 +3100,34 @@ explain select * from t where age=5;
- By default, Regions are split for a new table when it is being created in TiDB. After this variable is enabled, the newly split Regions are scattered immediately during the execution of the `CREATE TABLE` statement. This applies to the scenario where data need to be written in batches right after the tables are created in batches, because the newly split Regions can be scattered in TiKV beforehand and do not have to wait to be scheduled by PD. To ensure the continuous stability of writing data in batches, the `CREATE TABLE` statement returns success only after the Regions are successfully scattered. This makes the statement's execution time multiple times longer than that when you disable this variable.
- Note that if `SHARD_ROW_ID_BITS` and `PRE_SPLIT_REGIONS` have been set when a table is created, the specified number of Regions are evenly split after the table creation.

### tidb_server_memory_limit <span class="version-mark">New in v6.4.0</span>

- Scope: GLOBAL
- Persists to cluster: Yes
- Default value: 80%
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

0

TomShawn marked this conversation as resolved.
Show resolved Hide resolved
- Range:
- You can set the value in the percentage format, which means the percentage of the memory usage relative to the total memory. The value range is `[1%, 99%]`.
- You can also set the value in memory size in bytes. The value range is `[0, 9223372036854775807]`. The memory format with the units "KB|MB|GB|TB" is supported. The `0` value means no memory limit.
- This variable specifies the memory limit for a TiDB instance. When the memory usage of TiDB reaches the limit, TiDB cancels the currently running SQL statement with the highest memory usage. After the SQL statement is successfully canceled, TiDB tries to call Golang GC to immediately reclaim memory to relieve memory stress as soon as possible.
- Only SQL statements that use more memory than the `tidb_server_memory_limit_sess_min_size` limit are selected as the SQL statements to be canceled first.
TomShawn marked this conversation as resolved.
Show resolved Hide resolved
- Currently, TiDB cancels only one SQL statement at a time. After TiDB completely cancels a SQL statement and recovers resources, if the memory usage is still greater than the limit set by this variable, TiDB starts the next cancel operation.

### tidb_server_memory_limit_gc_trigger <span class="version-mark">New in v6.4.0</span>

- Scope: GLOBAL
- Persists to cluster: Yes
- Default value: `70%`
- Range: `[50%, 99%]`
- The threshold at which TiDB tries to trigger GC. When the memory usage of TiDB reaches the value of `tidb_server_memory_limit` \* the value of `tidb_server_memory_limit_gc_trigger`, TiDB will actively trigger a Golang GC. Only one GC will be triggered in one minute.
TomShawn marked this conversation as resolved.
Show resolved Hide resolved

### tidb_server_memory_limit_sess_min_size <span class="version-mark">New in v6.4.0</span>

- Scope: GLOBAL
- Persists to cluster: Yes
- Default value: `134217728` (which is 128 MB)
- Range: `[128, 9223372036854775807]`, in bytes
- After enabling memory limit, TiDB will terminate the SQL statement with the highest memory usage on the current instance. This variable specifies the minimum memory usage of the SQL statement to be terminated. If the memory usage of a TiDB instance is caused by many sessions with low memory usage, you can properly lower the value of this variable for more sessions to be canceled.
TomShawn marked this conversation as resolved.
Show resolved Hide resolved

### tidb_shard_allocate_step <span class="version-mark">New in v5.0</span>

- Scope: SESSION | GLOBAL
Expand Down
2 changes: 1 addition & 1 deletion tidb-configuration-file.md
Original file line number Diff line number Diff line change
Expand Up @@ -330,7 +330,7 @@ Configuration items related to performance.

> **Warning:**
>
> `server-memory-quota` is still an experimental feature. It is **NOT** recommended that you use it in a production environment.
> Since v6.4.0, this configuration item is deprecated and replaced by the system variable [`tidb_server_memory_limit`](/system-variables.md#tidb_server_memory_limit-new-in-v640).

+ The memory usage limit of tidb-server instances.
+ Default value: `0` (in bytes), which means no memory limit.
Expand Down