Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add docs for global memory control #10908

Merged
merged 13 commits into from
Nov 7, 2022
2 changes: 2 additions & 0 deletions TOC.md
Original file line number Diff line number Diff line change
Expand Up @@ -782,6 +782,8 @@
- [`INSPECTION_RULES`](/information-schema/information-schema-inspection-rules.md)
- [`INSPECTION_SUMMARY`](/information-schema/information-schema-inspection-summary.md)
- [`KEY_COLUMN_USAGE`](/information-schema/information-schema-key-column-usage.md)
- [`MEMORY_USAGE`](/information-schema/information-schema-memory-usage.md)
- [`MEMORY_USAGE_OPS_HISTORY`](/information-schema/information-schema-memory-usage-ops-history.md)
- [`METRICS_SUMMARY`](/information-schema/information-schema-metrics-summary.md)
- [`METRICS_TABLES`](/information-schema/information-schema-metrics-tables.md)
- [`PARTITIONS`](/information-schema/information-schema-partitions.md)
Expand Down
35 changes: 35 additions & 0 deletions configure-memory-usage.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,41 @@ In this configuration, when the memory usage of a tidb-server instance reaches 3
> + `server-memory-quota` is still an experimental feature. It is **NOT** recommended that you use it in a production environment.
> + The default value of `server-memory-quota` is `0`, which means no memory limit.

Since v6.4.0, you can use the system variable [`tidb_server_memory_limit`](/system-variables.md#tidb_server_memory_limit-new-in-v640) to set the threshold for the memory usage of a tidb-server instance.

For example, set the total memory usage of a tidb-server instance to 32 GB:

```sql
SET GLOBAL tidb_server_memory_limit = "32GB";
```

After you set this variable, when the memory usage of a tidb-server instance reaches 32 GB, TiDB will terminate the SQL operation with the largest memory usage among all running SQL operations in order, until the memory usage of the instance drops below 32 GB. The forcibly terminated SQL operation will return the `Out Of Memory Quota!` error to the client.

Currently, the memory limit set by `tidb_server_memory_limit` **DOES NOT** terminate the following SQL operations:

- DDL operations
- INSERT, UPDATE, and DELETE operations
- SQL operations that contain window functions and common table expressions

> **Warning:**
>
> + The global memory control of tidb-server instances is still an experimental feature. It is not recommended to use it in the production environment.
> + During the startup process, TiDB does not guarantee that the [`tidb_server_memory_limit`](/system-variables.md#tidb_server_memory_limit-new-in-v640) limit is enforced. If the free memory of the operating system is insufficient, TiDB might still encounter OOM. You need to ensure that the TiDB instance has enough available memory.
> + In the process of memory control, the total memory usage of TiDB might slightly exceed the limit set by `tidb_server_memory_limit`.
> + To ensure compatibility, when `tidb_server_memory_limit` is enabled, the system ignores the `server-memory-quota` value and uses `tidb_server_memory_limit` memory control mechanism. When `tidb_server_memory_limit` is disabled, the system uses the `server-memory-quota` value to control the memory usage of the tidb-server instance.

When the memory usage of a tidb-server instance reaches a certain proportion of the total memory (the proportion is controlled by the system variable [`tidb_server_memory_limit_gc_trigger`](/system-variables.md#tidb_server_memory_limit_gc_trigger-new-in-v640)), tidb-server will try to trigger a Golang GC to relieve memory stress. To avoid frequent GCs that cause performance issues due to the instance memory fluctuating around the threshold, this GC method will trigger GC at most once every minute.

## View the memory usage of the current tidb-server instance using the INFORMATION_SCHEMA system table

> **Warning:**
>
> The following system tables are introduced in v6.4.0. Currently, these tables are still experimental. The memory usage information provided is only for reference. It is not recommended to use the following system tables in a production environment to obtain memory usage information for decision-making.

To view the memory usage of the current instance or cluster, you can query the system table [`INFORMATION_SCHEMA.(CLUSTER_)MEMORY_USAGE`](/information-schema/information-schema-memory-usage.md).

To view the memory-related operations and execution basis of the current instance or cluster, you can query the system table [`INFORMATION_SCHEMA.(CLUSTER_)MEMORY_USAGE_OPS_HISTORY`](/information-schema/information-schema-memory-usage-ops-history.md). For each instance, this table retains the latest 50 records.

## Trigger the alarm of excessive memory usage

In the default configuration, a tidb-server instance prints an alarm log and records associated status files when the machine memory usage reaches 80% of its total memory. You can set the memory usage ratio threshold by configuring `memory-usage-alarm-ratio`. For detailed alarm rules, refer to the description of [`memory-usage-alarm-ratio`](/tidb-configuration-file.md#memory-usage-alarm-ratio-new-in-v409).
Expand Down
1 change: 1 addition & 0 deletions experimental-features.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ This document introduces the experimental features of TiDB in different versions
+ [FastScan](/develop/dev-guide-use-fastscan.md). (Introduced in v6.2.0)
+ [Extended statistics](/extended-statistics.md). (Introduced in v5.0.0)
+ [Randomly sample about 10000 rows of data to quickly build statistics](/system-variables.md#tidb_enable_fast_analyze) (Introduced in v3.0)
+ [Globally control the memory usage of a tidb-server instance](/configure-memory-usage.md#configure-the-memory-usage-threshold-of-a-tidb-server-instance). (Introduced in v6.4.0)

## Stability

Expand Down
63 changes: 63 additions & 0 deletions information-schema/information-schema-memory-usage-ops-history.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
---
title: MEMORY_USAGE_OPS_HISTORY
summary: Learn the `MEMORY_USAGE_OPS_HISTORY` information_schema system table.
---

# MEMORY_USAGE_OPS_HISTORY

The `MEMORY_USAGE_OPS_HISTORY` table describes the history of memory-related operations and the execution basis of the current TiDB instance.

```sql
USE information_schema;
DESC memory_usage_ops_history;
```

```sql
+----------------+---------------------+------+------+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+----------------+---------------------+------+------+---------+-------+
| TIME | datetime | NO | | NULL | |
| OPS | varchar(20) | NO | | NULL | |
| MEMORY_LIMIT | bigint(21) | NO | | NULL | |
| MEMORY_CURRENT | bigint(21) | NO | | NULL | |
| PROCESSID | bigint(21) unsigned | YES | | NULL | |
| MEM | bigint(21) unsigned | YES | | NULL | |
| DISK | bigint(21) unsigned | YES | | NULL | |
| CLIENT | varchar(64) | YES | | NULL | |
| DB | varchar(64) | YES | | NULL | |
| USER | varchar(16) | YES | | NULL | |
| SQL_DIGEST | varchar(64) | YES | | NULL | |
| SQL_TEXT | varchar(256) | YES | | NULL | |
+----------------+---------------------+------+------+---------+-------+
12 rows in set (0.000 sec)
```

{{< copyable "sql" >}}

```sql
SELECT * FROM information_schema.memory_usage_ops_history;
```

```sql
+---------------------+-------------+--------------+----------------+---------------------+------------+------+-----------------+------+------+------------------------------------------------------------------+----------------------------------------------------------------------+
| TIME | OPS | MEMORY_LIMIT | MEMORY_CURRENT | PROCESSID | MEM | DISK | CLIENT | DB | USER | SQL_DIGEST | SQL_TEXT |
+---------------------+-------------+--------------+----------------+---------------------+------------+------+-----------------+------+------+------------------------------------------------------------------+----------------------------------------------------------------------+
| 2022-10-17 22:46:25 | SessionKill | 10737418240 | 10880237568 | 6718275530455515543 | 7905028235 | 0 | 127.0.0.1:34394 | test | root | 146b3d812852663a20635fbcf02be01688f52c8d433dafec0d496a14f0b59df6 | desc analyze select * from t t1 join t t2 on t1.a=t2.a order by t1.a |
+---------------------+-------------+--------------+----------------+---------------------+------------+------+-----------------+------+------+------------------------------------------------------------------+----------------------------------------------------------------------+
2 rows in set (0.002 sec)
```

The columns in the `MEMORY_USAGE_OPS_HISTORY` table are described as follows:

* `TIME`: The timestamp when the session is terminated.
* `OPS`: "SessionKill"
* `MEMORY_LIMIT`: The memory usage limit of TiDB at the time of termination, in bytes. Its value is the same as that of the system variable `tidb_server_memory_limit`](/system-variables.md#tidb_server_memory_limit-new-in-v640).
* `MEMORY_CURRENT`: The current memory usage of TiDB, in bytes.
* `PROCESSID`: The connection ID of the terminated session.
* `MEM`: The memory usage of the terminated session, in bytes.
* `DISK`: The disk usage of the terminated session, in bytes.
* `CLIENT`: The client connection address of the terminated session.
* `DB`: The name of the database connected to the terminated session.
* `USER`: The user name of the terminated session.
* `SQL_DIGEST`: The digest of the SQL statement being executed in the terminated session.
* `SQL_TEXT`: The SQL statement being executed in the terminated session.
61 changes: 61 additions & 0 deletions information-schema/information-schema-memory-usage.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
---
title: MEMORY_USAGE
summary: Learn the `MEMORY_USAGE` information_schema system table.
---

# MEMORY_USAGE

The `MEMORY_USAGE` table describes the current memory usage of the current TiDB instance.

```sql
USE information_schema;
DESC memory_usage;
```

```sql
+--------------------+-------------+------+------+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+--------------------+-------------+------+------+---------+-------+
| MEMORY_TOTAL | bigint(21) | NO | | NULL | |
| MEMORY_LIMIT | bigint(21) | NO | | NULL | |
| MEMORY_CURRENT | bigint(21) | NO | | NULL | |
| MEMORY_MAX_USED | bigint(21) | NO | | NULL | |
| CURRENT_OPS | varchar(50) | YES | | NULL | |
| SESSION_KILL_LAST | datetime | YES | | NULL | |
| SESSION_KILL_TOTAL | bigint(21) | NO | | NULL | |
| GC_LAST | datetime | YES | | NULL | |
| GC_TOTAL | bigint(21) | NO | | NULL | |
| DISK_USAGE | bigint(21) | NO | | NULL | |
| QUERY_FORCE_DISK | bigint(21) | NO | | NULL | |
+--------------------+-------------+------+------+---------+-------+
11 rows in set (0.000 sec)
```

{{< copyable "sql" >}}

```sql
SELECT * FROM information_schema.memory_usage;
```

```sql
+--------------+--------------+----------------+-----------------+-------------+---------------------+--------------------+---------------------+----------+------------+------------------+
| MEMORY_TOTAL | MEMORY_LIMIT | MEMORY_CURRENT | MEMORY_MAX_USED | CURRENT_OPS | SESSION_KILL_LAST | SESSION_KILL_TOTAL | GC_LAST | GC_TOTAL | DISK_USAGE | QUERY_FORCE_DISK |
+--------------+--------------+----------------+-----------------+-------------+---------------------+--------------------+---------------------+----------+------------+------------------+
| 33674170368 | 10737418240 | 5097644032 | 10826604544 | NULL | 2022-10-17 22:47:47 | 1 | 2022-10-17 22:47:47 | 20 | 0 | 0 |
+--------------+--------------+----------------+-----------------+-------------+---------------------+--------------------+---------------------+----------+------------+------------------+
2 rows in set (0.002 sec)
```

The columns in the `MEMORY_USAGE` table are described as follows:

* MEMORY_TOTAL: The total available memory of TiDB, in bytes.
* MEMORY_LIMIT: The memory usage limit of TiDB, in bytes. The value is the same as that of the system variable [`tidb_server_memory_limit`](/system-variables.md#tidb_server_memory_limit-new-in-v640).
* MEMORY_CURRENT: The current memory usage of TiDB, in bytes.
* MEMORY_MAX_USED: The maximum memory usage of TiDB from the time it is started to the current time, in bytes.
* CURRENT_OPS: "shrinking" | null. "shrinking" means that TiDB is performing operations that shrink memory usage.
* SESSION_KILL_LAST: The timestamp of the last time a session is terminated.
* SESSION_KILL_TOTAL: The number of times sessions are terminated, from the time TiDB is started to the current time.
* GC_LAST: The timestamp of the last time Golang GC is triggered by memory usage.
* GC_TOTAL: The number of times Golang GC is triggered by memory usage, from the time TiDB is started to the current time.
* DISK_USAGE: The disk usage for the current data spill operation, in bytes.
* QUERY_FORCE_DISK: The number of times data is spilled to disk, from the time TiDB is started to the current time.
Loading